The convergence of information, technology and analytics in
the era of Big Data is revolutionizing how we need to approach our data
solutions. We now have capabilities to collect all data into a repository in a
scalable way which can serve people with information to make more informed
decisions. The challenge is that many companies have been building data
warehouses and reporting systems for years. Recent studies have shown that
organizations, both small and large are investing more in analytics and
maintaining or reducing their traditional data warehouse spending. To me this
seems an odd combination. How could one reduce data warehouse spending and
increase analytics? My conclusion is that companies are maintaining their
current data warehouses and are building new data solutions which are being
built to complement their current warehouses. So if we consider all things
maybe the true number is that data spending is increasing overall just in new
ways.
This split focus between analytics and data warehousing is
at odds with an enterprise approach to data. Organizations must consider how
they can evolve their existing data warehouses and enable better analytics not
through a revolution but rather an evolution. We must look at how Big Data is
impacting our business and data processes and adjust how we architect our data
warehouses.
According to Forbes, the average spending on data projects in
2015 was $7.4 million. Enterprise organizations spent $13.8M while SMB’s spend
an average of $1.6M. With this level of funding the value must be realized. So
how can we evolve our data warehouses? The key is finding the space where Big
Data makes the most sense.
Today many organization are at the some point towards the
development of data hubs and landing areas using Hadoop technology. We tend to concentration
on the landing and staging areas where we can make the most impact with the
least amount of disruption. By replacing these components in the data lifecycle,
we can build a new region where data is collected and prepared to meet with
analytic needs, replacing these areas which were based in a relational database
at a significant cost. The new expanded Landing and Staging areas are now built
with Big Data and analytics in mind in addition to the traditional needs for business
reporting. Data Architects like myself
are looking at creating an environment which collects all of the data and then
prepares it into a conformed arrangement where data can be served up in a
structured manner to supply data to the data warehouse while providing an
environment where unstructured and structured data can supply raw information
to the data scientists and data analysts for their analysis. This approach is
one which is quite intuitive but also one which enables a better data architecture
as we are separating the various parts of our solution. By separating the
landing and staging we can use the technology which best suits it today while
being mindful of the future. The same would apply to the high performance
analytic platforms. So we may choose an RDBMS like Oracle or Netezza which
today is the most appropriate platform for traditional BI but tomorrow could
bring us a new technology which will be too appealing to ignore. So by separating
the functionality and technology we can evolve our data warehouse in a more
agile way.
The use case of replacing your landing and staging with
Hadoop is one which serves many purposes including reducing costs and extending
capacity but primarily it creates a new environment to support modern advanced
analytics. This data evolution is needed to ensure that your data warehouse
changes with times or gets left behind in the highly competitive business world.
Now is the time to consider to renew your data warehouse architecture and see
how Big Data can help to elevate your business reporting and analytics
