Metadata is More Important than Ever

The challenge we all face in data today is ensuring that we use data as it is intended. Often data consumers are guilty of using data with little or no knowledge of its source or lineage. The more complex data gets; the bigger this problem of becomes. 

It is imperative for data organizations get a grip on what is meant by metadata and how to best include it within the processes which lead to data-driven decisions. Metadata as defined by many as data about data, but it is much more than that. It provides a description of the data including it attributes, relationships, business rules, along with data about how to retrieve, use and manage it. 

Metadata makes it easier for everyone to talk about data in a singular and consistent manner.

Therefore it is critical for long-term data success to get metadata collected and managed. This is not simply a one-time exercise but rather a continuous process. The task may seem daunting but in reality is that you must eventually bite the bullet and get started or you are doomed to failure or at least serious confusion. 

To begin the organization should look to the most business critical systems and clearly document their data. I would suggest that all organizations find a metadata tool to support this need. This can be a difficult as many of these tools are costly and often organizations have trouble justifying their cost. As an alternative you can manage it through a custom solution or Excel (if necessary) but find some way which data can be shared and utilized as desired. 

Metadata is a complex subject and it is made more complex by Big Data. In this area we often say that we only enforce a schema at the time we require the data. Although this is one of the powers of Big Data it is also one of its troubling approaches. In many ways this goes against the basic foundation for data governance but we cannot simply avoid it we must deal with it. I believe that any data which we collect can be documented. We can capture metadata based on this data. It should document the possible uses for the data and the alternative structures which are available. Although the data may be “schemaless” it still has structure and this should be captured and shared.

Ultimately we must realize that we cannot simply hope that our data is used as expected; we need to be sure it is.  

Compliance in the Age of Unlimited Data

During the recent CLS Conference in Toronto on Financial Industry compliance it struck me that a considerable amount of discussion was focused on the industry’s recognition of the scope and complexity that Compliance teams face in today’s world. We live in a world with vast quantities of data but we struggle to organize it in meaningful ways. There was discussion around the changes to regulations and strategies that various stakeholders are proposing to meet new and expanding reporting requirements and all of them require data. The message that came across clearly is that everyone from the regulator to the investor are part of the compliance lifecycle and can contribute to its management and enforcement, from my perspective I wondered about how well most of these organizations have harnessed their data to support compliance. There was even a discussion on how to turn compliance into a profit centre based on the savings for being simply compliant. Organizations can realize a cost savings by reduced legal proceedings and any potential fines.  From my perspective as a data professional I believe that the foundation of compliance, enforcement and identification is based in data and most of the dialogue mentioned source information but there was little discussion on the difficulty in collecting the data so that it is meaningful. From the compliance officer’s perspective, they need the data and IT just needs to provide it on a timely basis. As reflected by the speakers at the event this is often the most demanding and underestimated part of compliance. From an IT perspective we need to better serve the users by providing data in more standardized and consumable format for both the organization and the regulators.

There is a significant demand today for information, and compliance is one area in the financial industry where a truly comprehensive view of the data is needed more than most. The compliance group in all financial institutions need to generally report on all of the information across the entire institution. This requires that data is collected from numerous systems and organized in a way that is useful and meets the compliance requirements. Most financial companies have so many systems and so much data this task is daunting and challenging. Each financial institution have such a volume of data needed to support compliance they are looking at how to better store information such that it serves compliance needs but also provides additional value for analytics and prediction.  This is not simply a storage problem but also a problem of integration. This is where Big Data is lending a helping hand. We see many banks creating data repositories to service compliance requirements but with vision for a more robust platform for analytics in the future. Data is being stored and integrated to support the foundation of information for this need today of compliance and the future for improved analytics. These data repositories become valuable resources as for many this is the first time that data from across the various financial portfolios have been integrated and made available for use by analysts along with compliance teams.

The other challenge we observed was how these same organizations need to become more proactive to address problems before they become compliance issues. Most reporting today including compliance, present data at a point in time, this approach is great for reactionary actions but do little to prevent future incidents. Our data solutions need to provide more actionable insights and identify problematic trends in advance. One such example is finding trends in questionable trading.  Today most brokerages have systems which identify whether a single trade is potentially a violation of trading rules or a trade model such as layering or spoofing. Most receive many of these alerts per day and finding trends is difficult if not impossible. Based on a single trade alert, a compliance officer may identify it as a violation and take some action, but often with so much data, the noise of false alerts make it difficult to identify the true violation. However when grouped together we can see trends that at the individual level were not detectable but over a longer time frame are repeated and begin to form an identifiable and actionable trend. This approach to analytics can enable better and more proactive compliance. In this case it provided new analytics which complemented the compliance requirements.

The conference illustrated the complexity which all compliance teams face but also shared how organizations can better manage and implement it. At the core, financial institutions are finding better ways to address compliance and data is the foundation.

Time to Revitalize Your Data Warehouse

The convergence of information, technology and analytics in the era of Big Data is revolutionizing how we need to approach our data solutions. We now have capabilities to collect all data into a repository in a scalable way which can serve people with information to make more informed decisions. The challenge is that many companies have been building data warehouses and reporting systems for years. Recent studies have shown that organizations, both small and large are investing more in analytics and maintaining or reducing their traditional data warehouse spending. To me this seems an odd combination. How could one reduce data warehouse spending and increase analytics? My conclusion is that companies are maintaining their current data warehouses and are building new data solutions which are being built to complement their current warehouses. So if we consider all things maybe the true number is that data spending is increasing overall just in new ways.

This split focus between analytics and data warehousing is at odds with an enterprise approach to data. Organizations must consider how they can evolve their existing data warehouses and enable better analytics not through a revolution but rather an evolution. We must look at how Big Data is impacting our business and data processes and adjust how we architect our data warehouses.

According to Forbes, the average spending on data projects in 2015 was $7.4 million. Enterprise organizations spent $13.8M while SMB’s spend an average of $1.6M. With this level of funding the value must be realized. So how can we evolve our data warehouses? The key is finding the space where Big Data makes the most sense. 

Today many organization are at the some point towards the development of data hubs and landing areas using Hadoop technology. We tend to concentration on the landing and staging areas where we can make the most impact with the least amount of disruption. By replacing these components in the data lifecycle, we can build a new region where data is collected and prepared to meet with analytic needs, replacing these areas which were based in a relational database at a significant cost. The new expanded Landing and Staging areas are now built with Big Data and analytics in mind in addition to the traditional needs for business reporting.  Data Architects like myself are looking at creating an environment which collects all of the data and then prepares it into a conformed arrangement where data can be served up in a structured manner to supply data to the data warehouse while providing an environment where unstructured and structured data can supply raw information to the data scientists and data analysts for their analysis. This approach is one which is quite intuitive but also one which enables a better data architecture as we are separating the various parts of our solution. By separating the landing and staging we can use the technology which best suits it today while being mindful of the future. The same would apply to the high performance analytic platforms. So we may choose an RDBMS like Oracle or Netezza which today is the most appropriate platform for traditional BI but tomorrow could bring us a new technology which will be too appealing to ignore. So by separating the functionality and technology we can evolve our data warehouse in a more agile way. 

The use case of replacing your landing and staging with Hadoop is one which serves many purposes including reducing costs and extending capacity but primarily it creates a new environment to support modern advanced analytics. This data evolution is needed to ensure that your data warehouse changes with times or gets left behind in the highly competitive business world. Now is the time to consider to renew your data warehouse architecture and see how Big Data can help to elevate your business reporting and analytics