The New Data Warehouse 2.0.11

Its an interesting time for me and the rest of us at Thoughtcorp we continue to help organizations in innovative ways, using innovative techniques to deliver them. Over the past few years I have seen changes to the way systems are envisioned and how they are realized. So it also made me look at how I build data warehouse and reporting solutions today in 2011.

When I first began building reporting systems; they were not even call data warehouses yet. I build a system to do call detail analysis for a telecommunications company  using SQL*Plus as its reporting tool. Today I build systems that focus on providing business value and business insights. The key to build a flexible and reliable system which provide data in an easy to consume manner. We have converted tons of paper reporting into dashboards, Key Performance Indicators, modeling and other reporting capabilities which allow business to react to change and anticipate trends. Today organizations which do not have data warehouse technology in some form will quickly become business laggards.

So what has changed is how we deliver solutions. The key for a successful data warehouse/reporting solution is that the information provided in the required format provides the necessary information to create an action which will enhance the business. So the involvement of the stakeholders and users drive much of this. Based on information required in the business processes we design the warehouse to service these needs efficiently. However, in the “old” days we would build the entire data warehouse and expect that what we built will satisfy the requirements. Today by building using Agile techniques we build in smaller self-contained development unit which address all aspects including data, loading and reporting to provide quick wins an help the people get the product in their hands to work into a better solution.Today the biggest difference is data warehouse architects do not assume the solution, but build a framework in which a system can develop to meet needs and drive innovation and change. So out with the Big Bang data warehouse and in with the Agile Data Warehouse or DW 2.0.11.

Unexpected Learning

Last week I had the opportunity to take some time off and take a cruise. Considering how cold things have been around here lately it was a great time to get warm and enjoy some sun. As expected I did that, but unexpectedly I also learned some things along the way. This was a cruise with unexpected learning. And maybe I should have expected it. So what did I learn on my Winter vacation?

Well to start with my ship offered seminars on a variety of topics. One was an Improv class. Seemed like a better option than the Rumba class. So we attended and I learned lessons on working in a team, listening to your team and focusing on your task at hand and not lose that focus while others may struggle. The most important thing was to ensure the team continued to function. Through some exercises that I plan to bring back to my colleges we can bring these unexpected lessons home.

I also had the chance to meet someone in a similar field as us at Thoughtcorp, in that he did retail analysis. It was interesting to exchange stories on how we solved problems for our respective clients and how to better use retail data. Today’s challenges in retail are one that we focus on at Thoughtcorp and we really started to look at ways that retailers need to bring data together to enhance businesses.

Then I started reading a book, named “Start-up Nation” about how Israel has become a leader in innovation and technology. It is an interesting lesson on how the ability to be innovative and creative and not to take no for an answer when all around say you are bound to fail. I read about a company that approached PayPal about a new concept to capture fraud. The concept was that by separating people into good and bad people you can detect fraud. Good people leave a trail on the Internet; bad people don’t. Somehow a small Israeli company was able to deliver more accurate results in a exponentially short time frame compared to PayPal’s own analysts. It was impressive and was based upon the way that governments find terrorists. This was an approach that PayPal recognized as revolutionary. The stories continue and the lessons one can learn from a book like this can be translated to everyday business.

A great time on a great ship and met some great people who helped me make some unexpected discoveries.

BTW….the picture above is our ship the Celebrity Eclipse parked next the HMS Bounty….imagine the difference in technology of these two ships….hmmmm. 

New Year and New Challenges

As we begin a new year I start looking back and looking forward. How did I and how did the industry change in the past year and what should I now expect in the new year? In the past year at Thoughtcorp I look back at the solutions we created and how we are creating solutions 12 months later. I see a change coming and I look forward to what it will bring.

What I discuss is how we deliver business reporting and analytical solutions. At a basic level what organizations need today is no different than what they have always needed. Organizations need access to information on a timely basis to provide insightful information which can better enable the business to react to changing conditions. The new year also brings an approach which we have been developing over the past year around using Agile development methods in a data warehouse project. The key is that by going Agile, that your project will deliver in an expected manner with the business highly involved in the resulting solution. We have seen how information can transform a business. Agile has often been discussed in the context of BI, bur has rarely been used for an Enterprise data warehouse project. At Thoughtcorp we are now doing just that and have already seen the success and efficiencies it can bring to delivering project on time and more importantly meeting insightful business requirements.

So today I start to see that shift of using the Agile manifesto as a base for all data projects. As for reporting we start seeing organizations that previously had little integrated reporting can now start to mature and move ahead of basic reporting and truly embrace business reporting.