Big Day for Big Data at Oracle OpenWorld

When you come to Oracle OpenWorld you realize that the world is changing. I find that when Oracle embraces a technology they may not be first but they are there when it counts and that is true for Big Data and the Oracle ecosystem.

The Big Data is big news and Oracle has shown it’s ready to take on the challenge. Monday, Andy Mendelsohn, SVP Database Server Technologies, spoke to us about Oracle’s Big Data Strategy.

He discussed what Big Data is and how it is used. The explosion of information today is resulting in major changes to technology. We need to understand NoSQL databases, we need to embrace Hadoop and we need to change the way we think to find the information nuggets which will support business advancement.

So what has Oracle done…. let me count the ways

1. The Bog Data Appliance
2. The Oracle NoSQL database
3. The Oracle Loader for Hadoop
4. Oracle Data Integrator for Big Data
5. R Enterprise

This is significant for us in the data-space, the time for Big Data may be now or soon. It shows me that the technology  has been validated and it is time to start looking into this. This is very exciting technology and considering the mass of information we are collecting it will be important for us to use this data to achieve a competitive advantage.

Next April 2012 at COLLABORATE12, I will be running a bootcamp on Big Data, and it will be the first place that real-world experiences in Big Data will be shown.

These are Big times for Big Data!

If Someone Says “I Think”; Tell Them to Prove It! – IOUG Real World Performance: A Review

Seems like an odd statement. We always hear people say “I think…” and then they go on to tell you what they have hypothesized. Now you wonder is this story factual? Does it REALLY work? It is then time that you need to ask them to prove it. We don’t often challenge our peers to “Prove It”. We accept thoughts and then experiment on our own to see if this thought proves to be correct.

This past Friday I had the chance to attend the IOUG’s Real World Performance seminar which was held in Toronto. It was at this seminar led by 3 of most knowledgeable Oracle people in the world, where the proof showed us how to change the way we think. As they say there is no reason for things to run slowly today, but that instead that people have made the choice to run this way and not take advantage of new technology and new methods.

Andrew Holdworth, Tom Kyte and Graham Wood, led this amazing day of proof.  The image below is a picture from my chair… of the 3 of them up front in the room(as they were all day)

It was a day where the 3 Oracle database performance gurus showed us a new way to look at performance. They discussed data warehouses and operational systems. So what were some of teh highlights:

• Showed today’s data warehouses have undersized CPU’s and insufficient I/O
• Improve data loading, as data size will continue to expand
• Stop using SQL*Loader, it’s time has passed. Consider external tables
• Read compressed data if possible
• Statistics are critical to success
• Make sure cardinality is accurate in table statistics
• Use SQL*Monitor to investigate performance of SQL
• Manage your database and system resources. Use Instance Caging if needed
• Used set-based operations as often as possible
• Check your SQL statements for issues

The biggest thing takeaways I came away with is that today we should try to achieve full CPU utilization, we should optimize how we transfer data into a data warehouse, we need to make sure our SQL statements work as expected and that indexes and partitions can be your friends and your enemies.

It is amazing to me to see how we things have changed over time. Today we need to manage our data better, our databases more effectively and the improve the process of loading and retrieval. The tools are there for you, we just need to use them. As Tom Kyte said, “Why would people choose to run their database slowly?” 

There are more of the IOUG Real World Performance being held this year. If you get the chance to attend, you should; as it can change your life or at least the life of your database.

What do Data People do?

I find it an interesting time in the days of data. I see how companies are changing and how they are embracing the data revolution and some others are not. What I do see is that there is a thirst for data and for the consumption of data in a meaningful way.

The terminology we use is changing along with the technology we use. Last week someone asked me what I did. I said I was a data architect; this was caught with a blank stare. So I then change my tact and tell people if do “Business Reporting and Analytics”. This they seem to understand.. at least the reporting part. For most people analytics is some form of math which they may be learned in a Statistics class, but didn’t really understand how it worked. The key was they passed the test. For most they consider it something they heard about in the news.

So it got me thinking, what am I? What is my job? I think there are numerous terms that might describe what I data people do today or maybe we just want to make the job of pushing numbers all day sound sexy. Here’s what I and other have come up with:

• Data Warehouse Architect
• Data Guru
• Information Technician
• Data Scientist
• Data Analyst
• Analytics Geek
• Information Shark
• Information Jockey

The ideas are endless just like the information we use. As the non-data people begin to see the power of data as information becomes served up to the everyday person, I will need to explain what I do. No title will summarize it well. Basically I tell people I collect data and draw nice pictures with the data and that always seems to make people happy, which is a good thing.

Don`t forget to join me at Oracle OpenWorld in October for my presentation about Big Data and the challenges and outlook.

2011 IOUG/Oracle Real World Performance Tour in Toronto – Not Your Regular Seminar

There are some training events people should make every effort to attend because they are valuable and unique. This upcoming Toronto seminar is one of those occasions where you need to find some time. I hope many of you can join me at the upcoming IOUG Real World Performance Tour as it arrives in Toronto on September 9th, 2011.

Tickets are on sale now for 2011 the Toronto stop of the Oracle Real World Performance Tour – featuring Tom Kyte, author of the famed AskTom blog; Andrew Holdsworth, head of Oracle's Real World Performance Team; and Graham Wood, legendary Oracle Database performance architect.

Buy Tickets:

Friday, Sept 9 - Toronto

Radisson Toronto East

BUY TICKETS

Sound like an ordinary workshop? Think again.

This interactive performance engineering event features the rivaling perspectives of three Oracle rock stars and dueling screen projector presentations for a fun and different educational experience. Get a sneak peek by checking out the video from the first leg of the tour.

Past participants have praised the Real World Performance Tour for:

“Insight into how we handle different systems” Arizona fan review

“Got some ideas how to improve performance even without upgrading to 11 g” L.A. fan review

“Tuning nowadays vs. two versions ago; very reasonable price” L.A. fan review

Oracle Real World Performance Tour

Jam Sessions | 9 – 5 p.m. | $175 IOUG Members | $225 Non-Members

Discounted rates for COLLABORATE 11 attendees
All IOUG COLLABORATE 11 attendee will receive discounted access to these world-class experts - enter the code you received via separate email to get the special rate of $150 USD!

Discounted Group registration for 3+ Attendees
Register at least 3 days prior to the event (no onsite bookings)
www.ioug.org/rocks

Big Data is Curious

I was watching a TV show about Studs Terkel, a man who wrote about American life. His story is an interesting one. One statement I heard on the show was:

“Not everyone has a depth of curiosity.
And not everyone has depth of understanding”

I found this a interesting comment and one that I relate to in my everyday world. It is so true that data is expanding faster than our ability to analyze it. This is where Big Data comes in and together with an Agile approach to development can form and important part of a data solution. The ability to collect more structured and unstructured data gives us the ability to feed our curiosity. It is imperative that today’s analysts have the same curiosity as the ancient explorers. Curiosity is opportunity to learn; to discover, this is where real innovation happens. We must use Big Data to gather information to feed the need for understanding. We need to find meaning in this information and exploring it in a dynamic way can support this need. So we look to Big Data to provide this sandbox of data for analysis and knowledge.

There are concerns with Big Data which we also need to address. Security is one of the biggest concerns for most organization. So now we need to look at Big Data and how it is deployed. Often we look to create Clouds of data, but is this data secure? Is this data protected? The danger with collecting more information is how and what to protect? This is a new challenge and if we are to succeed in getting data more accessible and more useable we also need to ensure it is protected. We are seeing that as big data matures and Hadoop continues to become mainstream, we are seeing products needed to support these requirements.

So as the technology advances and our curiosity grows we will be able to create solutions which will provide a robust ability for analysts to gain business understandings.

Is Social Networking Becoming More or Less Social?

In the past few weeks it seems like the concept of social networking and social interactions has been changed. Google+ was recently launched and it has enhanced the social experience which was championed by Facebook. What is the major difference between the two? From my first impressions it is how we group our interactions or friends. Google+ provides one with the ability to group people into your personal groups or as they call them Circles like family, work, hockey buddies or any other group you can come up with. This allows for you to better focus your communications and share what is appropriate with the right friends.

As we know today when we comment on someone’s picture it may be possible for your own friends to access this image. The problem may be that the image of you drinking with your buddies may be very appropriate to share with your close friends, but not so for your business acquaintances. It appears that the paradigm of a friend is changing. The friends we have become influencers in the decisions we all make. When I go out to eat I check the web for recommendations and reviews. When I travel I check out hotels and airfares similarly, but when it comes to a final decision I look to my close friends. So now consider our use of social networking tools. We use Twitter, Facebook and now Google+ to communicate our pleasure and dissatisfaction, I even had a call directly from Dell when I experienced problems with customer service and tweeted about my misadventure. Finally I got action. In reality most people are unable to consume or interact with all of our friends and true communication happens in a much more focused manner.

In an presentation on social networking, the presenter states that on average we have 130 friends on Facebook. However we really only interact actively with 4 to 6 on a regular basis. This is true for all of interactions; consider you make 80% of phone calls to only 4 people and we Skype with 2 people 80% of the time. It is accepted that people generally have a strong influence on very few people, but their influence in decisions is staggering. So this is the concept which Google+ takes direct aim at. It looks to organize your life and “friends” into appropriate groups so that you can eliminate the chatter and allow for more focus. Google+ plus is not a revolution in social networking merely an evolution in much the same way as we live our lives in small, trusted groups of friends.

Bringing Big Data at Oracle OpenWorld

This week I learned that my seminar “Big Data: Are You Ready?” has been accepted for Oracle OpenWorld. This is always exciting for me, as it is the largest event of the Oracle Fall season. I will be presenting in the user group track as part of the IOUG data warehousing track. I hope to see many of you in San Fran between October 2nd and 6th. Below is my session listing from the Content Catalogue:

So this leads me to my thought today of Big Data and what is it all about. Today at Thoughtcorp we are building a lab which is all about Big Data. We are preparing an environment using Hadoop on a couple of servers which allow us to experiment with the product and the tools to see what works best and what does not. We are well along this path and we are looking at MapReduce for data access and how to best clone our VM to create deployable clones. All the fun technical stuff that I will discuss as we move through the process.

So the question we now sit around and ask ourselves is what are we going to use it for? What will our client need with Big Data? If I was working at Google or Yahoo, the question could be answered quickly. The vast amounts of data collected by these two companies is the reason we are discussing this technology. So we see the need that telecomm would have for this technology. Today’s telecomm companies are usually multiple lines of business with loosely related products. These products generally generate vast amounts of data, consider your smartphone or digital TV box. Consider analyzing each press of a button on your TV remote and the cable company analyzing the patterns of viewership via clicks. This information could be fed to networks to optimize the patterns in which television is offered based on the vary granular data. Television of tomorrow may be presented differently if companies better understand how viewers are really experiencing TV. This is not your parents Neilson Rating; this is something new. The digital set-top box is a Big Data candidate. So we look around at our customers and we are starting to help them understand how the collection of vast amounts of data can form a foundation for analysis.

In our world which generates more data than ever before, we need to find a way to use this information to allow businesses to optimize their services and customers optimize their experience. It will be one thing to collect the data, but how will it be used?

Data Security Do Companies Still Not Get It?>

The story today about the CitiGroup data breach just makes me wonder what companies are thinking?

I believe they understand the importance of security, but what are they doing about it. The tools to support what you need to do to secure your information are available. Security requires two things: focus and investment. You must focus your business on the requirement for security. Build the secure processes and enforce them. As Gene Kranz once said "Failure is not an option." People must be dedicated to an overall sense of information security.

The IOUG is currently running a security survey, what do you think?

Data is a corporate asset. It also contains sensitive personal information about customers, vendors, staff and corporate intelligence and we must make the investment. As a person who works with this type of information on a daily basis I never underestimate data security, but often companies do. They often secure externally facing data, but forget other points of access and ultimately face a breach. And what becomes of this data is the real fear. In the information age.... data is king. Time to roll out a security detail.

Data comes in many different forms and it is necessary to protect it throughout its lifecycle. You don't want to be the next headline in the USAToday

The State of the Oracle User

I have just returned from COLLABORATE 11, the Oracle user conference, and I have a lot of stories to tell. It was at THE event where the technology user group(IOUG), along with Quest and OAUG got together to share experiences with Oracle and Oracle-based technologies. I always believe if you are into Oracle technology there is no event like this one anywhere else in North America. This is where Oracle and users truly collaborate to educate the community.

The conference started in an inspirational manner, as Dan Thurmon, the author of Off Balance on Purpose spoke. He talked about how once we master something, we must challenge ourselves to change and get even better. Check out this video of Dan to see what he is about. It makes you realize that in today’s ever changing technology world.

This conference proved this to me. I even had a theme song playing in my head…. “The Times They Are A-Changin’”, in 1963 Bob Dylan seemed so insightful and this week was that very same to me. So on the week when I celebrated the same birthday as the song, I came the same realization. What is changing is how Oracle is delivering solutions. It showed that there are disruptive technologies which are ready for primetime.

I see two technologies which as Enterprise Data Architects we need to consider. First is Exadata and appliance solutions. And for me the second is Open-Source technology like MySQL. With Exadata, Oracle has brought together the best of Oracle database technology with Sun hardware to optimize data access. It’s not perfect, but it is coming and the wave is beginning. Larry Ellison just told us, that there is $2 Billion worth of Exadata orders in the pipeline. The secret sauce are the optimized disks and data retrieval. It was interesting to Ask Tom Kyte (at an IOUG lunch with Tom event), how he saw the Exadata technology is changing how we do our database deployments. He says its not quite as easy as the marketing team would like people to believe, but he did say that this machine in the right hands can achieve significant benefits. I also attended a presentation where someone basically used the Exadata very much like a traditional Oracle RAC implementation. So I begin to believe that the “Art of Exadata” will be the secret sauce. It will include people who understand architecture, database design, database deployments, SQL optimization and a little bit about hardware. Exadata’s time seems to have arrived

The other is open-source. Products like Java and MySQL are both open-source products which are now becomes part of the our corporate fabric. I am beginning to see much of the same needs we had 20 years ago when Oracle was doing the same thing for databases. The same management concerns need to be addressed to optimize the use of MySQL. Discussions on backups, optimization, replication and high-availability have become pressing concerns people need to address.

Of course the real value that I get from events like COLLABORATE 11, are the networking opportunities. I get a chance to meet and discuss interesting subjects with many people; each who add to much knowledge base. I get to know and understand how Oracle plans to move forward. One such example is Willie Hardie, the VP of Database Product Marketing and a great supporter of the IOUG, and who won the IOUG Ken Jacobs award recognizing the person at Oracle who has helped the user community in numerous ways. Here he is with IOUG President Andy Flower receiving his award:

We also had a chance to network in the World of Harry Potter and here I am getting on-board for the chance to share some fun with the user group.

And thanks to my friends at The Quest user group, who focus on PeopleSoft and JD Edwards who brought in Jim Lovell and Fred Haise from Apollo 13 frame to recount the experience from that flight, which blasted off in 1970 on my birthday, before encountering their infamous issues. Here they are today as they spoke during the event.

If you have never been to an event like this; it should be time to consider it. There is nothing like it. I made new friends and helped people who were attending for the first time feel welcome. It really is a community experience and one that Facebook or LinkedIn can not replicate. Plan to be at COLLABORATE 12 in Las Vegas, April 22, 2012, and get on the Hogwarts Express; it’s time.

Baseball and BI

The Major League baseball season started last weekend and the teams that should be winning are panicking (consider an 0-5 start for the Red Sox), while others are looking at what might happen after a quick start. The length of the baseball schedule, tends to be a big equalizer and as the Summer progresses into the Fall, the cream of the crop tends to come up to the top. So although the Toronto Blue Jays have started 4-1 and look to have some real pitching this year; I am not ready to get overly excited… it’s early. After all there are 157 games left. The playoff race is months away, but yet we get excited none the less. It is baseball season.

It is also the time when one of the most analyzed sports is changing the way they do business. I think I mentioned in the past that baseball is a sport that collects more information, which is at a very granular level, compared to other sports. It provides so much data that the science of sport not longer refers only to medicine but to data as well. Data may even be more important than ever in baseball and growing in all sports. Today, when someone \wants to sign a free agent or trade for a veteran or prospect player; the first thing a team will look at is their data.  Men like Bill James and Ari Kaplan have made analytics part of the game but also part of the business. By tracking how a player reacts to situations you can now extrapolate how they will fit into your type of offense and see in advance the benefit they will give to the team. The concept of “gut-free” still exists in baseball but data is quickly replacing it as part of multi-million dollar decisions.

So as we start the season we will be able to see if Mariano Rivera can continue to deliver heat in the strike zone, or the edge of the strike zone. We will that Derek Jeter likes to hit fastballs, but hits sliders even more effectively, all of this will be tracked. We all need to find the edge. Businesses have understood the value of data, but today sports are embracing them and changing the way they  make game and player decisions….it’s all in the stats.

Play Ball! Or Run the Report!

Go Expos!

Thoughtcorp or is it The Addams Family

This week one of my favourite people and client sent out an email about the people that work at Thoughtcorp and how we look like the Adams Family…. Now being someone who is being compared to Uncle Fester, I still think it is so funny. So let me introduce you to some of the Thoughtcorp Addams family. Thanks to John H for this fantastic comparison to the famous family.
First is me, Ian Abramson, as Uncle fester.

And next up is our Co-CEO David Bercovitch or should I say Lurch, or at least a shorter version of him.

And then is Val Hedow, of Director of HR, or as some will now refer to as Morticia…. It’s almost uncanny.
     
And of course, Gino Marckx, of Agile Practice Director, when it comes to Agile he is second to none, but amazing I think he will be known as Gomez Adams.


Finally the Dino Chronopoulos, the head of the Finance group at Thoughtcorp, seems to look a lot like Pugsley….interesting.

I love and To put a real spin on this. There are places where you have a chance to work where you can build or do great things, but you always have to remember to have some fun at work and let it loose. Thanks John for helping me understand that I now for the Addam’s Family.

Open Season on the Database

For years we have contemplated open-source databases and how they fit into our overall database technology strategy. The complexity of what is the right choice for an organization has many answers. It is no longer a simple question of choosing between Oracle or Microsoft or even IBM; no it’s a choice that needs to include MySQL (unless that counts as Oracle?), Postgres, or now Hadoop and Cassandra. The choices are as complex as the systems we need to support.  From my perspective it is now open-season on the database. The choice is no longer straightforward.

The emergence of “The Cloud” and the need to distribute more data more effectively has added new pressures. The need to analyze massive amounts of data more efficiently is a must-have. We look to the Oracle’s of the world they respond with a system that is scalable, efficient and leverages the power of Oracle to enable the business. The Oracle database on an Exadata server is a solid enterprise-grade solution that will support this need. It is robust and powerful, but it is costly. So organizations who need to similar functionality but want to do it using open-source technology or some lower cost-model of software now also have options. At Thoughtcorp we are leading providers in many areas of technologies and our team has varied knowledge and much of it focused on today’s latest technologies including wireless, apps and of course using open-source technologies to achieve their goals. The data group has seen this shift as well and now I am working on how products like Hadoop, which can be used for massive data analysis. It is like Google file system (GFS) and allows for the collection and analysis of data. From Hadoop other solutions have emerged. Hive provides a easy to use tool to work with Hadoop to simplify analysis. Now Hadoop has been combined Cassandra and open-source database from Facebook. What does this give us? A real-time database with big data capabilities. Wow…that’s a mouthful.

Of course these open solutions come today with some operational risks due to single possible points of failure. of course if you can manage the risk. You may have an alternative to today’s  enterprise databases. These new data options provide us more with choice and options and in a world where you consider…. If it’s good enough for Facebook and Google, is it something I should consider?

Building Data Warehouses for the Masses

Often data warehouse designers and architects are often accused of building systems which do not always serve the enterprise but only focused portions of the organization. I have seen this occur at numerous clients who did not invest in a long-term vision and long-term data strategy.

So how do we avoid this trap and ensure that we produce a system which does not go “silo” but sets a course to meet current and future needs at the enterprise level and not departmentally. I was recently working on a presentation about data warehousing design and the question of Ralph Kimball (Dimensional) versus Bill Inmon’s (Normalized) design approach of which is better. The question is that both work, if you keep to the big picture. Design your high-level design upfront and adhere to design and development standards, It is about setting a clear course for DW design and never allowing for an independent or silo’ed solution to be developed. The key is to always look for integration opportunities. Add to your core data warehouse; don’t built a new structure which is not tied in and used conformed dimensions.

Bill Inmon, said in referring to Dimensional modeling that “A 1000 minnows do not make a whale”. Build with a great purpose in mind  that you are building an integrated, stable data repository to support business reporting and analytics.

Data Warehousing and the DW Appliance

Today I attended a seminar presented by Oracle about the Exadata database machine. As usual it was presented as a compelling argument that a complete solution like a data warehouse appliance is one that ultimately reduces complexity and increases capacity.

During the session today, they quoted a study from TDWI which says that within 3 years 78% of respondents expect to be using a data warehouse appliance. I find this a staggering figure. Consider that today DW appliances have evolved and each of the major database vendors have their own flavour, but we still continue to see that traditional databases significantly outnumber appliances. I do admit that the argument for an appliance is one that needs to be considered. The idea that a single server or linked servers can come into a data centre and within days can be productive is incredible. The bigger question becomes which appliance do I choose? Do I buy Exadata, Teradata, Data Allegro, or Netezza? I guess that answer depends on what you want to achieve and the basic knowledge and comfort level you have with each vendor.

I find that the real differentiator is the functionality available within the appliance. As you might know Netezza uses Postgres as it’s base database. Although a great database, it is provided within only the basic SQL set of commands. To extend it out, you need to buy additional options and features. Then you look at a database like Oracle’s which has 30 years of development in it and now we are presented with a database that supports very advanced SQL. Oracle has provided an appliance that runs the database. It has not created a database which happens to run on an appliance. I often wonder how many people are aware of the advanced functions and analytics available in the Oracle database. The database has a significant number of statistical functions from the simple to the esoteric. The database contains a number of data mining functions as well. Although Oracle sells a great front-end tool in Oracle Data Mining (ODM), the functions which the tools use are available to all database users. So when you start to look at what appliance, you need to look at more than simple administration and technology, but at capability. The appliance who can do the most without any additional costs seems like a solid option. Now as to whether or not we will see a big increase  in the use of appliances will depend on how well and how cost effectively these appliances can replace our current hardware platforms.

Oracle User Group Season

As the winter starts to settle in and I ponder what I will be doing in the next few months I begin to plan my spring speaking season….and of course golf season.

We are a mere 2 1/2 months away from the the start of the COLLABORATE 11 conference. As well I am working with the Toronto Oracle User Group to help to put on another BI/DW day in March or April. So it’s time to get my materials together.

My session at COLLABORATE 11, is named Oracle Data Warehousing: Soup to Nuts. The focus is on providing a solid knowledge foundation in all of the areas which data warehouse architects need to be concerned with. I, along with my co-presenters Albert Hui and Jatinder Bhardwaj are planning to present details on how a data warehouse is designed and built. We will discuss how Agile development methodologies can be applied to data warehouse projects. This is something which organizations have not considered is if Agile can be applied to these projects. From the Thoughtcorp approach to Agile and warehousing has been combined and has now been shown in action to provide similar benefits to DW projects. In addition we will get technical and illustrate the features in Oracle which every data warehouse needs and when and why they should be used. This will be your toolkit for deployment. We will discuss ETL and how data should be loaded. This provides the attendee with knowledge on how you should use tools like OWB or ODI in your project. Finally we hope to touch on some of the benefits and successes of using the Exadata and how it can provide significant value in large data project.

I know it seems like a long way away, but in reality it could not be closer. So get on board. Get registered and come on out and see myself and others at COLLABORATE or at the next TOUG meeting and share your experiences along with ours.

The New Data Warehouse 2.0.11

Its an interesting time for me and the rest of us at Thoughtcorp we continue to help organizations in innovative ways, using innovative techniques to deliver them. Over the past few years I have seen changes to the way systems are envisioned and how they are realized. So it also made me look at how I build data warehouse and reporting solutions today in 2011.

When I first began building reporting systems; they were not even call data warehouses yet. I build a system to do call detail analysis for a telecommunications company  using SQL*Plus as its reporting tool. Today I build systems that focus on providing business value and business insights. The key to build a flexible and reliable system which provide data in an easy to consume manner. We have converted tons of paper reporting into dashboards, Key Performance Indicators, modeling and other reporting capabilities which allow business to react to change and anticipate trends. Today organizations which do not have data warehouse technology in some form will quickly become business laggards.

So what has changed is how we deliver solutions. The key for a successful data warehouse/reporting solution is that the information provided in the required format provides the necessary information to create an action which will enhance the business. So the involvement of the stakeholders and users drive much of this. Based on information required in the business processes we design the warehouse to service these needs efficiently. However, in the “old” days we would build the entire data warehouse and expect that what we built will satisfy the requirements. Today by building using Agile techniques we build in smaller self-contained development unit which address all aspects including data, loading and reporting to provide quick wins an help the people get the product in their hands to work into a better solution.Today the biggest difference is data warehouse architects do not assume the solution, but build a framework in which a system can develop to meet needs and drive innovation and change. So out with the Big Bang data warehouse and in with the Agile Data Warehouse or DW 2.0.11.

Unexpected Learning

Last week I had the opportunity to take some time off and take a cruise. Considering how cold things have been around here lately it was a great time to get warm and enjoy some sun. As expected I did that, but unexpectedly I also learned some things along the way. This was a cruise with unexpected learning. And maybe I should have expected it. So what did I learn on my Winter vacation?

Well to start with my ship offered seminars on a variety of topics. One was an Improv class. Seemed like a better option than the Rumba class. So we attended and I learned lessons on working in a team, listening to your team and focusing on your task at hand and not lose that focus while others may struggle. The most important thing was to ensure the team continued to function. Through some exercises that I plan to bring back to my colleges we can bring these unexpected lessons home.

I also had the chance to meet someone in a similar field as us at Thoughtcorp, in that he did retail analysis. It was interesting to exchange stories on how we solved problems for our respective clients and how to better use retail data. Today’s challenges in retail are one that we focus on at Thoughtcorp and we really started to look at ways that retailers need to bring data together to enhance businesses.

Then I started reading a book, named “Start-up Nation” about how Israel has become a leader in innovation and technology. It is an interesting lesson on how the ability to be innovative and creative and not to take no for an answer when all around say you are bound to fail. I read about a company that approached PayPal about a new concept to capture fraud. The concept was that by separating people into good and bad people you can detect fraud. Good people leave a trail on the Internet; bad people don’t. Somehow a small Israeli company was able to deliver more accurate results in a exponentially short time frame compared to PayPal’s own analysts. It was impressive and was based upon the way that governments find terrorists. This was an approach that PayPal recognized as revolutionary. The stories continue and the lessons one can learn from a book like this can be translated to everyday business.

A great time on a great ship and met some great people who helped me make some unexpected discoveries.

BTW….the picture above is our ship the Celebrity Eclipse parked next the HMS Bounty….imagine the difference in technology of these two ships….hmmmm. 

New Year and New Challenges

As we begin a new year I start looking back and looking forward. How did I and how did the industry change in the past year and what should I now expect in the new year? In the past year at Thoughtcorp I look back at the solutions we created and how we are creating solutions 12 months later. I see a change coming and I look forward to what it will bring.

What I discuss is how we deliver business reporting and analytical solutions. At a basic level what organizations need today is no different than what they have always needed. Organizations need access to information on a timely basis to provide insightful information which can better enable the business to react to changing conditions. The new year also brings an approach which we have been developing over the past year around using Agile development methods in a data warehouse project. The key is that by going Agile, that your project will deliver in an expected manner with the business highly involved in the resulting solution. We have seen how information can transform a business. Agile has often been discussed in the context of BI, bur has rarely been used for an Enterprise data warehouse project. At Thoughtcorp we are now doing just that and have already seen the success and efficiencies it can bring to delivering project on time and more importantly meeting insightful business requirements.

So today I start to see that shift of using the Agile manifesto as a base for all data projects. As for reporting we start seeing organizations that previously had little integrated reporting can now start to mature and move ahead of basic reporting and truly embrace business reporting.