What do Data People do?

I find it an interesting time in the days of data. I see how companies are changing and how they are embracing the data revolution and some others are not. What I do see is that there is a thirst for data and for the consumption of data in a meaningful way.

The terminology we use is changing along with the technology we use. Last week someone asked me what I did. I said I was a data architect; this was caught with a blank stare. So I then change my tact and tell people if do “Business Reporting and Analytics”. This they seem to understand.. at least the reporting part. For most people analytics is some form of math which they may be learned in a Statistics class, but didn’t really understand how it worked. The key was they passed the test. For most they consider it something they heard about in the news.

So it got me thinking, what am I? What is my job? I think there are numerous terms that might describe what I data people do today or maybe we just want to make the job of pushing numbers all day sound sexy. Here’s what I and other have come up with:

• Data Warehouse Architect
• Data Guru
• Information Technician
• Data Scientist
• Data Analyst
• Analytics Geek
• Information Shark
• Information Jockey

The ideas are endless just like the information we use. As the non-data people begin to see the power of data as information becomes served up to the everyday person, I will need to explain what I do. No title will summarize it well. Basically I tell people I collect data and draw nice pictures with the data and that always seems to make people happy, which is a good thing.

Don`t forget to join me at Oracle OpenWorld in October for my presentation about Big Data and the challenges and outlook.

2011 IOUG/Oracle Real World Performance Tour in Toronto – Not Your Regular Seminar

There are some training events people should make every effort to attend because they are valuable and unique. This upcoming Toronto seminar is one of those occasions where you need to find some time. I hope many of you can join me at the upcoming IOUG Real World Performance Tour as it arrives in Toronto on September 9th, 2011.

Tickets are on sale now for 2011 the Toronto stop of the Oracle Real World Performance Tour – featuring Tom Kyte, author of the famed AskTom blog; Andrew Holdsworth, head of Oracle's Real World Performance Team; and Graham Wood, legendary Oracle Database performance architect.

Buy Tickets:

Friday, Sept 9 - Toronto

Radisson Toronto East

BUY TICKETS

Sound like an ordinary workshop? Think again.

This interactive performance engineering event features the rivaling perspectives of three Oracle rock stars and dueling screen projector presentations for a fun and different educational experience. Get a sneak peek by checking out the video from the first leg of the tour.

Past participants have praised the Real World Performance Tour for:

“Insight into how we handle different systems” Arizona fan review

“Got some ideas how to improve performance even without upgrading to 11 g” L.A. fan review

“Tuning nowadays vs. two versions ago; very reasonable price” L.A. fan review

Oracle Real World Performance Tour

Jam Sessions | 9 – 5 p.m. | $175 IOUG Members | $225 Non-Members

Discounted rates for COLLABORATE 11 attendees
All IOUG COLLABORATE 11 attendee will receive discounted access to these world-class experts - enter the code you received via separate email to get the special rate of $150 USD!

Discounted Group registration for 3+ Attendees
Register at least 3 days prior to the event (no onsite bookings)
www.ioug.org/rocks

Big Data is Curious

I was watching a TV show about Studs Terkel, a man who wrote about American life. His story is an interesting one. One statement I heard on the show was:

“Not everyone has a depth of curiosity.
And not everyone has depth of understanding”

I found this a interesting comment and one that I relate to in my everyday world. It is so true that data is expanding faster than our ability to analyze it. This is where Big Data comes in and together with an Agile approach to development can form and important part of a data solution. The ability to collect more structured and unstructured data gives us the ability to feed our curiosity. It is imperative that today’s analysts have the same curiosity as the ancient explorers. Curiosity is opportunity to learn; to discover, this is where real innovation happens. We must use Big Data to gather information to feed the need for understanding. We need to find meaning in this information and exploring it in a dynamic way can support this need. So we look to Big Data to provide this sandbox of data for analysis and knowledge.

There are concerns with Big Data which we also need to address. Security is one of the biggest concerns for most organization. So now we need to look at Big Data and how it is deployed. Often we look to create Clouds of data, but is this data secure? Is this data protected? The danger with collecting more information is how and what to protect? This is a new challenge and if we are to succeed in getting data more accessible and more useable we also need to ensure it is protected. We are seeing that as big data matures and Hadoop continues to become mainstream, we are seeing products needed to support these requirements.

So as the technology advances and our curiosity grows we will be able to create solutions which will provide a robust ability for analysts to gain business understandings.

Is Social Networking Becoming More or Less Social?

In the past few weeks it seems like the concept of social networking and social interactions has been changed. Google+ was recently launched and it has enhanced the social experience which was championed by Facebook. What is the major difference between the two? From my first impressions it is how we group our interactions or friends. Google+ provides one with the ability to group people into your personal groups or as they call them Circles like family, work, hockey buddies or any other group you can come up with. This allows for you to better focus your communications and share what is appropriate with the right friends.

As we know today when we comment on someone’s picture it may be possible for your own friends to access this image. The problem may be that the image of you drinking with your buddies may be very appropriate to share with your close friends, but not so for your business acquaintances. It appears that the paradigm of a friend is changing. The friends we have become influencers in the decisions we all make. When I go out to eat I check the web for recommendations and reviews. When I travel I check out hotels and airfares similarly, but when it comes to a final decision I look to my close friends. So now consider our use of social networking tools. We use Twitter, Facebook and now Google+ to communicate our pleasure and dissatisfaction, I even had a call directly from Dell when I experienced problems with customer service and tweeted about my misadventure. Finally I got action. In reality most people are unable to consume or interact with all of our friends and true communication happens in a much more focused manner.

In an presentation on social networking, the presenter states that on average we have 130 friends on Facebook. However we really only interact actively with 4 to 6 on a regular basis. This is true for all of interactions; consider you make 80% of phone calls to only 4 people and we Skype with 2 people 80% of the time. It is accepted that people generally have a strong influence on very few people, but their influence in decisions is staggering. So this is the concept which Google+ takes direct aim at. It looks to organize your life and “friends” into appropriate groups so that you can eliminate the chatter and allow for more focus. Google+ plus is not a revolution in social networking merely an evolution in much the same way as we live our lives in small, trusted groups of friends.

Bringing Big Data at Oracle OpenWorld

This week I learned that my seminar “Big Data: Are You Ready?” has been accepted for Oracle OpenWorld. This is always exciting for me, as it is the largest event of the Oracle Fall season. I will be presenting in the user group track as part of the IOUG data warehousing track. I hope to see many of you in San Fran between October 2nd and 6th. Below is my session listing from the Content Catalogue:

So this leads me to my thought today of Big Data and what is it all about. Today at Thoughtcorp we are building a lab which is all about Big Data. We are preparing an environment using Hadoop on a couple of servers which allow us to experiment with the product and the tools to see what works best and what does not. We are well along this path and we are looking at MapReduce for data access and how to best clone our VM to create deployable clones. All the fun technical stuff that I will discuss as we move through the process.

So the question we now sit around and ask ourselves is what are we going to use it for? What will our client need with Big Data? If I was working at Google or Yahoo, the question could be answered quickly. The vast amounts of data collected by these two companies is the reason we are discussing this technology. So we see the need that telecomm would have for this technology. Today’s telecomm companies are usually multiple lines of business with loosely related products. These products generally generate vast amounts of data, consider your smartphone or digital TV box. Consider analyzing each press of a button on your TV remote and the cable company analyzing the patterns of viewership via clicks. This information could be fed to networks to optimize the patterns in which television is offered based on the vary granular data. Television of tomorrow may be presented differently if companies better understand how viewers are really experiencing TV. This is not your parents Neilson Rating; this is something new. The digital set-top box is a Big Data candidate. So we look around at our customers and we are starting to help them understand how the collection of vast amounts of data can form a foundation for analysis.

In our world which generates more data than ever before, we need to find a way to use this information to allow businesses to optimize their services and customers optimize their experience. It will be one thing to collect the data, but how will it be used?

Data Security Do Companies Still Not Get It?>

The story today about the CitiGroup data breach just makes me wonder what companies are thinking?

I believe they understand the importance of security, but what are they doing about it. The tools to support what you need to do to secure your information are available. Security requires two things: focus and investment. You must focus your business on the requirement for security. Build the secure processes and enforce them. As Gene Kranz once said "Failure is not an option." People must be dedicated to an overall sense of information security.

The IOUG is currently running a security survey, what do you think?

Data is a corporate asset. It also contains sensitive personal information about customers, vendors, staff and corporate intelligence and we must make the investment. As a person who works with this type of information on a daily basis I never underestimate data security, but often companies do. They often secure externally facing data, but forget other points of access and ultimately face a breach. And what becomes of this data is the real fear. In the information age.... data is king. Time to roll out a security detail.

Data comes in many different forms and it is necessary to protect it throughout its lifecycle. You don't want to be the next headline in the USAToday

The State of the Oracle User

I have just returned from COLLABORATE 11, the Oracle user conference, and I have a lot of stories to tell. It was at THE event where the technology user group(IOUG), along with Quest and OAUG got together to share experiences with Oracle and Oracle-based technologies. I always believe if you are into Oracle technology there is no event like this one anywhere else in North America. This is where Oracle and users truly collaborate to educate the community.

The conference started in an inspirational manner, as Dan Thurmon, the author of Off Balance on Purpose spoke. He talked about how once we master something, we must challenge ourselves to change and get even better. Check out this video of Dan to see what he is about. It makes you realize that in today’s ever changing technology world.

This conference proved this to me. I even had a theme song playing in my head…. “The Times They Are A-Changin’”, in 1963 Bob Dylan seemed so insightful and this week was that very same to me. So on the week when I celebrated the same birthday as the song, I came the same realization. What is changing is how Oracle is delivering solutions. It showed that there are disruptive technologies which are ready for primetime.

I see two technologies which as Enterprise Data Architects we need to consider. First is Exadata and appliance solutions. And for me the second is Open-Source technology like MySQL. With Exadata, Oracle has brought together the best of Oracle database technology with Sun hardware to optimize data access. It’s not perfect, but it is coming and the wave is beginning. Larry Ellison just told us, that there is $2 Billion worth of Exadata orders in the pipeline. The secret sauce are the optimized disks and data retrieval. It was interesting to Ask Tom Kyte (at an IOUG lunch with Tom event), how he saw the Exadata technology is changing how we do our database deployments. He says its not quite as easy as the marketing team would like people to believe, but he did say that this machine in the right hands can achieve significant benefits. I also attended a presentation where someone basically used the Exadata very much like a traditional Oracle RAC implementation. So I begin to believe that the “Art of Exadata” will be the secret sauce. It will include people who understand architecture, database design, database deployments, SQL optimization and a little bit about hardware. Exadata’s time seems to have arrived

The other is open-source. Products like Java and MySQL are both open-source products which are now becomes part of the our corporate fabric. I am beginning to see much of the same needs we had 20 years ago when Oracle was doing the same thing for databases. The same management concerns need to be addressed to optimize the use of MySQL. Discussions on backups, optimization, replication and high-availability have become pressing concerns people need to address.

Of course the real value that I get from events like COLLABORATE 11, are the networking opportunities. I get a chance to meet and discuss interesting subjects with many people; each who add to much knowledge base. I get to know and understand how Oracle plans to move forward. One such example is Willie Hardie, the VP of Database Product Marketing and a great supporter of the IOUG, and who won the IOUG Ken Jacobs award recognizing the person at Oracle who has helped the user community in numerous ways. Here he is with IOUG President Andy Flower receiving his award:

We also had a chance to network in the World of Harry Potter and here I am getting on-board for the chance to share some fun with the user group.

And thanks to my friends at The Quest user group, who focus on PeopleSoft and JD Edwards who brought in Jim Lovell and Fred Haise from Apollo 13 frame to recount the experience from that flight, which blasted off in 1970 on my birthday, before encountering their infamous issues. Here they are today as they spoke during the event.

If you have never been to an event like this; it should be time to consider it. There is nothing like it. I made new friends and helped people who were attending for the first time feel welcome. It really is a community experience and one that Facebook or LinkedIn can not replicate. Plan to be at COLLABORATE 12 in Las Vegas, April 22, 2012, and get on the Hogwarts Express; it’s time.

Baseball and BI

The Major League baseball season started last weekend and the teams that should be winning are panicking (consider an 0-5 start for the Red Sox), while others are looking at what might happen after a quick start. The length of the baseball schedule, tends to be a big equalizer and as the Summer progresses into the Fall, the cream of the crop tends to come up to the top. So although the Toronto Blue Jays have started 4-1 and look to have some real pitching this year; I am not ready to get overly excited… it’s early. After all there are 157 games left. The playoff race is months away, but yet we get excited none the less. It is baseball season.

It is also the time when one of the most analyzed sports is changing the way they do business. I think I mentioned in the past that baseball is a sport that collects more information, which is at a very granular level, compared to other sports. It provides so much data that the science of sport not longer refers only to medicine but to data as well. Data may even be more important than ever in baseball and growing in all sports. Today, when someone \wants to sign a free agent or trade for a veteran or prospect player; the first thing a team will look at is their data.  Men like Bill James and Ari Kaplan have made analytics part of the game but also part of the business. By tracking how a player reacts to situations you can now extrapolate how they will fit into your type of offense and see in advance the benefit they will give to the team. The concept of “gut-free” still exists in baseball but data is quickly replacing it as part of multi-million dollar decisions.

So as we start the season we will be able to see if Mariano Rivera can continue to deliver heat in the strike zone, or the edge of the strike zone. We will that Derek Jeter likes to hit fastballs, but hits sliders even more effectively, all of this will be tracked. We all need to find the edge. Businesses have understood the value of data, but today sports are embracing them and changing the way they  make game and player decisions….it’s all in the stats.

Play Ball! Or Run the Report!

Go Expos!

Thoughtcorp or is it The Addams Family

This week one of my favourite people and client sent out an email about the people that work at Thoughtcorp and how we look like the Adams Family…. Now being someone who is being compared to Uncle Fester, I still think it is so funny. So let me introduce you to some of the Thoughtcorp Addams family. Thanks to John H for this fantastic comparison to the famous family.
First is me, Ian Abramson, as Uncle fester.

And next up is our Co-CEO David Bercovitch or should I say Lurch, or at least a shorter version of him.

And then is Val Hedow, of Director of HR, or as some will now refer to as Morticia…. It’s almost uncanny.
     
And of course, Gino Marckx, of Agile Practice Director, when it comes to Agile he is second to none, but amazing I think he will be known as Gomez Adams.


Finally the Dino Chronopoulos, the head of the Finance group at Thoughtcorp, seems to look a lot like Pugsley….interesting.

I love and To put a real spin on this. There are places where you have a chance to work where you can build or do great things, but you always have to remember to have some fun at work and let it loose. Thanks John for helping me understand that I now for the Addam’s Family.

Open Season on the Database

For years we have contemplated open-source databases and how they fit into our overall database technology strategy. The complexity of what is the right choice for an organization has many answers. It is no longer a simple question of choosing between Oracle or Microsoft or even IBM; no it’s a choice that needs to include MySQL (unless that counts as Oracle?), Postgres, or now Hadoop and Cassandra. The choices are as complex as the systems we need to support.  From my perspective it is now open-season on the database. The choice is no longer straightforward.

The emergence of “The Cloud” and the need to distribute more data more effectively has added new pressures. The need to analyze massive amounts of data more efficiently is a must-have. We look to the Oracle’s of the world they respond with a system that is scalable, efficient and leverages the power of Oracle to enable the business. The Oracle database on an Exadata server is a solid enterprise-grade solution that will support this need. It is robust and powerful, but it is costly. So organizations who need to similar functionality but want to do it using open-source technology or some lower cost-model of software now also have options. At Thoughtcorp we are leading providers in many areas of technologies and our team has varied knowledge and much of it focused on today’s latest technologies including wireless, apps and of course using open-source technologies to achieve their goals. The data group has seen this shift as well and now I am working on how products like Hadoop, which can be used for massive data analysis. It is like Google file system (GFS) and allows for the collection and analysis of data. From Hadoop other solutions have emerged. Hive provides a easy to use tool to work with Hadoop to simplify analysis. Now Hadoop has been combined Cassandra and open-source database from Facebook. What does this give us? A real-time database with big data capabilities. Wow…that’s a mouthful.

Of course these open solutions come today with some operational risks due to single possible points of failure. of course if you can manage the risk. You may have an alternative to today’s  enterprise databases. These new data options provide us more with choice and options and in a world where you consider…. If it’s good enough for Facebook and Google, is it something I should consider?

Building Data Warehouses for the Masses

Often data warehouse designers and architects are often accused of building systems which do not always serve the enterprise but only focused portions of the organization. I have seen this occur at numerous clients who did not invest in a long-term vision and long-term data strategy.

So how do we avoid this trap and ensure that we produce a system which does not go “silo” but sets a course to meet current and future needs at the enterprise level and not departmentally. I was recently working on a presentation about data warehousing design and the question of Ralph Kimball (Dimensional) versus Bill Inmon’s (Normalized) design approach of which is better. The question is that both work, if you keep to the big picture. Design your high-level design upfront and adhere to design and development standards, It is about setting a clear course for DW design and never allowing for an independent or silo’ed solution to be developed. The key is to always look for integration opportunities. Add to your core data warehouse; don’t built a new structure which is not tied in and used conformed dimensions.

Bill Inmon, said in referring to Dimensional modeling that “A 1000 minnows do not make a whale”. Build with a great purpose in mind  that you are building an integrated, stable data repository to support business reporting and analytics.

Data Warehousing and the DW Appliance

Today I attended a seminar presented by Oracle about the Exadata database machine. As usual it was presented as a compelling argument that a complete solution like a data warehouse appliance is one that ultimately reduces complexity and increases capacity.

During the session today, they quoted a study from TDWI which says that within 3 years 78% of respondents expect to be using a data warehouse appliance. I find this a staggering figure. Consider that today DW appliances have evolved and each of the major database vendors have their own flavour, but we still continue to see that traditional databases significantly outnumber appliances. I do admit that the argument for an appliance is one that needs to be considered. The idea that a single server or linked servers can come into a data centre and within days can be productive is incredible. The bigger question becomes which appliance do I choose? Do I buy Exadata, Teradata, Data Allegro, or Netezza? I guess that answer depends on what you want to achieve and the basic knowledge and comfort level you have with each vendor.

I find that the real differentiator is the functionality available within the appliance. As you might know Netezza uses Postgres as it’s base database. Although a great database, it is provided within only the basic SQL set of commands. To extend it out, you need to buy additional options and features. Then you look at a database like Oracle’s which has 30 years of development in it and now we are presented with a database that supports very advanced SQL. Oracle has provided an appliance that runs the database. It has not created a database which happens to run on an appliance. I often wonder how many people are aware of the advanced functions and analytics available in the Oracle database. The database has a significant number of statistical functions from the simple to the esoteric. The database contains a number of data mining functions as well. Although Oracle sells a great front-end tool in Oracle Data Mining (ODM), the functions which the tools use are available to all database users. So when you start to look at what appliance, you need to look at more than simple administration and technology, but at capability. The appliance who can do the most without any additional costs seems like a solid option. Now as to whether or not we will see a big increase  in the use of appliances will depend on how well and how cost effectively these appliances can replace our current hardware platforms.