Data Quality Using R

Data Quality is a valuable approach to making sure that your data meets your needs and provides you with reliable information when it is needed. The problem is that most organizations are hesitant to embark on the data quality journey due to the costs. The challenge is how can we reduce the cost of data quality? Most organizations realize the value data has but the cost of software, people and time, end many initiatives before they begin.

Companies put together data management plans which require them to make investments. These investments tend to be in the area of data quality tools. Organization consider Data Quality products from Informatica, SAS, SAP, Trillium and Oracle, all provide tools that support this need using a well curated approach. These tools are add-ons or options to existing ETL toolsets, and generally they are expensive and complex to implement. However the advantage they bring is tight integration with ETL tools. Some companies may find that the barriers to entry are too significant and therefore often ignore or reduce their data quality efforts.

We need to find a better way or at least one which is more cost effective. Many enterprises today are enamored with open-source solutions. These are the companies who are actively embarking on open-source projects like Hadoop and the various projects within its ecosystem. One of those tools which has emerged as a tool that is commonly paired with big data is R. As defined by Wikipedia and the Inside-R Community; R is a programming language and software environment for statistical computing and graphics supported by the R Foundation for Statistical Computing. R is a programming language: you do data analysis in R by writing scripts and functions in the R programming language. R is a complete, interactive, object-oriented language: designed by statisticians, for statisticians. The language provides objects, operators and functions that make the process of exploring, modeling, and visualizing data a natural one. Complete data analyses can often be represented in just a few lines of code. The challenge is that R is limited by memory. So you do need to consider this when you choose the right sample for your tests.

At the recent Collaborate16 conference, I attended a session by Michelle Kolbe, of Red Pill Analytics, about using R for Data Quality. This idea was one I thought to be an eye-opener. A viable solution to our problem was right in front of us. She proposed that R provides the necessary features and analytics to support a robust data quality approach. R packages which are applicable to data quality include summary, distribution, histograms, regressions and dataqualityR. These packages allow us to profile data quickly and easily. The summary package, profiles each column in the file or table and provides information including min, max, median, distributions and other useful measures. By combining graphical and text mining packages allows for advanced visualizations like word clouds and clustering which can be more telling by providing visual clues leading to outliers and other questionable data. This functionality within R can empower a Data Quality program, by defining a standard framework, one can build a library of data quality processes with you only having to provide the file or table you wish to analyze. 

The choice to use R will probably be one which IT will initiate and is one which should support business needs and data requirements but it will not serve as a stand-alone solution rather a solution. We can look at R as a low cost alternative to the mainstream DQ tools, but it is not a complete solution and will require a data processing product such as Python. The idea I propose is that R could act as the initial tool to profile and identify data issues.  With a construction of a framework to make it easy to use, you will have the start of an advanced approach to quality. 

The age of Big Data has created the opportunity which R presents and you should consider it for your organization. Data Quality is an ancient problem that we have news ways to find and fix an old problem. Recently the CTO at Canadian Tire stated “We aren’t interested in best practices. That’s for our competitors. What we’re looking at is next practices.” This is how we need to look at R; it is the next practice which may become a best practice.

If Someone Says “I Think”; Tell Them to Prove It! – IOUG Real World Performance: A Review

Seems like an odd statement. We always hear people say “I think…” and then they go on to tell you what they have hypothesized. Now you wonder is this story factual? Does it REALLY work? It is then time that you need to ask them to prove it. We don’t often challenge our peers to “Prove It”. We accept thoughts and then experiment on our own to see if this thought proves to be correct.

This past Friday I had the chance to attend the IOUG’s Real World Performance seminar which was held in Toronto. It was at this seminar led by 3 of most knowledgeable Oracle people in the world, where the proof showed us how to change the way we think. As they say there is no reason for things to run slowly today, but that instead that people have made the choice to run this way and not take advantage of new technology and new methods.

Andrew Holdworth, Tom Kyte and Graham Wood, led this amazing day of proof.  The image below is a picture from my chair… of the 3 of them up front in the room(as they were all day)

It was a day where the 3 Oracle database performance gurus showed us a new way to look at performance. They discussed data warehouses and operational systems. So what were some of teh highlights:

• Showed today’s data warehouses have undersized CPU’s and insufficient I/O
• Improve data loading, as data size will continue to expand
• Stop using SQL*Loader, it’s time has passed. Consider external tables
• Read compressed data if possible
• Statistics are critical to success
• Make sure cardinality is accurate in table statistics
• Use SQL*Monitor to investigate performance of SQL
• Manage your database and system resources. Use Instance Caging if needed
• Used set-based operations as often as possible
• Check your SQL statements for issues

The biggest thing takeaways I came away with is that today we should try to achieve full CPU utilization, we should optimize how we transfer data into a data warehouse, we need to make sure our SQL statements work as expected and that indexes and partitions can be your friends and your enemies.

It is amazing to me to see how we things have changed over time. Today we need to manage our data better, our databases more effectively and the improve the process of loading and retrieval. The tools are there for you, we just need to use them. As Tom Kyte said, “Why would people choose to run their database slowly?” 

There are more of the IOUG Real World Performance being held this year. If you get the chance to attend, you should; as it can change your life or at least the life of your database.

Oracle Users and Customers Back Oracle at the EU

As I have been discussing over the past few days; I have been deeply involved in helping Oracle respond to objections that were made by the European Commission surrounding the loss of competition in the database market place by Oracle acquiring Sun. The experience has been one that has proven to me one important factor; computer people are passionate. Sure we may not know how to say hello to a girl, but when it comes to the software we work with on a daily basis we now how to get passionate for that.

I became involved about a week ago after receiving a call from Oracle’s CCO; Jeb Dasteel, from there the experience began. I have already talked about some of the anticipation and if you have been following on Facebook you would have seen commentary I provided during the entire experience, as much for the Oracle community as my kids who are busy studying for finals at McMaster University. Give me a break I’m a proud Dad. So how did things go?

It all began Wednesday afternoon in Brussels, Belgium with meetings at Oracle’s lawyer’s offices, which conveniently were right next to the Conrad Hotel. (BTW, nice hotel and it just so happens that the Italian President was in to speak at the EC and he was staying on my floor, we had a quick spaghetti party on 6.) So, off to the lawyers to discuss and finalize our presentation for the next day. It was determined that since the IOUG and the UKOUG (represented by Chairman Ronin Miles), were both there that we should combine statements. So for the entire afternoon we reviewed the updated materials and took it even further. I expect to have the slides posted in the next few days.

You should understand that these hearings are held in private and without the media. However, I asked the folks at Oracle if they felt this might be an issue and they did not. I can provide you information on what I said but not what I learned during the hearing as they are confidential.

So we were now ready. The power team that had been brought together to address these concerns were all top-notch. Oracle brought its President Safra Catz and the top legal minds inside and outside of Oracle (could have been 20 lawyers in the room, just on Oracle’s side). There was the leadership from Oracle EMEA with Loic le Guisquet, the person who runs the entire practice in Europe, the Middle East and Africa, and a couple of other very senior people who really helped me directly on getting our message together, including Doug Kehring and Wayne Waters. There were economists on both sides. Not sure what economists really do, but they had a lot to say. On top of all the Oracle and legal muscle, there was Miles and me representing the great user community as well as a number of Oracle’s biggest and brightest customers.

The session room was filled to near capacity. Here is a view from my seat in the hearing room.

It included the Competition Commission and the case team working on this file. In addition to what seemed like 100 people associated with Oracle, there were people who were supporting the objection. Some of those groups included: Microsoft, SAP and MariaDB. As well there were representatives from 11 member states of the EC and one guys from US DOJ, I assume they all were here to watch this historical purchase work through the process All of these had a stake in the decision, but for different reasons. Rumours were even swirling that SAP might announce that it now runs on MySQL. I never heard if it does as they spoke on Friday and I assume they were announcing that it was certified to run on that database platform. I only attended the session on Thursday when Oracle made its primary case along with Sun. There and was a great job explaining how the MySQL product really works and why this objection may not make sense. The first was the idea that the EC’s objections may not be based on fact but in an interpretation or twisting of facts. One concern that I heard and stated at the session, was that the commission had questioned the validity of the hundreds of letters sent to the commission by customers and users and user groups. I can reiterate again, that the letter the IOUG sent was created by us without input from Oracle. They provided us a sample but our letter was done by the time we received the sample. Basically we used the address for our letter....over and over each of us pledged that we wrote these on our own without prodding. If it was like the IOUG; we began talks about objecting to the objection as soon as it was released, I expect most people were the same. Rumour even had it that this case may have garnered the most email and direct mail support of any purchase/merger ever for the EC.

The discussions were definitely heated at times. Questions and answered were almost like bombs being thrown from one group to the other. The Commission asked great questions and Oracle’s team were able to respond clearly and quite reasonably. There were many questions that included some emotional concerns, people need to remember that this is business.

As I said during my presentation to the committee, IT professional are innovators and rebels and that nothing that Oracle would do with MySQL would change competition. If someone in IT sees the opportunity to create something new or create a fork of MySQL. The chances are there and it will not be stopped. 

I just found the whole process where your accusers are also your jury an interesting concept but this how government work. I don’t know how this will turn out but I can tell you that it is far from over and I expect Oracle will fight this until a just decision is made. Considering that we expected that the deal to close in the summer, it is becoming a real issue. Decisions that need to be made cannot due to the uncertainty. People need to know what the company plans for its staff as it is integrated. This delay is costing people, companies and confidence. I hope that by being there as the President of the IOUG, a user group member, as a Thoughtcorp employee and as an Oracle customer and user, that in some way we have helped them see the light and let this deal go though.

The Adventure Begins

Today is the day that I have had little time to really mentally prepare for as my trip to the European Commission. I still find it amazing that I have this opportunity to speak as part of the government process to complete this multi-billion dollar deal. Somehow I am now in the middle of it and today is the day that I finally fly to Brussels.

The entire trip to Brussels is a first for me, I have been many places and seen many great things. I find that breathing in a great city from its architecture so important when I go to places with great history and Brussels seems to have held onto its ancient roots. As for my presentation. Another first, I have spoken in front of many audiences in my day but for most of those talks it has generally alright to open with a joke. I expect that after the team gets through reviewing my content they may decide to pull the joke and go with the more serious stuff.

I am actually very exciting to present some preliminary information about the types of databases that people are running. The IOUG along with Unisphere Research we have been conducting a survey on virtualization and we have some results already that are interesting. Turns out that of the people who took the survey close to 90% run Oracle as their enterprise database and of that almost 90% of those companies also run MS SQL Server and then 44% use MySQL. The results are interesting and deeper than this but some of it will form the basis of some of are existing arguments. I together with Ronin Miles, the Chairman of the UKOUG, will present the position of the user communities and it all begins in just a couple of days.

On the lighter side I was thinking about some of the great adventures and places that I have been part of my life already. And in no particular order:

• Discovering Israel (wow….that’s history)
• Exploring Rome
• Living in London for 6 months
• Exploring the ruins of Pompeii
• St Petersburg
• Dubrovnik, Croatia
• Attending the 1976 Olympic in Montreal
• Meeting Prime Minister Pierre Trudeau
• Being at the 1994 NY Rangers Game 7 win in the conference finals
• Nos Expos (enough said)
• Attending 3 Stanley Cup parades
• Being there as my home province of Quebec was considering separation; I heard one of the greatest speeches in Quebec history as Rene Levesque called on Quebecers to rally after the loss for the separatists
• Seeing the kids grow up….that’s the greatest adventure
• And now on to: Speaking at the EC

I am sure there are many others……it’s amazing to look back and see where we have been and how much we have already done. Of course there is still so much to go.

Another Day Closer to Brussels

Friday and another day closer to me going to Brussels to speak at the EU. In a way it seems a little surreal at this point in time.I continue to work on my presentation to get ready for next week and still have to do my day job to do and boy are things getting busy these days at Thoughtcorp as we build momentum in the Enterprise Data Group during this time of a BI explosion. Always good to be busy.

My logistics are done and I am looking forward to seeing Brussels if only for a short time. I’ve been to Belgium before. I plan to see what I can in the evenings, this is a country with long history and we are going to be in the core of Brussels close to the old city. There are a number of UN Heritage Sites in the city which I look forward to seeing during the holiday period. I always loved the way London comes alive during the holidays. I seem to always time my European trips during the Fall and Winter, not sure how it works out, but one day. A few years back I had the chance to attend the UKOUG conference in Birmingham and if I am not mistaken it rained 8 out 9 days. A perfect week in the UK in December, so hopefully it will be better next week in Brussels.

One thing that has been interesting to see the reaction of people when you let them know that you are going away next week. This must be important as I have to miss hockey next week and I don’t like to miss hockey. Being the goalie you need to be there. So I told the team last night, and this is a team where I am on the older side of the Bell curve and most of the guys are in their 20’s. When I tell the story of how I was going to speak at the EU, the reactions ranged from disbelief to amazement. I play hockey with these guys but they don’t really understand what I do in my job let alone the IOUG. I am still trying to figure out how I am going to be part of this process but one thing is that I am going to soak up the experience.

Right now my wife is planning her days while I will be sequestered at the EU. Belgium and Brussels of course is known for its chocolate. I think she plans a tour and tasting of each in the city. And Belgium has a connection to Canada via the poem that in Canada symbolizes the sacrifice of our soldiers have made for us and I can remember hearing it every Remembrance Day growing up. We even have a part of the poem on our $5 bill. Finally they tell me that I need to eat waffles, French fries and beer……so I look forward to a fun gastronomic experience.

So the wheels are in motion and the experience of a lifetime is getting closer. At least I have two hockey games between now and when I leave to keep my mind clear. The weekend pack and plan before I know it I will be looking in the souvenir shops looking to add to my travel snow globe collection.

Standing Behind Your Words

I am about to embark on an experience which I would never have imagined would have been on my path in life. Next week I will be going to the European Union’s Competition Committee to present the position of the IOUG on the pending acquisition of Sun by Oracle. As you may have already read previously in my blog; the IOUG’s leadership is of the opinion that the purchase will not affect competition in the database market. We feel that the technology community are innovators and they will respond to the needs and desires of the marketplace.

So I have been asked by Oracle, on behalf of the IOUG, to go to Brussels and be a part of their presentation to the EU. This has all evolved very quickly. I have to thank everyone at my company, Thoughtcorp, for their support during a hectic time. To me it illustrates the leadership that companies like ours can show in today’s diverse technology landscape.

So believe it or not I am flying to Brussels (Home of the EU) next Tuesday (arriving Wednesday morning) and then on to lunch and meetings…..wonder if they will be serving waffles?

This entire process is one that illustrates to me how you need to be prepared for anything when you make a statement. You need to be confident in yourself and the great team around you to make such a curveball into one of the great experiences of ones life. My segment before the commission will be around 15 minutes and basically present the position the IOUG published in our recent press release. I have to thank the Board and staff at the IOUG who all took part in building our position on the subject. I am ready to tell the EU why the IOUG made the statement that we did and why we support the deal, sometimes you need to stand up for what you feel is right and this is about as big a political stage they could find.

So here I go; I get to be part of something that for me may be a once in a lifetime experience. I am excited and anxious about how it will all come together, but I am confident in my knowledge and abilities to represent the IOUG community in a responsible manner. After all this is done, it would be nice to say that the Oracle database user community stood up and made a stand and helped to influence the direction of tomorrow’s technology landscape.

I look forward to sharing as much of my experience with you as I can; as Ian’s Excellent Adventure begins.

Responding to the EU Objection to Oracle’s purchase of Sun

The past few days have been an exciting and interesting series of events. Recently the European Competition issued objections to the purchase of Sun by Oracle. Remember that this deal has already been approved by the US Competition Commission. So what’s the hold-up?

It appears that the EU feels that Oracle acquiring Sun will limit competition in the database marketplace. On the surface this seems like an valid observation based on market share alone. Consider that Oracle is the #1 database in enterprise environments and that MySQL is leading in smaller database environments. So that means that Oracle will be the #1 database vendor. Does it limit competition when you dominate the marketplace? Does it stop companies like IBM, Microsoft and others to develop and innovate in their own product stacks. The IOUG Board discussed our response and it was a consensus opinion that the acquisition of Sun and MySQL will not limit competition. Considering that we are the user group who focuses solely on Oracle database and middleware technology that we were well suited to respond in a meaningful way.

We feel that the delay is causing even more damage by adding a level of uncertainty to the company’s future and will impact profits and dollars. Considering that Oracle has purchased other databases in the past illustrates how well Oracle continues to support and innovate in these various DBs. Oracle purchased RDB years ago, and more recently TimesTen, Berkeley DB and Essbase, and each have been supported and extended. Oracle purchases products that help them in being leaders in both proprietary databases as well as open ones.

For me this is really a question about innovation and not competition. As we often say…”If you build a better mousetrap…”, and it is not different in this case. MySQL was created as an inexpensive alternative to the big databases. It was built by a private group named MySQL AB who felt that a small cheap and sometimes free database would be able to grab market in the competitive database field. Somehow they did capture marketshare from companies that felt they were untouchable. To go on and become the leader in many verticals especially on the web. So where is the next MySQL and how will step up and compete and how does this purchase stop it. Competition is driven by innovation, differentiation and market needs, and the IOUG feels that the EU objection is ungrounded.

The IOUG at Oracle Open World 2009 – A Retrospective

A week has passed since I was last in San Francisco and I felt that now that I was over jetlag it would be a good time to look back at how the week went from mine and the user perspective.

The week for me has always been about networking and sharing and this week was no different. Although it looked like attendance was down, it was great to see the people who could make it plus the lines tended to be shorter. Which when you want to get to the corn dog at the Big Bash it was a good thing. The other advantage is that this was one year when I did not hear about people being unable to attend sessions due to room capacities. The people really do make this event. Below is a picture of me and a couple of IOUG Board members, Michelle Malcher and Todd Sheetz, at the IOUG booth at Open World. They along with the rest of the Board really did such a great job with the IOUG led programming that I am sure it helped everyone who attended one of out sessions.

The week as with most OpenWorld was about product direction and product understanding. I think as always Oracle presented their direction clearly for what they know and can talk about. The question of the finalization of the Sun acquisition was  handing like a big lead balloon over the entire event. I expect that when the EU finally allows this deal to go through, Oracle will have a strategy that will quickly integrate both businesses and communities together. The IOUG had some meeting with some folks from the various Sun communities and I feel that we have a number of synergies that will help both groups to benefit from the additional topics that become part of technology education.

I also met with a number of senior Oracle technology and Business Intelligence executives to see how they planned to focus their products in the next year and how the IOUG can help our users get ready for this technology. The focus for 2010 includes BI and Analytics and we are focused on providing the best Oracle BI event in conjunction with COLLABORATE10 event coming in April 2010 in Las Vegas. BTW, if you want to submit an abstract for COLLABORATE10, today is you last day….so click here to submit and abstract. The focus on driving down costs was clearly stated by all of the Oracle folks. The key is that through better more effective use of shared and open technology you can drive down costs. From a DW perspective the Exadata and Database Machine seems like the flavour of the day. I realize that this is some great technology and can provide some customers with some real performance, so don’t get me wrong about this, the other DW appliance manufacturers must be concerned. The latest version of the machine from Oracle and Sun show us where this company is going and how technology will make a difference. The speed of the basic machine along with Flash Memory provides the power not only for data warehousing but now for OLTP as well. Ray Roccaforte talked about how he expected this flash memory improvement to help DWing in the future, but for now it provides OLTP with the additional power of The Database Machine. I have to admit when I first discussed this machine with the VP in charge of this, I said I was surprised that Oracle had not gone after the OLTP market with this approach and now a year later here it is.

Lots of discussion around OBIEE and ODI/OWB, since this was my focus for the week and it was great to see how people are truly exploiting this technology. Considering how late Oracle arrived into the BI and DW tool game, they are making some solid moves into this domain and their tools can really provide the performance and results that companies are looking for.

Also met with some users involved with Z/Series Linux, a group that runs Oracle on Linux on IBM mainframes. This is a small but passionate group that was looking to the IOUG to support their networking and educational goals, and I expect that we will see many of these folks in Las Vages at COLLABORATE10 as we develop a conference-within-conference for the community.

The last evening was Aerosmith, Roger Daltrey, The Wailer and 3 Dog Night. I made it to 3 of 4, only missing the Dogs. Aerosmith was its usual polished self, with Steven Tyler being award and IOUG Lifetime achievement award. Daltrey was the biggest surprise as he delivered a great set of well known and lesser known Who material. He was in fine form and he voice in its 60th year was able to keep up. The Wailers were awesome. I grew up in Montreal with the music of Bob Marley and The Wailers and I had seen them perform before, so it was great to hear the wonderfully energizing music fill the San Fran nights.

The show wrapped and by the end of it many of us were happy to return back to share the word. I expect that 2010 will be an exciting year whether you are a Oracle or Sun customer, as we learn how the next evolution for Oracle and the user communities.

User Groups Kick Off Open World

Today in San Francisco begins the annual Oracle led customer event; Oracle OpenWorld. The conference is one where people can learn Oracle’s direction for technology on the near and long-term. Sunday is a very valuable day for people who attend the event. It is Sunday that is day when the user communities run sessions that are technically focused on various components of the Oracle database and middleware.

The IOUG is front and center in this activity starting the day with important topics including security, business intelligence among many others.  Of course if you are here come by the IOUG booth in Moscone West (2nd floor) and meet others with similar interests as you.

I started my Oracle Openworld experience with the security topic. The session is a panel discussions with individuals from the user community sharing real world experiences and with the people from Oracle who are responsible for delivering the security technology to us. Below is Andy Flower of LoganBrittan and Tanya Baccam of Baccam consulting that re giving great insights into the current state of security and providing tips on how to improve it.

It is interesting that current issues that people are experriencing and little people are doing with regard to security. The IOUG survey has illustrated some many issues that people are experiencing. With a little work DBAs and SysAdmins we can do a lot. Organization need to look to encryption, audit and options like Oracle’s Data Vault.

One small question I have….When is a blogger not a blogger? The answer is when Oracle decides that a Blog is not a Blog…so my adventure starts without Blogger credentials…let’s see if I can beat the system and still get they access I need this week...that’s all for now…..and on to the rest of the week.

The Best You Can Be; Can Be the Best You Are With

I always look to Friday as a day that should be something more than work. It is a day I usually reflect and look back at how the week was good and how I could have made it better. This week I realized something about team and being a teammate. I know I have gone on in the past about team work and my hockey team, but today I realized something about work that sparked today’s blog entry.

I work for Thoughtcorp, we are a reasonably sized solutions company based in Toronto. These guys have been around a long time and at one point were a competitor of mine, so this type of business is nothing new to me. What I realized this week is that this is a team with great talent (which I already knew) but what I didn’t get this week was the energy this team had and the focus it puts into doing business. I realized that some of the old companies I worked for had lost it and I too missed the signs that it had happened. The concept of a team in those organization was lost. We lost the ability to motivate each other, we got into a rut. Those companies may not even realize they have lost the internal challenge to be great, but I see it now. It is critical for an organization to survive it needs to be its best. the motivation I found this week made me realize that work again is like hockey and golf

How do we get better at any sport? We learn from and play against people who are more skilled than us. Taking up the challenge and not shirking away from it is how we can harness the motivation to be better or at least better than our competitors. In a team environment it becomes a way of making the entire team better….if you don’t constantly challenge yourself to be as good as those around you you will never be great.

The user community is the same way in an Oracle sort of way. I look at how my career has been impacted by the user community. Would I have authored a book? Become President of the IOUG? Helped to create two companies? Created solutions that matter? Maybe but I feel that my taking the chance to take an active role in the IOUG has made it all happen in some way. It was the motivation of meeting the people in the community and learning what was possible. The people in this community are leaders and by using them as examples and mentors the value to an individual and organization it limitless. We need to use this motivation to make us better individuals. It seems so clear and easy but you need to see it and realize the value to all those around you. Then you become the motivator for others. Make it a great weekend!

Oracle 11g Release 2 is Coming

Being part of the user group like the IOUG does have it's privileges. One of the things we get to do is take part in the Oracle database Beta program. As most people know; Oracle has been feverishly working to complete the latest release of the database (Oracle 11g Release 2). The user group was able to help Oracle test this release of the database, by suggesting customers who wanted to take part in the Beta testing. This is truly a value to the individual, their organization and Oracle. Together they work to bring a solid product to general release.

So what is most compelling about the upcoming release of the database? Of course I can't answer the specifics of that until Oracle releases the product some time in the near future, but what I can say is that this is the release of the 11g database that people should find to provide the stability and performance. The old joke goes how can you tell if someone is a Junior or Senior DBA? The answer goes that when the first version of the software arrives the Junior DBA wants to install it straight to production and the Senior DBA throws out the CD and waits for Release 2. Sadly the reality of this joke has some truth in it as organization often wait until the second release before installing the latest product. So here we are with Oracle 11g Release 2, and based on what I have seen I would suggest that as the information about this release is disseminated you should start to consider the adoption of the this release. It will increase performance, increase features and increase the return on your database investment by doing things better and smarter.

It is time to put on your Oracle Database 11g radar and start collecting the information that will help you decide if the time is right for you to enter into the 11g universe and start to using it to help your business be better and more effective

User Group and Oracle Loyalty

Recently the IOUG and Oracle embarked on a survey that collected the value that people receive from user groups like the IOUG and Oracle. I mentioned this survey previously and I hope that many of you took the time to complete it.

So what did this survey tell us? What did it tell Oracle? It told us that generally people are very satisfied. The survey showed that 80% of people who are associated with one of the participating in a user group are Very Satisfied with this relationship. This is no surprise for people who are already members, and has now confirmed this with Oracle.

So what did you identify as the most valuable reason for being a member? I think that question is quite obvious; it is Networking. Again over 75% of respondents said that Networking was a very valuable benefit of being a member. In addition similar numbers were seen in the Education provided by the user community. This confirms what we see as being our value. The user groups are the best place to go to meet people who have similar Oracle interests. The value of in-person and virtual networking are two of the foci for the coming year for the IOUG. The only place you can get unbiased information about Oracle technology comes from the user communities and this survey has confirmed this value.

The key is providing value. People today understand that they need to do more with less. So user groups need to provide this type of cost-effective education. The IOUG is looking at working on a BI event in the coming months. We expect this will be a virtual event in the late fall and one that will be delivered at a very advantageous price point. The IOUG is listening and our program is changing to meet our member needs.

One interesting tidbit that as mention today was mentioned was that according to some folks at Oracle, it seems like User Group members log less Service Requests for support. So now how will Oracle help the user groups is the question as we already know how user groups help its members and Oracle. As some people say “Membership Has It’s Privileges”…..take some time and join in. (www.ioug.org)

Communicating Success

The fact that we work close to people on a daily basis does not necessarily mean that we are communicating on the same levels. We need to consider communications internally and externally with a project team as one of the critical factors to a successful project.

We often work as part of a project team that needs to become the well oiled machine where all are participating actively in all aspects of the project; especially communications. Projects often set up communications plans that include a number of items. These plans will include regular meetings (daily and weekly), project dashboards, project wiki’s and other documentation and methods. These methods of communication work. A project that has clear communication plans and processes are projects that have a communication foundation for doing this well.

The key to any plan that you may enact really require cooperation rather that communication. The idea is that people need to understand the value of this type of information exchange. The team must be willing to share knowledge and information. The members of the team must be ready to share success and to know when to ask for advice. So it important to provide a facility where this information exchange can occur.

Communications are not simply about telling people what is going on and what is going to happen but it should also include a component of knowledge exchange. During any development project the team members must be willing to share information about the better way to do things. The ability to share best practices at a project level can be a critical factor in a project achieving its goals. We need to make sure we are not simply informed by that we are more knowledge about the project and the methods that we should use on each and every project.

Good communications during a project will help you and your project succeed. Just remember talk about things and make things better. The only real problem or issue on any project is one that is unspoken so make sure your project team is open and involved.

The Data Warehouse Team

Building the right team is a subject that I have discussed here before, but I wanted to talk about how to put the right team together and what skills are needed to build a data warehouse or business intelligence solutions.

So who is on the team?

• Project Manager
• Data Warehouse Architect
• Business Analysts
• ETL Lead Developer
• BI Lead Developer
• QA Analyst
• Various other developers and analysts.

So where do we start? The key to me in any successful data warehousing project is having someone who can be the Data Warehouse Architect. This is the person who will drive the solution and ensure that the most efficient solution is implemented. This person must have the ability to understand DW/BI technology and understands how to translate business needs into a solution. This person will directly impact the project’s success so choose carefully and choose experience.

DW and BI projects need standard project management controls, as many projects are complex and have many moving parts. This person should be knowledgeable in the DW and BI arena. Just another project to manage.

The ETL and BI Leads are also critical to the ultimate success of the project. These people will create and establish the approach that will be used to deploy the system. These people are generally the most experienced technology people on the team and can provide leadership to the rest of the development team. These people may also be development leads, but generally we try to focus these roles on ensuring that development is executed as needed.

Finally you need you QA, analysts and developers who will fill the remainder of the team.

Ultimately the team will need to work well and the leads are important people who must ensure that communication occurs between various people so that everyone understands their role in the overall project and can see how it contributes to success.

So build the right team with the right people who want to build solutions that provide organizations with competitive intelligence capabilities.

IOUG Forum at COLLABORATE09 (Day 1)

Wow...finally. The latest and greatest Oracle user event has finally started. It is amazing to see all of the folks that are here. There is nothing that truly replaces the face to face interaction that these types of events offer.

The conference kicked off with the general session that was delivered by Charles Phillips. Of course the Presidents of each user group (me for the IOUG, Ray Payne for OAUG and Sue Shaw for Quest) were all up there introducing Charles and thanking everyone for coming. Considering the reduction in business travel these days we appreciate the folks that were able to attend.

Charles Phillips opened the event by discussing numerous items including new release of Beehive (Oracle’s collaboration tool), BI Applications, more and free extended support of Oracle products and a new eBusiness Suite release. And Charles restated Oracle’s support of the user community and its shown by their active partnership at the event. The picture above are all the user group Presidents (courtesy of Peter Smith) doing our introductions.

In addition the IOUG is offering to all those individuals who could not attend in person the chance to be part the event. We are hosting 35 sessions that run throughout the conference to people in-person here and streaming it in an interactive manner to people in the virtual world. This is the first hybrid event and on day one it has been working great.

On to a BI Panel for the BIWA SIG. Later

Oracle acquires Sun

Mondays seem to bring a lot of things. I met someone this morning dreading the start of the week as she was off to a meeting. I suggested that it was a great chance to get your week off on the right foot. It was an opportunity. Then I arrive at my desk and open my email and what do I find that Oracle has purchased Sun for $7.4 Billion. This is what makes today a great opportunity.

So today starts a new evolution in the Oracle world. For years Larry Ellison has created a company that provides almost all of the technology stack. They provided applications, middleware and database. To a certain extent they also supported hardware by providing Linux support and the launch of the Oracle/HP Database Machine. This now changes a lot of things. Firstly Oracle now like IBM can control their entire technology stack and provide single-vendor solutions from soup to nuts. How will this change the direction of the Exadata is not known, but I expect that in some form it will continue, it looks like we should expect some changes in this area.

Of course one of the biggest prizes from Oracle’s perspective must be the open source components of the company. Oracle now becomes the Java company, they have always been a market leader in the area and now will own the basic building blocks. Consider Ellison’s statement from this morning conference call announcement where he said `` Java is the foundation of Oracle’s Fusion Middleware and the single most important software we have acquired``. In addition there is MySQL, what will happen here? How will Oracle deal with this database which is a rival of its own?

This acquisition raises some interesting scenarios but at the basic level we are look at what Oracle strategy is for buying companies is are the best. Larry stated it today by saying `` One of the reasons Oracle is so successful is that we buy companies that are number one.``. We have seen it with Peoplesoft, BEA among many other where the company was the leader. It will also be great to see how Larry Ellison and Scott McNealy get along now that they have to work together everyday? These pioneers have always seemed to be allies; now that Scott works for Larry it will be fascinating to see how it works out. These guys have lots of energy and are superior thought-leaders, I expect that this combination could be a very powerful and visionary one, or it could upset the balance of power in the executive ranks. Acquisitions can be a tricky thing.

From an IOUG perspective we look forward to welcoming the Sun technology community. As this acquisition progresses the IOUG will keep everyone in the loop. It will be exciting to see how the to this purchase will change things at Oracle.

I love Mondays...you just never know what can happen.

Lots of Anticipation

With spring in the air I am filled with lots of anticipation about a lot of different things. This is a time of renewal and a time when we can leave the cold and look forward to some heat....at least in Canada.

What am I excited about at this time of year? First off is the upcoming IOUG Forum at COLLLABORATE09, where the best and latest techniques will be presented by the most renowned experts in the world. Lately I have talking to many people about the upcoming event and the great content and networking opportunities that attendees will have. This is our best chance to get together and I hope that at a time of economic stress people see the value that an event like ours provides to ensure that you keep your jobs and help your organizations survive the challenges.

Then I am excited about my current project. We are about to deliver the design for our solution. It is a solution that provides the customer with a great framework to deliver the information and value that they need. In addition the use of Oracle 11g and Oracle Warehouse Builder (OWB) is an exciting technical implementation. In the past 2 years, I have seen OWB going from being nowhere on the local ETL scene to today where Thoughtcorp is involved with 3 different organizations using OWB. The emergence of the product is exciting for companies looking for cost-effective and productive ETL solutions.

Finally, the excitement is for the start of golf season. For those of you who live in climates that don’t allow you to play golf 12 months a year understand this form of anticipation. The grass is turning green and I prepare to hit the links. Over that past few years I have encouraged my passion for the game. I enjoy the game, the company and being out on a great course. My game provides lots of potential that if and when I can harness it can actually look like golf. My golf course has yet to open but it should be soon. This is the truest meaning of anticipation.

Oracle Warehouse Builder - It is a Real Choice

Recently I have begun work on a project for my company, Thoughtcorp. On this project we are implementing a Campaign Management Tool for a marketing organization. The project is one that is reasonably complex, and one that will ultimately provide new value to the company. It is always exciting to be involved in projects where you get to see technology, most of it Oracle technology at work. My project is using Oracle 11g as its database and as a result of this choice the project is also using Oracle Warehouse Builder (OWB).

OWB has been around Oracle for a long time. This was the ETL tool that Oracle developed to enter into this market. Since that time the product was released in 200o, it has undergone many changes and evolutions. In addition Oracle also purchased a product named Sunopsis, which has since become Oracle Data Integrator (ODI). The goal is to merge much of the functionality of OWB and ODI to provide options in ETL choices while providing robust implementation options. Since OWB was released in Oracle 10g Release 1, its functionality has been available to the Oracle database community. This functionality has been lurking within your database and for my project it is time to use it.

The approach when starting a new project is to determine what tools you will be using to develop your solution. In many situations these choices are already made for you when you first start. The database is often a standard, like Oracle, the reporting tool, the ETL tool and others are often defined. In a scenario like my latest project was one that had a database and reporting infrastructure defined, but no ETL tool had previously been used or purchased. The solution could have taken two distinct paths, we could have gone out and acquired a new ETL tool, as there are many options, but the decision was to leverage the existing investment in Oracle and use OWB to act as our ETL tool. This decision provided the organization with a solid tool that is part of Oracle integration direction of the future and one that can support the requirements of our loading effort.

This decision is one that we now are working with and one that so far has shown to be a solution that can support our project’s and the organization’s ETL needs. It’s great to see Oracle’s wide breadth of technology to help to solve our challenges in a way that truly is cost effective. By leveraging the technology contained within our existing investment, we can help our organizations to be more effective without spending any new monies and in these times that could provide a very compelling reason to adopt this type of technology.

The Network

Monday night hockey has come and gone and I sadly report that my team fell to defeat 2-1. The team fought hard, but our opponent on that night scored more goals, but did not defeat us.
My teammates and friends would have loved to be the Champions, but it was not to be. Being part of this team is special already. The guys are all unique and bring a closeness and honesty (except for Jimmy the Liar) that I have rarely found in many people. These guys love the game of hockey, but as we all get older we also realize that we come on Monday nights to be part of the event that is hockey. We talk about everything and exchange stories (some real and some you can’t believe are real) and have some fun, these are the reasons that really bring us out each week.

Here is a picture of the Remax All Stars.....a great group of friends.




What does my team illustrate to me is that here is a network of people you spend your time with, and they contribute to your success and you to their's. I look at the network of Oracle professionals I know and I feel impressed. I know and feel comfortable asking the best DBAs, Architects, Developers, Trainers and DW people questions that I cannot answer and they will help me. I look to the community to help me be better. I look to my hockey team to make us all winners, I look to my playing partner in golf to make us better and get some more putts and lower scores, I look for my Oracle friends to make us both better at using the Oracle technology. The true value of a network is the value that it brings to your personal and business life. I like to hang out with my hockey buddies and my Oracle buddies, my network is at work all year long.


I also want to remind you to register for the IOUG Forum at COLLABORATE09 in Orlando. The early bird value pricing ends this THURSDAY so sign up and make your Oracle network part of your winning team.

IOUG Elects New Board Members

Every year the IOUG elects people to the Board of Directors and right now our election is underway. If you are a member of the IOUG you can cast your Ballot now for the slate of great people who have been nominated for consideration. So who are these people who are coming to the Board? These are all individuals who have a long and deep commitment to the success of the user community.

The first person on this year's slate is Judi Doolittle, Judi is the current EVP of the IOUG and bring a strong technical and managerial leadership to the group. She continues to provide the group with leadership from an Oracle customer perspective. New to the Board is will be John Matelski, Todd Sheetz and Jon Wolfe. Many of you will have already have met some or all of the three new candidates. They have been long time and active volunteers in the IOUG and great Oracle community. John Matelski comes to us after a recent stint of leading the International Oracle User Council (IOUC) and works as the CIO for Gwinnett County in Georgia. He will provide the IOUG membership with strong strategic leadership and will aid us in working with Oracle to represent the user community. Todd Sheetz is a hands-on Oracle technologist; he has recently been leading the IOUG Best Practices SIG and has been an important part of the growth and expansion of the community. Finally is Jon Wolfe. Jon has been the technical session chair for the past few COLLABORATE conferences, under his guidance he has worked with the Conference Committee to provide education that improves each and every year. Education and networking are the core values of the user community and Jon is an integral part of these offerings.

These are great people who have seen the value to be involved in the user community. User group members are statistically more knowledgeable, happier and richer than non-members. In these times of uncertainty, it’s time to get involved just like Judi, John, Todd and Jon.