Data in the Time of Change

Back at it... I have been working on various other social forums of blogging and writing but I am back at it in my original blog. Time to get data to work.

Data is what I continue to focus on today. All aspects of data and how it should be used for the greater good.

Here is something I wrote recently for the IOUG and thought you might find interesting. I am also finishing up some ideas for 2015 which should be coming soon. I resolve for 2015 to be a better blogger.

Data in the Time of Change

By Ian Abramson, EPAM Systems Inc.

. We look at changes in technology, processes changes, professional changes and others which are impacting us as Oracle and Data Professionals. I find it intriguing that in my everyday job at EPAM I am one of those who helps companies change how they treat and use their data. Big Data and advanced analytics is all about how organizations and individuals are changing to the new data paradigm and I am lucky enough to be part of it. We are empowering the individual to make choices based on an unprecedented access to information which we can now provide. 

The change has been in many ways a gradual change and in other ways very sudden. The technology in Big Data ecosystem is one which changes faster than Superman in a phone booth.  We see releases from Apache, the various Hadoop distributions along with new projects which surround the technology becoming available each and every week. In August Apache released Spark for general release. Spark allows you to now interact with Hadoop up to 100 times faster. They do this by changing how they get data from the cluster as they reduce overhead and do as much work in memory as possible. Why is this important? It is important as existing Hadoop installations will want to find better performance compared to the existing tools. So now you have a choice. Do you begin using Spark or if you are using Cloudera should you use Impala? You know that you need to change but the question is which change will bring the most benefit. Big Data requires one to embrace change.

Besides the technology evolution we are seeing more desire for businesses to change. Big Data has opened up new opportunities within business. We now are providing them with access to information which previously had been unavailable or difficult to access and analyze. So we need to be sure we control this access because with great power comes great responsibility. Embrace change but make changes to improve the experience or the organization not simply to change.

The following infographic illustrates how data and analytics is changing and how we can make a difference if we can all evolve and change:

The Everything of Data

When do things change from hype to reality? Or when is it just more hype?  I have previously discussed how Big Data is changing everything; it had moved from simple hype to complicated reality producing very interesting results. The latest hype is the concept or the reality of interconnecting all of our data via the Internet of Things (IoT). This concept is one which should improve how things work. Not just businesses but I mean machines, distribution systems and people. I have to admit that I truly love the idea of being able to share information in a way that is meaningful and impactful through sharing insights and recommendations and ultimately improving our lives. We live in a complex world; now imagine your data becoming your life optimizer. 

The notion that all the data morsels we generate in our everyday lives and all of the sensors in our machines generate lots of information but generally these components never talk to each other. This connection between individuals, public sources, businesses and in our infrastructure grid can connect in a seamless way on to work together for a better individual experience and a improved global performance.  We can have simple things like our home dryers waiting until the power grid has a lower demand before starting to dry your clothes. Or that a doctor in a remote part of Canada can search a medical database of all medical symptoms; their treatment an outcomes to tailor a healthcare program for a sick patient. The interactions and benefits which we can ultimately achieve is limitless.

The question of hype is one which truly valid for IoT. We need to consider how can we share and share in a way that benefits people, governments and businesses. We have the technology today to begin the sharing but in reality we also live in a society where sharing information in this way scares people. They ask why should their confidential information be shared? I say why not. To quote Mr. Spock; “Logic clearly dictates that the needs of the many outweigh the needs of the few.”. We need to consider the up side and manage the risk of sharing.

Big Data has brought with it new challenges and new problems. The security of big data in Hadoop clusters is one which is evolving on a daily basis. We have options to encrypt, to restrict and to limit who can see what but the security of Hadoop is also quite basic and requires additional products like Kerabos to help in providing some security, but in many ways Big Data is available and if someone wants to get at it they will have an easier time then trying to break into a well-secured Oracle database. Then again Oracle does have a 25 year head start on Hadoop.

So is the Internet of Things hype? I don’t think so, I think it’s more of a dream state. One where data can live together and provide value well beyond its original intention. So it’s one dream that I hope to help bring to reality.