The-Data-Guy: Make data work for you

Is your data an asset or a liability?

Some time ago, I read an article discussing how many companies are in fact losing revenue because they are not exploiting the data they have: Companies are losing revenue by not exploiting data

Image by Marius B via FlickrI see many articles, tweets, and blogs about the big data and the sheer volumes of data that are being stored at many enterprises.

I wonder, if those companies storing Big Data are using the data effectively?

How much of that data feeds into a Key Performance Indicator for the organization?

Even if it is not being used as a KPI, is it being used in data mining research?

English: kingstonsmith@husmail.com datamining graph. (Photo credit: Wikipedia)

Determining whether your data is an asset is something that can be a huge topic, but in order to simplify the question I will propose it this way:

Does your data work for you, or do you work for your data?

Data that is pertinent to an organization, after it is used in the customer facing application, should be used for analytical purposes. This could be Business Intelligence, Data Science, Statistical studies, process modeling or any number of names that may be leveraged.

One way to tell how far along in making data work is understanding your maturity. However, it is not always necessary to create a data warehouse in order to make your data work.

Depending on the volume of your data, you may be able to use R to do some statistical analysis to find the pertinent data. A small Hadoop cluster can collect and store all of you application data extracted and stored in HDFS, then RStudio server can read the data for some basic profiling.

Once this is done your local data scientist, or statistician can advise you on the best method for exposing this data to the rest of the organization. A small data mart may be useful in this case.

Without some of these fundamental tools, getting the answer to sophisticated questions that were not originally thought of during the application develop lifecycle may quite difficult to answer.

Simply relying on application developers to do data analysis and create a data product takes time and money away from the project they are working on. This makes you work for your data.

A proper data architect organizes the data so that getting answers from the data is a reasonably simple process.

It may take a bit of time to get all of the pieces( Hadoop, or data warehouse) in place, but once they are there they make data work for you.