Make data work for you

Is your data an asset or a liability?

Some time ago, I read an article discussing how many companies are in fact losing revenue because they are not exploiting the data they have: Companies are losing revenue by not exploiting data

Image by Marius B via FlickrI see many articles, tweets, and blogs about the big data and the sheer volumes of data that are being stored at many enterprises.

I wonder, if those companies storing Big Data are using the data effectively?

How much of that data feeds into a Key Performance Indicator for the organization? 

Even if it is not  being used as a KPI, is it being used in data mining research?

English: kingstonsmith@husmail.com datamining graph. (Photo credit: Wikipedia)

 
 


Determining whether your data is an asset is  something that can be a huge topic, but in order to simplify the question I will propose it this way:

Does your data work for you, or do you work for your data? 

Data that is pertinent to an organization, after it is used in the customer facing application, should be used for analytical purposes. This could be Business Intelligence, Data Science, Statistical studies, process modeling or any number of names that may be leveraged.

One way to tell how far along in making data work is understanding your maturity. However, it is not always necessary to create a data warehouse in order to make your data work.

Depending on the volume of your data, you may be able to use R to do some statistical analysis to find the pertinent data. A small Hadoop cluster can collect and store all of you application data extracted and stored in HDFS, then RStudio server can read the data for some basic profiling.

Once this is done your local data scientist, or statistician can advise you on the best method for exposing this data to the rest of the organization. A small data mart may be useful in this case.

Without some of these fundamental tools, getting the answer to sophisticated questions that were not originally thought of during the application develop lifecycle may quite difficult to answer.

Simply relying on application developers to do data analysis and create a data product takes time and money away from the project they are working on. This makes you work for your data.

A proper data architect organizes the data so that getting answers from the data is a reasonably simple process.

It may take a bit of time to get all of the pieces( Hadoop, or data warehouse) in place, but once they are there they make data work for you.

Related articles

• Reaping the Benefits of Big Data – New Organizational Approach
• PAW 2016 Predictive Analytics Conferences
• A data scientist shares his passions
• The Modern Law Firm and Data

Centrality-and-Architecture

How does centrality affect your Architecture?

Some time ago, I was responsible for a data architecture I had mostly inherited. There were a number of tweaks I worked to on to refine the monolithic nature of the main database. It was a time of upheaval in this organization. They had outgrown their legacy Computer Telephony Interface application. It was time to create something new. 

A large new application development team was brought in to develop some new software.

There was a large division of labor and processing where some things were handled by the new application, and another thing was developed to handle the data. Reporting, cleansing, analysis, ingress feeds, egress feeds, all of these went through the “less important” system. 

This was the system I was responsible for. 

In thinking about how best to explain a Data Structure Graph, I spent some time revisiting this architecture and brought it into a format that could be analyzed with the tools of Network Analysis. 

After anonymizing the data a bit, and limiting the data flows to only the principle data flows, I constructed a csv file to load into Gephi for analysis.

Source	Target	Edge_Label
Spider	ODS	Application
ODS	Spider	Prospect
Vendor1	ODS	Prospect
Vendor2	ODS	Prospect
Vendor3	ODS	Prospect
ODS	Servicing	Application
Legacy	ODS	Application
ODS	Legacy	Prospect
ODS	Dialer1	Prospect
ODS	Dialer2	Prospect
Gov	ODS	DNC
ODS	Spider	LegacyData1
ODS	Spider	LegacyData2
ODS	Spider	LegacyData3
Spider	ODS	LegacyData1
Spider	ODS	LegacyData2
Spider	ODS	LegacyData3
ODS	ThirdParty	Prospect
ThirdParty	ODS	Application
Legacy	ODS	Application
Legacy	ODS	DialerStats
Dialer1	ODS	DialerStats
Dialer2	ODS	DialerStats

I ran a few simple statistics on the graph, then did some partitioning to color the graph to make it apparent the degree of a node this is the first output of Gephi:

The actual statistics Gephi calculated are in this table:

Id	Label	PageRank	Eigenvector Centrality	In-Degree	Out-Degree	Degree
Vendor1	Vendor1	0.01991719	0.00000000	0	1	1
Vendor2	Vendor2	0.01991719	0.00000000	0	1	1
Vendor3	Vendor3	0.01991719	0.00000000	0	1	1
Gov	Gov	0.01991719	0.00000000	0	1	1
Spider	Spider	0.08121259	0.44698155	1	1	2
Servicing	Servicing	0.08121259	0.44698155	1	0	1
Legacy	Legacy	0.08121259	0.44698155	1	1	2
Dialer1	Dialer1	0.08121259	0.44698155	1	1	2
Dialer2	Dialer2	0.08121259	0.44698155	1	1	2
ThirdParty	ThirdParty	0.08121259	0.44698155	1	1	2
ODS	ODS	0.43305573	1.00000000	9	6	15

From the Data Architecture perspective, which “application” has the greatest impact to the organization if there were a failure?

Which “application” should have the greatest degree of protection, redundancy, and expertise 

associated with it? 

Let's cover in detail the two metrics in the middle of the last table PageRank, and Eigenvector Centrality. 

I will have to create individual blog entries for both PageRank and Eigenvector Centrality to discuss the actual mechanism for how these are calculated. The math for these can be a bit cumbersome, and each algorithm should be given due attention on its own.

The point of this analysis is to determine which component of the architecture should have additional resources devoted to it. For any customer facing application, it should be given due attention, and infrastructure. However, one question I have seen many of my clients struggle with is what is the priority of the back-end infrastructure? Should once component of the architecture be given more attention than another? I have 90 databases throughout the organization, which one is the most important?

These centrality calculations show unequivocally which component of the architecture has the most impact in the event of an outage, or where the most value can be provided for an upgrade.
 
This type of analysis can begin to shed light on the answers to these questions. A methodical approach to an architecture based on data, rather than the division that screams the loudest can give insight into how an architecture is truly implemented.

I call these artifacts a Data Structure Graph