Question The Answers.

Some time ago I wrote an entry on the difference between Data Science and Business Intelligence: https://bit.ly/DataSciencevsBusinessIntelligence

I recently came across this quote:

“Advances are made by answering questions. Discoveries are made by questioning answers” —Bernard Haisch.

I think there is a relationship between this quote and that previous post.

In essence what I was attempting to say was that Business Intelligence is generally a process that your data flows through that enriches application Data and prepares it such that it can be used to answer questions. These questions may be simple:

1. How many widgets did this business unit produce last quarter? 
2. How many did that business unit sell last quarter? 
3. Which sales person sold what percentage last quarter? 
4. What is the recurring cost of this Customer? 

These are all important questions. However, this same data should be used as part of any predictive effort. If you are using different data for your data science efforts and your business intelligence efforts then as you chart new territory through Data Science, your Business Intelligence platform will assist in showing the value of the Data Science effort. 

These two sides of a similar coin can and should be complementary. 

Business Intelligence will drive your business forward, Data Science will show you the direction you should go. 

Practical Text for the Data Professional.

Recently, I have been having conversations about text analysis. 

Before we get into the details, why would you want to do Text analysis?  Do you

• Collect survey data?  
• Customer feedback? 
• Complaint forms? 
• Market Content?
• Solicit feedback through Social platforms?
• Perform SEO?

These are just the tip of the iceberg when it comes to analyzing the text you deal with every day.

Text analysis, by itself, can be a little intimidating. So, I put together a small R notebook using some off the shelf CRAN packages to parse PDF files, and create some metrics that can be analyzed by Tableau and Gephi. The PDF files are a collection of books that I have downloaded from various sources over the years. Many of these are the PDF companions of hardback books I have purchased for my own learning of a given topic. Some are PDF conversions of Power Points from presentations I have attended.

The R notebook can be found on  RPubs, and the Tableau workbook can be found on Tableau Public.

Each cell of the R notebook can be a topic in and of itself. The process I followed for this outline is to

·         Simply (emphasis on simply) parse the document

·        Break the document into sections (not chapters)

·         Calculate the lexical score for each section

·         Calculate the Sentiment for each section

·         Annotate the text

·         Pull out the most frequently used Nouns, adjectives, Verbs, and Keyword phrases.

In the notebook I only show a single PDF that I parsed, I also create a “batch” process to create CSV’s for each of these. In addition to the csv files I also prepped the data into files that could be loaded into Gephi for Graph analysis. 

The individual CSV files, I loaded into Tableau for some different visualizations. 

This is an example graph of the smaller Automatic Keyword Extraction Graph created.

This shows the relationship between Documents and Sections that have the same keywords.

 If two documents use the same keyword that has been extracted from the raw text, there is a line or edge between the nodes which are the documents and sections.

The code I wrote is stored on my Github . 

Any of these features that are generated from the text could also be considered a feature to be used in a Machine Learning application as well depending on your use case for the text analysis.

I will be writing and speaking in much more detail about this process in the coming months, I will update this page when I have a link to where you can get more information. 

In the meantime, if you have questions, please comment below, and I will both answer and incorporate your questions into future work.

Enjoy!

The Little Data Science Checklist

There are lots of sources online and in book stores that will teach a person how to do Machine Learning, Regression, Text analysis, or any other fascinating topic related to Data Science.

But is that all there is?

Just know apply an algorithm, and you improve the bottom line of your company?

What if you need to justify a project? How do you demonstrate that there is a process that can be followed for data science?

Recently with some colleagues we were discussing this precise topic.

Here is the answer that we came up with:

Question:

            What is the question?

            Who asked the question?

            When do they want it?

            How does it provide business value?

Hypothesis:

            Validation Criteria

            Identify the data

            Collect the data

            Transform, Merge, Munge

            Analyze and explore

            Iterate

            Peer Review

            Visualize and Communicate

            Generate new directions.

            Produce Finding

Finding:

            TRUE

            FALSE

            Inconclusive

            Experiment Required

Supporting Data:

            Raw Data

            Chart

            Visualization

I will pick a section and write more about each of these sections, but this little checklist is a beginning step in applying a Data Science process for an Enterprise. 

Related articles

• Data Science 101, now online
• Going Deeper into Regression Analysis with Assumptions, Plots & Solutions
• Deep Learning Prerequisites: Linear Regression in Python - education save coupon

Show Your work

"Show your work!
Yes, you got the answer right.
I am still marking it wrong since I don't know how you did it."

I hated that phrase.

Yes, I heard it more than once.

Teachers never explained to me the point of showing your work is so that if you do in fact come up with the wrong answer they can assist you in how to tweak your algorithm.

I don't think any teacher would have ever explained things this way.

Honestly, most of my teachers I am not convinced they actually understood this is what they were doing.
They really only knew one method of solving a given problem, and in order for them to supply help the students had to follow their methods.

(The majority of my early education was in religion based schools, so these teachers were not Scientists, Mathematicians, etc...)

Now to today, I am a Data Scientist. Which means, I work with numbers, algorithms, Data, business users, technical experts, Architects, Statisticians, and in Domain experts.

I spend much of my day munging data into data structures, or algorithms that provide insight into our data, our users, and our customers. One thing I see Data Scientists doing, and not necessarily talking about is the whole "Show your Work" philosophy.

If you are going to make a claim about an insight you have about data, you should show how you got to that conclusion. In many papers it may be in a section like "Methods and Assumptions", but I think this is important in even internal presentations to business users. You should be able to show others how you got to whatever conclusion you have come to.




It may not be "Page 1", (actually Page 1 should really be your final conclusion, but that is another story), but for any presentation in addition to citing your sources, you should touch on any methodologies you followed.

For example, in my current research, I am evaluating Markov chain sequences of behavior patterns. I will write a separate blog about Markov chains at some point, but part of my foundation work has been to show how to both collect the data, then munge my observations into a probability transition matrix.

This has re-iterated to me that we in the data science community have a responsibility to be able to show our work.

Not everyone will want to go into the same weeds, rabbit holes, and other detailed work that you have done. But after taking any number of wrong turns during your analysis, you should be able to show from first principles how you arrived at your final conclusion.  Even if that is saved for a smaller presentation for those willing to go through your process with you.

Ad astra!

Related articles

• 5 Peer-to-Peer Learning Resources for SMBs
• Understanding Data Science for building Predictive Analytics Solutions by Francesca Lazzeri
• Computational Thinking for All Students
• Probabilistic data structures in Python

Becoming a Data Scientist

Everyone's path to becoming a Data Scientist is different.

This makes it difficult to recommend to someone outside the academic world how re-invent themselves into the image of a Data Scientist.

 Joel Grus makes a quite succinct overview of what a Data Scientist should be able to do. I first saw this image on twitter, and searching around found his presentation on Slideshare.

Joel makes some great points, about learning Data Science.

Science is a tool for understanding the world around us.

"Science is a systematic enterprise that creates, builds and organizes knowledge in the form of testable explanations and predictions about the universe."

Some people may think of Data Science as part of the tools used by Scientists wearing white coats in pristine labs. You can begin doing Science in your own kitchen. I am much more a follower of Bill Nye, rather than Brian Cox.

One of the earliest Data predictions I did was to answer the following question: How big should this database be?

I created a Data Model in ERWin, then did Volumetric estimation based on daily table growth.

I had a model (the Entity Relationship Diagram). I had some assumptions (we are adding this many records every day). I was able to do forecasting(another word for prediction.).

Using this information we knew how to size the database server for this application, and use that information as part of the infrastructure pricing estimates given to management.

While this may not be considered "proper data science" by many people this is a simple example of how to begin using Science in even the most mundane way.


Sure there are some foundation components like Math, Data Management, Programming skills, Business acumen, and data visualization that need to be learned and understood. Try to solve a problem in your universe.

So the question before us is: How do I become a Data Scientist?

Here are some steps to follow:

1. Pick a problem that interests you. (Ask an interesting question.)
2. Learn all you can about the problem. (What is already known about the problem?) 
3. Collect as much data as you can about the problem. (From as many sources as you can.)
4. Make a prediction about the problem. (Don't worry if others have already done this.)
5.  Be wrong! (This is most important. Fail first, Fail fast, and Fail Often!) 
6.  Figure out what you did wrong, and correct it. (Then iterate. Go back to learn more (2), or get more data(3), or update your prediction(4) based on new information.)
7.  Be right! (Finally! Now show why you were correct, and how you can apply this to other domains.)

If you are just starting on the journey to become a Data Scientist, do this a few times. The idea is to learn this process. The tools will change based on your environment, the specific problem you are trying to solve, and what tools your employer will allow you to use.

The point is, as Joel Grus, Bill Nye, and Nike says: Just do it!

Bootstrap analysis

Bootstrap Analysis

There are times in the life of a Data Scientist where you have a bit of data, and the broad directive of: "See what this says."

Recently I have faced this, and came up with this concept - Doing a Bootstrap analysis. 

Spend a very small time-period. (Small is relative to your initial understanding of the data). Pull the data together, and munge it into your tool of choice. 

Then come up with broad features that describe the data set at a high-level. These features are an enrichment of the data set itself. So in R terms it would be adding a column to a data frame. In SQL terms, creating a new table with a ranking column based on some grouping criteria for example. 

Once you have this enriched data set perform some simple trending analysis on your features compared to some outcome variable in the original data set. If you see something interesting, that is the thread you can pull to find out more details. 

Further contextualizing the data will give further insight, but this initial analysis will give you an idea of what further contextual information would be useful.

This "limited" analysis should enable you to answer the initial question of: How long will it take to get something useful out of this data?

This is a different concept from "bootstrapping ", others have explained in far more detail, and far better than I ever could about that technique. This is just a way to communicate to stakeholders that there needs to be some time given to initial analysis of a data set to get an idea of how deep the rabbit hole goes...

Related articles

• Data Science Skills for 2016
• What skills should data scientists have in 2016?