Poor Planning

Poor Planning on your part does not constitute an Emergency on my part.  

(Except when it does.)

Recently I was involved in a major hardware failure at work.

There had been indications our disk storage was under duress for quite some time. It had been in my status reports for months, I finally quit reporting on it since none of our leadership team even acknowledged the concern.

They would say, yeah we know we need to do something there. Or even better: Your predecessor complained about that as well. 

Over a long weekend involving most of my team, teams from other groups, vendors from infrastructure support and hardware vendors we fought for our customers not management.

War room calls were set up to run around the clock with people stepping in and out of the meeting we finally were able to get things ironed out. 

But at what cost? 

To proactively take care of this issue would have required potentially spending a bit of money up front to either get new infrastructure or upgrade it.

However to be down for the amount of time that we were down we broke faith with our customers.

I don't know the long term impacts of that on our organization. 

In cases like this it is never a good idea to say "I told you so."

A current joke going around says: "At the start of every disaster movie there's a scientist being ignored"

When you are in the middle of a disaster even if you were the one that could have prevented it, the only thing that you can do is to work hard on the recovery.

There is a tendency to shoot the messenger when dealing with issues, but that should be the last thing on anyones mind. The messenger is the one who knew about the problem the longest. Usually they are the person that has been kept up at night stressing over what to do.

The messenger probably has the most ideas about how to solve the problem.

Knee-jerk reactions do not create long term solutions. Careful design, planning, testing, and proactively building in resiliency and stability create long term solutions.

None of these are cheap, but as the saying goes: "Pay me now, or pay me later."

Proactive "pay me now" situations appear to be expensive.

Reactive "pay me later" situations make the proactive look like nickels and dimes.

When you are getting warnings about things that need to be addressed, don't ignore them. If you are the ones giving the warnings, don't give up.

Keep warning.

Keep telling.

Above all, Keep planning.

When the disaster strikes, someone has to be the voice of reason.

Question The Answers.

Some time ago I wrote an entry on the difference between Data Science and Business Intelligence: https://bit.ly/DataSciencevsBusinessIntelligence

I recently came across this quote:

“Advances are made by answering questions. Discoveries are made by questioning answers” —Bernard Haisch.

I think there is a relationship between this quote and that previous post.

In essence what I was attempting to say was that Business Intelligence is generally a process that your data flows through that enriches application Data and prepares it such that it can be used to answer questions. These questions may be simple:

1. How many widgets did this business unit produce last quarter? 
2. How many did that business unit sell last quarter? 
3. Which sales person sold what percentage last quarter? 
4. What is the recurring cost of this Customer? 

These are all important questions. However, this same data should be used as part of any predictive effort. If you are using different data for your data science efforts and your business intelligence efforts then as you chart new territory through Data Science, your Business Intelligence platform will assist in showing the value of the Data Science effort. 

These two sides of a similar coin can and should be complementary. 

Business Intelligence will drive your business forward, Data Science will show you the direction you should go. 

How to create a mathemaical model

One day I was speaking to a friend of mine we were telling stories about some of the previous jobs we had held. He had been a teller at a bank. He asked me how they train tellers to recognize counterfeit money. It's really easy, he said, they never give them counterfeit money to work with. They always work with real bills when they practice counting and such. Then when you get a counterfeit it feels funny.

I recently recalled this conversation when I was looking for a pattern in some numbers.
As I copied the numbers into excel, I realized the progression kinda looked like a logistic progression with a base of 20. This tiny insight allowed me to do some further searches and find that there is a mathematical model that already represents the data I was looking at: Watts & Strogatz_model

By no means does this make me an expert in this area, but it did drive home for me the value of studying formal mathematical models.

I think the more that one studies formal models, whether they are in your current domain or not,  the more familiar you become with various types of models the better you will be at creating models yourself.

After all, a mathematical model is a set of rules that describe the behavior of data. Understanding how data behaves in various scenarios will improve your ability to recognize a pattern.

Manifesto for Stable Infrastructure and Data Management

Our goal should be Stability, not Agility. 

There has been much written about Agile Software Development techniques. Some even say people are overdoing the use of the Agile name (https://www.infoq.com/presentations/agile-2018).

There is a time and a place for everything. A time for Agile, a time for Stability. Stability is simply the act of making your ecosystem S.T.A.B.L.E. Everyone should want to have achieve a stable environment that is a Scalable Technology Architecture By Layering Everything

In light of this, I propose the following Manifesto for Stable Infrastructure and Data Management.


We are uncovering better ways of growing business operations by doing it and helping others do it.
Through this work we have come to value:

Steady over explosive change.

Individual expertise over title or position.

Shared responsibility over admonishment.

Ecosystem availability over new applications.

 We follow these principles:

Our highest priority is to keep the business working while ensuring that we maintain a foundation of high quality tools and technology.

The modern business is built around a wide variety of software and applications.

No one application is more important than the holistic environment.

Not all applications will be built and maintained by our organization.

Treating all projects as if they were a software development project diminishes the value of the project itself.

Projects will come and go, new applications will join our ecosystem, old applications will be shutdown, but the data should be available forever.

The stability of the Enterprise, and the ability to fulfill the mission is paramount; this takes precedence over new needs.

Use out of the box tools and applications to drive using data for a different purpose than originally captured in a custom application.

Business continuity takes precedence over deployment of new applications.

Adopting new tools and technologies should be done with adequate time, training, and trials such that the original adopters are not the only ones capable of supporting it.

A minority of technologists are software developers; the Data of the organization is co-equal in importance to any algorithm that may be developed.

Practical Text for the Data Professional.

Recently, I have been having conversations about text analysis. 

Before we get into the details, why would you want to do Text analysis?  Do you

• Collect survey data?  
• Customer feedback? 
• Complaint forms? 
• Market Content?
• Solicit feedback through Social platforms?
• Perform SEO?

These are just the tip of the iceberg when it comes to analyzing the text you deal with every day.

Text analysis, by itself, can be a little intimidating. So, I put together a small R notebook using some off the shelf CRAN packages to parse PDF files, and create some metrics that can be analyzed by Tableau and Gephi. The PDF files are a collection of books that I have downloaded from various sources over the years. Many of these are the PDF companions of hardback books I have purchased for my own learning of a given topic. Some are PDF conversions of Power Points from presentations I have attended.

The R notebook can be found on  RPubs, and the Tableau workbook can be found on Tableau Public.

Each cell of the R notebook can be a topic in and of itself. The process I followed for this outline is to

·         Simply (emphasis on simply) parse the document

·        Break the document into sections (not chapters)

·         Calculate the lexical score for each section

·         Calculate the Sentiment for each section

·         Annotate the text

·         Pull out the most frequently used Nouns, adjectives, Verbs, and Keyword phrases.

In the notebook I only show a single PDF that I parsed, I also create a “batch” process to create CSV’s for each of these. In addition to the csv files I also prepped the data into files that could be loaded into Gephi for Graph analysis. 

The individual CSV files, I loaded into Tableau for some different visualizations. 

This is an example graph of the smaller Automatic Keyword Extraction Graph created.

This shows the relationship between Documents and Sections that have the same keywords.

 If two documents use the same keyword that has been extracted from the raw text, there is a line or edge between the nodes which are the documents and sections.

The code I wrote is stored on my Github . 

Any of these features that are generated from the text could also be considered a feature to be used in a Machine Learning application as well depending on your use case for the text analysis.

I will be writing and speaking in much more detail about this process in the coming months, I will update this page when I have a link to where you can get more information. 

In the meantime, if you have questions, please comment below, and I will both answer and incorporate your questions into future work.

Enjoy!