The-Data-Guy: Data Mining

Showing posts with label Data Mining. Show all posts

2018-06-12

Practical Text for the Data Professional.

Recently, I have been having conversations about text analysis.

Before we get into the details, why would you want to do Text analysis? Do you

Collect survey data?
Customer feedback?
Complaint forms?
Market Content?
Solicit feedback through Social platforms?
Perform SEO?

These are just the tip of the iceberg when it comes to analyzing the text you deal with every day.

Text analysis, by itself, can be a little intimidating. So, I put together a small R notebook using some off the shelf CRAN packages to parse PDF files, and create some metrics that can be analyzed by Tableau and Gephi. The PDF files are a collection of books that I have downloaded from various sources over the years. Many of these are the PDF companions of hardback books I have purchased for my own learning of a given topic. Some are PDF conversions of Power Points from presentations I have attended.

The R notebook can be found on RPubs, and the Tableau workbook can be found on Tableau Public.

Each cell of the R notebook can be a topic in and of itself. The process I followed for this outline is to

· Simply (emphasis on simply) parse the document

· Break the document into sections (not chapters)

· Calculate the lexical score for each section

· Calculate the Sentiment for each section

· Annotate the text

· Pull out the most frequently used Nouns, adjectives, Verbs, and Keyword phrases.

In the notebook I only show a single PDF that I parsed, I also create a “batch” process to create CSV’s for each of these. In addition to the csv files I also prepped the data into files that could be loaded into Gephi for Graph analysis.

The individual CSV files, I loaded into Tableau for some different visualizations.

This is an example graph of the smaller Automatic Keyword Extraction Graph created.

This shows the relationship between Documents and Sections that have the same keywords.

If two documents use the same keyword that has been extracted from the raw text, there is a line or edge between the nodes which are the documents and sections.

The code I wrote is stored on my Github .

Any of these features that are generated from the text could also be considered a feature to be used in a Machine Learning application as well depending on your use case for the text analysis.

I will be writing and speaking in much more detail about this process in the coming months, I will update this page when I have a link to where you can get more information.

In the meantime, if you have questions, please comment below, and I will both answer and incorporate your questions into future work.

Enjoy!

2016-01-29

Can you Kaggle?

Kaggle?

Do you even Kaggle, Bro?

English: Kaggle logo (Photo credit: Wikipedia)

Kaggle is a platform for predictive modelling and analytic competitions on which companies and researchers post their data and statisticians and data miners from all over the world compete to produce the best models. This crowdsourcing approach relies on the fact that there are countless strategies that can be applied to any predictive modelling task and it is impossible to know at the outset which technique or analyst will be most effective. - Kaggle Site Definition

Basically, Kaggle is a site where they peridocially have competitions for Machine Learning, and Data Mining Practitioners.

Generally the format of the competition is this:

Here is a training dataset

Here is a test data set

Here is a sample submission file

The training and test data set have the "key" features that the people designing the competition recognizes as being important. One row per observation with some particular "training" or "outcome" variable. The biggest difference between the two is that the test data set does not have the "training" or "outcome" variable.

This is what you need to accurately predict.

The sample submission file is generally a file with 2 columns an ID variable, and the Response variable.

The ID from the submission is a unique identifier from the test data set, and the response is what you, and your algorithm predict.

There is an evaluation process behind the scenes. Once you submit your prediction, your submission is compared to a set of "known" predictions. The predictions behind the scenes are the ones generally considered to be accurate by the group hosting the competition.

An evaluation score is calculated on your prediction versus the accepted predicted values. Some mechanism like F-Statistic, or RMSE, or area under the ROC are used as a score.

These competitions show a wide variety of industries, and some of the competitions allow you to win money.

Since there is money on the line, there are some groups that take these things incredibly seriously.

They will dedicate resources and time to winning a competition. As such the leaderboard may or may not be an accurate representation for how well someone knows an algorithm or methodology, rather it may represent the amount of time and resources dedicated to winning the competition.

For example, in one competition recently Springleaf the top score on the leaderboard is: .80427 second place is: .80394 a difference of .00033

The difference between my score and the leader was .04759!

These competitions can be fun, exciting, and provide for opportunities to try new tools, methods, and algorithms. However, unless you dedicate quite a decent amount of time and energy into them you may not come in first place.

2011-05-27

Analytical Skills?

One particular skill listed on many job descriptions I have seen gives me more questions than answers.

The skill they would like the candidate to have is listed as : Analytical Skills

How is this quantified? How do you know that someone has analytical skills?

Wiki defines Analysis as the ability to break down a complex topic or substance into smaller parts to gain a better understanding of it.

To what level of detail does this particular job need Analytical Skills? Will the candidate need to analyze the Enterprise accounting functions? Does it need to be the analysis of the physical server infrastructure? Does it need to be organizational layout? Will the candidate be analyzing forensic evidence?

Analytical skill is such a broad topic, and once a person does their analysis what happens to it? Is this a position that has the ability to not just analyze data, but also act on it? Is the analysis required in the position for recommendation purposes, educational purposes, or will the candidate be making decisions based on data provided by others?

Many people have analytical skills, but do they have good analytical skills? Do decision makers listen to and follow their recommendations after an analysis is done?

Data Management professionals are constantly analyzing data. The raw data can represent many diverse topics. Some of the key topics for analysis that come to mind are People, Processes and Things.

People

The analysis of people, their motivations, and their interactions is covered by subjects like anthropology, psychology, sociology and other behavioral sciences. Some people are naturally gifted people readers and can understand others with limited formal training. The analysis of people is useful to many groups within an organization, human resources, marketing, sales, even executive leadership.

Process

There are many types of processes in our lives, a process for getting a drivers license, getting married, fulfilling a product order, shipping a product, and many others. Understanding and recommending improvements to the nature of the processes that we interact with on a daily basis can be very valuable.

Things

Things can be companies, human languages, computer languages, web pages, corporate ledgers, computers, cars, religions, money, inventories, nature. Every "thing" can be studied and analyzed. The more we understand things the more data we generate about those things in order to contribute to human knowledge, self knowledge or our corporate enterprise.

Data

All things that are analyzed have one thing in common. Data. In all analysis data is what is collected, stored, manipulated, reported, recommended and decided upon.

There are best-practices for data management that can assist every type of analysis of every subject. "Pure" analytical skill is seldom used in a vacuum. Databases store data, operational systems collect data, a data warehouse helps in the correlation of data amongst multiple systems that are gathering data. Data Management by its very nature is an analytical skill.

When I see a position for a data management professional that requires analytical skills, I still find it humorous. Because the analytical skills that we can provide in both our own analysis and in guiding the analysis of others should go without saying.

Pages