Pages

Showing posts with label R. Show all posts
Showing posts with label R. Show all posts

2016-03-07

Recency_Frequency_Intensity

RFI Analysis

Expanding on my prior post about Bootstrap Analysis, this post is a demonstration of one particular technique for learning more about how deep the rabbit hole goes when you are exploring a new data set

RFM analysis is a way to summarize the interactions your customers have with your organization.  Using the three concepts of Recency, Frequency and Monetary Value, and their various interactions and combinations you can create a very simple statistical model of the value of a given set of customers. 

Now, what if your customers are not spending money? Changing Monetary Value to the Intensity of their interactions with your organization, store, or web-site is one way to use the common tools available for RFM analysis as an RFI analysis. 

Recency - How recently, according to some definition, did an interaction occur? 

Frequency - How frequently does a particular customer or set of customers perform an interaction? 

Intensity - How intently did the customer interact? 


Each of these metrics needs to be interpreted in the context of your particular brand.


Here is a small example visualization using some random data I generated: 

I generated a list of customers, randomly assigned them to 5 different stores. After that I set up a random number generator in Excel to populate the three metrics of Recency, Frequency, and Intensity. 

In this particular case Recency could be how recently a customer visited a store in a Month. So if a customer visited on the last day of the month, the Recency would be high. Frequency is how frequently this customer visited this store during the month. Intensity could be how much money, or how much time they spent in the store.

So long as the metric is applicable to your use-case, and the definitions are consistent throughout the analysis, each of them

This analysis tool can be applied in a variety of settings where there is recurring interactions between a customer base, and a set of customers. 

With the proper visualization of data similar to the above image, many data points can be represented. In this case we represent the RFI metric itself as a point in the geometric cube. The shape of the point represents the store. The color represents the customer. 

So in this simple diagram there are 5 dimensions of data represented. 

The R code to do this is: 

#Load the correct packages for 3d representation and reading csv files
library(scatterplot3d)
library(readr)

#read the data
rfm.df <- read_csv('rfm.csv')

#Define some shapes based on unique store
shapes = c(16,17,18,19,20)
shapes <- shapes[as.numeric(rfm.df$Store)]
rfm.df$Customer <- as.factor(rfm.df$Customer)
s3d <- scatterplot3d(rfm.df$Recency,rfm.df$Frequency,rfm.df$Monetary,
                     xlab="Recency",ylab="Frequency",zlab="Monetary Value",pch=shapes,color=as.integer(rfm.df$Customer))
legend(s3d$xyz.convert(.9,1,0.8),legend=levels(rfm.df$Customer),text.col=as.integer(rfm.df$Customer),cex=1)



2016-02-03

R on RHEL 7 in AWS

Tux, the Linux penguin

R on RHEL 7 in AWS


If I have start to run some software, and it ends up being less than straightforward, I always try to make a checklist of the procedure for how to build it. 

Recently I was testing the performance of some R code. 


I launched an AWS server Red Hat Linux 7 (RHEL 7) and found R was not installed. 

Ok, so this is a simple yum install, right? 

Not so fast. 

Here is everything I needed to do, in order to get R and rstudio running on my server: 

sudo su -c 'rpm -Uvh http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm'
sudo yum update
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/lapack-devel-3.4.2-5.el7.x86_64.rpm
sudo yum locallinstall lapack-devel-3.4.2-5.el7.x86_64.rpm
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/blas-devel-3.2.2-5.el7.x86_64.rpm
sudo yum locallinstall blas-devel-3.2.2-5.el7.x86_64.rpm
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/texlive-epsf-svn21461.2.7.4-38.el7.noarch.rpm
sudo yum locallinstall texlive-epsf-svn21461.2.7.4-38.el7.noarch.rpm
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/texinfo-tex-5.1-4.el7.x86_64.rpm
sudo yum locallinstall texinfo-tex-5.1-4.el7.x86_64.rpm
sudo yum install R

wget https://download2.rstudio.org/rstudio-server-rhel-0.99.491-x86_64.rpm
yum localinstall --nogpgcheck rstudio-server-rhel-0.99.491-x86_64.rpm
service rstudio-server
service rstudio-server status

All of this came about from google searches on stackoverflow, and other places where I found hiccup after hiccup just getting everything installed. 

These commands will save you some time. 


2016-02-02

My introduction to R

R

English: Logo for R
Some time ago, I began down a path of learning statistics. This was not a topic I had studied in detail before. I had learned a bit of statistics over time while doing other things but never formally. 

I recognized I needed to learn more formally. 

During a number of the lectures I followed multiple professors referenced the R language. 

In my experience I had heard about SAS, and even worked with people that needed data from the various data repositories I managed imported into SAS. 

So I called one of my friends who is a big SAS user, and asked him about R. 

His response, R is basically a cleanroom version of SAS, but you have to write a lot of code to do the same things SAS does. 

Now this is something I could get into. 

So formally from the wiki: 
"R is a programming language and software environment for statistical computing and graphics supported by the R Foundation for Statistical Computing. The R language is widely used among statisticians and data miners for developing statistical software and data analysis. Polls, surveys of data miners, and studies of scholarly literature databases show that R's popularity has increased substantially in recent years." -- R programming language


I have been a DBA, and a data architect for quite some time. Generally, the types of systems I had built up to that point in time were analytical frameworks. The need for these is apparent with the majority of the Business Intelligence tools that are out there.

As a general rule, the performance of a BI tools is almost entirely dependent on the data model (dimensional) that the BI tool reads from.

There are a few exceptions to this rule, but it has been a guiding rule for the majority of my career.

Now, with R there is not as much of a need for the structuring of the data to support the analysis. R is a programming language, as such you can do the Data Munging necessary within your code.

And since R is a vector based language you can do set operations which are incredibly faster than doing for loops, cursors and the like.

R has many various packages for doing various types of analysis. Machine learning, Sentiment Analysis, Data Mining, various types of regressions.

All of which, only a few years ago I would have needed a SAS license to be able to attempt.

Since R is open-source, I am able to download it and run with it with no "request" and "approval" process. I don't have to justify an expenditure to get a tool that helps me do my job.

If you are in a data architect, DBA, or other DataOps  I encourage you to check-out R. You will find a new powerful tool in your toolbox.

I will be writing a bit more about R over time as well.


2016-01-16

Patches, we don't need no stinking patches

Are your packages up to date?


If you write software with open source tools you have a number of hidden, or even not so hidden dependencies on what you develop. 

Scala (programming language)
Scala (programming language) (Photo credit: Wikipedia)
The power of Spark, Scala, Java, R, Python, and Perl is generally in the packages and libraries that provide functionality.

These packages and libraries are more robust than just sample code.

They are made up of code, and sometimes even data examples that implement a particular algorithm or method to solve a particular problem.

It is generally a best practice to stay current with the version of a particular package that your code may be reliant on. This, at times requires your coders to update their code to reflect changes in the released packages.

As it so happens some of these packages, in turn, have requirements for particular packages that may need to be installed on the server, at the operating system level, where this code needs to run.

When there is a potential for conflict between two things that may need a similar package requirement, the most effective pattern I have seen recently is to separate the dependencies to run on individual machines.

The rise of virtual machines for code separation makes it easier to implement these things.

By using a virtual machine, the process of creating a niche environment for a specific process to run without conflicting with another process can be automated. Tools such as puppet, and Chef ease this process.

This automation, returning to my earlier point, becomes very straightforward if everyone uses the most current packages, libraries and dependencies.

This may seem obvious, and pointless to actually write this down.

However, I have worked at and been a consultant in more than one production environment where more effort was applied to maintaining old functionality than simply updating dependencies and  code to reflect the most current releases of dependent libraries.

There is a cost to this.

The cost of missed opportunities.

By  spending so much time "patching" your code to reflect older functionality, you are missing new opportunities to meet the needs of your customers.


How current are you?








2016-01-04

Writing for 30 days challenge

Writing for 30 days

Can you do it? 


Quite by happenstance, I found myself writing at the beginning of  this year. 

I did not set a new years resolution to write more. 

Writing
Writing (Photo credit: Wikipedia)
On New Years day, I did get inspired to create a few R packages. I have a feeling I will need some tools like this in the near future, so I started putting the basics together to use later. 

Then I found something odd. 

It involves three different tools, so I thought others might have seen something similar I wrote about it and posted it here. Then the next day, I wrote about my little package so I could send it out to a few people to take a look at.

After posting that I realized I had just written every day this year. 2 for 2, but still it's 100% :) 


I looked through some of my draft unpublished posts I have started over the years.

I have 50 posts unpublished.

So, I updated one, revised it, finished it, and published it.

That was Yesterday.

Day 3.

Here I go, I am going to publish something at least once a day till the end of January. I will be going through all of my drafts and cleaning them out. Some will be deleted, others will be updated, but at the end of the 30 days I will post how many of my drafts I have left.

I have always attempted to keep this blog relevant to data type topics, or at least professional topics.

In order to write every day for 30 days I will bend this guideline I have set for myself a bit.

I will write at least a paragraph, sometimes even longer. But I want this to be more than a tweet, or a facebook post. I may not always say something insightful, data related, professional, but I will write and publish a post for all to see every day for 30 days.

I encourage you to write as well.

P writing blue
P writing blue (Photo credit: Wikipedia)


   

LinkedIN, Facebook, these are both great platforms for topical comments about work or play.



YOU write something. If you never post on Facebook, hold your breath, bite your tongue and write "Hello World!" on Facebook.

Even a small Verse of a paragraph is a start.


Maybe something you say will start a conversation.


 You never know.

In the immortal words of Robin Williams: "What will your verse be?"



2015-06-12

SparkR on Windows 7

Apache SparkR on Windows 7

Like many people, I have been looking forward to working with SparkR. It was just released yesterday, you can find the full Apache Spark download here: Apache Spark Download

I have set up SparkR on a couple machines to run locally for testing. I did have a few hiccups getting it working, so I will document what I did here.

1. JDK 1.8_045
2. R 3.1.3
3. Apache Spark prebuilt with Hadoop-2.6
4. hadoop-common-2.2.0-bin-master
5. Fix logging. 
6. Paths
7. Run as Administrator.

1. Install the JDK 1.8_045 available here: jdk 1.8
2. Install R 3.1.3 available here: R 3.1.3

The first two items are standard Windows based installs, so they don't have to be put in any particular location. For the following downloads, I recommend creating a directory called Spark under your My Documents folder (C:\users\\Documents\Spark)

3.  Download and unzip Spark1.4 prebuilt with hadoop-2.6 available here: Spark 1.4 to the spark directory just referenced.


4. This link describes the winutils problem pretty well. Unzip the hadoop-common-2.2.0-bin.master, set up a HADOOP_HOME environment variable. I created this under the Spark directory mentioned above.


5. Copy spark-1.4.0-bin-hadoop2.6\conf\log4j.properties.template to spark-1.4.0-bin-hadoop2.6\conf\log4j.properties.
Edit the file.  log4j.properties
Change:
log4j.rootCategory=INFO, console
to:
log4j.rootCategory=ERROR, console

6. Make sure your path includes all the tools you just set up. A portion of my path is:
C:\Program Files\Java\jdk1.8.0_45\bin;C:\Users\<yourname>\Documents\Spark\spark-1.4.0-bin-hadoop2.6\bin;C:\Users\<yourname>\Documents\Spark\spark-1.4.0-bin-hadoop2.6\sbin;C:\Users\<yourname>\Documents\R\R-3.1.3\bin\x64

7. Run a command prompt as Administrator.

If I missed any of these steps, I had a number of issues to get things to work properly.

Now you can run sparkR


Have fun with Apache SparkR. I know I have much to try!

Good luck.