Big Data - HADOOP on Amazon Customer Review - AWS

Further on the Road to Big Data  (Amazon Customer Review Dataset - AWS)

Project Goal and Objective:

As introduced in the first  Project Page, this was a self teaching HADOOP project(s) that I created to expand my skills. This is a  second project was create to learn about the big data tools now being used in the market.  The goal was to read the below refrenced maternal and generate big data analytical results.

** This is a second chapter of my Big Data journey, the first part was covered under the NOAA data  effort:  Big Data Hadoop NOAA

Working Through the book:

As mentioned im my inital discover project (link above) the White-Sarfi Hadoop book was greatly used.   After the NCDC/NOAA weather dataset example, there was second Amazon Customer Review Datasets under AWS that made real world and tangible sense to me.  In addition, Amazon is starting to rule the retail world, why not get more exposure to them and learn what you can to enhance marketable skills.

Web Research Points  ---> In the White-Safri book there was an AWS sample with the reference to the AWS data links.  (Amazon Customer Dataset- AWS )

• So, I am a frequent user Amazon to a degree and was curious who or what actually gets purchased on the site.
• Two AWS- Customer data sets the I found interesting where the Retail set (2 gigs -unzipped) and Garden set (1 gigs - unzipped)
•  The files are in tsv format (tab), the differs from the standard CSV on most text file sources.

Problem Statement:

USE THE AWS DATA SETS FIND THE HIGHEST VOLUMENT ITEMS PURCHASED AND CAN THOSE TOTALS BE ROLLED INTO PARENT PRODUCT GROUPS  (ie: Shoes vs. Shirts)

Solution Steps:

• Look into the AWS sample data sets provided.

• I choose not to review at the AWS sample code referenced, since I run Apache open source Hadoop on my own hardware.

• New challenge was data in is tab separated format, soad to reference Java to find how to read tsv files using standard libaries.

Analysis Steps:

• Review data sent in a text editor for some quick findings (On Ubuntu, I found the mousepad text editor best to handle 1 Gig files)
• AWS sets are build on customer feedback and the star rating customer provided on the items purchased
• I can count each of the data rows by the unique product code in data sets, drop customer number.
• In the data sets it was found the parent product code was not helpful, I did not find a useful pattern. (ie: assumed the higer digests in parent number would seperate shoes from shits.) 
• I was able to use the AWS product description to lookup Actual Amazon products and confirm other assumptions/approaches

Question on the Data Sample Qualtiy:

I started to wonder on the data quality that was being provided.  It was nice and very thoughtful that that Amazon provided these data sets for free with a login registration process. Why would Amazon give other retailer insight into their site and how sales feedback or volume is collected?

Second question, how was this sample data chosen (single time slice steam or selective choice option, -- ie: tech guy in the background playing a data god role) 

AWS feedback: I sent an email off to AWS and they responded, it was confirmed that do collect a continues stream of data.  that was very good to hear.  There is a time stamp is present in each data row, so it can be use to explain some seasonal oddities or 'funnies' in the data.  {Still some data funnies found, results found that 1,0000+ set of socks sold {fair and possible}, but then 750+ rubber costume horse-head</font> items sold same period {very odd}.

HADOOP Logic:

Main goal of extract was to roll-up data for unique product count sold. So, individual data rows fields like customer description feedback comments at the purchase level were dropped (large text field, happy to drop).  I could always reference the full extract file in the mousepad or excel to get the the actual record detail  if needed. 

Using the main key as "product_id" and then to build a value string of multiple fields in the mapper was my design goal.  The idea here was to possibly further work or process the data once an Hadoop reduce output out file was created.  I could parse or further process the value field, once if was out of Hadoop (ie: Excel, My-Sql,  or Tableau )

I did keep the individual purchase feedback votes field in the hadoop mapper at the  individual at the record level. The idea for this was to add these up during reducer stage A latrer secondary goal was to calculate an average of approval by unique product_id by adding total_votes feedback divided  by total unique product purchased count.

Working on tsv files on both data set was able to pick these AWS key data fields for Hadoop processing/extract:

• product_category
• product_parent
• product_title
• total_votes
• product_id

Mapper key fields:

• key.set(product_id.trim());
• val.set(product_category   + "," + product_parent + "," + product_title + "," + total_votes + "," + product_id);

Reducer code sample :

Table of results:

I was able to export the Hadoop output file into Excel for better data review and trend analysis finding.

 My-SQL data check:

Since there were some unexpected values found (ie: three wolf t-shirt sales), I loaded the full data set into My-SQL and ran some queries,  The results did check or correlate with the Hadoop program unique product count purchases. 

Note: The 'three wolf t-shirt sales' (orange rows above) was an online purchase hoax with fake reviews that drove sales to record high.  (pls check, lookup in wiki: 'three wolf shirt sales')

Some other odd item total count purchase count total appeared to needed validation. One product item under two separate product descriptions different sizes did validate to positive results. A second prodcut item suspect was the total number purchased costume rubber horse heads.  By check the 'product title' in the row and also looking up the item under Amazon, it the same product but by different manufactures. 

• ---

  SELECT 
      `asw_garden_sales`.`product_id` , count(*) AS a,
      `asw_garden_sales`.`product_parent`,
      `asw_garden_sales`.`product_title`,
      `asw_garden_sales`.`product_category`
  FROM `mysql`.`asw_garden_sales`
  GROUP by `asw_garden_sales`.`product_id`
  order by a desc;