The Fifth Elephant 2017

Many applications we use today are powered by cloud and mobile. One of the critical components that drives engagement for the platforms on cloud is the recommendation engine. Recommendation systems are becoming all-pervasive. The transactions/interactions we have with the platform decide the next set of recommended items. As both users and the number of products offered on the platform scale, we are hit with two kinds of challenges - engineering and machine learning.

This talk is about how we designed a real-time recommendation engine at Red Hat when the transactions data is both Big Data and Wide data. We define wide data as those whose number of items in a transaction basket is greater than 1000. Some examples of big and wide data are: Financial Instruments traded by portfolio manager in a day; Products shipped from a warehouse; software components in a cloud platform etc.

The standard approaches have been market basket analysis(frequent pattern mining), collaborative filtering(matrix factorization) and currently deep learning.

Apache Spark lends itself nicely to build the data science pipeline: ingestion, data processing and machine learning. Out of the box, we have a parallel implementation of FP Growth algorithm for mining frequent itemsets. But as our data became wider, model training performance took a hit. This talk discusses how we used another popular recommendation algorithm in Spark - Alternating Least Squares to generate frequent itemsets. The new approach was faster and scaled well for big and wide data.

The engineering and data science approaches are novel and the attendees will learn how to build recommendation systems on the cloud, some of the challenges and some ideas on how to overcome them.

Who is this presentation for:
Data Scientists, Product Managers

Audience Level:
Beginner

Takeaway:
Attendees will learn how to build recommendation systems for wide data. Attendees will also learn how to use Apache Spark to build ML pipeline for recommendation system.

Pre-req knowledge
Interest in data science. It will help if attendees know some examples of recommendation systems.

Outline

1. Introduction to the problem (recommendations for big and wide data)
2. Overview of common approaches (Frequent Pattern mining)
3. Implementation of a common approach on Apache Spark (FP Growth)
4. Challenges with traditional models for recommendation systems
5. How we tested/measured the bottlenecks
6. Other approach considerations: Alternate Least Squares, Deep Learning
7. Pros and Cons of those approaches
8. Use Apache Spark ALS to generate frequent patterns
9. Performance comparisons - model training and in real-time predictions
10. The chosen model architecture
11. Demo and closing thoughts

Speaker bio

Harjinder Mistry is currently a member of Developer-Tools team in RedHat, where he is incorporating data science into next-generation developer tools powered by Spark. Prior to RedHat, he was a member of IBM Analytics team and he developed Spark-ML pipeline components of IBM Analytics Platform. Earlier, he had spent several years in DB2 SQL Query Optimizer team building and fixing the mathematical model that decides the query execution plan. He holds M.Tech. degree from IIIT, Bangalore, India.

Slides

https://speakerdeck.com/harjinderhari/recommendation-engine-for-wide-transactions