The Fifth Elephant 2017

Ensemble learning is the process by which multiple machine-learning models are evaluated and combined to help build a combined model that provides better results. Building these models require experimenting with not just multiple Machine-Learning models, but also with various model-parameters that help build good individual models.

In this talk, we will share how did we built an automated machine-learning pipeline to help evaluate multiple machine learning models and model parameters. The purchase propensity model used multiple ML techniques, ranging from regression techniques to Random-Forest based classifiers and helped build a machine-learning ensemble model over 100’s of millions of transaction data-points. The system that was built provided an ability to scale, both for the various modelling combinations available, and for the size of the datasets involved. We will discuss on how did we employ best practices in Spark during every step of building scalable models.

Outline

Performing Exploratory Data Analysis using Spark.
Discussion on commonly encountered issues during feature engineering.
Discussion over various classification techniques including-
Logistic regression (experimenting with regularization parameters to avoid overfitting)
Random forests
GBM
Addressing technical challenges in performing K-fold cross-validation.
Search for optimal parameters for modeling using Grid-search
Ensemble based approaches (bagging & self-training) using Spark.

Speaker bio

Padma Chitturi is Lead Engineer at Fractal Analytics Pvt Ltd and has over five years of experience in large scale data processing. She has authored the book “Apache Spark for Data Science Cookbook”. Currently, she is part of capability development at Fractal and responsible for solution development for analytical problems across multiple business domains at large scale. Prior to this, she worked for an Airlines product on a real-time processing platform at Amadeus Software Labs. She has worked on realizing large-scale deep networks (Jeffrey dean’s work in Google brain) for image classification on the big data platform Spark at Impetus. She works closely with Kafka, Spark, Storm, Cassandra, Hadoop, Deep Learning, Computer Vision and Real-time streaming. She was an open source contributor to Apache Storm.
www.linkedin.com/in/padmachitturi

Links

• https://www.amazon.in/Apache-Spark-Data-Science-Cookbook-ebook/dp/B01CGKAJ1Q
• http://www.datasciencecentral.com/profiles/blogs/implementing-a-distributed-deep-learning-network-over-spark
• http://signalprocessingsociety.org/newsletter/2015/02/implementing-a-distributed-deep-learning-network-over-spark/

Slides

https://drive.google.com/file/d/0B-rWMe2CC0Z4UnFrNEtRQlNTRlU/view?usp=sharing