The Fifth Elephant 2017

On data engineering and application of ML in diverse domains


Optimising Model performance using automated ML pipeline for predicting purchase propensity @ Fractal Analytics

Submitted by PadmaCh (@padmach) on Tuesday, 25 April 2017

Preview video

Section: Full talk for data engineering track Technical level: Advanced


Ensemble learning is the process by which multiple machine-learning models are evaluated and combined to help build a combined model that provides better results. Building these models require experimenting with not just multiple Machine-Learning models, but also with various model-parameters that help build good individual models.

In this talk, we will share how did we built an automated machine-learning pipeline to help evaluate multiple machine learning models and model parameters. The purchase propensity model used multiple ML techniques, ranging from regression techniques to Random-Forest based classifiers and helped build a machine-learning ensemble model over 100’s of millions of transaction data-points. The system that was built provided an ability to scale, both for the various modelling combinations available, and for the size of the datasets involved. We will discuss on how did we employ best practices in Spark during every step of building scalable models.


Performing Exploratory Data Analysis using Spark.
Discussion on commonly encountered issues during feature engineering.
Discussion over various classification techniques including-
Logistic regression (experimenting with regularization parameters to avoid overfitting)
Random forests
Addressing technical challenges in performing K-fold cross-validation.
Search for optimal parameters for modeling using Grid-search
Ensemble based approaches (bagging & self-training) using Spark.

Speaker bio

Padma Chitturi is Lead Engineer at Fractal Analytics Pvt Ltd and has over five years of experience in large scale data processing. She has authored the book “Apache Spark for Data Science Cookbook”. Currently, she is part of capability development at Fractal and responsible for solution development for analytical problems across multiple business domains at large scale. Prior to this, she worked for an Airlines product on a real-time processing platform at Amadeus Software Labs. She has worked on realizing large-scale deep networks (Jeffrey dean’s work in Google brain) for image classification on the big data platform Spark at Impetus. She works closely with Kafka, Spark, Storm, Cassandra, Hadoop, Deep Learning, Computer Vision and Real-time streaming. She was an open source contributor to Apache Storm.



Preview video


  • Zainab Bawa (@zainabbawa) Reviewer 2 years ago

    Thanks for this proposal, Padma. Please share a draft slide deck, detailing the contents of the proposed talk. Also share a preview video explaining what this talk is about and key takeaways for participants. We need this information by May 23 to close review on this proposal.

  • PadmaCh (@padmach) Proposer 2 years ago

    Hi, Thanks ! I ll upload them shortly

  • PadmaCh (@padmach) Proposer 2 years ago

    Hi I have uploaded the slides

  • PadmaCh (@padmach) Proposer 2 years ago

    @zainabbawa Uploaded slides and the video. Please let me know if any issues with accessibility.

  • padma priya chitturi 2 years ago

    @zainabbawa: Sent the final presentation. Unable to find the edit link to update the same in the website

  • PadmaCh (@padmach) Proposer 2 years ago

    @@zainabbawa: Uploaded the final version of the presentation

Login with Twitter or Google to leave a comment