The Fifth Elephant 2017

In this talk I will talk about how we used Apache Spark to build a self service data platform at Indix that helped democratise access to several datasets at Indix to our customers and the internal engineering and data science teams. I will also share some of the lessons learnt while optimizing performance and tuning Spark jobs that run on these datasets.

Outline

Indix is building the world’s largest catalog of structured product information and to achieve this mammoth goal we’ve built various data pipelines using Hadoop, Kafka, Akka Streams and Apache Spark.
We started using Apache Spark well before its 1.0 release with mixed success but over the last couple of years, it has become the framework of choice for building data pipelines at Indix.

My talk will consist of two sections

Section I - Building a self-service data platform on Apache Spark

Every day the Indix Data platform teams generate several terabytes of data, what we call as internal datasets. This includes crawled HTML pages, semi-structured product data extracted from these pages and the output of several ML algorithms on these datasets. These datasets are available on HDFS, S3 or on Kafka topics, have different schemas and are encoded in Avro, Thrift, CSV or Parquet formats.
18 months ago, accessing these datasets, processing them or running ad-hoc queries on them was a painful process and users had to be conversant with all these tools and technologies to make sense of it. Since then we have built a data platform on top of Apache Spark to make it easier for anyone to access these anytime. We used our own abstractions and married them with the abstractions like Datasets of Apache Spark to provide a seamless and self service experience to our internal teams. The first section will state the problem, the abstractions built and finally show a demo of the tool we built.

Section II - Optimizing Spark jobs on large datasets

If you have tried your hands at tuning Spark jobs on large datasets, you would know that it is a dark art as you run into a number of hidden Spark pitfalls that only surface once your datasets grow into several terabytes and your jobs are running across 100s of executors. One also needs a deeper understanding of Sparks internals especially around memory management, shuffle and partition management when running joins or aggregates. The second section of the talk will talk about the common Spark piftalls, the scenarios they usually occur in and best pratices to adopt for optimal performance.

Whether you are just starting out with Spark or you are a seasoned developer, by the end of the talk, you should learn how to build data products on top of Apache Spark with tips and tricks to help make your jobs run faster.

Speaker bio

I work as a software engineer at Indix on the data platform team where I am part of the core team that built and manages the self service data platform on Apache Spark.

Links

• Talk on optimizing spark jobs (section II) at Chennai Geek Night in Mar 2017 - https://www.youtube.com/watch?v=eW82yoGoKFY
• Wrote a blog post on the subject that was published by “Towards Data Science” - https://medium.com/towards-data-science/lessons-from-using-spark-to-process-large-amounts-of-data-part-i-421ba36002f3

Slides

https://speakerdeck.com/reema/optimizing-spark-jobs