The Fifth Elephant 2017

On data engineering and application of ML in diverse domains

Tarun Gupta


Processing mission critical events in real time

Submitted Apr 30, 2017

If you have an event driven mission-critical application, you are always worried about such application failing and leading to opportunity or revenue loss. For a data based adtech company like Zapr Media Labs, one such application is deducting costs in real time for displayed advertisements and stop displaying when daily or hourly caps are reached. Such applications have challenges of scalability, exactly once guarantee, de-duping, replayability and zero data loss during application upgrades.

In this talk, I will discuss the design choices we have made to tackle these challenges, various technology evaluated. Finally, how we used Spark and Kafka with some neat tricks to achieve this and learnings from the journey of this applications.


Talk will be structured in the following format

  1. Requirements of our application and other similar use cases
  2. Challenges with such applications
  3. Design choices to tackle each challenge
  4. Technology stack evaluated and used
  5. Lessons Learned

Speaker bio

Technical Lead at Zapr Media Labs. Leading infrastructure team at Zapr to build scalable infrastructure that helps profiles millions of users and bridge between TV to Mobile advertisment campaigns. Before this Worked at Qualcomm as Engineer with Linux USB team working on development of USB function drivers, USB controller drivers, Linux USB Host and Device Stack for Snapdragon based chip-sets.



{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

All about data science and machine learning