The Fifth Elephant 2017

On data engineering and application of ML in diverse domains


Processing mission critical events in real time

Submitted by Tarun Gupta (@tarung) on Sunday, 30 April 2017

Preview video

Section: Crisp talk for data engineering track Technical level: Intermediate


If you have an event driven mission-critical application, you are always worried about such application failing and leading to opportunity or revenue loss. For a data based adtech company like Zapr Media Labs, one such application is deducting costs in real time for displayed advertisements and stop displaying when daily or hourly caps are reached. Such applications have challenges of scalability, exactly once guarantee, de-duping, replayability and zero data loss during application upgrades.

In this talk, I will discuss the design choices we have made to tackle these challenges, various technology evaluated. Finally, how we used Spark and Kafka with some neat tricks to achieve this and learnings from the journey of this applications.


Talk will be structured in the following format
1) Requirements of our application and other similar use cases
2) Challenges with such applications
3) Design choices to tackle each challenge
4) Technology stack evaluated and used
5) Lessons Learned

Speaker bio

Technical Lead at Zapr Media Labs. Leading infrastructure team at Zapr to build scalable infrastructure that helps profiles millions of users and bridge between TV to Mobile advertisment campaigns. Before this Worked at Qualcomm as Engineer with Linux USB team working on development of USB function drivers, USB controller drivers, Linux USB Host and Device Stack for Snapdragon based chip-sets.



Preview video


  • Zainab Bawa (@zainabbawa) Reviewer 2 years ago

    The learnings slide, which is the most critical aspect of the presentation, is blank.

  • Zainab Bawa (@zainabbawa) Reviewer 2 years ago

    Share a two-min preview video explaining what this talk is about and why participants should attend in order to complete review of this proposal.

Login with Twitter or Google to leave a comment