If you have an event driven mission-critical application, you are always worried about such application failing and leading to opportunity or revenue loss. For a data based adtech company like Zapr Media Labs, one such application is deducting costs in real time for displayed advertisements and stop displaying when daily or hourly caps are reached. Such applications have challenges of scalability, exactly once guarantee, de-duping, replayability and zero data loss during application upgrades.

In this talk, I will discuss the design choices we have made to tackle these challenges, various technology evaluated. Finally, how we used Spark and Kafka with some neat tricks to achieve this and learnings from the journey of this applications.

Outline

Talk will be structured in the following format
1) Requirements of our application and other similar use cases
2) Challenges with such applications
3) Design choices to tackle each challenge
4) Technology stack evaluated and used
5) Lessons Learned

Speaker bio

Technical Lead at Zapr Media Labs. Leading infrastructure team at Zapr to build scalable infrastructure that helps profiles millions of users and bridge between TV to Mobile advertisment campaigns. Before this Worked at Qualcomm as Engineer with Linux USB team working on development of USB function drivers, USB controller drivers, Linux USB Host and Device Stack for Snapdragon based chip-sets.

Links

• https://www.linkedin.com/in/tarunsgupta/

Slides

https://docs.google.com/presentation/d/16HItPQ-DqvBwCILC95T6pVsvQG7mh40_xSGgSH7WIlw/edit?usp=sharing