The Fifth Elephant 2017

On data engineering and application of ML in diverse domains

Processing mission critical events in real time

Submitted by Tarun Gupta (@tarung) on Sunday, 30 April 2017

videocam
Preview video

Technical level

Intermediate

Section

Crisp talk for data engineering track

Status

Submitted

Vote on this proposal

Login to vote

Total votes:  +5

Abstract

If you have an event driven mission-critical application, you are always worried about such application failing and leading to opportunity or revenue loss. For a data based adtech company like Zapr Media Labs, one such application is deducting costs in real time for displayed advertisements and stop displaying when daily or hourly caps are reached. Such applications have challenges of scalability, exactly once guarantee, de-duping, replayability and zero data loss during application upgrades.

In this talk, I will discuss the design choices we have made to tackle these challenges, various technology evaluated. Finally, how we used Spark and Kafka with some neat tricks to achieve this and learnings from the journey of this applications.

Outline

Talk will be structured in the following format
1) Requirements of our application and other similar use cases
2) Challenges with such applications
3) Design choices to tackle each challenge
4) Technology stack evaluated and used
5) Lessons Learned

Speaker bio

Technical Lead at Zapr Media Labs. Leading infrastructure team at Zapr to build scalable infrastructure that helps profiles millions of users and bridge between TV to Mobile advertisment campaigns. Before this Worked at Qualcomm as Engineer with Linux USB team working on development of USB function drivers, USB controller drivers, Linux USB Host and Device Stack for Snapdragon based chip-sets.

Links

Slides

https://docs.google.com/presentation/d/16HItPQ-DqvBwCILC95T6pVsvQG7mh40_xSGgSH7WIlw/edit?usp=sharing

Preview video

https://www.youtube.com/watch?v=gvkb0DVVvdU

Comments

  • 1
    Zainab Bawa (@zainabbawa) Reviewer a year ago

    The learnings slide, which is the most critical aspect of the presentation, is blank.

  • 1
    Zainab Bawa (@zainabbawa) Reviewer a year ago

    Share a two-min preview video explaining what this talk is about and why participants should attend in order to complete review of this proposal.

Login with Twitter or Google to leave a comment