The Fifth Elephant 2013

Whether a startup or a big organization, everyone faces the challenge of keeping track of data in real time. Data availability in real time has significant advantages, particularly in the search engine marketing and analytics space, where most decisions are made on-the-fly. However real-time systems often are assumed to need the latest and greatest in technology.

The agenda for this session is to familiarize the audience with how we built a real time tracking system at scale with just MySQL, Apache2, Perl and Java.

Outline

At Sokrati we track 2B+ params per month, all of this in real time. All of this managed with just 11 boxes, and we wouldnt need any additional non-storage box even if this number triples.

In this talk I will talk about the following:

• How did our architecture evolve to achieve sub second latencies.

• How did we achieve this with help of simply Apache2, Perl, MySQL and Java.

• How does this work with no downtime.

• How do we handle schema changes with terabytes of data without significant downtime.

Speaker bio

Nikil Doshi is Senior Software Design Engineer at Sokrati, a digital marketing and analytics startup in Pune with a mission to delight advertisers globally. He currently heads the team that not only captures tracking data in real time but also implements algorithms which take decisions on this real time data to achieve client goals. His interests include data tracking, making it available in real time, building scalable systems.