The Fifth Elephant 2013

An Event on Big Data and Cloud Computing

Tracking 2B parameters/month in real time - with just MySQL!

Submitted by Nikil Doshi (@nikildoshi) on Tuesday, 4 June 2013

videocam_off

Technical level

Intermediate

Section

Storage and Databases

Status

Submitted

Vote on this proposal

Login to vote

Total votes:  +14

Objective

Whether a startup or a big organization, everyone faces the challenge of keeping track of data in real time. Data availability in real time has significant advantages, particularly in the search engine marketing and analytics space, where most decisions are made on-the-fly. However real-time systems often are assumed to need the latest and greatest in technology.

The agenda for this session is to familiarize the audience with how we built a real time tracking system at scale with just MySQL, Apache2, Perl and Java.

Description

At Sokrati we track 2B+ params per month, all of this in real time. All of this managed with just 11 boxes, and we wouldnt need any additional non-storage box even if this number triples.

In this talk I will talk about the following:

  • How did our architecture evolve to achieve sub second latencies.

  • How did we achieve this with help of simply Apache2, Perl, MySQL and Java.

  • How does this work with no downtime.

  • How do we handle schema changes with terabytes of data without significant downtime.

Speaker bio

Nikil Doshi is Senior Software Design Engineer at Sokrati, a digital marketing and analytics startup in Pune with a mission to delight advertisers globally. He currently heads the team that not only captures tracking data in real time but also implements algorithms which take decisions on this real time data to achieve client goals. His interests include data tracking, making it available in real time, building scalable systems.

Comments

Login with Twitter or Google to leave a comment