The Fifth Elephant 2014

What should you know if you want to integrate DynamoDB into your BigData application ?

In this talk, I wish to share experiences my team and I had building a BigData application using DynamoDB. Through this, we had the opportunity to explore various options you would have with using DynamoDB, the issues you would face, and the trade-offs you may need to make. By sharing these experience, I hope it with equip developers with what to watch out for under similar circumstances. Also, it will serve to highlight some of the underlying concepts of a NoSQL technology such as DynamoDB.

Outline

Mobile device vendors and app writers try to capture very fine grained application events from millions of devices around the world. They then mine this data using BigData technologies to extract insights around device and application usage. We worked with a client building an ETL application for mining this data using AWS technologies. An interesting part of this application was handling a de-duplication step for events being captured. We used DynamoDB for building the de-dup step.

In this talk, I’d like to share the team’s experiences in building the application with DynamoDB. It’s not so much of a “What’s DynamoDB” talk as it is about “What do you need to know to use DynamoDB”, particularly when you are integrating it from applications like Elastic MapReduce. In the process, I will explain some underlying concepts and capabilities of DynamoDB.

Speaker bio

I have had a long relationship with BigData technologies - dating back to early days of Hadoop at Yahoo!, where I was a committer to the Hadoop MapReduce & (now retired) Hadoop on Demand components since Hadoop 0.16 (long before it became Hadoop 1.0). Since 2010, I have been at ThoughtWorks, where I’ve been helping our clients on delivery projects involving BigData solutions with Hadoop and AWS components. I am passionate about distributed computing and keeping in touch with its rapid growth is a constant struggle I am happy to go through. I also love to share whatever knowledge I gain with others via talks at conferences and my blog at yhemanth.wordpress.com.