The Fifth Elephant 2014

A conference on big data and analytics

De-dup @ Scale : Experiments with DynamoDB

Submitted by Hemanth Yamijala (@yhemanth) on Thursday, 22 May 2014

Section: Full talk Technical level: Intermediate


What should you know if you want to integrate DynamoDB into your BigData application ?

In this talk, I wish to share experiences my team and I had building a BigData application using DynamoDB. Through this, we had the opportunity to explore various options you would have with using DynamoDB, the issues you would face, and the trade-offs you may need to make. By sharing these experience, I hope it with equip developers with what to watch out for under similar circumstances. Also, it will serve to highlight some of the underlying concepts of a NoSQL technology such as DynamoDB.


Mobile device vendors and app writers try to capture very fine grained application events from millions of devices around the world. They then mine this data using BigData technologies to extract insights around device and application usage. We worked with a client building an ETL application for mining this data using AWS technologies. An interesting part of this application was handling a de-duplication step for events being captured. We used DynamoDB for building the de-dup step.

In this talk, I'd like to share the team's experiences in building the application with DynamoDB. It's not so much of a "What's DynamoDB" talk as it is about "What do you need to know to use DynamoDB", particularly when you are integrating it from applications like Elastic MapReduce. In the process, I will explain some underlying concepts and capabilities of DynamoDB.

Speaker bio

I have had a long relationship with BigData technologies - dating back to early days of Hadoop at Yahoo!, where I was a committer to the Hadoop MapReduce & (now retired) Hadoop on Demand components since Hadoop 0.16 (long before it became Hadoop 1.0). Since 2010, I have been at ThoughtWorks, where I've been helping our clients on delivery projects involving BigData solutions with Hadoop and AWS components. I am passionate about distributed computing and keeping in touch with its rapid growth is a constant struggle I am happy to go through. I also love to share whatever knowledge I gain with others via talks at conferences and my blog at


  • Govind Kanshi (@govindsk) 5 years ago

    Dear Hemanth Thanks, Did you use DynamoDB for final storage/ingestion or as an intermediate storage. I am assuming you will cover the whole planning face where solutions like them have high requirements on thought process for read/write rates? Indexing - various kind of indexing useful for different scenarios or challenges you faced with evolving requirements for queries? At times this becomes challenging with platforms like them.

  • Hemanth Yamijala (@yhemanth) Proposer 5 years ago

    @vinodsan - Thanks !

    @govindsk - DynamoDB was used as an auxiliary store for the de-duplication process. The store for the transformed data was S3. The talk will cover the following aspects: Deciding on IOPS, consistency issues, interoperating with EMR, etc. Indexing will be covered as an option considered for (but ultimately not chosen) for data expiry.

  • Vinod Sankaranarayan (@vinodsan) 5 years ago

    good topic. will be interested to know more.

Login with Twitter or Google to leave a comment