The Fifth Elephant 2012

Finding the elephant in the data.

Lessons Learned : Scaling Hadoop and BigData in Cloud (Amazon EMR )

Submitted by Vijay Rayapati (@amnigos) on Tuesday, 20 March 2012

videocam_off

Technical level

Intermediate

Section

Big Data Infrastructure & Processing

Session type

Lecture

Status

Submitted

Vote on this proposal

Login to vote

Total votes:  +14

Objective

High level technology and business perspective around BigData including how to and why leverage cloud based platforms like Amazon EMR along with Map Reduce for data analysis.

Expect to learn concepts, insights, challenges in problem solving, scaling, dealing with data and performance tuning.

Description

I will be talking about Hadoop, Map Reduce in general and how to leverage Cloud based platforms like Amazon EMR for Hadoop Map Reduce jobs. Will be sharing lessons learned from managing hyper scale production Hadoop clusters and tuning for performance in general – Think 68400 GB RAM, 26000 CPUs and 1700000 GB Disks :)

This talk is aimed at providing insights and challenges while building large scale data analysis platforms using Hadoop and technical challenges involved in scaling algorithms to data storage to json parsers to in-memory data stores to managing 100's of jobs.

Requirements

Basic understanding of Hadoop will be good.

Speaker bio

An Engineer (aka CTO) working at Kuliza on Platforms, Cloud and BigData. Previously at #Startups #GizaPage #Trilogy #eFoodlet #Michelin.

Have worked on large scale web platforms and distributed systems for enterprises and consumer web. At Kuliza, we work on PetaByte scale EMR clusters with 100's of nodes running Hadoop for data analysis. Our cloud team runs 500+ production servers with every possible #technical #stack #configuration out there!

Links

Comments

  • 1
    Govind Kanshi (@govindsk) 6 years ago

    Thanks for doing this talk, this will be of great help to many folks whether they use on premise or otherwise offerings.

Login with Twitter or Google to leave a comment