Submitted Jul 15, 2016

Hadoop & Cloud Storage: Object Store Integration in Production

Section: Crisp talk Technical level: Intermediate

This proposal has been added to the schedule

Today’s typical Apache Hadoop deployments use HDFS for persistent, fault-tolerant storage of big data files. However, recent emerging architectural patterns increasingly rely on cloud object storage such as S3, Azure Blob Store, GCS, which are designed for cost-efficiency, scalability and geographic distribution. Hadoop supports pluggable file system implementations to enable integration with these systems for use cases such as off-site backup or even complex multi-step ETL, but applications may encounter unique challenges related to eventual consistency, performance and differences in semantics compared to HDFS.

Outline

In this session, I am going to explore challenges mentioned in abstract and present recent work to address them in a comprehensive effort spanning multiple Hadoop ecosystem components, including the Object Store FileSystem connector, Hive, Tez and ORC. Our goal is to improve correctness, performance, security and operations for users that choose to integrate Hadoop with Cloud Storage. We use S3 and S3A connector as case study.

Speaker bio

Rajesh Balamohan is a “Member of Technical Staff” in Hortonworks. He has been working on Hadoop for last couple of years. Recently he has been concentrating on Tez performance at scale. Rajesh is a committer and PMC in Apache Tez project.

Slides

https://drive.google.com/open?id=19snM3dNusd7TqonJXrQZ-Vtx0UmHGsRv0lVoFlITCUg

Comments

Up next

Meet the needs of content marketing with the power of NLP

Content Marketing is one of the recent buzz in the space of digital marketing. Content Marketing broadly refers to focusing on providing quality and useful content to customers for engaging and attracting customers towards a brand. With the proliferation of channels where these content can potentially be delivered, there is an increasing demand from content writers to provide content that can be adapted to the needs of each of these channels. Further, the speed at which the content needs to be generated poses a unique challenge and calls for ways to aid in creating and adapting these content quickly for different channels. In this talk, I will walk through some of the recent advances in Natural Language Processing to aid in effective content authoring to meet the needs of a content writer.

Jul 13, 2016

The Fifth Elephant 2016