The Fifth Elephant 2015
A conference on data, machine learning, and distributed and parallel computing
rt
ravi teja
@ravi_teja
Think Incremental with hive.
Submitted Jun 15, 2015
Section:
Crisp Talk
Technical level:
Intermediate
Session type:
Demo
Hive is a data warehouse infrastructure over hadoop for summarization, query, and analysis of data.
We propose a incremental processing approach for hive.
This would optimise the data processing speeds upto ~70% .
Outline
Data is IP.
In the current day scenario, data processing at scale ,has become a necessity for every company and every system.
With scale comes challenges and need for optimisations. Incremental is one of it!
To process with scale, the de-facto tools used are MR, Hive ,Pig and Spark.
Hive holds a big chunk in these!
Running computations over the complete data every time would be very inconvenient and is inefficient way to process in many cases.
Instead computing the complete data, computing only the newly arrived data and then merging the result would be of a great use.
This approach would enable the processing to happen only the new feed into the system instead of re-processing the complete available data set.
Incremental read ,incremental joins and sidelines will be the main aspects of this approach.
Sideline of join failures for the data which hasn’t arrived yet while incremental read.
Speaker bio
Ravi Teja Chilukuri works as a Software developer at Flipkart’s Data Platform. He has been working on the processing and web-analytics layer at Flipkart. Prior to Flipkart he has been working with the data teams for Paypal and Huawei.
The speaker has given company wide talks on Yarn and Mapreduce earlier in PayPal and Huawei and is a Mapreduce and Yarn contributor at Apache Software Foundation.
Siddhartha Reddy is an Architect at Flipkart. Siddhartha Reddy and Ravi are working together on the incremental pipeline in the Data platform.
{{ gettext('Login to leave a comment') }}
{{ gettext('Post a comment…') }}{{ errorMsg }}
{{ gettext('No comments posted yet') }}