The Fifth Elephant 2015
A conference on data, machine learning, and distributed and parallel computing
MP
Madhukara Phatak
@phatak_dev
Anatomy of RDD : A Deep dive into Spark RDD Data structure.
Submitted May 6, 2015
Section:
Full Talk
Technical level:
Advanced
RDD is the core abstraction of Apache Spark. So understanding RDD in depth is very
crucial to use spark very effectively. This talks aims to take audience a deep
dive into RDD to make them understand why it’s so powerful.
Outline
This is an Advanced talks aimed toward people who already know Spark. This talk
tries to deconstruct RDD abstraction to peek inside. We will be discussing about
- Immutability and Distribution
- Partitions
- Partition API’s like mapParittions, lookUp etc
- Implementation of Laziness
- RDD dependency hierarchy
- Transformation and Action implementation
- Caching implementation
All the above topics are discussed with real code.
Requirements
- Prior experience of Working with Spark
Speaker bio
Madhukara phatatak is a Bigdata consultant @ Datamantra. He has been actively working in Hadoop,Spark and its ecosystem projects from last 5 years.
He was lead developer of Nectar, a ML library for hadoop.He also contributed to hadoop source code to improve cyclic checks in Jobcontrol api.With raise of Apache Spark, he with his team has open sourced courseera machine learning course examples on spark here. He blogs on spark here. Also he runs a Spark meetup group in Bangalore.
Links
- Apache Spark Architecture : https://www.youtube.com/watch?v=65aV15uDKgA
- Extending Spark API : https://www.youtube.com/watch?v=2T1Pub7IxCU
- MindMap : https://drive.google.com/file/d/0B_j4gJdtrf_QMFJyREF6SVJNTHc/view?usp=sharing
{{ gettext('Login to leave a comment') }}
{{ gettext('Post a comment…') }}{{ errorMsg }}
{{ gettext('No comments posted yet') }}