The Fifth Elephant 2017

On data engineering and application of ML in diverse domains

##Theme and format
The Fifth Elephant 2017 is a four-track conference on:

  1. Data engineering – building pipelines and platforms; exposure to latest open source tools for data mining and real-time analytics.
  2. Application of Machine Learning (ML) in diverse domains such as IOT, payments, e-commerce, education, ecology, government, agriculture, computational biology, social network analysis and emerging markets.
  3. Hands-on tutorials on data mining tools, and ML platforms and techniques.
  4. Off-the-record (OTR) sessions on privacy issues concerning data; building data pipelines; failure stories in ML; interesting problems to solve with data science; and other relevant topics.

The Fifth Elephant is a conference for practitioners, by practitioners.

Talk submissions are now closed.

You must submit the following details along with your proposal, or within 10 days of submission:

  1. Draft slides, mind map or a textual description detailing the structure and content of your talk.
  2. Link to a self-record, two-minute preview video, where you explain what your talk is about, and the key takeaways for participants. This preview video helps conference editors understand the lucidity of your thoughts and how invested you are in presenting insights beyond your use case. Please note that the preview video should be submitted irrespective of whether you have spoken at past editions of The Fifth Elephant.
  3. If you submit a workshop proposal, you must specify the target audience for your workshop; duration; number of participants you can accommodate; pre-requisites for the workshop; link to GitHub repositories and documents showing the full workshop plan.

##About the conference
This year is the sixth edition of The Fifth Elephant. The conference is a renowned gathering of data scientists, programmers, analysts, researchers, and technologists working in the areas of data mining, analytics, machine learning and deep learning from different domains.

We invite proposals for the following sessions, with a clear focus on the big picture and insights that participants can apply in their work:

  • Full-length, 40-minute talks.
  • Crisp, 15-minute talks.
  • Sponsored sessions, of 15 minutes and 40 minutes duration (limited slots available; subject to editorial scrutiny and approval).
  • Hands-on tutorials and workshop sessions of 3-hour and 6-hour duration where participants follow instructors on their laptops.
  • Off-the-record (OTR) sessions of 60-90 minutes duration.

##Selection Process

  1. Proposals will be filtered and shortlisted by an Editorial Panel.
  2. Proposers, editors and community members must respond to comments as openly as possible so that the selection processs is transparent.
  3. Proposers are also encouraged to vote and comment on other proposals submitted here.

Selection Process Flowchart

We will notify you if we move your proposal to the next round or reject it. A speaker is NOT confirmed for a slot unless we explicitly mention so in an email or over any other medium of communication.

Selected speakers must participate in one or two rounds of rehearsals before the conference. This is mandatory and helps you to prepare well for the conference.

There is only one speaker per session. Entry is free for selected speakers.

##Travel grants
Partial or full grants, covering travel and accomodation are made available to speakers delivering full sessions (40 minutes) and workshops. Grants are limited, and are given in the order of preference to students, women, persons of non-binary genders, and speakers from Asia and Africa.

##Commitment to Open Source
We believe in open source as the binding force of our community. If you are describing a codebase for developers to work with, we’d like for it to be available under a permissive open source licence. If your software is commercially licensed or available under a combination of commercial and restrictive open source licences (such as the various forms of the GPL), you should consider picking up a sponsorship. We recognise that there are valid reasons for commercial licensing, but ask that you support the conference in return for giving you an audience. Your session will be marked on the schedule as a “sponsored session”.

##Important Dates:

  • Deadline for submitting proposals: June 10
  • First draft of the coference schedule: June 20
  • Tutorial and workshop announcements: June 20
  • Final conference schedule: July 5
  • Conference dates: 27-28 July

For more information about speaking proposals, tickets and sponsorships, contact or call +91-7676332020.

Hosted by

All about data science and machine learning

Vimal Sharma


Apache Atlas Introduction: Need for Governance and Metadata management

Submitted May 26, 2017

Apache Atlas is the one stop solution for data governance and metadata management on enterprise Hadoop clusters. Atlas has a scalable and extensible architecture which can plug into many Hadoop components to manage their metadata in a central repository. Vimal Sharma will review the challenges associated with managing large datasets on Hadoop clusters and demonstrate how Atlas solves the problem. Vimal will focus on Cross Component lineage tracking capability of Apache Atlas. Vimal will also discuss the upcoming features and roadmap of Apache Atlas.

The talk is intended to introduce Apache Atlas and its capabilities to audience. It is also intended to invite potential developers to contribute to the Apache Atlas project.


Apache Atlas Project Introduction
Data Governance challenge and use case scenarios
Atlas architechture
Cross component lineage capability of Atlas
Apache Ranger integration to enforce tag based policies
Atlas TypeSystem
Model Spark Dataframe as an Atlas type
Demo based on the above model
Invitation to contribute

More details on Order of presentation

Why Apache Atlas(What are the use cases)
Enterprises have 100s of ETL pipelines wherein developers take the source data, apply transformations and persist the result into the warehouse. Now, if an upstream pipeline breaks/fails, how does the owner of current dataset narrow down on the cause and culprit ETL pipeline. Further, if the current pipeline breaks, the owner has no mechanism to alert the owners of downstream processes. A tool which could keep track of the provenance/lineage/impact of a dataset would solve this issue. Atlas has the capability to track lineage of the datasets.

ETL redundancy is another striking issue in current enterprise Hadoop deployments. Many developers process data and persist it to the warehouse. They don’t have any mechanism to detect if the result they need is already computed and resides in a dataset. Using the lineage diagram and classification feature of Atlas, developers can look into the details of derived datasets and skip the expensive processing if the information is already available in one of the derived datasets.

Further, enterprises need to adhere to compliance policies which span multiple datasets across components like Hive, HBase, HDFS etc. How can the business make sure that a particular policy is enforced across datasets in these components. Datasets can be tagged in Atlas and Ranger can use its Tag based policy feature to enforce constraints

Cluster admin may need to periodically clean up the unused/dormant datasets from the warehouse. How can the admin narrow down on the candidate datasets for archival. Atlas is useful in determining the relevance of a dataset on the basis of the number of tags and downstream datasets derived from it.

What is Apache Atlas
Apache Atlas is the governance and metadata framework for Hadoop. Atlas has a scalable and extensible architecture which can plug into many Hadoop components to manage their metadata in a central repository. By virtue of its extensible TypeSystem, any arbitrary component(not necessarily a Hadoop component) can be modelled to capture the metadata of its datasets and events. The metadata events can then be classified using tags which can further be used to enforce security policies by Ranger. When a dataset derives from another dataset, the event can be registered and Atlas will capture the lineage relationship.

Atlas provides inbuilt support for some Hadoop components like Hive, Storm and Sqoop. This means that whenever new datasets and events are created in these components, Atlas captures the metadata of those events. For new components like Spark, first the model of the metadata to be captured needs to be defined and registered with Atlas. Once the model is in place, datasets and events occuring in that component can be registered with Atlas using its rich REST API.

The demo for the presentation will cover these parts. First, Spark datasets will be modeled and registered with Atlas. Then, a realistic use case will be considered where we will capture a lineage relationship across components like HDFS and Kafka. We will then go to Atlas UI and inspect lineage and other features like tag based classification, search and advanced search

Speaker bio

Vimal Sharma is Apache Atlas PMC and Committer at Hortonworks. Vimal graduated from IIT Kanpur with a B.Tech in Computer Science. Vimal is highly passionate about Hadoop stack and has previously worked on scaling backend systems at WalmartLabs using Spark and Kafka.

Vimal was a speaker at ApacheCon BigData 2017



{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

All about data science and machine learning