##Theme and format
The Fifth Elephant 2017 is a four-track conference on:
- Data engineering – building pipelines and platforms; exposure to latest open source tools for data mining and real-time analytics.
- Application of Machine Learning (ML) in diverse domains such as IOT, payments, e-commerce, education, ecology, government, agriculture, computational biology, social network analysis and emerging markets.
- Hands-on tutorials on data mining tools, and ML platforms and techniques.
- Off-the-record (OTR) sessions on privacy issues concerning data; building data pipelines; failure stories in ML; interesting problems to solve with data science; and other relevant topics.
The Fifth Elephant is a conference for practitioners, by practitioners.
Talk submissions are now closed.
You must submit the following details along with your proposal, or within 10 days of submission:
- Draft slides, mind map or a textual description detailing the structure and content of your talk.
- Link to a self-record, two-minute preview video, where you explain what your talk is about, and the key takeaways for participants. This preview video helps conference editors understand the lucidity of your thoughts and how invested you are in presenting insights beyond your use case. Please note that the preview video should be submitted irrespective of whether you have spoken at past editions of The Fifth Elephant.
- If you submit a workshop proposal, you must specify the target audience for your workshop; duration; number of participants you can accommodate; pre-requisites for the workshop; link to GitHub repositories and documents showing the full workshop plan.
##About the conference
This year is the sixth edition of The Fifth Elephant. The conference is a renowned gathering of data scientists, programmers, analysts, researchers, and technologists working in the areas of data mining, analytics, machine learning and deep learning from different domains.
We invite proposals for the following sessions, with a clear focus on the big picture and insights that participants can apply in their work:
- Full-length, 40-minute talks.
- Crisp, 15-minute talks.
- Sponsored sessions, of 15 minutes and 40 minutes duration (limited slots available; subject to editorial scrutiny and approval).
- Hands-on tutorials and workshop sessions of 3-hour and 6-hour duration where participants follow instructors on their laptops.
- Off-the-record (OTR) sessions of 60-90 minutes duration.
- Proposals will be filtered and shortlisted by an Editorial Panel.
- Proposers, editors and community members must respond to comments as openly as possible so that the selection processs is transparent.
- Proposers are also encouraged to vote and comment on other proposals submitted here.
We will notify you if we move your proposal to the next round or reject it. A speaker is NOT confirmed for a slot unless we explicitly mention so in an email or over any other medium of communication.
Selected speakers must participate in one or two rounds of rehearsals before the conference. This is mandatory and helps you to prepare well for the conference.
There is only one speaker per session. Entry is free for selected speakers.
Partial or full grants, covering travel and accomodation are made available to speakers delivering full sessions (40 minutes) and workshops. Grants are limited, and are given in the order of preference to students, women, persons of non-binary genders, and speakers from Asia and Africa.
##Commitment to Open Source
We believe in open source as the binding force of our community. If you are describing a codebase for developers to work with, we’d like for it to be available under a permissive open source licence. If your software is commercially licensed or available under a combination of commercial and restrictive open source licences (such as the various forms of the GPL), you should consider picking up a sponsorship. We recognise that there are valid reasons for commercial licensing, but ask that you support the conference in return for giving you an audience. Your session will be marked on the schedule as a “sponsored session”.
- Deadline for submitting proposals: June 10
- First draft of the coference schedule: June 20
- Tutorial and workshop announcements: June 20
- Final conference schedule: July 5
- Conference dates: 27-28 July
For more information about speaking proposals, tickets and sponsorships, contact email@example.com or call +91-7676332020.
Lessons Learnt building and optimizing a self service Data Platform on Apache Spark at Indix
In this talk I will talk about how we used Apache Spark to build a self service data platform at Indix that helped democratise access to several datasets at Indix to our customers and the internal engineering and data science teams. I will also share some of the lessons learnt while optimizing performance and tuning Spark jobs that run on these datasets.
Indix is building the world’s largest catalog of structured product information and to achieve this mammoth goal we’ve built various data pipelines using Hadoop, Kafka, Akka Streams and Apache Spark.
We started using Apache Spark well before its 1.0 release with mixed success but over the last couple of years, it has become the framework of choice for building data pipelines at Indix.
My talk will consist of two sections
Every day the Indix Data platform teams generate several terabytes of data, what we call as internal datasets. This includes crawled HTML pages, semi-structured product data extracted from these pages and the output of several ML algorithms on these datasets. These datasets are available on HDFS, S3 or on Kafka topics, have different schemas and are encoded in Avro, Thrift, CSV or Parquet formats.
18 months ago, accessing these datasets, processing them or running ad-hoc queries on them was a painful process and users had to be conversant with all these tools and technologies to make sense of it. Since then we have built a data platform on top of Apache Spark to make it easier for anyone to access these anytime. We used our own abstractions and married them with the abstractions like Datasets of Apache Spark to provide a seamless and self service experience to our internal teams. The first section will state the problem, the abstractions built and finally show a demo of the tool we built.
If you have tried your hands at tuning Spark jobs on large datasets, you would know that it is a dark art as you run into a number of hidden Spark pitfalls that only surface once your datasets grow into several terabytes and your jobs are running across 100s of executors. One also needs a deeper understanding of Sparks internals especially around memory management, shuffle and partition management when running joins or aggregates. The second section of the talk will talk about the common Spark piftalls, the scenarios they usually occur in and best pratices to adopt for optimal performance.
Whether you are just starting out with Spark or you are a seasoned developer, by the end of the talk, you should learn how to build data products on top of Apache Spark with tips and tricks to help make your jobs run faster.
I work as a software engineer at Indix on the data platform team where I am part of the core team that built and manages the self service data platform on Apache Spark.
- Talk on optimizing spark jobs (section II) at Chennai Geek Night in Mar 2017 - https://www.youtube.com/watch?v=eW82yoGoKFY
- Wrote a blog post on the subject that was published by “Towards Data Science” - https://medium.com/towards-data-science/lessons-from-using-spark-to-process-large-amounts-of-data-part-i-421ba36002f3