The Fifth Elephant 2017
On data engineering and application of ML in diverse domains
AP
Ananth Packkildurai
@ananthpackkildurai
Search Infrastructure @ Slack using Lambda Architecture
Submitted Apr 27, 2017
Section:
Full talk for data engineering track
Technical level:
Intermediate
Slack is a collaboration tool for teams. We’re on a mission to make your working life simpler, more pleasant, and more productive. Search is the core feature of Slack offerings as Slack itself is an acronym for “Searchable Log of all conversation & knowledge”.
At Slack, we experiment frequently with various machine learning models to improve search experience so rebuilding search indexes are critical for search infrastructure.
This talk will centre around lambda architecture, common pitfalls of lambda architecture and best practices, an overview of Slack search infrastructure and our experience in building Solr offline indexing at scale.
Outline
This talk will be around,
- Lambda Architecture
- Common pitfalls of Lambda Architecture
- Design patterns to handle common pitfalls of Lambda Architecture
- Apache Solr offline indexing
- Apache Solr search infrastructure.
Requirements
Some basic understanding of mapreduce and stream processing.
Speaker bio
Ananth Packkildurai is currently working as a Senior Data Engineer at Slack Technologies, San Francisco and has over thirteen years of experience in building systems at scale. Prior to Slack, Ananth worked as a senior data engineer at Bazaarvoice Inc to build large scale consumer reviews analytical platform. He works closely with Hadoop, Apache Crunch, Kafka, Apache Solr, Apache Spark, Druid and other big data platform tools.
Linkedin: https://www.linkedin.com/in/ananthdurai
Links
Slides
https://speakerdeck.com/vananth22/search-infrastructure-using-lambda-architecture
Login to leave a comment
Damon Young
@damonyoungdf
Hi, Ananth. Thanks for the great talk. Can I ask: why did you choose Solr over, say, Elastisearch, for the main search platform at Slack?
Zainab Bawa
@zainabbawa
Thanks for submitting this proposal, Ananth. Have you presented this talk earlier? We require draft slide deck detailing the content that you will cover in the talk and structure of the presentation. You also need to upload a two-min preview video explaining what this talk is about, and what is the key takeaway for participants.
Ananth Packkildurai
@ananthpackkildurai Submitter
Hi Zeinab, I've uploaded the draft slide and a link from Slack engineering blog that talks about search in general. This talk will touch upon some of the aspect mentioned in the blog but mostly centered around data infrastructure. I will follow up with short 2 min video on it.
Zainab Bawa
@zainabbawa
Ananth, update on the preview video? We are closing decisions on proposals on 10 June. The editorial team needs the video before 10 June.
Ananth Packkildurai
@ananthpackkildurai Submitter
Hi there,
I've updated a preview video and requirement for the talk as well.