The Fifth Elephant 2019

The eighth edition of India's best data conference

Participate

The art of abstraction to handle database and storage system chaos

Submitted by Ashish Tadose (@ashishtadose) on Sunday, 14 April 2019

Abstract

With growing data volumes and varying needs of data storage and access patterns gave rise to adoption of diverse databases such as key value, wide columns, document, graph and so on. Also, with increasing adoption of public clouds organizations started leveraging flexible storage mediums such as HDFS & Object stores. There is a dire need of query engine in the analytics platform which can query across these federated systems without creating copies of data. In this session we will go over segment of Walmart’s data analytics stack which tackled this use case by leveraging open source solutions.

Outline

  • Database chaos - diverse SQL/NoSQL systems, accept it there’s no one stop solution!!
  • Hetrogenious storage system - on prem distributed systems HDFS, public and private object stores
  • Need for unified querying layer to query across all hetrogenious systems, even better if multiple federated systems can be queried in a single query.
  • Presto overview - open source distributed SQL engine for running fast analytic queries against various data sources
  • Alluxio overview - open source tiered caching storage which unifies data access to different systems, and seamlessly bridges computation frameworks and underlying storage
  • How Presto and Alluxio together powers scalable and performant analytical platform @ WalmartLabs
  • Enterprise security integrations and tools built over open source Presto @ WalmartLabs

Speaker bio

Ashish is passionate about BigData technologies and designing data analytics systems for large scale. He has been working on Hadoop processing pipelines since 2009. At his current role @WalmartLabs as Staff Engineer he focuses on building data products to power Walmart’s DataLake and also contributing to overall data and cloud strategy.
Prior to that he worked as Senior Data Architect at PubMatic where he contributed to setup Lambada architecture pipeline for large scale AdTech data processing for reporting, analytics & machine learning.
He is frequent speaker at BigData meetups & conferences, Apache committer and FOSS lover.

Comments

  • Ravi Ramchandran (@rramchan) 2 months ago

    Should be an interesting share.. Looking forward.

  • Zainab Bawa (@zainabbawa) Reviewer a month ago

    Thanks for the submission, Ashish.

    For this proposal to be interesting for the audience at The Fifth Elephant, you have to share draft slides which explain:

    1. Problem statement/context, which the audience can relate to and understand. The problem statement cannot be about Walmart’s data analytics stack because this is not generalizable across participants (not all of who will have Walmart’s scale and challenges). The problem statement has to be a problem (based on this context) that can be generalized for all.
    2. What were the tools/options available in the market to solve this problem? How did you evaluate these, and what metrics did you use for the evaluation? Why did you decide to build your own tool?
    3. Why did you pick the option that you did? If you chose open source solutions to solve your problem, why was this a better option than adopting a proprietary solution?
    4. Explain how the situation was before the solution you picked/built and how was life in Walmart after implementing the solution you picked and built? Show before-after scenario comparisons.
    5. What compromises/trade-offs did you have to make in this process?
    6. What is the one takeaway that you want participants to go back with at the end of this talk? What is it that participants should learn/be cautious about when solving similar problems?

    Next steps: we need to see detailed slides, covering the about points, by or before 21 May, in order to close the decision on your proposal.
    If we don’t receive slides by 21 May, we will move the proposal for consideration for a future conference, since the timelines for The Fifth Elephant are tight.

Login with Twitter or Google to leave a comment