With growing data volumes and varying needs of data storage and access patterns gave rise to adoption of diverse databases such as key value, wide columns, document, graph and so on. Also, with increasing adoption of public clouds organizations started leveraging flexible storage mediums such as HDFS & Object stores. There is a dire need of query engine in the analytics platform which can query across these federated systems without creating copies of data. In this session we will go over segment of Walmart’s data analytics stack which tackled this use case by leveraging open source solutions.

Outline

• Database chaos - diverse SQL/NoSQL systems, accept it there’s no one stop solution!!
• Hetrogenious storage system - on prem distributed systems HDFS, public and private object stores
• Need for unified querying layer to query across all hetrogenious systems, even better if multiple federated systems can be queried in a single query.
• Presto overview - open source distributed SQL engine for running fast analytic queries against various data sources
• Alluxio overview - open source tiered caching storage which unifies data access to different systems, and seamlessly bridges computation frameworks and underlying storage
• How Presto and Alluxio together powers scalable and performant analytical platform @ WalmartLabs
• Enterprise security integrations and tools built over open source Presto @ WalmartLabs

Speaker bio

Ashish is passionate about BigData technologies and designing data analytics systems for large scale. He has been working on Hadoop processing pipelines since 2009. At his current role @WalmartLabs as Staff Engineer he focuses on building data products to power Walmart’s DataLake and also contributing to overall data and cloud strategy.
Prior to that he worked as Senior Data Architect at PubMatic where he contributed to setup Lambada architecture pipeline for large scale AdTech data processing for reporting, analytics & machine learning.
He is frequent speaker at BigData meetups & conferences, Apache committer and FOSS lover.