The Fifth Elephant 2019

Gathering of 1000+ practitioners from the data ecosystem

Data Quality Management @Walmart Data Lake

Submitted Feb 27, 2019

Session type: Full talk of 40 mins

Erroneous decisions made from bad data are not only inconvenient, but also extremely costly. According to Gartner research, “the average financial impact of poor data quality on organizations is $9.7 million per year.”
In additional research for organizations that Gartner has surveyed, the analyst firm “estimate that poor-quality data is costing them on average $14.2 million annually.” Definetely, Bad data is bad for business.
In a Data Lake environment, Robust Ecosystem Products are required to govern Data Quality.Since Data Lakes are based on an ELT model, quality is much more difficult to govern. With legacy systems, and also an existing lake, it becomes a more uphill task.
In this talk I will take the audience on the journey to build DQAF (Data Quality Assessment Framework), which is Walmarts Product for Continuous Data Quality Assessment, and how we are changing the entire way we look at quality. DQAF is based on Wang’s Philosophy of TDQM. We believe To increase productivity, organizations must manage information as we manage products.

Outline

Key areas I will cover

Data Quality Management - Overview and Challenges

Here I will define Data Governance, and how Data Quality Management ties into the same
Different facets of Data Quality Management,
Problem Areas
Solution Areas

The DQAF (Data Quality Assessment Framework) - Providing Data Quality Management for Walmart Data Lake

Overall Data Quality Management Journey for Walmart
Introduce DQAF, and Definintion
Architecture Overview
Architecture Blocks Details

DEMO

How DQAF Solved our Problem Areas, and Built into our Solution Areas
Quality Improvement is a Continous Journey ! - Journey Going Forward
Thinking of Quality in your Data Lake - Tips on how to baby step it

Requirements

Internet Access

Speaker bio

Ravi is an Senior Architect with the GDAP(Global Data and Analytics Platforms) Group in Walmart Labs. Ravi started, building products for primarily for banks, and governments. He developed a passion for scaling large scale products, early on when he was involved with Gujarat Govt for creating workflow solutions. Looking at ways to impact lives directly, he moved towards healthcare, where was build API and Cloud Products for PHC, IoT Devices, MIC, Data Pipelines, Data Conversational Engines
Data being the next frontier, Ravi has been dabbling in all things Data Platform since the last couple of years. He was instrumental in developing the IoT Platform, for GE Healthcare. Currently Ravi is responsible for key oversight of Walmarts next gen Data Platform on hybrid cloud, primarily in areas of Quality, Governance, Privacy and Operability.

Slides

https://drive.google.com/open?id=1wq2_IJKxWv5COUTQ_wVIyMtBULmO9jiA