The Fifth Elephant 2019

Each week 275 million people shop at Walmart, generating multi-terabytes of interaction and transaction data. In Customer Backbone team, we enable extraction, transforming and storing of data to be served to teams such as Ads and Personalisation for building various customer-centric machine learning models such as bid models, fraud detection and omnichannel reorder. At 5 Billion events/day our Kafka Streams cluster processes events from various channels(web/online/mobile) and triggers the models when a type of event is ingested by our platform to which each model is subscribed to. In this talk, I will share an overview of architecture, leveraging Core Kafka, Kafka Streams, and Kafka Connect, for processing events and executing models, which is inbuilt for scalability and reliability. I will discuss the below points in detail.
•Ensuring fairness among the model runs
•Providing isolation and reusing features/inferences across models at the same time
•Dynamically updating global data(such as product catalog) on each node needed to run models using Kafka Connect and global stores
•Customizing models to either trigger them on each event or as batch after frequent time intervals using features such as windowing stores and suppress
•Implementing data archival/TTL policies and other features developed to be cost efficient
•Increasing availability of the system by enabling read via replicas and while restoring/rebalancing states
•Advantages and limitations of the platform

Key Takeaways:
•Leveraging Kafka Streams for model inferencing
•Lessons learned from productionising a Kafka Streams cluster while making it cost-efficient

Outline

Key Takeaways:
•Leveraging Kafka Streams for model inferencing
•Lessons learned from productionising a Kafka Streams cluster while making it cost-efficient

Requirements

Basic understanding of distributed systems, Kafka and Kafka Streams

Speaker bio

Navinder is working as a data engineer in Walmart Labs where he has been working with Kafka ecosystem for the last couple of years, especially Kafka Streams and created a new platform on top of it to suit their needs to process billions of events per day in real time and trigger models on each event. He has been active in contributing back to Kafka Streams and has patented few features as well. He is interested in solving complex problems and distributed systems and likes to spend time in gym and boxing ring in his spare time.

Links

• https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/77038

Slides

https://speakerdeck.com/navinder/building-a-multitenant-data-processing-and-model-inferencing-platform-with-kafka-streams?slide=4