The Fifth Elephant 2018

Data is gold. It is at rest or in motion, is transient or reasonably permanent, is being written or read, is expensive or cheap to store and so on.
While we are mostly concerned about Data stores and processing engines, the impact of Data in motion is usually ignored - on the data centre infrastructure, across System interactions, in analyzing User or System behavior.

In this talk, I will use eCommerce as an example to discuss Data in motion - User generated(Transactions, Click-stream), Cross-System calls, Data flowing in and out of Decision Systems and Data movement between Transactional and Archival systems. I will cover challenges in dealing with Data in motion, at scale. The talk will then address these challenges in an evolutionary manner starting with obvious methods and move on to less frequent ones. In this talk I will use real-world examples from our experience at Flipkart to discuss the problems and solutions - many that are ongoing.

Outline

• Define Data in motion, with examples
• Types of Data in motion and their relative importance
• Challenges in handling Data in motion - infrastructure(network), data serialization, guarantees, impact due to scale
• All data is not equal - differentiating by importance, velocity and volume and treating them differently
• Methods used in dealing with Data in motion - Data stores, Transfer protocols, Smart proxies and Observability of systems, Service Mesh
• Learnings from experience at Flipkart - with examples

Speaker bio

Regunath is an open source developer, engineer who built Aadhaar and currently works on Retail and Marketplace systems at Flipkart. His work and interests include Data stores & processing frameworks.