The Fifth Elephant 2024 Annual Conference (12th &13th July)

Real world data contains multiple records belonging to the same customer. These records can be in single or multiple systems and they have variations across fields, which makes it hard to combine them together, especially with growing data volumes. This hurts customer analytics - establishing lifetime value, loyalty programs, or marketing channels is impossible when the base data is not linked. No AI algorithm for segmentation can produce the right results when there are multiple copies of the same customer lurking in the data. No warehouse can live up to its promise if the dimension tables have duplicates.

With a modern data stack and DataOps, we have established patterns for E and L in ELT for building data warehouses, datalakes and deltalakes. However, the T - getting data ready for analytics still needs a lot of effort. Modern tools like dbt are actively and successfully addressing this. What is also needed is a quick and scalable way to build the single source of truth of core business entities post Extraction and pre or post Loading.

This talk will cover the problem of entity resolution, its challenges and practical approaches to overcome them.

Outline

• The data stack tools and technologies
• The problem of entity resolution
• Business impact of entity resolution
• Challenges in building ER systems
• Some approaches to solve ER challenges

Impact

The talk will equip the data and egineer and data scientist to think about entity resolution holistically as part of their data stack and use the appropriate tools and techniques to build scalable and accurate ER systems.