The Fifth Elephant round the year submissions for 2019

As a provider of Competitive Intelligence as a Service to eCommerce businesses and consumer brands, DataWeave aggregates and analyses product catalog data from eCommerce websites each day at massive scale. Once aggregated, this data is fed into a complex process of extraction, transformation, machine learning, and analyses. These operations are performed on a consistent basis to provide our customers with easily consumable and actionable insights.
To be precise, we aggregate over 200 million data points across 2000+ web sources to deliver 200+ reports each day.
The web sources span across multiple verticals ranging from eCommerce, travel, classified listings, mobile apps, and more. Having a generic aggregator to aggregate data from multiple websites across multiple domains is a significant challenge.

Outline

Why do we need data aggregators?
A singapore-based VC firm wants to analyze how its portfolio businesses were performing in India and wants to crawl the mobile apps of a few of its businesses, along with that of their competitors.
An investment firm in the US wants to take stock of how a web-based B2C business is growing every quarter, thereby enabling an informed buy/sell decision before an earnings call.
A ‘brand’ wants to analyze their share of voice on eCommerce websites and track pricing violations on online marketplaces.
Smart organizations are looking for ways to capture and store data (both internal and third-party) at scale, process it efficiently, and generate actionable insights consistently. This talk will throw some light on how we aggregate data at massive scale and convert unstructured Web data to consumable insights. We will also talk about several problems we encounter along the way and how we solve them.

Talk flow:

1. Evolution of the platform
2. Data Collection at scale
   • horizontal scaling
   • politeness policy
   • bot blocking
3. Mobile App crawling
4. Correctness and completeness
5. Types of datasets
   • HTML
   • CSV/JSON
   • Images
   • Managing the datasets

Speaker bio

Mithun, Data Architect, DataWeave

I work as an architect in the data platforms team at DataWeave, a provider of Competitive Intelligence as a Service for eCommerce businesses and consumer brands. I design and manage data aggregation at scale, which involves writing crawlers, extracting structured data, and more.
I have 10 years of experience in the software industry, with extensive experience in building web crawlers for complex web environments.

Slides

https://drive.google.com/file/d/1n_Dv8ROi0e9imYDIXfPC0fwhxwI21DJq/view?usp=sharing