Note: slides I’ve submitted are from a previous talk which I wasn’t super happy with. Will be more or less rewritten.

Talk is about autonomous organizations - organizations making as many decisions as possible without human intervention.

• Simpl is a case study; if data science team all dies, credit underwriting/fraud checks/etc should continue.
• Automated trading strategies are another example (bots playing the shares markets)

Discussion of what good engineering is:

• Loosely coupled (web frontend writes to Kafka, different backend systems read from Kafka, backend crashes don’t affect frontend)
• Modular (web + backend systems are well separated)
• Single responsibility principle (each system does one well known thing)

Data science isn’t good engineering:

• Everything is connected. A system affecting marketing alters behavior of users downstream.
• A model is a giant mess of spaghetti code. Edge cases on top of edge cases.
• Data and relationships change over time. Data often goes bad.
• Biggest problems are conceptual bugs, not coding bugs - e.g., leaking future data into the present, or allowing multiple comparisons into your testing procedure.
• Data collection management is key. If I have a team member picking up a project in March, I need to start collecting data in Jan. Time is precious, don’t waste it!

Data science interacts with the world in which it lives:

• Selection effects: a model trained on professional basketball players finds no relation between height and player quality. Should you ignore player height when picking a team?
• Adversarial effects: if you lock down a fraud vector then fraudsters stop attacking it. Data says “no fraud attempts, safe to unlock.”
• Market adjustments: if you’re interacting with a market you will often transmit information to it, and it will adjust to you.

What you can do about this mess:

• Badly formatted data can be fixed with enough code. Missing data is gone. Always focus on collection.
• Data quality monitoring - if you have trouble parsing any data, pagerduty should blow up.
• Alerts about relationships - if your predictive model assumes a correlation between X and Y, raise an alert if that correlation vanishes.
• Amateurs focus on which model to use. Professionals think about how to backtest the model.
• Continuous monitoring - how quickly can you know that your model stopped making money?

1. Be clear of Requirements and Constraints
   • Having a scalable system for data ingestion
   • Data design (Specific or Generic)
   • Querying interface - why stick to SQL?
2. Take time to Design Data
   • Walking through example of generic table design
3. Sort out Data production part first
   • Identify all possible data producers (and understand requirements). In our case -
   • Android/iOS app
     • Cannot keep sending each event over network
     • Cannot lose data even if app crashes or is killed
     • Keep out of context from the application itself
   • Microservice(s)
     • Cannot keep sending each event over network
     • Keep data collection agnostic of microservice itself
4. Design v1.0 of Data pipeline
   • How and why we chose “anti-pattern”
5. Choose/Design Data warehouse
   • Data design in Redshift
   • Compression ON for certain columns
   • Tuning for scale
   • Taking care of Querying patterns of Product Managers and Data scientists
6. Open up: Enable many Data Interfaces
   • On demand Data loading and querying: OnDemand Table(s)
   • Flexibility for complicated analysis: Adhoc redshift cluster(s)
7. Understand, Tune & Repeat
8. Optimize for Usage
   • Added more columns at generic level e.g.
   • More examples
9. Optimize for Cost & Ops
   • Retention policies of data
     • Not all events are of same importance
     • But all events should be accessible if required
10. Upgrade to v2.0 of Data pipeline

The topics we will be covering in this talk:
1. Introduction - Briefly provide business context to appreciate the need to solve this problem, and challenges involved.
2. The factors driving the decision to choose Cosmos DB as our backend store.
3, Key insights into what drives cost of the store, and various gotchas involved when designing such a system.
4. How to optimize the cost and bring intelligence to enable auto-scalability.
5. The need for building a multi version concurrency control and how to achieve it to enable parallel writes with multiple schema versions for the same record.
6. The tradeoff between readability and storage cost, and how to get the best of both worlds by building an avro library to enable inflight abbreviated compression.

In this talk we will take care of below questions and explain the same followed by a demo is the system build.
What is business motivation to build Spark like(or better) distributed processing framework in-house?
Why distributed frameworks like Spark will not work for us in long run and why we need something else?
What are basic design layers, data structures and algorithms required to build one such a system?
What are the benchmark results and how it works better than Spark for us?
Demo run of the framework.

• Introduction
• Need of Multicloud and multi tenant models
• Lessons learned while building Einstein platform
• How traditional machine learning works
• Introducing TransmogrifAI
• Type Hierarchy
• Automatic Feature Engineering across text, categorical, numerical, spatial features
• Handling label leakage
• Autmatic Model Selection and hyper parameter tuning
• Models supported currently
• Demo
• Uses cases being solved in production
• Summary

This is a call for the tech community to come together and create open source technlogy which can help fight misinformation/disinformation. The technology created will be useful not only in the Indian context but also in the global context.

Will cover a few case studies covering how we adapted to important business milestones:
1. Mobile App traffic surpassing Desktop
2. Scaling From a single seller (WS Retail) to tens of thousands and from handful of categories to thousands.
3. Launch of new private label business line.

PPT Covering Outline : https://docs.google.com/presentation/d/1FNFjq74PhW8KTStHZLadhmNzhLaHotp6WfsO5LcQeKw/edit?usp=sharing

The flow would look like this

• The Need for a Message Bus in building a data processing pipeline
• For the events generated in the Message Bus, the need for a contract for data control (with examples of showing how we messed up and learnt from it).
• • explain in more detail of what a contract is
• • how it can be implemented
• • • starts with hierarchical modeling of data. relations between objects
• • • what are tools other there to store this complex relationship between entites

• Discuss the gains from implementing contract control for any data that flows in the data pipeline

• • from a business perspective of improving business logic, joining with other data sets
• • from a technological ease -
• Schema extendibility of fields in data,
• predictability of development,
• back dated processing - backward and forward compatibility
• Able to break down pipeline by responsibility - teams can work on different component of the pipeline - Implementing the above for multi step data processing (enrichment)

Additional Advantages * Cost wise * Data cleaning * Data consistency * Linear pipeline

1. Description of the context at the Flipkart delivery hub.
2. Solution constraints - customer time window, maximum number of shipments per vehicle.
3. Defining a non-standard cost function - total travel time, route outliers, compactness of routes.
4. Formulating the problem as a variant of VRPTW (vehicle routing problem with time windows).
5. Computational complexity: NP-hard (generalization of Traveling Salesman Problem)
6. Overview of some exact algorithms.
7. Heuristics - (a) construction of routes and (b) route improvement
8. Description of our construction step and iterative computational procedure to improve solutions.
9. Discussion of results

Rough Outline:

• Objectives of a feature engineering platform (5 mins)

  • Reduce time to market
  • Enhance robustness of models
  • Enable explainability

• Points of friction & required capabilities (20 mins)

  • What is in my data? (catalog)
  • Is my input data complete and correct? (health)
  • How do I link existing side information (augment/enrich)
  • How to capture tacit knowledge/signal (labeling)
  • How do I reliably prepare my training datasets (pipelines)
  • How do I check audit & validate what has been computed (audit)
  • How do I discover what is being computed and used? (marketplace)
  • How do I export and track exported discovered features for model dev (search)
  • How do I link the features to performance? (monitor)
  • How do I reuse the features in the streaming path? (library)

• Economics of Feature Engineering (5 mins)

  • Feature computation expensive, and each has a price
  • Amortization happens over time & across models
  • Process discipline required
  • Questions to ask:
    1. How many models will I have over time?
    2. How defensible should they be?
    3. How available should they be?
    4. How many features will they need?

• Approaches to building one (5 mins)

  • FEAST (Go-JEK; Thought through but tied to GCP)
  • Combine standalone components (OSS exists but incur integration costs)
  • Thirdparty (Move fast but incur platform costs)

We would be presenting answers to the following…
Why just any generic approach would not have worked?
How our data source and structure left us with no previously adapted choices?
Why we the project was necessary to meet the end goals of the company?
And how did we tackle a number of problems on the way?

Company’s Goals:

Following are the primary product-goals of the company which are relevant to ADAM project.:
1)Universal Product Catalog
2)Aggregation and Market Analysis
3)Self evolving Knowledge Graph

Raw Dataset:

Introduction, Structure and The good, bad and the ugly of the data-set.

WHY ADAM (Automatic Detection and Annotation Module)… A deep NN model:

Definition,
Usecases::
1) Enrichment of Knowledge graph
2) For Analytics

Components of ADAM:

1)Smart Automatic Training Data Generation
2)State of the Art Sequence Tagging Model
3)Active Learning approach

Why the above architecture is chosen:

1)Zero ground truth and no training data available whatsoever.
2)Multi-Independent Source of data generation thus imagine the variance
3)Short representations and extremely noisy
4)Prone to Extreme human error (not bias but error!!!)

Finally details of the architecture and WHY they were necessary:

1)Smart Automatic Training Data Generation:

<>How we leveraged the structure of dataset (Stock-item and Stock-group)?
<>How we used the existing knowledge-base AKA (CREGS)?
<>How we improvised using the information from other sources like Amazon and GS1?

2)State of the Art Sequence Tagging Model:

<>Why we created our own word embeddings and how it helped us?
<>Why BiLSTM and CRF were used and why are they state of the art?
<>Why this specific architecture was needed and why anything else wouldn’t work
<>What was the accuracy and how well did the model performed?

3)Active Learning:

<>Why Active learning when we can generate labels automatically?
<>How we integrated and designed Manual annotation model ourselves?
<>How well did we reach maturity with the minimum data-points manually labelled?
<>Why extrinsic sampling or intrinsic sampling used?

Conclusion that we will showcase as per 5^th Elephant:

1)How to tackle the noisy data problem in case of textual data?
2)Why a deep NN model plays an important role in generalisation?
3)Why Active Learning is a really important concept for dealing with the problem of no label data?