Rough outline

1. Business Requirements
2. Usecase
   • Having a scalable system for data ingestion
   • Data design - Specific or Generic?
   • Querying interface - why stick to SQL?
   • Query interface users - skills, requirements and expectations
3. Data ingestion
   • High throughput stats service
   • Thin client: Badger
   • High throughput Ingestion backend
   • Hot loading to Redshift
4. Data Warehousing
   • Data design in Redshift and data lake
   • Tuning for scale
   • Taking care of Querying patterns of Product Managers and Data scientists
5. S3 as Data Lake
   • On demand Data loading and querying: OnDemand Table(s)
     • Gotchas
   • Flexibility for complicated analysis: Adhoc redshift cluster(s)
     • Gotchas
6. Scaling up
   • Typical bottlenecks and solutions we tried
7. Learnings

1. Feature Engineering Overview
2. Typical Feature Engineering Cycle
   • Trends
3. Detailed Cost Drivers
   • Examples: Reconciliation & auditing, change management
4. Indicative Quantitative Improvement
5. Detailed discussion of each driver

Data Governance session outline will cover,
- systematic approach to identifying enterprise data assets, who owns them and who can access them - data protection approach - data catalogue, data profiling and data quality

<Work in progress>

Motivation
Approach
Learnings

1. Introduction and Context
2. The Capacity Problem - what is it; why it is important?
3. The Efficiency Problem - what, why and the necessary trade-offs
4. Data and its Nature
5. Challenges with Accurate Data Capture
6. Challenges with high Variance
7. Real-time Vs. historical data
8. Representing Capacity
9. Aggregated capacity (Zone-level)
10. Point-in-time-capacity (Order-level)
11. Journey and Results: Solving for Capacity
12. Efficiency Levers
13. Predictions and accounting for errors
14. Trade-offs
15. Optimal Assignment
16. Batching
17. Aggregate Analysis Vs. Specific Analysis
18. Pitfalls of Aggregate Analysis
19. Conclusions

• Ingestion Patterns
  • Unified Ingestion Proxy
  • Schema Definitions
  • In-flight enrichments
  • Highly Available
• Storage Patterns
  • Decouple storage and compute
  • Query Lineage & Optimization
  • Noisy Neighbour
• Consumption Patterns
  • Single GUI and a programmatic interface. All the magic underneath
  • Parity between streaming and stationary data

• Introduction to credit scoring
• Limitation of credit scoring
• Alternative credit scoring
• Data Science challenges in alternative credit scoring
• How Trusting Social works?