Meta Refresh 2013

In this talk I will motivate the need for an experimentation platform. I shall discuss the various modules of an experimentation platform and share how to go about designing or building such a platform. I shall also work on the basic mathematics to develop the intuition (not just share the final formula) of when a result is statistically significant.

Outline

The greatest tool for rapid innovation is rapid experimentation. The true crystal ball for all experiments is the greatest designer of all - the internet. Actions of people on your site tell you what your audience desires or even craves.

In an A/B testing platform we require: 1) Cohort-aware bucketing 2) Stickiness beyond a session and translated to an user 3) Ability to fix the percentage of users who will see an experiment e.g. 1% 4) Determining statistical significance of results (Gaussian error and beyond) 5) Ability to rapidly deploy the changes to production

In this talk we shall talk about discuss all 5 layers and include basic mathematics to develop the intuition behind the formulae for statistical significance.

I will also discuss how to extend this beyond

Requirements

Basic understanding and knowledge of experiments

Speaker bio

Ashok Banerjee is VP of Data Platform and Analytics Engineering at Flipkart and has to date 22 patents approved and counting. Prior to Flipkart Ashok has worked at Twitter in San Francisco and Google in Mountain View.

Experience Summary (reverse chronologically)

Ashok today leads the technology team for Data Platform and Analytics at Flipkart and has also led the largest online Supply Chain infrastructure in India (Flipkart) - At Google he led a large scale Datawarehouse infrastructure which converts SQL (approximately) into execution on a platform built on MapReduce, GFS, Columnar compressed data using block oriented computing. This was at the scale of many billion rows added per day (cannot disclose how many billions) - At Google Ashok had led the payment processing infrastructure which processes payments for Adwords, Adsense, Checkout and Google Apps At BEA he worked on WebLogic Server and led infrastructure teams on EJB Container, Web Container, Classloading, Application Deployment within a Server etc. - At Oracle Ashok led the Oracle Application Server Clustering infrastructure and also worked on EJB container and RMI-IIOP Protocols

Ashok takes interest in Experimentation platforms and Innovation Enablers, Large Data Systems (Databases and alternative databases - NOSQL, Message Systems), Parallel Computing, Distributed Systems, Fault Tolerant Computing, Database, Recommendation Systems, Supply Chain and Mathematical Models and Investments.

On the non-work side Ashok enjoys - sailing, wind surfing, horse riding, german shepherd dogs and soccer.