Meta Refresh 2013

The design and engineering of user interface on the web

A/B testing Platform

Submitted by Ashok Banerjee (@ashokbanerjee) on Jan 2, 2013

Section: Process Technical level: Beginner Session type: Lecture Status: Submitted


In this talk I will motivate the need for an experimentation platform. I shall discuss the various modules of an experimentation platform and share how to go about designing or building such a platform. I shall also work on the basic mathematics to develop the intuition (not just share the final formula) of when a result is statistically significant.


The greatest tool for rapid innovation is rapid experimentation. The true crystal ball for all experiments is the greatest designer of all - the internet. Actions of people on your site tell you what your audience desires or even craves.

In an A/B testing platform we require: 1) Cohort-aware bucketing 2) Stickiness beyond a session and translated to an user 3) Ability to fix the percentage of users who will see an experiment e.g. 1% 4) Determining statistical significance of results (Gaussian error and beyond) 5) Ability to rapidly deploy the changes to production

In this talk we shall talk about discuss all 5 layers and include basic mathematics to develop the intuition behind the formulae for statistical significance.

I will also discuss how to extend this beyond


Basic understanding and knowledge of experiments

Speaker bio

Ashok Banerjee is VP of Data Platform and Analytics Engineering at Flipkart and has to date 22 patents approved and counting. Prior to Flipkart Ashok has worked at Twitter in San Francisco and Google in Mountain View.

Experience Summary (reverse chronologically)

Ashok today leads the technology team for Data Platform and Analytics at Flipkart and has also led the largest online Supply Chain infrastructure in India (Flipkart) - At Google he led a large scale Datawarehouse infrastructure which converts SQL (approximately) into execution on a platform built on MapReduce, GFS, Columnar compressed data using block oriented computing. This was at the scale of many billion rows added per day (cannot disclose how many billions) - At Google Ashok had led the payment processing infrastructure which processes payments for Adwords, Adsense, Checkout and Google Apps At BEA he worked on WebLogic Server and led infrastructure teams on EJB Container, Web Container, Classloading, Application Deployment within a Server etc. - At Oracle Ashok led the Oracle Application Server Clustering infrastructure and also worked on EJB container and RMI-IIOP Protocols

Ashok takes interest in Experimentation platforms and Innovation Enablers, Large Data Systems (Databases and alternative databases - NOSQL, Message Systems), Parallel Computing, Distributed Systems, Fault Tolerant Computing, Database, Recommendation Systems, Supply Chain and Mathematical Models and Investments.

On the non-work side Ashok enjoys - sailing, wind surfing, horse riding, german shepherd dogs and soccer.


  • Sunil Pai (@threepointone) 7 years ago

    Wow, this talk needs more upvotes. Ashok, would love to hear you talk on this. Eager to hear about the flipkart setup for A/B testing.

    • Ashok Banerjee (@ashokbanerjee) Proposer 7 years ago

      Thank you Sunil - excited about this topic!

  • Rahul Gonsalves (@rahul) 7 years ago (edited 7 years ago)

    Ashok, I’m looking forward to this talk. I think that the ability to continuously experiment is one of the most useful (and hardest) to a designer. The ability to constantly question one’s assumptions and receive an answer cannot be overstated.

    As an aside, have you seen Dan McKinley’s recent talk on Continuous Experimentation at Etsy ( I suspect that there are many overlapping themes in both your talks.

    • Ashok Banerjee (@ashokbanerjee) Proposer 7 years ago

      Hi Rahul,
      Thanks for the pointer to Dan’s talk - will definitely listen to it. Thank you for sharing the topic is very similar.


  • jaseem abid 7 years ago (edited 7 years ago)

    The one talk I’m going to miss :( My upvote :)

  • Sriram Karra (@karra) 7 years ago (edited 7 years ago)

    I am very interested in a roundup of the necessary mathematical tools for meaningful A/B testing - which seems the flavour of the current season. I am glad the organizers decided against taking their brief on focussing on the ‘why’ as against the ‘how’ too seriously.

  • MohanRajGaneshan (@iammohanraj) 7 years ago

    Hi Arpan !

    Could you please share with your contact details


Login to leave a comment