JSFoo 2012

It’s like kung fu with JavaScript and objects! All about building full size apps in JavaScript.

A/B Testing Platform and when is the result significant

Submitted by Ashok Banerjee (@ashokbanerjee) on Monday, 10 September 2012

Section: Browser Technical level: Beginner Session type: Lecture


In this talk I will motivate the need for an experimentation platform. I shall discuss the various modules of an experimentation platform and share how to go about designing or building such a platform. I shall also work on the basic mathematics to develop the intuition (not just share the final formula) of when a result is statistically significant.


The greatest tool for rapid innovation is rapid experimentation. The true crystal ball for all experiments is the greatest designer of all - the internet. Actions of people on your site tell you what your audience desires or even craves.

In an A/B testing platform we require: 1) Cohort-aware bucketing 2) Stickiness beyond a session and translated to an user 3) Ability to fix the percentage of users who will see an experiment e.g. 1% 4) Determining statistical significance of results (Gaussian error and beyond) 5) Ability to rapidly deploy the changes to production

In this talk we shall talk about discuss all 5 layers and include basic mathematics to develop the intuition behind the formulae for statistical significance.

I will also discuss how to extend this beyond


Basic understanding and knowledge of the Web world

Speaker bio

Ashok Banerjee is VP of Data Platform and Supply Chain Engineering at Flipkart and has to date 22 patents approved and counting. Prior to Flipkart Ashok has worked at Twitter in San Francisco and Google in Mountain View.

Experience Summary (reverse chronologically)

Ashok today leads the technology team for Data Platform and the largest online Supply Chain infrastructure in India (Flipkart) - At Google he led a large scale Datawarehouse infrastructure which converts SQL (approximately) into execution on a platform built on MapReduce, GFS, Columnar compressed data using block oriented computing. This was at the scale of many billion rows added per day (cannot disclose how many billions) - At Google Ashok had led the payment processing infrastructure which processes payments for Adwords, Adsense, Checkout and Google Apps At BEA he worked on WebLogic Server and led infrastructure teams on EJB Container, Web Container, Classloading, Application Deployment within a Server etc. - At Oracle Ashok led the Oracle Application Server Clustering infrastructure and also worked on EJB container and RMI-IIOP Protocols

Ashok takes interest in Experimentation platforms and Innovation Enablers, Large Data Systems (Databases and alternative databases - NOSQL, Message Systems), Parallel Computing, Distributed Systems, Fault Tolerant Computing, Database, Recommendation Systems, Supply Chain and Mathematical Models and Investments.

On the non-work side Ashok enjoys - sailing, wind surfing, horse riding, german shepherd dogs and soccer.


  • Kamal Govindraj (@gkamal) 6 years ago

    How relevant do you think this is to javascript deverlopers? Are you going to show code / implementation details or is it going to cover the concepts?

  • Ashok Banerjee (@ashokbanerjee) Proposer 6 years ago

    Ok heard from Zainab that there was a question and I saw no email notification for the same.

    This is not about the language Javascript per-se. In fact typically most of the actual code for an experimentation platform tends typically to be in Java. That said in UI both static & interaction design people have ideas a dime a dozen and yet objective evaluation of results in non trivial.

    This talk will cover the concepts, including the mathematics of when something is Gaussian noise versus when there is conclusive evidence. This talk will help interpret conclusions.

Login with Twitter or Google to leave a comment