The Fifth Elephant 2023 Monsoon

On AI, industrial applications of ML, and MLOps



This video is for members only

Schaun Wheeler


What do you do when A/B tests aren't enough? Validation of massively-parallel adaptive testing using dynamic control matching

Submitted Jun 14, 2023

A/B testing is a widely-used paradigm within marketing optimization because it promises identification of causal effects, because it is implemented out of the box in most messaging delivery software platforms, but mostly because it is held up as a “gold standard” for evaluating options. This talk will explain why A/B tests are not a particularly sound method, why businesses rarely choose better (adaptive) methods, and outline the workings of a full-fledged testing approach that relies on reinforcement learning to allow for thousands or even tens of thousands of simultaneous tests on overlapping samples of users. In particular, we will focus on a method for disentangling causal effects of intermeshed tests under conditions of continuous test adaptation, using a matched-synthetic control group that adapts alongside the tests.

Presentation can be found here:


{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hybrid access (members only)

Hosted by

All about data science and machine learning

Supported by

E2E Cloud is India's first AI hyper scaler, a cloud computing platform providing accelerated cloud-based solutions at maximum optimization and lowest pricing