The Fifth Elephant 2023 Monsoon

A/B testing is a widely-used paradigm within marketing optimization because it promises identification of causal effects, because it is implemented out of the box in most messaging delivery software platforms, but mostly because it is held up as a “gold standard” for evaluating options. This talk will explain why A/B tests are not a particularly sound method, why businesses rarely choose better (adaptive) methods, and outline the workings of a full-fledged testing approach that relies on reinforcement learning to allow for thousands or even tens of thousands of simultaneous tests on overlapping samples of users. In particular, we will focus on a method for disentangling causal effects of intermeshed tests under conditions of continuous test adaptation, using a matched-synthetic control group that adapts alongside the tests.

Presentation can be found here:
https://drive.google.com/file/d/1dow7WeCB9qJwIsQcYx8mcfLUelL3rslL/view?usp=drive_link