Skip to main content

Bayesian Inference

Analyzing randomized experiment results is often difficult. At times you get stuck with drawing a business decision out of results provided by experimentation tools.

Suppose you ran a test with two variations and you observe the following:

  • A: Impressions 500, Revenue $5, CPM $10
  • B: Impressions 500, Revenue $5.2, CPM $10.4

It is tempting but hard to conclude that B is the best overall. The observed performance difference might be merely due to chance rather than because of changes that are being tested.

In bayesian the fill rate and CPM are thought in terms of probability distribution which represents our belief based on observed data. The spread of probability distribution curve represents how precisely the experiment has measured the fill rate and CPM. As we gather more data, we refine the probability distribution for all the variants and the control. We quantify the probability of a random sample from any of the variant probability distributions would have a higher fill rate or eCPM over a random sample picked from the control probability distribution. This tells how likely the variant is best overall. This approach is way better than looking at mean values of eCPMs and making a decision.

Suppose you ran a test with the 3 variations and you observe the following:

  • A: Impressions 500, Revenue $5, CPM $10
  • B: Impressions 500, Revenue $5.2, CPM $10.4
  • C: Impressions 500, Revenue $6, eCPM $12

From the probability distribution of the observed results of A,B and C we calculate the following:

  • Chances of A beating B & C = 10%
  • Chances of B beating A & C = 80%
  • Chances of C beating A & B = 20%

While variant C is most appealing the chances of C realizing the effect is only 20%. The recommended variant here would be B, since there is a 80% chances of it realizing the effect.