The Basics
Getting Started
Open Auction
Audiences
Experiments
Ad Quality
Integration
Reports
Analyzing randomized experiment results is often difficult. At times you get stuck with drawing a business decision out of results provided by experimentation tools.
Suppose you ran a test with two variations and you observe the following:
It is tempting but hard to conclude that B is the best overall. The observed performance difference might be merely due to chance rather than because of changes that are being tested.
In bayesian the fill rate and CPM are thought of in terms of probability distribution which represents our belief based on observed data. The spread of the probability distribution curve represents how precisely the experiment has measured the fill rate and CPM. As we gather more data, we refine the probability distribution for all the variants and the control. We quantify the probability of a random sample from any of the variant probability distributions would have a higher fill rate or eCPM over a random sample picked from the control probability distribution. This tells how likely the variant is best overall. This approach is way better than looking at the mean values of eCPMs and making a decision.
Suppose you ran a test with the 3 variations and you observe the following:
From the probability distribution of the observed results of A, B and C we calculate the following:
While variant C is most appealing the chances of C realizing the effect is only 20%. The recommended variant here would be B since there is an 80% chance of it realizing the effect.
