Last Updated on: 19 Sep, 2023
Multivariate testing is a randomized experimentation process, where we split users into more than 2 groups, and expose each group to different app configurations. If you have a hypothesis, which requires you to change multiple variables and measure the effect of each combination of variables, this is where you start.
You can use Multivariate testing to validate any hypothesis by adding ad units, segments (definition/behavior), add networks, bidding vs. waterfall, open auction optimizations, timeouts, etc...
To help you measure the effect of a change, we use Bayesian inference to state the probability of any one variant being the best overall. This way as an experiment generates more data, we will make a recommendation on the variant that is best overall and how likely is the variant to be best overall. This helps you take an informed and accurate decision quickly, without making the mistake of acting on early data.
We also help you understand how experiments are impacting other KPIs like retention rate and LTV as well as making experiments and variants available in Cohort Analysis.
Create an Experiment
To create an experiment first open the App Workspace. To open your App Workspace, click on All Apps, and then Select the App from the menu. Then go to Reports/Experiments, and click on Create Experiment. To create an A/B experiment, select Multi-variate.
To define an experiment:
- Start with the hypothesis
- Create randomized user groups with uniform probability (Uniform Choice) or weighted probability (Weighted Random)
- Configure the control group, size of the user group, and the targeting group
The size of the user group defines what percentage of eligible users will be exposed to a particular treatment. Please note users who are mapped to a particular treatment once are always exposed to the same treatment until the experiment is Live.
The targeting group selected for the control is the current active targeting group to that all eligible users are exposed.
To define a variant:
- Configure the variant group, size of the user group, and the targeting group
- Configures the variant targeting group, primarily networks, adlines, and open auctions.
For accurate evaluation we recommend, that variant adline credentials (adunit id, placement id, zone id, tag id, etc..) are set up afresh with the respective networks.
To add ad lines for nonexistent networks, first, go to Partner / Networks and enable the network. Create new ad lines for the variant by clicking on Configure for each variant.
Experiments can be saved in the Draft state. To start an experiment click on Start Experiment.
- An experiment cannot be started without a variant.
- Once an experiment is started (Live), neither the size of user groups nor targeting groups can be changed.
- It is recommended that networks, ad lines, and open auction settings not be changed after an experiment is started.
- A Live experiment can be Paused (II) or Concluded.
To view experiment results click on the results icon from Results/Experiments for the respective experiment.
The following are reported for each variant:
||Count of DAU that have been exposed to a treatment, either control or any variant.
||Count of ad requests that have been exposed to a treatment, either control or any variant.
||Count of impressions that have been shown to users exposed to a treatment, either control or any variant.
||Total revenue reported for treatment, either control or any variant.
||Fill rate reported for treatment, either control or any variant.
||eCPM reported for treatment, either control or any variant.
||The range in which revenue/impression will fall with a probability of 0.95 for treatment, either control or any variant.
||The likelihood that a variant will beat all other variants
Conclude an Experiment
To conclude an experiment, select the variant that is delivering the best results either on the basis of the aggregate metrics or on the basis of the highest P(Beat All). Click on Scale Variant & Calculate.
Start a Follow-Up Experiment
To start a follow-up experiment from a concluded experiment, go to Reports / Experiments and click on the create icon. This will create a copy of the Concluded experiments in the Draft state. In the draft, state experiments can be edited.