A/B testing

The MAX A/B Testing feature allows you to use real-time data to test, and to measure the impact of, any change that you make to your waterfall configuration.

You can experiment with different ad monetization strategies and identify opportunities that will maximize LTV. Such strategies might include adding or removing a network to see incrementality, adding a new price point, or experimenting with regional waterfall optimizations.

Creating an A/B test

Take the following steps to create an A/B test:

1. Select MAX > Mediation > Manage > Ad Units in your MAX Dashboard.
2. Click the ad unit for which you would like to establish an A/B test. The Edit Ad Unit window appears.
3. From the ⋯ menu in the Default Waterfall tab, select Create A/B Test.
4. You then see a small pop-up dialog that describes the process. Check the “Copy existing ad unit configuration…” checkbox and click Create A/B Test to copy your existing waterfall to your new test.

1. When you create a new A/B test, by default MAX applies your Test configuration to 50% of your users. You can modify this value, for example if you want to run tests longer at a lower volume of traffic. To do so, change the value of Test Group Allocation when you create your A/B test.

2. You see a new toggle called Test in the Default Waterfall tab. In this toggle setting, you see your test configuration (which, at first, is the same as the waterfall configuration you copied). Type in a new A/B Test Name for your test, and start changing the waterfall according to what you want to test.

   AppLovin recommends that you name your A/B tests with a taxonomy that reflects the start time for the A/B test (for example, “0812_01UTC_FBBiddingAdded”). This makes it easier for you to keep track of changes and to monitor the results.

   

3. After you make the changes that distinguish your test waterfall, click Save. The MAX ad unit page updates with your new A/B test.

4. When you return to your ad unit page, your default view will still be your original control waterfall. Click the Control toggle to switch to Test and compare your two waterfalls.

   You can visualize each waterfall setting by selecting Preview Waterfall from the ⋯ menu in the Default Waterfall tab. Then, in the Waterfall Preview dialog, choose the country for which you want to visualize your waterfall. You then see a visual representation of the non-bidder ad networks in your waterfall for that country (and also a list of any bidder networks, which supplement your waterfall).

   

Monitoring results

Monitor the impact of your test on requests, impressions, and revenue, for 24–48 hours after the test begins. Do this in MAX > Mediation > Analyze > Advanced Reporting or MAX > Mediation > Analyze > A/B Tests. You then can either promote your test waterfall to all your users, or you can deprecate it and revert to the original control configuration. Your A/B test should appear in production and be present in your reports no more than 20 minutes after you save it.

Any change that you notice reflected in requests, impressions, and revenue indicates a change in the IMP/DAU, REQ/DAU, and ARPDAU. This is probably a result of the changes you made in your test waterfall.

The example screenshot above shows a 28% drop in IMP/DAU, a 16% decrease in REQ/DAU, and a 27% drop in ARPDAU. If these results continue after a few hours (to generate a larger sample), this test would be a good one to deprecate, due to the potential revenue loss.

To deprecate or promote a test:

1. Go to the Default Waterfall tab of your ad unit page.
2. Set the Control/Test toggle to Test.
3. Select End A/B Test from the ⋯ menu.
4. In the End A/B Test dialog that appears, click either Deprecate Test or Promote Test.
5. Click Save. Your changes will be live in production traffic in a few minutes.

Understanding results

MAX provides an A/B test result page for each test that generates over 10,000 impressions in the test group. This page shows results the day after the test generates these impressions. The A/B Tests page (MAX > Mediation > Analyze > A/B Tests) lists and links to these pages.

You can read this A/B test result page to understand the Network Revenue share-of-voice changes, the ARPDAU lift anticipated with the test, and many other data points that will help you make the right decision to promote or deprecate the test.

• Lucas Yalap, CTO of YSO Corp, discusses how they A/B test their monetization setup
• “What is LTV?” from AppLovin’s Blog.

MAX copies your existing waterfall structure into the test structure you create. You only need to make those changes that you want to test in your new Test Ad Unit. You do not have to build a complete new waterfall from scratch.

AppLovin recommends that you promote changes that have been stable for 24 hours, above 10,000 impressions per day, and above 2% incrementality, for sustained improvement in your ARPDAU. Deprecate any test that does not result in a lift.