About A/B Testing

Compare Creatives

A variant is a creative that you are testing. A minimum of two variants is required before you can publish an A/B test task. An A/B test task cannot contain more than 20 variants. A champion variant is the control variant in the A/B test task. The performance of other variants that are included in the task is compared against the performance of the champion in order to determine the overall variant winner. For example, you might have one creative that is currently in use on your website and three new creatives that have not been delivered yet. You can create a web A/B test task that compares the current creative and the new creatives. In the A/B test task, you can add all of the creatives and label the current creative as the champion variant.

In your A/B test task, there is an option that enables the system to optimize the distribution of the variants while the test is running. This option enables the system to change the distribution of variants as the A/B test progresses so that the best-performing variant gets a larger portion of the distribution.

In task optimization, the initial weights are the weights that are assigned at the time of the optimization. Optimizing the distribution of the variants prevents new segments from being discovered by the analytics.

Find the Winning Creative

The variant that performs the best against the target audience is denoted as the overall variant winner. However, before the overall variant winner can be determined, a minimum estimated sample size must be met. The minimum estimated sample size is the minimum number of unique participants needed in a test before analytics can detect a difference in conversion rates.

Note: When estimating sample size, analytics uses the number of variants and the value of the Minimum detectable difference. To set the Minimum detectable difference, navigate to AdministrationTasks. Click Test in General.

After a minimum confidence level is reached, the system identifies the overall variant winner. You can then decide to end the test task and create a new task with the winning creative.

As the A/B test progresses, the system's analytics might discover that one variant out-performs the overall variant winner for a subset of your target audience. This subset of people is called a discovered segment, and you can access the profile for a discovered segment on the Insights tab. The profile includes details about the discovered segment, including which criteria distinguish the discovered segment from the segment defined in targeting.

Customize Test Settings

By default, A/B tests use the settings that are defined in the AdministrationTasks. Click Test in General. These settings affect how winners are identified and the size of the recommended sample size.

You can override the application defaults and specify custom values for this task by clicking Test Settings in the Variants page of the Content tab. These changes do not impact other tasks.

Set Task Priority

Although you cannot set a priority for an A/B test task, it always takes priority over a task that is not an A/B test task. For example, if you have a Web task and a Web A/B test task that deliver content to the same spot, the Web A/B test task always delivers its content, regardless of the priority of the Web task.

Create a Standard Task from an A/B Test Task

You can create a standard task that contains the same targeting, spot, and metrics as an A/B test task. Open the A/B test task and click . Select Create new task, and then select the variant that you want to use in your new task.

You can also create a standard task from the winning variant.