Test Email Subjects

Setting Up the Test

To enable A/B testing of your email subject line:

1. Hover over the Testing section and click Edit.
2. Enable the Subject line A/B test option.
   Note: Enabling this option affects what actions can be taken after the task is activated. Activating a task without a schedule also runs it. Scheduled tasks are activated and run when the start date is reached.

   • After the task is activated, this option cannot be changed.
   • When this option is enabled, tasks can be activated only one time. After the task runs, it cannot be renamed or updated.
3. Select the number of hours for the test Duration. The duration defines how long data is collected before test results are finalized. The test starts after the task is run and the first email is sent. By default, this duration is set to 4 hours.
4. Select a primary metric.
   Note: When you enable A/B subject line testing, any metrics or schedule options that were selected previously are cleared. Only primary metrics are supported for A/B testing. In addition, the metric is set on the Header section instead of the Metrics section.
   • Open rate: The email must be opened at least one time.
   • Click rate: At least one link must be clicked one time.
   Note: These metrics are calculated based on the recipient’s email address. SAS Customer Intelligence 360 considers the user identity as a unique identifier.
5. Set the percentage for the Baseline conversion. This value is the expected conversion rate based on the average open or click rate for all emails sent from this tenant in the past 90 days, excluding bounced emails.
6. Click Test Settings to customize the test settings for this task.

   By default, A/B tests use the settings that are defined in the Test page. To see these settings, navigate to AdministrationTasks and click Test in General. These settings affect how winners are identified and the size of the recommended sample size. You can override the application defaults and specify custom values for this task. These changes do not impact other tasks.

   1. Customize the values that impact the recommended winner. These are the values for the minimum confidence and lift that must be met in order for the application to recommend an overall variant winner in A/B test tasks.
      • Enter a value for the Confidence interval and threshold. This value represents the likelihood that a variant's conversion rate differs from the winner's conversion rate. The valid range is from 75% to 99%.
      • Enter a value for the Lift. The lift is calculated as (variant conversion rate – champion conversion rate) / champion conversion rate. The variant with the largest lift that exceeds both the confidence and the lift becomes the recommended winning variant. The valid range is from 0% to 100%.
   2. Enter a value for the Minimum detectable difference. This value represents the minimum difference between the winner's conversion rate and a variant's conversion rate. The valid range is from 1% to 5%.

   To restore the test settings to the values defined in the adminstration settings, navigate to AdministrationTasks. Then click Test in General, and click Restore application settings. This restores the settings to the values defined in the Test page at the time that the link is clicked.

7. Hover over the Header section and click Edit.
8. (Optional) Enter a hypothesis. This is a short description of what you are testing.
9. Enter the two subjects to test.
10. (Optional) Use generative AI (GenAI) to suggest alternative subjects. The suggestions match the tone of the original subject. Merge tags are supported.
    TipReview the GenAI Best Practices for Generating Subjects when you are first using this option.
    1. Click Generate.
    2. Select the subject that you want to use.
    3. Select Apply to variant B or Apply to variant A depending on which subject you want to replace.
    IMPORTANT It is your responsibility to review any output generated with AI to ensure that the AI-generated content that you select is appropriate and accurate for its intended purpose.

To preview each subject line, select Responsive previews from the Content section.

Running the Test

The test begins after the first email message is sent. If the application can identify a winner, the winning variant is selected based on the data that is collected during the test duration. The processing time for some data might extend beyond the test duration. For example, confidence and lift are not calculated until the application finishes processing the data that was collected throughout the duration of the test.

To see the performance of the variants, as well as other metrics about the test, select the Task Insights view. The task continues to run after the duration. However, the data that is collected after the duration does not impact whether a winning variant is found. The updated data is displayed on the Task Insights page.

Keep these points in mind when running an A/B subject line test in a bulk email task:

• Activating a task without a schedule also runs it. Scheduled tasks are activated and run when the start date is reached.
• Recurring schedules are not supported.
• When this option is enabled, tasks can be run only one time. After the task runs, it cannot be renamed or updated.
• Scheduled tasks do not have the option to be run manually. The exception to this rule is when a scheduled task is paused before the start date and the start date has passed.
• If the occurrence fails, the task cannot be run again. However, you can save a copy of the task and run the copy.