Last modified March 21, 2019 by John Mcgrath

How do I A/B test campaigns with a control group?

After creating a Resource A/B test with a control group, target the variants in your marketing campaigns. This enables you to compare the effectiveness of your campaign(s) against a true control group.

How to include a control group in campaigns

Step 1: In the Campaigns center, select Create campaign.

Step 2: On the Choose your marketing channel screen, select In-app message as your campaign channel, then select Continue.

Step 3: Complete the Overview and Add Content screens of the in-app message workflow (for more information on completing these screens, see Creating in-app messages).

Step 4: On the Set Target screen, under the Define Target Audience section, select Custom Audience and then define the following values:

  • Under User Profile Filters, select Resource Tests.
  • Select the test from the list of available resource A/B tests.
  • In the next list, select have been exposed to.
  • Select one of the non-control variants you created (for example, Activities Page).
    • Note: When creating your A/B Resource test, the control population size should be set to ‘100’ minus the percentage of users you want to expose to the A/B test campaigns. For example, if you want to expose 20 percent of your user base to the campaigns, set the control population size to ’80’.
  • To add the audience filter, select Apply filter.

A/B test for localized campaign

Step 5: In the Set Display Triggers section, select the event that you want to trigger the in-app message (for example, Session Start or activity_page_opened).

Set display trigger


Step 6: In the Select Campaign Goals section, select any additional campaign goal you want to track.

Step 7: Select Save as Draft: Preview & Test to proceed, and then schedule and launch the campaign as required.

Step 8: Create and launch another in-app message campaign. Follow the same steps, but this time use the non-control variant you didn’t use in Step 4 (Messages Page).

Step 9: After users have had a chance to engage with the app, return to the Resource A/B Testing screen and select the A/B test you created to compare how each of the variants are performing.

resource A/B test details control