Last modified April 25, 2022 by John Mcgrath

Campaign holdouts

Measure the value of your campaigns by comparing key metrics between users who are exposed to a campaign versus those who aren’t with Swrve’s campaign-level holdouts.

Swrve provides you with two ways to measure the effectiveness and impact of your campaigns:

  • Holdout experiments – Exclude a percentage of new users from receiving all campaigns and compare the projected uplift in key metrics between users not exposed to any marketing campaigns versus those who are. For more information, see Holdout experiments.
  • Campaign holdouts – Apply a control group to specific campaigns and only target users who are eligible to receive that campaign, providing a more precise representation of campaign effectiveness.

At the campaign level, use the control group controls to customize the percentage of users you want to hold back from receiving the campaign. For example, you might choose not to apply a control group to types of campaigns that have already proven successful, but to apply a bigger control group to a category of campaigns that your team is rolling out for the first time.Enable campaign holdout toggle switch and control group size slider control

Campaign holdouts are currently only available for push notification campaigns as part of our Early Access program. To have this feature enabled for your app, contact your CSM at support@swrve.com.

Add a holdout to your campaign

The campaign holdout controls are available in the Audience builder step of the campaign workflow. To enable a campaign holdout:

  1. In the Audience block of the campaign builder, define your campaign’s target audience. You have the option to target all users or create a custom audience using the profile and behavior audience filters. For more information, see Targeting campaign audiences.
  2. For significant results, we recommend having a control and treatment size of at least 1000 users each. To check how many users are eligible to receive your campaign based on your audience filters, select Refresh estimate.
  3. To add a control group to the campaign, turn on the Campaign holdout toggle.
  4. To define the size of the control group, drag the Control group size slider to reflect the percentage of the audience you want to hold back from receiving the campaign.
  5. Review the estimated number of users in the control and treatment groups. Adjust the size of the control group as needed to ensure both groups contain a minimum of 1,000 users each.
  6. To save your audience and return to the campaign builder, select Save.

The control and treatment group figures represent the estimated number of users in each group for a single day. If running a recurring push campaign over multiple days, the number of users who enter the control and treatment groups might exceed the estimate.

Once the campaign starts, it is not possible to disable the campaign holdout or modify the control group size. Users in the control and treatment group remain in their respective groups until the campaign ends.

Campaign holdout results

To access the campaign holdout results, on the campaign build page select View Report, and then in the report, select the Holdout tab.

The Holdout report displays the campaign holdout details and platform-based holdout metrics for your campaign, if relevant. It also displays Swrve’s default tracking metrics and results for any custom goals you selected when creating the campaign.

To view platform-specific metrics for multi-platform campaigns, select the related platform.Campaign holdout report page

Audience

The Audience section of the report displays the number of users in the control and treatment groups.

  • Control – Displays the percentage of the control group and the total number of users who would have been sent a push notification if they were not in the control group.
  • Treatment – Displays the percentage of the treatment group and the total number of users who were sent the notification.
The treatment group size is equal to the audience figure reported in the Summary report. However, the control group and treatment group figures do not take into account messages rejected by the push notification services (APNS, GCM, ADM, HMS). Therefore, the Summary report’s sent count will always be lower than the treatment group count.

Holdout metrics

The Holdout metrics section displays holdout results for the default metrics and custom goals included in your campaign.

For every metric, the metric cards displays the following information for the campaign’s control and treatment groups:

Name Description
Control The holdout metric projected for 100% of users in the campaign. For example, if revenue for a control group of 15% was $300, then the projected revenue would be $2000 ($300 times 100, divided by 15).

Note: If the metric is already a ratio, for example, time in app per user, the metric is not projected for 100% of users, but instead is divided by the number of users in the respective group.

Treatment The treatment metric projected for 100% of users in the campaign. For example, if the total count of X event was 1700 for a treatment group of 85%, then the projected event count would be 2000 (1700 times 100, divided by 85).

Note: If the metric is already a ratio, for example, time in app per user, the metric is not projected for 100% of users, but instead is divided by the number of users in the respective group.

Total difference The difference, or uplift, between the totals for the treatment and control. For example, if the control group projects to 560 purchases and the treatment group projects to 700 purchases, then the uplift is 140 purchases.
Projected difference (percentage) The projected difference as a percentage of the projected control. For example, if the control projects to 560, and the uplift is 40, then the percentage is 7.1% (40 times 100, divided by 560).

Note: If the metric is already a ratio, for example, time in app per user, this represents the ratio % of the total difference and control group value (100 times the difference, divided by the holdout).

Campaign holdout metrics are calculated the same way as Holdout experiment results. However, campaign holdout metrics only consider users who Swrve attempted to send the push notification to (that is, users who are opted into push notifications and meet the campaign’s audience criteria).

Attribution period

Swrve tracks campaign metrics and reports on them for the attribution period you select when first setting up the campaign. However, the attribution periods used by each report differ.

  • The Summary report only displays the metrics for users who have engaged with or been influenced by a push notification.
  • The Holdout report displays the metrics for all users Swrve sent the push notification to (treatment group) or would have sent the push notification to (control group).

Therefore, the Holdout report treatment group figures will be larger than the Summary report figures.

In recurring push campaigns, the attribution period only applies to the first push engagement, influence, send (or in the case of the control group, would-be send). User behavior taken after any subsequent engagements, influences, or sends are not attributed to the campaign.

Statistical significance

To help achieve statistical significance, the Holdout report does not display metrics for campaigns that have fewer than 1,000 unique users in each of the control and treatment groups.

If the campaign is ongoing (recurring), the number of users in the control and treatment groups may increase over time. If results are not immediately available, return to the report at a later time to check if the campaign has reached the minimum audience threshold. If the campaign ends without reaching the 1,000 user per group minimum, Swrve deems the results as inconclusive and does not display the Holdout report.


Next steps

  • To measure the success of your campaign, set custom goals. For more information, see Campaign goals.
  • Review your campaign results. For more information, see Campaign reports.