Campaign holdouts
Swrve provides you with two ways to measure the effectiveness and impact of your campaigns:
- Holdout experiments – Exclude a percentage of new users from receiving all campaigns and compare the projected uplift in key metrics between users not exposed to any marketing campaigns versus those who are. For more information, see Holdout experiments.
- Campaign holdouts – Apply a control group to specific campaigns and only target users who are eligible to receive that campaign, providing a more precise representation of campaign effectiveness.
At the campaign level, use the control group controls to customize the percentage of users you want to hold back from receiving the campaign. For example, you might choose not to apply a control group to types of campaigns that have already proven successful, but to apply a bigger control group to a category of campaigns that your team is rolling out for the first time.
Add a holdout to your campaign
The campaign holdout controls are available in the Audience builder step of the campaign workflow. To enable a campaign holdout:
- In the Audience block of the campaign builder, define your campaign’s target audience. You have the option to target all users or create a custom audience using the profile and behavior audience filters. For more information, see Targeting campaign audiences.
- For significant results, we recommend having a control and treatment size of at least 1,000 users each. To check how many users are eligible to receive your campaign based on your audience filters, select Refresh estimate.
- To add a control group to the campaign, turn on the Campaign holdout toggle.
- To define the size of the control group, drag the Control group size slider to reflect the percentage of the audience you want to hold back from receiving the campaign.
- Review the estimated number of users in the control and treatment groups. Adjust the size of the control group as needed to ensure both groups contain a minimum of 1,000 users each.
- To save your audience and return to the campaign builder, select Save.
Once the campaign starts, it is not possible to disable the campaign holdout or modify the control group size. Users in the control and treatment group remain in their respective groups until the campaign ends.
Group size estimates
The figures represented by the audience estimates depend on the campaign channel.
Push notifications
The control and treatment group figures represent the estimated number of users in each group for a single day.
If running a recurring push campaign over multiple days, the number of users who enter the control and treatment groups might exceed the initial estimate.
In-app message & Embedded campaigns
The control and treatment group figures represent the estimated number of users in each group across seven days, and do not take display trigger criteria into account. The actual number of users who view the campaign will depend on the campaign’s display criteria. For example, if you choose to display the message on app launch, more users are likely to see the campaign than if you select a specific event as the trigger.
Campaign holdout results
To access the campaign holdout results, on the campaign build page select View Report, and then in the report, select the Holdout tab.
The Holdout report displays the campaign holdout details and platform-based holdout metrics for your campaign, if relevant. It also displays Swrve’s default tracking metrics and results for any custom goals you selected when creating the campaign.
To view platform-specific metrics for multi-platform campaigns, select the related platform.
Audience
The Audience section of the report displays the number of users in the control and treatment groups.
- Control – Displays the percentage of the control group and the total number of users who would have been sent the push notification or shown the in-app message or embedded campaign if they were not in the control group.
- Treatment – Displays the percentage of the treatment group and the total number of users who were sent the push notification or shown the in-app message or embedded campaign.
There are slight differences in how Swrve calculates the audience metrics on the Summary tab versus the Holdout tab, depending on the campaign channel.
Push notifications
The push notification treatment group size is equal to the Audience figure reported in the Summary report. However, the control group and treatment group figures do not take into account messages rejected by the push notification services (APNS, FCM, ADM, HMS). Therefore, the Summary report’s Sent count will always be lower than the treatment group count.
In-app messages & Embedded campaigns
The in-app message and embedded campaign treatment group size is equal to the User figure in the Summary report. However, the treatment and control group figures do not take into account messages blocked by message display rules. Therefore, the Summary report’s User count will typically be lower than the treatment group count.
Holdout metrics
The Holdout metrics section displays holdout results for the default metrics and custom goals included in your campaign.
For every metric, the metric cards displays the following information for the campaign’s control and treatment groups:
| Name | Description |
|---|---|
| Control | The holdout metric projected for 100% of users in the campaign. For example, if revenue for a control group of 15% was $300, then the projected revenue would be $2000 ($300 times 100, divided by 15).
Note: If the metric is already a ratio, for example, time in app per user, the metric is not projected for 100% of users, but instead is divided by the number of users in the respective group. |
| Treatment | The treatment metric projected for 100% of users in the campaign. For example, if the total count of X event was 1700 for a treatment group of 85%, then the projected event count would be 2000 (1700 times 100, divided by 85).
Note: If the metric is already a ratio, for example, time in app per user, the metric is not projected for 100% of users, but instead is divided by the number of users in the respective group. |
| Total difference | The difference, or uplift, between the totals for the treatment and control. For example, if the control group projects to 560 purchases and the treatment group projects to 700 purchases, then the uplift is 140 purchases. |
| Projected difference (percentage) | The projected difference as a percentage of the projected control. For example, if the control projects to 560, and the uplift is 40, then the percentage is 7.1% (40 times 100, divided by 560).
Note: If the metric is already a ratio, for example, time in app per user, this represents the ratio % of the total difference and control group value (100 times the difference, divided by the holdout). |
Attribution period
Swrve tracks campaign metrics and reports on them for the attribution period you select when first setting up the campaign. For recurring campaigns, the attribution period only applies to the first engagement, influence, send (or in the case of the control group, would-be send). Any action a user takes after any subsequent engagements, influences, or sends is not attributed to the campaign.
Push notifications
The attribution periods used by each report differ for push notifications.
- The Summary report only displays the metrics for users who have engaged with or been influenced by a push notification.
- The Holdout report displays the metrics for all users Swrve sent the push notification to (treatment group) or would have sent the push notification to (control group).
Therefore, the Holdout report treatment group figures will be larger than the Summary report figures.
In-app messages & Embedded campaigins
For in-app message and embedded campaigns triggered on user behavior, the attribution period used by the Summary and Holdout reports is the same. The reports display metrics for users who have viewed the in-app message or have been delivered the embedded campaign.
Statistical significance
To help achieve statistical significance, the Holdout report does not display metrics for campaigns that have fewer than 1,000 unique users in each of the control and treatment groups.
If the campaign is ongoing (recurring), the number of users in the control and treatment groups may increase over time. If results are not immediately available, return to the report at a later time to check if the campaign has reached the minimum audience threshold. If the campaign ends without reaching the 1,000 user per group minimum, Swrve deems the results as inconclusive and does not display the Holdout report.
Next steps
- To measure the success of your campaign, set custom goals. For more information, see Campaign goals.
- Review your campaign results. For more information, see Campaign reports.