Last modified April 6, 2017 by Shelly Wolfe

Campaign Holdout Experiments (New!)

Swrve’s Holdout Experiments enable you to automatically exclude a percentage of new users from new campaigns to compare the projected uplift in key metrics between users not exposed to marketing campaigns (holdout group) versus those who are (treatment group).

To access the Holdout Experiments screen, on the Campaigns menu, click Holdout Experiments.

Holdout Experiments

To start or stop a holdout experiment or add metrics to the Holdout Vs. Treatment Results, you must have a Company Admin user account.

Creating Holdout Experiments

To create a holdout experiment:

Step 1: Under the Holdout Group Settings section, in the Holdout Experiment field, enter the experiment name.

For segmentation purposes, the holdout group name is prefixed with Swrve Holdouts. The Holdout Experiment field enables you to include an additional identifier, otherwise the group is identified with a date/time stamp.

Step 2: In the Holdout Group Percentage box, enter the percentage of your new user base you want to exclude from new campaigns.

The set percentage applies only to new users from the start of the experiment and forward, and does not include your existing user base. This ensures a fair comparison to the group that is exposed to campaigns, as it compares users that haven’t seen any Swrve campaigns with users that have seen them. Existing users may have already been exposed to prior campaigns.

Step 3: Click Start Experiment.

Step 4: A dialog box is displayed and prompts you to confirm you want to start the experiment. Select the check box and click Start Experiment.

After you start the experiment, the status changes to Experiment is Live and the results of the experiment are displayed under the Holdout vs. Treatment Results section. The holdout group is automatically excluded from all subsequent campaigns, but you can manually override this on a campaign level.


Holdout Experiment Segments

Swrve automatically creates segments for the holdout group and treatment group that you can then use to target your audience in subsequent campaigns. The default format of the segment names are Swrve Holdout – [Experiment Name] and Swrve Treatment – [Experiment Name]. The segments are available in the following areas:

  • On the Segments tab when targeting audiences during campaign creation.
  • On the Segments tab when creating new segments.
  • In the Segments filter when viewing any dashboard reports, for example, KPI reports.

The segments track the following users:

  • Holdout – Any new users after the experiment starts that are allocated to the holdout group and excluded from campaigns.
  • Treatment – Any new users after the experiment starts that are allocated to the treatment group and exposed to campaigns.
Both the holdout and treatment segments include only new users from the start of the experiment and do not include existing users, as they may have already been exposed to prior campaigns. If you use either segment to create an audience, be aware that it only includes new users.

Stopping Holdout Experiments

To stop a holdout experiment:

Step 1: In the Holdout Group Settings section, click Stop Experiment.

Step 2: A dialog box is displayed and prompts you to confirm you want to stop the experiment. Select the check box and click Stop Experiment.

After you stop the experiment, the users added to the holdout and treatment groups remain in those segments, but no new users are added. The results for those segments apply to only the time period covered by the experiment. Any campaigns still running when the experiment is stopped continue to exclude the holdout segment.


Holdout Experiment Results

The Holdout Vs. Treatment Results section of the screen displays a summary of results for the current experiment. If no experiment is currently running, you can view the results of previous experiments by selecting the experiment from the list of available holdout groups.

For every metric included, the results table displays the following information for the holdout and treatment groups in the experiment:

Column Description
Holdout Total Displays the percentage of the holdout group and the total count of the metric for the users in the holdout segment.
Treatment Total Displays the percentage of the treatment group and the total count of the metric for the users in the treatment segment.
Projected Holdout The holdout metric projected to 100% of users. For example, if revenue for a holdout group of 15% was $300, then the projected revenue would be $2000 ($300 times 100, divided by 15). Note: This column replicates the holdout value if the metric is already a percentage or ratio, since it has no unit of measure (for example, retention).
Projected Treatment The treatment metric projected to 100% of users. For example, if the total count of event X in for a treatment group of 85% was 1700, then the projected event count would be 2000 (1700 times 100, divided by 85). Note: This column replicates the treatment value if the metric is already a percentage or ratio, since it has no unit of measure (for example, retention).
Projected Difference The difference, or uplift, between the projected totals for the treatment and holdout. For example, if the holdout projects to 560 purchases and the treatment projects to 700 purchases, then the uplift is 140 purchases. Note: If the metric is already a percentage or ratio, for example, retention, then this is the percentage difference of the treatment minus the holdout.
Projected Uplift The uplift as a percentage of the projected holdout. For example, if the holdout projects to 560, and uplift is 40, then percentage is 7.1% (40 times 100, divided by 560). If the metric is already a percentage or ratio, for example, retention, then this is the ratio % of the difference and holdout percentages (100 times the difference, divided by the holdout).

You can add metrics to the displayed results as needed. To add a metric to the table of results:

Step 1:  In the Holdout Vs. Treatment Results section, click Add Metric.

Step 2: Select the type of metric you want to add — KPIs, Event Counts, IAPs or Virtual Purchases.

Step 3: Select the specific metric; for example, the KPI, event, item purchase (or any purchase).

Step 4: Click Add Metric.

The metrics you select are displayed by default for any subsequent experiments you create. To remove a metric from the table, click the delete icon () for the selected metric.


Creating New Campaigns

If a holdout experiment is running, any new campaigns you create automatically include an audience filter to exclude the holdout segment from receiving the campaign. Additionally, you can create an audience that targets a current or preexisting experiment, using either the holdout or treatment segments, or both, as part of the audience (but remember the segments only include new users included in that experiment). This is useful if you want to run a follow-up campaign against these users for any reason.

New campaigns automatically exclude the holdout segment of new users, which means they target eligible users from both the treatment segment of new users and historical users who joined prior to the experiment. However, only the treatment users impact the experiment report, to ensure like-to-like comparisons with the holdout.

Operational Campaigns

In some instances, you may need to send an urgent message to all users, for example, a status update or service maintenance window. To ensure a campaign goes to all users, manually remove the holdout group filter on the Set Target screen of the campaign workflow.


Managing Existing Campaigns

Holdout experiments only apply to new users from the time the experiment starts. It is important to manage any existing campaigns, since new users could be exposed to existing campaigns. If you want the experiment to cover all campaigns, complete the following steps:

  1. Pause the active campaigns.
  2. Start the holdout experiment.
  3. Edit the target audience of each paused campaign to exclude the holdout segment.
  4. Restart the campaigns.

Similarly, after an experiment is stopped, users in the holdout segment continue to be excluded from any campaigns that were created during the experiment. If you want those users to now be eligible for those campaigns, either stop all campaigns before ending the experiment, or complete the following steps:

  1. Pause all campaigns.
  2. Stop the holdout experiment.
  3. Edit the target audience of each paused campaign to remove the holdout segment filter.
  4. Restart the campaigns.