Swrve provides you with two ways to measure the effectiveness and impact of your campaigns:

• Holdout experiments – Exclude a percentage of new users from receiving all campaigns and compare the projected uplift in key metrics between users not exposed to any marketing campaigns versus those who are.
• Campaign holdouts – Apply a holdout group to specific campaigns and only target users who are eligible to receive that campaign, providing a more precise representation of campaign impact. For more information, see Campaign holdouts.

To access the Holdout Experiments screen, on the Analytics menu, select Holdout experiments.

Creating holdout experiments

To create a holdout experiment:

1. Under the Holdout Group Settings section, in the Holdout Experiment field, enter the experiment name.
   For segmentation purposes, the holdout group name is prefixed with Swrve Holdouts. The Holdout Experiment field enables you to include an additional identifier, otherwise the group is identified with a date/time stamp.
2. In the Holdout Group Percentage box, enter the percentage of your new user base you want to exclude from new campaigns.
   The set percentage applies only to new users from the start of the experiment and forward, and does not include your existing user base. This ensures a fair comparison to the group that is exposed to campaigns, as it compares users that haven’t seen any Swrve campaigns with users that have seen them. Existing users may have already been exposed to prior campaigns.
3. Select Start Experiment.
4. A dialog box is displayed and prompts you to confirm you want to start the experiment. Select the check box and select Start Experiment.

After you start the experiment, the status changes to Experiment is Live and the results of the experiment are displayed under the Holdout vs. Treatment Results section. The holdout group is automatically excluded from all subsequent campaigns, but you can manually override this on a campaign level.

Holdout experiment segments

Swrve automatically creates segments for the holdout group and treatment group that you can then use to target your audience in subsequent campaigns. The default format of the segment names are Swrve Holdout – [Experiment Name] and Swrve Treatment – [Experiment Name]. The segments are available in the following areas:

• On the Segments tab when targeting audiences during campaign creation.
• On the Segments tab when creating new segments.
• In the Segments filter when viewing any dashboard reports, for example, KPI reports.

The segments track the following users:

• Holdout – Any new users after the experiment starts that are allocated to the holdout group and excluded from campaigns.
• Treatment – Any new users after the experiment starts that are allocated to the treatment group and exposed to campaigns.

Stopping holdout experiments

To stop a holdout experiment:

1. In the Holdout Group Settings section, select Stop Experiment.
2. A dialog box is displayed and prompts you to confirm you want to stop the experiment. Select the check box and select Stop Experiment.

After you stop the experiment, the users added to the holdout and treatment groups remain in those segments, but no new users are added. The results for those segments apply to only the time period covered by the experiment. Any campaigns still running when the experiment is stopped continue to exclude the holdout segment.

Holdout experiment results

The Holdout Vs. Treatment Results section of the screen displays a summary of results for the current experiment. If no experiment is currently running, you can view the results of previous experiments by selecting the experiment from the list of available holdout groups.

For every metric included, the results table displays the following information for the holdout and treatment groups in the experiment:

You can add metrics to the displayed results as needed. To add a metric to the table of results:

1. In the Holdout Vs. Treatment Results section, select Add Metric.
2. Select the type of metric you want to add—KPIs, Event Counts, IAPs or Virtual Purchases.
3. Select the specific metric; for example, the KPI, event, item purchase (or any purchase).
4. Select Add Metric.

The metrics you select are displayed by default for any subsequent experiments you create. To remove a metric from the table, select the delete icon  for the selected metric.

Creating new campaigns

If a holdout experiment is running, any new campaigns you create automatically include an audience filter to exclude the holdout segment from receiving the campaign. Additionally, you can create an audience that targets a current or preexisting experiment, using either the holdout or treatment segments, or both, as part of the audience (but remember the segments only include new users included in that experiment). This is useful if you want to run a follow-up campaign against these users for any reason.

Operational campaigns

In some instances, you may need to send an urgent message to all users, for example, a status update or service maintenance window. To ensure a campaign goes to all users, manually remove the holdout group filter on the Audience screen.

Managing existing campaigns

Holdout experiments only apply to new users from the time the experiment starts. It is important to manage any existing campaigns, since new users could be exposed to existing campaigns. If you want the experiment to cover all campaigns, complete the following steps:

1. Pause the active campaigns.
2. Start the holdout experiment.
3. Edit the target audience of each paused campaign to exclude the holdout segment.
4. Restart the campaigns.

Similarly, after an experiment is stopped, users in the holdout segment continue to be excluded from any campaigns that were created during the experiment. If you want those users to now be eligible for those campaigns, either stop all campaigns before ending the experiment, or complete the following steps:

1. Pause all campaigns.
2. Stop the holdout experiment.
3. Edit the target audience of each paused campaign to remove the holdout segment filter.
4. Restart the campaigns.