Last modified February 5, 2019 by Shelly Wolfe

Targeting users in A/B tests

One of the main features of A/B testing is the ability to target specific users in a test.

Target users in resource A/B test

You can target the following types of users on the Build A/B Test screen of the resource A/B test workflow:

  • All Users – anybody who uses the app is a participant.
  • Only new users who join after the start of the A/B test.
  • Only users who belong to a particular segment.
  • Only users of a specific version of your app.

Additionally, you can set a cap on the number of participants in a test or assign a resource A/B test to user partitions in order to run resource A/B tests in parallel with no overlap of users between tests.

Targeting new users

You can restrict deployment of resource A/B tests to users who are new since the time the test was started. This feature ensures that the test is relatively unaffected by users’ pre-conceived notions of the app or by any historic impression of app content.

To use this option, under the Target Users section, select Only new users who join after the start of the A/B test. When you start the A/B test, you can specify the date from which to include new users.

To enable the targeting of new users, your development team must ensure that the app’s request for users’ A/B test resource differences are accompanied by the joined parameter. For more information, see the A/B Test API guide.

Targeting users by segment

To target users by segment, under the Target Users section, select Only users who belong to the following segment and then select the required user segment from the segment list.

If users who are part of the test group change membership of the selected segment, they still remain in the A/B test until it is finished.

Targeting users by app version

You can A/B test against a specific version of your app by specifying the app version you want to target in the App version field. You can enter multiple versions by adding a comma after each version string. The app version string(s) you enter in the App version field must match the version string sent by your development team using the Swrve A/B Test API. If you miss just one digit, your test has no participants.

Setting a cap on A/B test participants

This setting enables you to minimize the risk of resource A/B tests by capping the number of participants. To use this feature, under the Target Users section, select Set a maximum of X participants, and then enter the number of participants in the box.

If you set a cap, you can change or remove it later, while the test is running. This enables you to minimize the risk of tests where the impact of some of the variants may be unknown.

Once the cap is reached, no new participants are allowed enter the test. However, all existing participants continue to be served the correct variant from the test.

At any stage, you can change, remove or re-add the participant cap by selecting the Edit link next to the number of Max participants.

Participant cap

Alternatively, you can limit the test to the existing participants by selecting the Freeze link next to the current Participants count. To allow participants to be added to the test again, select Unfreeze.

Assigning a test to user partitions

Partitioning enables you to run A/B tests in parallel with no overlap of users between tests. This is useful if you want to guarantee exclusivity between tests; for example, if you want to test changes that can directly affect other tests which are running at the same time.

Swrve enables you to create partition groups to manage test exclusivity. Each partition group contains 100 partitions. You can assign an A/B test to a percentage of the partitions that are currently available (and unassigned) in the partition group. For example, if you assign A/B Test A to 80 partitions (80%) of Partition Group X, then you can only assign A/B Test B to a maximum of 20 partitions (20%) of Partition Group X. This ensures that your A/B test is only served to users who have not already been served other A/B tests.

The Partition Group option is available on the Build A/B Test screen of the resource A/B test workflow, under the Target Users section. To assign an A/B test to a percentage of partitions within a partition group:

Step 1: If the partition group you require is not already available in the Partition Group drop down list, create a new group. Otherwise, proceed to Step 2.

To create a new partition group:

  • Select Create Partition Group.
  • In the Create Partition Group dialog box, enter the Name of the partition group.
  • Select Create Partition Group to save the partition group.

Step 2: Select the partition group for the current A/B test from the Partition Group drop-down list. The percentage of the partitions that are currently available for assignment within the partition group displays below. Swrve also displays and provides links to any other A/B tests that are currently using the partition group so that you can modify these tests if required to free up partitioning space.

Step 3: In the partitions (n available) field, enter the percentage of the available partitions to which you want to assign the current A/B test.

If you created a partition group prior to Swrve Release 67, you manually defined the number of partitions in a partition group. Your user base was then split evenly among the number of partitions you selected when creating your A/B test. This split still applies for any partition groups you created prior to 67. For partition groups that you define from 67 onwards, the new behavior of variably-sized user bases applies.

Next steps