One of the main features of A/B testing is the ability to target specific users in a test.
You can target the following types of users on the Build A/B Test screen of the resource A/B test wizard:
- All Users – anybody who uses the app is a participant.
- Only new users who join after the start of the A/B test.
- Only users who belong to a particular segment.
- Only users of a specific version of your app.
Additionally, you can set a cap on the number of participants in a test or assign a resource A/B test to user partitions in order to run resource A/B tests in parallel with no overlap of users between tests.
Targeting New Users
You can restrict deployment of resource A/B tests to users who are new since the time the test was started. This feature ensures that the test is relatively unaffected by users’ pre-conceived notions of the app or by any historic impression of app content.
To use this option, under the Target Users section, select Only new users who join after the start of the A/B test. When you start the A/B test, you can specify the date from which to include new users.
Targeting Users by Segment
To target users by segment, under the Target Users section, select Only users who belong to the following segment and then select the required user segment from the segment list.
Targeting Users by App Version
You can A/B test against a specific version of your app by specifying the app version you want to target in the App version field. You can enter multiple versions by adding a comma after each version string. The app version string(s) you enter in the App version field must match the version string sent by your development team using the Swrve A/B Test API. If you miss just one digit, your test has no participants.
Setting a Cap on A/B Test Participants
This setting enables you to minimize the risk of resource A/B tests by capping the number of participants. To use this feature, under the Target Users section, select Set a maximum of X participants, and then enter the number of participants in the box.
If you set a cap, you can change or remove it later, while the test is running. This enables you to minimize the risk of tests where the impact of some of the variants may be unknown.
Once the cap is reached, no new participants are allowed enter the test. However, all existing participants continue to be served the correct variant from the test.
At any stage, you can change, remove or re-add the participant cap by clicking the Edit link next to the number of Max participants.
Alternatively, you can limit the test to the existing participants by clicking the Freeze link next to the current Participants count. To allow participants to be added to the test again, click Unfreeze.
Assigning a Test to User Partitions
Partitioning enables you to run A/B tests in parallel with no overlap of users between tests. This is useful if you want to guarantee exclusivity between tests; for example, if you want to test changes that can directly affect other tests which are running at the same time.
Swrve enables you to create partition groups to manage test exclusivity. Each partition group contains 100 partitions. You can assign an A/B test to a percentage of the partitions that are currently available (and unassigned) in the partition group. For example, if you assign A/B Test A to 80 partitions (80%) of Partition Group X, then you can only assign A/B Test B to a maximum of 20 partitions (20%) of Partition Group X. This ensures that your A/B test is only served to users who have not already been served other A/B tests.
The Partition Group option is available on the Build A/B Test screen of the resource A/B test wizard, under the Target Users section. To assign an A/B test to a percentage of partitions within a partition group:
Step 1: If the partition group you require is not already available in the Partition Group drop down list, create a new group. Otherwise, proceed to Step 2.
To create a new partition group:
- Click Create Partition Group.
- In the Create Partition Group dialog box, enter the Name of the partition group.
- Click Create Partition Group to save the partition group.
Step 2: Select the partition group for the current A/B test from the Partition Group drop-down list. The percentage of the partitions that are currently available for assignment within the partition group displays below. Swrve also displays and provides links to any other A/B tests that are currently using the partition group so that you can modify these tests if required to free up partitioning space.
Step 3: In the partitions (n available) field, enter the percentage of the available partitions to which you want to assign the current A/B test.