You can deploy any resource A/B test to specific QA users of your app. This enables you to try as many tests as you want in your production environment without deploying it to your user base (that is, without starting the test) or having to recreate the test in a sandbox environment.
There are two main steps involved in QA testing a resource A/B test:
- Set up the QA device.
- Assign the QA device to your resource A/B test.
Setting up a QA Device
For information about setting up QA devices, see How Do I Set up QA Devices?
Assigning a QA Device
Once you’ve set up a QA device, add it to your A/B test. You must have already built your A/B test and clicked Create A/B Test on the Confirm A/B Test Details screen of the resource A/B test wizard. For more information about creating resource A/B tests, see Creating Resource A/B Tests.
To assign a QA device to your A/B test:
Step 1: Navigate to the QA Users section on the Resource A/B Test Details screen of your test.
Step 2: Select Control or the variant you want to test on your QA device. You can also click Disabled to exclude the device from the selected test.
What displays on the device of the QA user depends on the option you select and on the A/B test status, as detailed in the table below:
|A/B Test Status||Disabled||Control||Variant|
|Staging||QA user is treated as if they are not part of the A/B test and are served default resources.||QA user is served the control.||QA user is served the variant.|
|Started||QA user is treated as any other user.||QA user is served the control.||QA user is served the variant.|
|Finished||QA user is served the A/B test winner.||QA user is served the A/B test winner.||QA user is served the A/B test winner.|
If the A/B test is in Staging status, QA user(s) only see the control or the variant if they are in the control or variant group. Your app users are not affected by the A/B test. The image below illustrates what is seen by the various audiences of the app.
- If the A/B test is in Started status, both QA user(s) and users of your app who qualify for the test see one of the A/B test resources. Disabling a QA user who qualifies for the test does not prevent them from seeing the A/B test resources.
- If the A/B test is in Finished status, both the QA user(s) and the users of your app see the A/B test winner if their app version matches the A/B test restrictions.
Note that it takes up to 30 seconds for the A/B test server to update.
- View A/B testing best practices. For more information, see A/B Testing Best Practices.
- Find information about targeting users in A/B tests and running tests in parallel with no overlap of users. For more information, see Targeting Users in A/B Tests.
- Find information about aborting, deleting and archiving A/B tests. For more information, see Aborting and Deleting A/B Tests.