Last modified June 28, 2023 by Shelly Wolfe

QA testing A/B tests

You can deploy any resource A/B test to specific QA users of your app. This enables you to try as many tests as you want in your production environment without deploying it to your user base (that is, without starting the test) or having to recreate the test in a sandbox environment.

There are two main steps involved in QA testing a resource A/B test:

  1. Set up the QA device.
  2. Assign the QA device to your resource A/B test.

Setting up a QA device

For information about setting up QA devices, see How do I set up QA devices?

Assigning a QA device

Once you’ve set up a QA device, add it to your A/B test. You must have already built your A/B test and selected Create A/B Test on the Confirm A/B Test Details screen of the resource A/B test wizard. For more information about creating resource A/B tests, see Creating resource A/B tests.

To assign a QA device to your A/B test:

Step 1: Navigate to the QA Users section on the Resource A/B Test Details screen of your test.

A/B Test QA

Step 2: Select Control or the variant you want to test on your QA device. You can also select Disabled to exclude the device from the selected test.

For any tests using the same resource, you can only assign a QA device to one test at a time. If you assign a QA device to a subsequent test, the QA device in the first test is automatically set to Disabled.

What displays on the device of the QA user depends on the option you select and on the A/B test status, as detailed in the table below:

A/B Test Status Disabled Control Variant
Scheduled QA user is treated as if they are not part of the A/B test and are served default resources. QA user is served the control. QA user is served the variant.
Staging QA user is treated as if they are not part of the A/B test and are served default resources. QA user is served the control. QA user is served the variant.
Started QA user is treated as any other user. QA user is served the control. QA user is served the variant.
Finished QA user is served the A/B test winner. QA user is served the A/B test winner. QA user is served the A/B test winner.
  • If the A/B test is in Scheduled or Staging status, QA user(s) only see the control or the variant if they are in the control or variant group. Your app users are not affected by the A/B test. The image below illustrates what is seen by the various audiences of the app.
  • If the A/B test is in Started status, both QA user(s) and users of your app who qualify for the test see one of the A/B test resources. Disabling a QA user who qualifies for the test does not prevent them from seeing the A/B test resources.
  • If the A/B test is in Finished status, both the QA user(s) and the users of your app see the A/B test winner if their app version matches the A/B test restrictions.

QA A/B Test

Note that it takes up to 30 seconds for the A/B test server to update.

Next steps