Last modified January 14, 2020 by Shelly Wolfe

A/B testing in-app messages

Use Swrve’s in-app message A/B testing feature to test and track up to 10 variations per message. When creating a new in-app message, use the A/B test feature to:

  • Create up to 10 message variants and measure user response to different background images or call to action (CTA) buttons.
  • Track and compare how variants are performing and adjust your in-app messages accordingly.

To access the A/B test feature, on the Add Content screen of the in-app message workflow, select A/B test this message. For information on how to access the campaign workflow and create in-app messages, see Creating in-app messages.

A/B testing of in-app messages vs. resources

The A/B testing of in-app messages feature is for testing variations of the content and layout of your in-app messages only. For more information on A/B testing specific aspects of your app using variants of different resources, see Intro to resource A/B testing.

Create A/B test for in-app message

To create an A/B test of an in-app message:

Step 1: On the Overview screen of the in-app message campaign workflow, specify the message name, description, template and orientation and then select Next: Add Content to proceed.

Step 2: On the Add Content screen, add the content of your control message, based on the template selected in Step 1; for example, background and button images and actions.

Step 3: Select A/B test this message. The A/B test variant tabs appear and by default include a copy of the control message labeled as Variant B.

A/B test in-app message

Step 4: To create additional variants:

  • Select the add icon next to the current variant. A copy of the current message is created and the banner displays the percentage of the target audience that will see that message. The target audience is divided evenly, based on the number of variants you create. For example, if you create four variants, 25% of the target audience will see each variant.
  • Select the edit icon or variant name to enter a specific name for the variant. The name displays on the tab above as you type.
  • Change the content of the message variant as required (for example, different background, button text or button position).
  • After you have created your message variants, select Next: Set Targeting to proceed.

Step 5: On the Set Target screen, specify the target audience, display triggers and campaign goals. The target audience is split evenly among the number of your variants, however each variant includes the same display trigger and campaign goals for all users. Select Save as Draft: Preview & Test to proceed.

Step 6: On the Preview & Test screen, perform the following actions:

  • If required, QA test your in-app message before launching it. We recommend you check each A/B test variant on at least one QA device. For more information, see QA testing in-app messages.
  • Select each variant tab for a preview of the in-app message. To launch the message:
    • If you want to launch the message immediately, select Launch as soon as possible.
    • If you want to define the message activation period, enter a start and end date/time in the Schedule launch section and then select Launch on <scheduled date>.

If you launch the message (without scheduling), its status is set to active. If you schedule the message to be deliverable from a particular date/time, its status is set to scheduled until this time is reached.

Your message now appears on the In-App Messages screen.

You can create a copy of your message by selecting the Duplicate icon on the message overview on the In-App Messages screen. In this instance, a copy of your message and all the variants is created in draft status and displays on the In-App Messages screen. You can edit and launch the copy according to your needs.

Viewing A/B test results

To view the individual in-app message screen for a particular message, select the campaign on the Campaigns screen. For in-app messages in any status other than draft or scheduled, the individual message screen displays a detailed report of your in-app message campaign results. For more information on the individual in-app message screen, see Managing in-app messages.

At present, you cannot add or delete variants after a message is active. If you want to add or remove variants based on their performance, you need to end the message, create a duplicate draft copy of the message and then edit or delete the variants as required. For more information, see Managing in-app messages.

Next steps