Last modified November 22, 2014 by Gordon Glauser

A/B Testing In-App Messages

Swrve’s A/B testing of in-app messages feature enables you to test and track up to 10 variations per message. When creating a new in-app message, use the A/B test feature to:

  • Create up to 10 message variants and measure user response to different background images or call to action (CTA) buttons.
  • Track and compare how variants are performing and adjust your in-app messages accordingly.

To access the A/B test feature, on the Add Content screen of the in-app message wizard, click A/B test this message. For information on how to access the wizard and create in-app messages, see Creating In-App Messages.

A/B Testing of In-App Messages vs. Resources

The A/B testing of in-app messages feature is for testing variations of the content and layout of your in-app messages only. For more information on A/B testing specific aspects of your app using variants of different resources, see Intro to Resource A/B Testing.


Create A/B Test for In-App Message

To create an A/B test of an in-app message:

Step 1: On the Overview screen of the in-app message wizard, specify the message name, description, template and orientation and then click Next: Add Content to proceed.

Step 2: On the Add Content screen, add the content of your control message, based on the template selected in Step 1; for example, background and button images and actions.

Step 3: Click A/B test this message. The A/B test variant tabs appear and by default include a copy of the control message labeled as Variant B.

ABTestIAM

Step 4: To create additional variants:

  • Click the add icon next to the current variant. A copy of the current message is created and the banner displays the percentage of the target audience that will see that message. The target audience is divided evenly, based on the number of variants you create. For example, if you create four variants, 25% of the target audience will see each variant.
  • Click the edit icon or variant name to enter a specific name for the variant. The name displays on the tab above as you type.
  • Change the content of the message variant as required (for example, different background, button text or button position).
  • After you have created your message variants, click Next: Set Targeting to proceed.

Step 5: On the Set Target screen, specify the target audience, display triggers and campaign goals. The target audience is split evenly among the number of your variants, however each variant includes the same display trigger and campaign goals for all users. Click Save as Draft: Preview & Test to proceed.

Step 6: On the Preview & Test screen, perform the following actions:

  • If required, QA test your in-app message before launching it. We recommend you check each A/B test variant on at least one QA device. For more information, see QA Testing In-App Messages.
  • Click each variant tab to view a preview of the in-app message. To launch the message:
    • If you want to launch the message immediately, click Launch as soon as possible.
    • If you want to define the message activation period, enter a start and end date/time in the Schedule launch section and then click Launch on <scheduled date>.

If you launch the message (without scheduling), its status is set to active. If you schedule the message to be deliverable from a particular date/time, its status is set to scheduled until this time is reached.

Your message now appears on the In-App Messages screen.

You can create a copy of your message by clicking the Duplicate icon on the message overview on the In-App Messages screen. In this instance, a copy of your message and all the variants is created in draft status and displays on the In-App Messages screen. You can edit and launch the copy according to your needs.


Viewing A/B Test Results

To view the individual in-app message screen for a particular message, click the message on the In-App Messages screen. For in-app messages in any status other than draft or scheduled, the individual message screen displays a detailed report of your in-app message campaign results. For more information on the individual in-app message screen, see Managing In-App Messages.

At present, you cannot add or delete variants after a message is active. If you want to add or remove variants based on their performance, you need to end the message, create a duplicate draft copy of the message and then edit or delete the variants as required. For more information, see Managing In-App Messages.

Next Steps