LogoLogo
  • Home
  • Integration & SDKs
    • Web / Browser Push
      • Wordpress Integration Steps
      • Wix Integration Steps
      • Safari
        • Safari on Desktop
        • Safari on Mobile (iOS / iPadOS)
      • SDK
        • SDK Methods
        • SDK Events
        • Customizing Prompt CSS
        • AMP Support
        • E-Commerce / Abandoned Cart
    • Native App Push
      • Apple / iOS
        • P8 Key or P12 Cert Setup
        • SDK: Swift / Obj-C
          • Activity Tracking
          • Deep Linking
          • Live Activities
          • App Messages
          • Identity Synchronization
          • SDK Methods
          • SDK Delegates
        • SDK: React Native
        • SDK: Flutter
        • Advanced
          • Self-Managed Integration
        • iOS SDK Changelog
      • Android
        • Firebase App Setup
        • SDK: Kotlin / Java
          • Deep Linking
          • Activity Tracking
          • App Messages
          • Identity Synchronization
          • SDK Methods
          • SDK Callbacks
          • Live Activities
        • SDK: React Native
        • SDK: Flutter
        • Advanced
          • Self-Managed Integration
        • Android SDK Changelog
      • React Native
        • Android: Firebase App Setup
        • iOS: P8 Key or P12 Cert Setup
        • SDK: React Native
          • Deep Linking
          • Activity Tracking
          • Subscriber Matching
          • Live Activities
          • App Messages
          • SDK Methods
      • Flutter
        • Android: Firebase App Setup
        • iOS: P8 Key or P12 Cert Setup
        • SDK: Flutter
          • Deep Linking
          • Activity Tracking
          • Subscriber Matching
          • Live Activities
          • App Messages
          • SDK Methods
  • API
    • API Access Management
    • API Reference
  • Platform
    • Dashboard
    • Notifications
      • A/B Testing
      • Custom Buttons
      • Notification Previews
      • Inline Segmentation
      • Notification Templates
      • Macros
      • Native App Push Notifications
    • App Messages
    • Segments
      • Custom Geo Segmentation
    • Campaigns
    • Insights
    • User Management
    • Organizations
      • Multi-Factor Authentication
      • Single Sign On
        • Google Workspace
        • Microsoft Azure AD
        • Okta Workforce
      • Multi-Domain Notifications
      • Multi-Domain Segments
      • API Access
    • Multi-Channel Notifications
  • Info Center
    • Launch Guide & Best Practices
      • Landing Domain Whitelist
    • Web / Browser Push
      • Common Questions
      • Browser Support
      • Retrieve Push User ID
      • Not Getting Prompted
      • Not Receiving Notifications
      • How to Unsubscribe
    • Native App Push
      • Retrieve Push User ID
    • Workflow Planning
Powered by GitBook
On this page
  • Enabling A/B Testing
  • Test Name
  • Test Type
  • Content Testing
  • Delivery Testing
  • Variant Distribution
  • Split Delivery
  • Sample Delivery
  1. Platform
  2. Notifications

A/B Testing

A/B Testing allows you to test visual and behavioral changes within a single notification against portions of your audience to determine what works best.

PreviousNotificationsNextCustom Buttons

Last updated 1 year ago

Enabling A/B Testing

To enable A/B Testing during notification creation simply click the switch in the A/B Testing well at the top of the page:

Test Name

Naming your test is important for future identification of notifications that comprise a test as well as identifying performing within the Insights section of the platform.

A test name cannot be used more than once. We suggest adding dates to test names if you want to a single test multiple times.

For example: Image Test / 2020-02-10

Test Type

There are 3 types of test to choose from depending on what elements of the notification you want to test against each other:

  1. Content: Variants can be created with separate title, body, image, and url.

  2. Delivery: Variants can be created with separate delivery times and strategies.

  3. Content & Delivery: Variants allow testing all Content and Delivery components of the notification.

Content Testing

When content testing is enabled additional options will be shown within the Content section. A new bar will appear that contains each variant that will be tested for this notification. Up to four variants can be created by pressing the plus sign (+).

You must enter a Title and Primary Landing URL before creating additional variants.

Delivery Testing

When delivery testing is enabled additional options will be shown within the Delivery section. A new bar will appear that contains each variant that will be tested for this notification. Up to four variants with different delivery types and strategies may be created.

Variant Distribution

Once you've created each variant you want to test, the Distribution section of the page provides options that allow you to determine how your test will be deployed to your audience.

Split Delivery

Split Delivery allows you to choose a percentage of your audience that each variant should receive. This distribution type allows you to test new variants on small percentages of your audience while ensuring clickthrough rates stay high with what you already know works.

Sample Delivery

Sample Delivery delivers a sampling percentage of your audience to each variant before deploying the best performing notification to the rest of your audience.

Use the sliders to determine what percentage of your audience each variant and the winner will receive. Each variant must receive at least 5% of the audience's delivery.

When Sample Distribution is chosen a Winner Strategy section appears that allows you determine when the variant with the best performance is deployed to the rest of your audience. There are two ways to configure this timing:

  1. Time Elapsed After Send: Specify the number of hours that have elapsed after the initial variants were sent.

  2. Specific Time: Schedule the winning notification to be delivered at a specific date and time in the future.

This option is not available for audiences with fewer than 5,000 subscribers.

Performance Constraint (Sample Delivery Only)

When running a Sample Delivery test, you have the option to activate a performance constraint. If none of your deployed variants in the A/B Test reach the CTR value placed in the performance constraint, then no Notification will be deployed to the remaining audience.