Skip to main content

​
Overview

An experiment sends Version A and Version B of a campaign to small test groups and waits for results. Then it sends the winning version to the rest of the audience. You set it up on the Experiment step of the campaign wizard (Marketing → Campaigns → Text & Email Campaigns → New Campaign, /campaigns/new). Follow it under Experiment results on the campaign page (/campaigns/:id).
In short: 1) Turn on the test and write Version B. 2) Choose how to split your audience. 3) Pick how the winner is decided.
Time to complete: ~10 minutes  ·  Difficulty: Intermediate  ·  Available on Send Now and Schedule for Later campaigns — not on recurring ones.

​
Before you start

  • Finish the Message step first: Version A is whatever you wrote there.
  • To pick a winner by Click rate, a text campaign needs a link with Track this link on, and an email needs a link in its body.
  • Sending the winner (automatically or by hand) requires permission to send campaigns. If you don’t have it, ask an Owner or Admin. See Users & roles.

​
Set up an experiment

Turn it on

On the Experiment step, switch on Test two versions of this campaign.

Choose what to test

Under What do you want to test? pick Subject line (email only) or Message content. A Designed email can only test the subject line — switch to a Plain email to test the body.

Write Version B

Version A is shown for reference. Type Version B’s subject line (up to 500 characters) or Version B’s message. Text messages show the character and SMS segment count. Merge tags (placeholders like {{first_name}}) work as usual.

Or generate ideas

Click Generate ideas, optionally pick a tone — Professional, Friendly, Short, Urgent, or Sales — and click Generate. You get up to five ideas, each with a one-line reason; click one to use it as Version B. Text ideas longer than your original are flagged “Uses N texts”. Generating ideas uses AI credits from your plan.

Split the audience

Under How should we split your audience? choose one. Smart test (Recommended) — 10% get A, 10% get B, and the winner goes to the remaining 80%. Even split — 50/50, nothing held back. Custom — your own Version A %, Version B %, and Winner % adding up to 100.

Decide how to pick the winner

Under How should we pick the winner? choose Open rate (email only), Click rate, or I’ll pick the winner myself. Set How long should the test run? to 30 minutes, 1 hour, 3 hours, 6 hours, or 24 hours (the default). Turn on Ask me before sending the winner to approve it yourself.
A hint below shows “~N contacts per version”. Under 200 per version reads “Small test group — results may be directional.” The summary panel lists the experiment (“Testing subject line, smart 10/10/80 split”) before you click Send Campaign.
For email, Open rate can be inflated by Apple Mail’s privacy protection — Click rate is usually the more reliable signal.

​
How the test runs

Each contact is placed in the Version A, Version B, or winner group by a fixed rule based on their record. The groups never overlap, and a contact can’t move between them if a send is retried. The two test groups are sent as separate runs; the winner group waits. Once both test sends finish, the test duration you chose starts. A check runs every minute, and when the timer is up the experiment is decided:
  • Open rate or Click rate compares unique opens (or clicks) divided by delivered messages for each version.
  • The result is High confidence when each version delivered at least 30 messages and there were at least 5 opens or clicks in total. The difference must also be big enough that there’s only a 5% chance it’s down to luck (statistically significant at 95%). Otherwise the leader still wins, marked Small audience — directional result. A tie goes to Version A.
  • With I’ll pick the winner myself, Ask me before sending the winner, a failed test send, or nothing delivered, the experiment moves to Waiting for your decision. You get an “Experiment needs your decision” notification.

​
Send the winner

When the winner is decided automatically, the winning version is sent to the remaining audience right away and you’re notified with “Winner sent”. If the experiment is waiting on you, open the campaign, click Send this version on the card you prefer, and confirm. Click Don’t send to remaining audience to end the experiment without sending more messages.
The campaign must still be Running. Pausing or cancelling it cancels the experiment, and no winner is sent.

​
Read the results

The Experiment results panel on the campaign’s Overview tab shows a timeline — Testing, Deciding (“We’ll pick a winner around …”), then Winner sent, Waiting for your decision, or Cancelled. While a winner group is pending it also shows “Winner goes to ~N more people”. Each version card lists Delivered, Open rate, Click rate, and Unsubscribes. While the test runs, the leading card says “Version B is ahead so far”. Afterwards the winner gets a Winner badge and a tag: High confidence, Small audience — directional result, or Picked manually. Delivery totals for the whole campaign appear in the metric cards above.

​
FAQ

No — an experiment is always Version A against Version B.
No. Experiments are set up when the campaign is created and can’t be added or edited afterwards.
No. Even split sends both versions to the whole audience, so there is nobody left to send the winner to; the results panel simply reports which version did better.
Exactly the winning version — its subject, preview text, or message — sent from the same senders to the contacts that were held back.

​
Troubleshooting

Text campaigns need a tracked link (turn on Track this link in the message step); email campaigns need a link in Version A or Version B.
You’re using a Designed email. Switch to Plain email to test the body, or test the Subject line instead.
A test send failed or nothing was delivered, so no automatic decision was possible. Review the cards and click Send this version or Don’t send to remaining audience.
Switch the schedule to Send Now or Schedule for Later.

Campaign wizard

The other steps of creating a campaign.

Campaign analytics

Delivery and engagement across campaigns.