How to share test subject lines like a newsletter with 2.5 million subscribers
From Kelly Forst April 20, 2021
Morning Brew hit 1,000,000 openings using this split test strategy. Also, learn how to do this in AWeber.
Stuck between two subject line ideas?
It’s impossible to know which will give the best results without doing a split test – commonly referred to as an A / B test. A split test gives you real feedback on the subject line that attracted your audience more to open.
This is how the most successful marketers optimize their open rates.
Take Morning Brew, for example. Morning Brew is a popular newsletter with more than 2.5 million subscribers. Their daily newsletter features business and technical news of the day written for an enterprising millennial audience.
In August 2020, Morning Brew reached a major milestone – 1,000,000 openings of its daily newsletter.
But how did the newsletter achieve this success?
Morning Brew’s Secret to 1,000,000 Emails Opens
Morning Brew author Roby Howell spilled the beans on Twitter over how the popular newsletter tests the subject lines for every mail it sends.
This is how it works:
- The Morning Brew team tests 4 subject lines every morning. These subject lines are based on catchy, interesting sentences from the news of the day.
- Each of the 4 subject lines are sent to 3% of the audience at 5 a.m.
- The winning subject line will be sent to the remaining 88% of their subscribers an hour later.
This strategy allows you to quickly test your subject lines against a small segment of your audience and then send the “winner” to the remaining audience.
It’s a smart way to get more openings for every email.
Would you like to follow in Morning Brew’s footsteps to set up your own split tests like this one? Everything you need to know is here.
Psst … If you want to keep your split testing simple, start with a simple 50/50 split test. This will help you learn the ropes and try split tests before jumping into more advanced testing strategies like this one.
connected: The Ultimate Guide to AB Testing
How to Share Test Subject Lines Like Morning Brew
Testing subject lines like Morning Brew is pretty easy. However, there are a few steps that need to be taken outside of your email service provider to ensure that your results are statistically significant.
That means you need to make sure you have a large enough audience for the first part of your show. If your audience is too small for Groups A and B, the results will not be significant. If the percentage is too high, you could be missing out on the opportunity to send the winning subject line to more subscribers.
How to set the breakout percentage
To find out how many subscribers to get Version A and Version B of your newsletter, you need to look at your historical open rates and the size of your email list.
For this example, let’s say your email list has 10,000 subscribers and you’re seeing 20% open rates historically.
Use the Optimizely Calculator to change the “Baseline Conversion Rate” to 20% to take into account the historical open rate and calculate the ideal audience size for A&B.
If groups A and B must each have 1,100 subscribers, it means that each group is 11% of your total audience.
So in this case we want to do an 11% / 11% / 78% split test.
connected: Your guide to writing the world’s best email subject lines
Determination of a subject line winner
Once versions A and B have been sent, the next step is to determine the winner. The winner will then be sent to the remaining 78% of your audience.
To evaluate what would go to the remaining group, look at the results of each email and score them on a split test calculator.
For example, let’s say the first results are:
Group A: 1,100 emails sent, 278 emails opened, 21 bounces
Group B: 1,100 emails sent, 245 emails opened, 33 bounces
If you look at the numbers, it’s obvious that Version A gives better results than Version B. However, you should verify that there is enough data to determine whether the results are statistically significant (and not random).
The Neil Patel AB test calculator allows you to record and plug in the results. Make sure you subtract the number of bounces from your total emails sent. With roving bounces you can measure the actual opening rate. When an email bounces there is no chance of opening.
Ideally, you want to aim for statistical significance above 90%. But sometimes you have to move fast. In this case you should aim for statistical significance of 80% or more.
Now you can be sure that you are sending Version A to the remaining 78% of your audience
This way you can maximize the number of openings. In this example, we would get 234 more openings if the winning email went to the remaining 78% of the subscriber list.
We came to this conclusion by applying the open rate (or the conversion rate in Neil Patel’s calculator above) to the remaining audience.
7800 * .26 = 2028
7800 * .23 = 1794
Those 234 openings could mean the difference between additional sales.
How to share test subject lines like Morning Brew with AWeber
Want to set up a subject line split test like Morning Brew for your own email marketing? Follow the steps below and get better results from your email marketing
Note: Split tests in AWeber are available to AWeber Pro users. Update your account to unlock this feature or contact our Customer Solutions team.
Step 1: compose your emails
The first step in split testing is to make sure that two versions of your email have already been created in the message editor.
Since this is a split test for subject lines, all other variables in your email should be exactly the same.
Connected: Email split tests you can set up in 1 minute
Step 2: set up your split test
To create a split test, hover over the Messages tab and click Split Tests. Then click the green “Create” button.
Next, name your split test. It is helpful to include a descriptive language in the name to distinguish one split test from another. This is also a quick way to get the takeaway from each split test so you can apply the findings to other emails.
Then select a segment of your recipients (if you don’t want to send the emails to your entire list).
Finally, define your split segments using the formula used above. And click “Save”.
Now it’s time to choose your designs. Make sure your designs match what you have defined in your subject line name.
You can then schedule your broadcast split test message for segment A and segment B.
Once you’ve sent your emails, you can set a specific waiting time to use the steps above to determine a “winner”.
You can then send this winning email to segment C.
Click here for more step-by-step instructions on how to test subject lines in AWeber.