Last Friday, after publishing the post about Google's quality raters handbook (which I'd found through a post on "Pot Pie Girl's" blog), I spent a little time looking over her older posts and found an interesting one showing the results of an email subject line test she ran back in March.

I thought I'd share the results with a little speculation about why the results may have come out as they did.

The email was about a webinar which was going to start about an hour from the time they were sent. In the prior post, which introduced the test, she mentioned a few points worth noting here:

  • These lists WERE previously told about the upcoming webinar and many had already registered
  • ...
  • I picked 8 different lists and used one unique subject line on each with the same content inside the email body

The first bullet, I include just FYI to establish the context of the test. The other bullet indicates that the test wasn't perfectly randomized, since the results may have been influenced by differences between the people on the different lists. If each of the different headlines had gone to a mix of people from each of the lists, the results would have been more conclusive.

That said, here are the email subject lines and the open and click-through rates that each produced:

Subject Line Open Rate Click Rate
[Your Registration] Webinar starting soon 17.1% 5.9%
Don't let your spot go to someone else (1 hour notice) 11.5% 4.3%
You're invited to a webinar (we're starting soon!) 8.7% 3.5%
We're about to be LIVE - are you coming? 8.2% 2.9%
- you in? 8.3% 2.6%
Your Personal Invitation (1 hour notice) 10.5% 2.5%
Hurry! We're starting soon... 6.3% 1.8%
1 Hour Notice Is All I Can Give 5.6% 1.7%

Let's take a look at each of the subject lines and why it might have performed as it did.

"[Your Registration] Webinar starting soon"

This subject line was the clear winner. It starts off by suggesting that the email about something the recipient already took action on and registered for -- not something they needed to weigh and make a new decision about. This may have been less effective if many of the recipients hadn't already registered, and even less if it were about something they hadn't even heard about. Under these circumstances, it worked well.

"starting soon" isn't specific about how long till the webinar started. But it really doesn't have to be. Nobody says something you attend online is "starting soon" if it's in five hours, tomorrow, or next week.

"Don't let your spot go to someone else (1 hour notice)"

This subject line does a great job of working with scarcity. By saying "your spot", it suggests that the recipient already has a spot, but that they're at risk of losing it if they don't act quickly.

In his book, Predictably Irrational (read my review here), Dan Ariely suggests that people value things more highly once they own them than they do before owing them. Although in my review, I disagree with some of Dan's interpretation of the research he cites, I do think the point has validity. People avoid loss. If something was worth having in the first place, it must be worth hanging on to.

In this subject line, which isn't specific about what it's about, being specific about "1 hour notice" was likely effective in increasing open rates.

"You're invited to a webinar (we're starting soon!)"

This subject line's strengths are that it's specific about what it's about (a webinar) and clear about the fact that it's starting "soon" -- not talking about one that's being held next Tuesday. As with the winning subject line, under these circumstances, it's not necessary to be more specific about how soon.

"We're about to be LIVE - are you coming?"

This subject line performed slightly less well that the previous one, perhaps reflecting it's lower specificity. Something live was about to begin, but it doesn't say exactly what. However, most recipients could probably guess that it was something online, like a webinar or teleseminar.

When speaking of "going live", "about to be" is probably specific enough about the timeframe.

"- you in?"

This subject line got a slightly higher open rate than the previous one, but a lower click-through rate. The question probably aroused some curiosity regarding what it was about, but a relatively low percentage of those who opened the email decided that what they found in it was worth clicking through to.

The problem with a vague subject line like this is that even if it achieves a high open rate, it's almost always going to get less clicks per open than a subject line that's more specific, and perhaps even a lower overall click-through rate. Why? Let's make up some hypothetical numbers.

Let's assume that you're emailing 100 people. 30 of them (30%) would be interested in what the email is about. Let's say that if your subject line is clear about what the email is about, 20 of them would open it, all of whom are among those who'd be interested, and 10 of them would click through (so half of the people who would be interested AND opened the email clicked through).

On the other hand, if your subject line is non-specific but arouses curiosity, perhaps 50 recipients would open it. Since it's going to be a random mixture of those interested in your offer and those who aren't, you're going to get your email opened about about 15 people who are interested (30% of 50). And if about half of those click through, you'll get about 7 or 8 clicks -- less than you would with a more specific subject line.

Even if some of the disinterested recipients click through, they're not going to convert once they get to your sales page, so pumping up click-throughs by getting some from people who won't convert only skews your numbers.

"Your Personal Invitation (1 hour notice)"

This may be the most interesting result -- high open rate, but low click-through. The subject line claims that the email is personal, which prompts people to open it. But once they do, they see that it's really a bulk message, not "personal" in the way they thought, or at least hoped it was.

I personally dislike subject lines like this. They feel deceptive, even if they weren't intended that way.

Judging from the low click-through rate, it's possible that as soon as those who opened the message discovered that it wasn't really "personal", they stopped reading and closed it immediately. This subject line may be effective in getting people to open the message, but it may be causing a backlash that significantly reduces how many actually read the message. And it certainly didn't produce a level of response in line with its open rate.

"Hurry! We're starting soon..."

Two quick comments for this one: it's too non-specific (it doesn't even imply that what's starting soon is something online the way "going live" did in a previous subject line), and no one likes to be rushed.

"1 Hour Notice Is All I Can Give"

I'm not surprised that this subject line came in last. First, it focuses too much on the sender. Second, it implies that the sender doesn't want to give. And third, it could be interpreted as "all I can spare for you is one hour, because you're not important enough to give more to."

Thanks to Pot Pie Girl for sharing these test results. I hope you learned something from them that you can use to increase response rates for your emails.