Bounce handling is hard

Sometimes I find it hard to find a new topic to write about. I decide I’m going to write about X and then realize I did, often more than once. Other times I think I can blog about some issue only to realize that it’s too complex to handle in a quick post. There are concepts or issues that need background or I have to work a little harder to explain them.
One thing I haven’t blogged about before is bounce handling. That particular topic falls into the other category of posts that take a lot of time to write and need a significant amount of work to make sense. I was even joking with my fellow panel members at EEC a few months ago about how that’s a post that so needs to be written but I’m avoiding it because it’s so hard. There’s so much to be conceptualized and explained and I realize it’s not a blog post but multiple blog posts, or a white paper or even a book.

So let’s start with some simple definitions.  Those of you who work at ISPs are probably thinking of bounces in terms of accept than reject, that’s not exactly what I’m talking about here. I’m writing these for senders, who usually call rejects during the SMTP transaction bounces.

What is bounce handling?

In the bulk mail space, bounce handling describes the process of what to do with future email to addresses that have not accepted one or more emails.
Most ESPs and senders segregate bounces into two categories: hard and soft. I’ll be honest, I’ve never been able to get a good definition for what a hard bounce vs. a soft bounce is in this context. I think every ESP defines them a little differently. So what I’m going to do here is describe them as I understand most people to use them. Anyone with different ideas or definitions, feel free to address it in comments.

What is a hard bounce?

A hard bounce is an email address that is not accepting email and is unlikely to accept email in the future. The most common example is address not found or unknown user responses from the ISP.

What is a soft bounce?

A soft bounce that is an email address that did not accept an email but is likely to accept email in the future. These can be things like spam blocks or temp failures or rate limiting by the receiving mail server.

That sounds so simple.

Well, yeah, it does. But at what point do you make the determination that an address is good or bad? We’re trying to make decisions about what to do in the future based on SMTP response codes. SMTP response codes do not address what to do with future mail to an email address. They just tell us what to do with that message.
In order to deal with bulk email, senders take the SMTP response codes and the text of the response and try to infer what to do with future emails. Each ESP and bulk SMTP server handles the interpretation differently. Incidentally the interpretation of SMTP codes for future mails is a not a feature found in most of the open source SMTP software. That software implements the RFCs. Anyone using these packages for bulk mail needs to build their own bounce handling. That’s part of why open source servers are a bad match for bulk mail.

What are SMTP response codes?

SMTP response codes are the ways mail servers communicate while sending mail. A sending server basically issues a command (HELO, EHLO, MAIL FROM, DATA) and the receiving server responds to those commands with 3 digit codes. The first number in the code (with one exception) a 2, 4 or 5.
Any response that starts with a 2 means: Yup! We’re good!  Receiving servers respond with 2 codes throughout the SMTP transaction. After the sending server completes the send and says “that’s the whole email” the final 2xx means that the message was received and it’s no longer the sending server’s responsibility. Usually these are 250 responses, but there are others.
Any response that starts with a 5 means: Woah! Stop right there! This response can be due to a number of things from the address not existing to a protocol error to a spam block. Errors starting with a 5 can also happen any time during the SMTP transaction. When receiving a 5xx the sending server should stop the transaction and not try to send that message again.
Any response that starts with a 4 means: Stop, briefly, let me catch my breath! These are temporary errors. When the sending server gets a 4xx code, it should stop the transaction, queue the mail and attempt it in the future.

A hard bounce starts with 5 and a soft starts with 4, right?

Sorta. In the SMTP world a 5xx is a hard bounce and means that message will never be delivered. A 4xx is a soft bounce and means the message can be queued and reattempted at a later date. In the bulk mail world a hard bounce means that no further mail should ever be sent to that address from that sender. A soft bounce means that future emails can be sent to that address. Because we’re trying to map apples onto oranges, there are some grotty corners.

You lost me.

I think I lost me. (This is usually the point where I start pacing around the office and deciding this is not a blog post I want to write. I hit that about half way through the 2xx description…)
Here’s the thing, there is no published or standard for how a receiver should alert a sender as to what to do with future emails to an email address that’s currently undeliverable. All the RFCs talk about is what to do with the current message. We’ve tried to interpret those messages to make sane decisions about how to send mail. But there is no right way to do it.

Let’s add new codes!

Yeah, no. That’s not going to work. First, some folks have proposed some changes in the past, and that’s never gotten anywhere through the IETF. Second, it’s complicated. I can come up with half a dozen reasons for why this is a challenge. Some of them start with agreeing on the problem space. Others involve having to update SMTP services across the internet. It’s hard and it’s complicated and email is such an entrenched protocol making substantive changes to the SMTP transaction is really a non-starter.

So… where now?

ESPs do their best at classifying response codes and phrasing to make good decisions for what to do with future email. But there is no real right way to do it. Everyone processes bounces a little differently. Sometimes addresses that have bounced off a list will still be deliverable. It happens. There are any number of “right” things to do with the address, depending on why it initially bounced off the list.
There is no one way to do things, but the better informed you are about how your ESP handles bounces the better you can deal with the issues.