Bounces and bounce handling is one of those topics I’ve avoided writing about for a long time. Part of my avoidance is because there are decades of confusing terminology that hasn’t ever been really defined. Untangling that terminology is the first step to being able to talk sensibly about what to do. Instead of writing a giant long post, I can break it into smaller, more focused posts.
What is a bounce?
A bounce is when an email is undelivered. In the deliverability field, we use the term bounce to describe mail that was not delivered during the SMTP transaction and mail that was later returned as undeliverable.
Why does mail bounce?
There are a large number of reasons mail is undeliverable. One of the common reasons mail is undeliverable is because the recipient address doesn’t exist. Mail can bounce for other reasons, too. Some of those reasons are technical: the mail is malformed, or the receiving server is having a problem. Other reasons are filter related, like the sending IP has a poor reputation or there is a block against content in the mail.
What should we do when mail bounces?
That depends on the specifics of the bounce. In the SMTP protocol there are 3 answers a receiving mail server can given when presented with an email:
- I accept this message,
- I will never accept this message
- I don’t accept this message, but ask me again later.
MTAs handle the individual emails just fine. If they get a “never accept” (5xy) response, they immediately stop trying to deliver the message. If they get a “tray again later” (4xy) response, they will attempt to redeliver for a certain period of time.
Where’s the confusion?
The confusion is that we have a situation where we need to make decisions about trying the next email based on extremely limited data. There’s also a very diverse usage of SMTP response codes among MTAs. The same 3 digit codes can mean different things and the same messages are assigned different 3 digit codes. All in all, a situation that is not easily handled mechanically.
You haven’t mentioned soft or hard bounces?
I have, sort of. 5xy (no, never) responses are sometimes called hard bounces while 4xy (try again later) responses are sometimes called soft bounces. This isn’t what bulk senders mean when they use the terminology. Generally, hard bounces are used to mean emails that will never deliver and that usually means the address does not exist but can be something technical (DMARC fail, technical problems, etc) that needs to be fixed on the senders end before attempting to mail again. Soft bounces mean emails that might deliver in the future, so mostly every other bounce type. In these cases soft and hard terminology does not map onto the SMTP 4xy and 5xy response codes.
What is a soft bounce?
A soft bounce typically describes a case where a message was not accepted by the receiving server, but there’s no evidence the next message won’t deliver.
What is a hard bounce?
A hard bounce typically describes an email address not existing. No future message will deliver because the address does not exist.
What’s the problem? Just suppress hard and send to soft.
It’s not that simple. There are cases where something breaks and ISPs response “user unknown” to every email, even those that are valid addresses. In these cases it’s worth attempting to send a few times before suppressing the address from future mails.
There are also cases where a soft bounce is actually permanent. Take the “mailbox full” case. In the environment of multi gigabyte mailboxes and spam folders that don’t count towards a mail quota, it’s very rare for mailboxes to actually fill up. It can happen, but in most cases of mailbox full the mailbox has been totally abandoned. Maybe the recipient lost the password, maybe they passed away, whatever it is, the mail is never going to deliver.
Bounce handling recommendations
For user unknown the sooner you stop sending the better. In the early 2000s there was a handshake agreement between a group of ESP representatives and ISP representatives that address that bounce will be removed after 3 consecutive bounces in a period of more than 15 days. In the years since then, this has been a standard recommendation. Many ESPs have their own bounce handling rules, and they’re all within the spirit of the agreement.
The short version of all of this is that bounces are a fact of life, and that if mail repeatedly fails to deliver to an address then it’s time to suppress the address from future mailings.
RSS - Posts
This is a very helpful article. Thank you Laura!
I think the most confusing thing for my clients is the different types of 4xx bounces. Some mean try again in a few minutes like Yahoo’s typical:
“451 mta4311.mail.gq1.yahoo.com Resources temporarily unavailable. Please try again later [#4.16.1].”
These are potentially retried many times in a single send.
Some should not be retried like this from Icloud:
450 4.2.2 : user is overquota
That message should be removed from the queue.
Moreover, some of those should be counted with the 3 strike rule that many senders use to determine a mailbox is invalid, while some should not.
So when a client asks about whether we “retry”, as if it’s some fancy feature, the only good answer is “sometimes”.
The “handshake agreement” was between some US ESP and some US ISPs, in the early 2000s indeed. It’s a very long time ago, MANY things evolved since, on both sides, and considering a hard bounce really is hard only after a few tries is a behavior from the past. I’m confident not many ISPs would still agree to this today.
In the past 5 years, I’ve seen twice a case where an ISP (the same) provided “hard” replies for valid email addresses. So we had to manually un-hard the addresses sent to this ISP during the issue.
You should apply the “X bounces over a minimum of Y days” only to soft bounces, and remove hard ones immediately.
@Laura: you only mention replies from the receiving server, but you don’t mention addresses that don’t exist (at the moment of the submission) because the domain doesn’t exist, the DNS is down, there’s no MX (or A but honestly …) record, the MX server is down, i.e: there’s no reply from the expected server. To me, “hard” is when the receiving server said “this particular email address does not exist”, and “soft” is all other reasons.
At multi-gigabyte providers (e.g., Google) the mailbox is just part of the 15GB or whatever of storage allowed for the account. I believe this means that “mailbox full” doesn’t so much mean that the mail spool itself is full, necessarily, but rather that the account is using all the storage space allotted to it.
See also http://support.google.com/mail/answer/6374270
Good point, it may not all be mail. But the “mailbox full” still means no mail will be accepted until the user goes under quota.
In fact, that’s exactly how old school mailboxes worked. My very first ISP account offered 10MB of data – and that included email and any content I published on my personal website. I would purge mail from my account monthly to stay under quota, particularly after I started actually putting images on my website. I think my website used 8MB and mail got 2 or so.
you only mention replies from the receiving server, but you don’t mention addresses that don’t exist (at the moment of the submission) because the domain doesn’t exist, the DNS is down, there’s no MX (or A but honestly …) record, the MX server is down, i.e: there’s no reply from the expected server.
That is correct, I didn’t mention it. As I said, I made a deliberate choice to focus on certain aspects of bounce handling. In this post it was about the SMTP transaction itself. Some of the modern bulk mail MTAs will create a fake SMTP response so that these addresses can be processed with regular bounce handling. However, none of these are failures occur during the SMTP transaction and they don’t involve an actual response from the receiving mailserver.
There are multiple other posts to be written about this issue, delivery failures outside the SMTP transaction is one of them.
And thank you for those posts to come!
Sometime “when it’s appropriate” is the only rational response. It’s not always a good one.
A few years ago I had a client using multiple ESPs wanting their in house database to be the database of record. My task was to look at how each individual ESP was bounce handling and come up with a sane and sensible ruleset they could use to apply to all the ESPs. Getting the rules was impossible (although I’d consulted on bounce handling for one of the ESPs more than a decade ago so I had a really old copy of their detailed bounce handling rules from then). We did our best, but each ESP does things so individually that it was tough. I did spend about 3 days working on the venn diagram of bounces that I should clean up and share sometime.
I remember 552/5.2.2 being the error code I usually saw with ‘mailbox full’/’quota exceeded’-type messages. Has there been a big move to 4.2.2? I know that 5.2.2 is treated as a temporary error by at least some mailing list software.
RFC 3463 seems to indicate that both 4.2.2 and 5.2.2 are acceptable, and doesn’t seem to recommend one over the other.