Rate Limiting
Technology does not trump policy when it comes to delivery
Recently Ken Magill wrote an article looking at how an ESP was attempting to sell him services based on the ESPs ‘high deliverability rates.’ I commented that Ken was right, and I still think he is.
Ken has a followup article today. In the first part he thanks Matt Blumberg from Return Path for posting a thoughtful blog post on the piece. Matt did have a very thoughtful article, pointing out that the vast majority of things affecting delivery are under the control of the list owner, not under the control of the ESP. As they are both right, I clearly agree with them. I’ve also posted about reputation and delivery regularly.
Delivery can be counter-intuitive
We all know that receiving ISPs rate limit incoming email. With the volumes of mail that they’re currently dealing with they must do that in order to keep their servers from falling over.
A client was dealing with rate limits recently. These were not typical rate limits, in that the recipient ISP was 4xxing mail. Instead, the recipient ISP was not accepting any incoming connections. The client was having a bit of a difficult time understanding what was happening and why the problem wouldn’t be solved by increasing the rate at which they were trying to send to the ISP.
Imagine if you will, that at every ISP there is a reception desk that manages the incoming calls. The receptionist is under orders from the to limit the number of calls coming in. When the phone rings, the receptionist can do any of the following:
1) answer the phone and put the call through (250 message accepted)
2) answer the phone and put the caller on hold (connection hangs or delivery is slow)
3) answer the phone and tell the caller to call back later (4xx message deferred)
4) fail to answer the phone (no connection at all)
The delay the client was seeing was #4, in that they were attempting so many connections at once that the ISP was just not answering.
In this case, reducing the number of connections attempted worked. The “receptionist” was not so overwhelmed by the number of ringing lines that she was able to actually answer all the calls and put them right through.
While lowering the rate at which the client was attempting to delivery seems counter intuitive to getting improved delivery, because we understood the mechanism we could lower rate and get an increase in delivery.
Soft bounces and rate limiting
What is your policy for handling soft bounces? What do you consider a soft bounce? What is the right thing to do about soft bounces?
The first step in talking about soft bounces is to define them. When I talk about soft bounces, I mean mail that has been rejected with a 4xx response during the SMTP transaction. As described by RFC5321, when a recipient MTA responds with a 4xx it is telling the sending MTA “Wait! I can’t take this mail right now. Come back a little later and try again.” The sending MTA will then continue to attempt to deliver the message until either it is delivered or until it hits the max delivery time, usually 3 – 5 days.
In a well behaved and RFC compliant MTA, messages that have reached the maximum time without delivery due to 4xx rejections will be converted to permanent rejections (5xx). With a correct MTA, this means too many emails in a row timing out shoud result in an email address being removed from future mailings.
For a number of reasons some ISPs, notably Yahoo, are using 4xx responses to slow down mail from some senders. Many senders treat this as a inconvenience and a frustration and try to figure out how to get around the rate limiting. The UK DMA published an article on soft bounces with the following words of wisdom.