A decade or so ago I was offering whitelisting services to clients. It was pretty simple. I’d collect a bunch of information and do an audit on the customer’s sending. They’d get a report back identifying any issues that would limit their chances at acceptance. Then I’d go and fill in the forms on behalf of the client. Simple enough work, and it made clients feel better knowing their mail was whitelisted at the various ISPs.
When email filters were less complex and more binary, whitelists were a great way for receivers to identify which senders were willing to stand up and be held accountable for their mail. Over time, whitelists became much less useful. Filtering technology progressed. Manual whitelisting wasn’t necessary for ISPs to sort out good mail from bad.
The era of whitelisting is over.
In fact, three of the major whitelist providing ISPs were AOL, Yahoo, and Verizon; all three are now a part of OATH. The Verizon whitelist page now redirects to postmaster.aol.com. New requests to signup for the AOL whitelist are rejected with the message that AOL whitelisting is no longer available or necessary. Yahoo has a “new IP review” form rather than a whitelisting form.
Whitelisting is dead.
Even the various certification and whitelisting services have mostly gone away. Both Habeas and Goodmail failed to achieve a profitable exit event. Of course, Return Path is still around, but they have built a platform of tools and services unrelated to whitelisting or certification.
Now senders are going to have to focus on sending mail that people ask for and want in order to make it to the inbox.
It is morning in the channel. The regular crowd is around discussing the usual.
JK, smart, competent head of deliverability at an ESP asks: Anyone familiar with SECTOOR EXITNODES listings and have insight into what’s going on if listed?
ME: Uh, that’s the Tor Exit Nodes list. They think your IP is used by Tor. That’s all sorts of weird. Let me do some digging.
5 minutes of google searches, various dig commands and a visit to the now non-existent sectoor.de website show that the sectoor.de domain expired and is now parked.
ME (back in channel): It looks like the blacklist domain expired and is now parked. So they’re listing the world and nothing to worry about. Not your problem, and not anything you can fix.
JK: Like a UCEProtect fiasco – not just us but everyone?
ME: No, more like the spamcannibal fiasco. The domain expired and so it’s listing the world.
ME: The world would be a better place without MXToolbox worrying about every stupid blocklist. Or even if they would follow the blocklist RFC check for expired domains before panicking the world.
It is a regular occurrence that senders ask filters and ISPs for mitigation. But there seems to be some confusion as to what mitigation really means. I regularly hear from senders who seem to think that once they’ve asked for mitigation that they don’t have to worry about filtering or blocking at that ISP for a while. They’re surprised when a few weeks or even days after they asked for mitigation their mail is, one again, blocked or in the bulk folder.
The words What Makes You Special on a badge, asking the question of what characteristics set you apart as an individual as different, unique, distinguished or better than the rest.
What is mitigation?
Think of mitigation as a flag that tells spam filters to ignore the history for an IP or domain. The history isn’t deleted or removed, it’s still there. But the “start date” is moved to the mitigation date. If I am a sender that’s been using an IP for a few years and I have a few bad months of sends in the middle, I can ask the ISP to mitigate the effect of those bad months on my reputation. The sender starts over fresh, with none of the bad history.
Mitigation is not a get out of jail free card.
Mitigation is not a get out of spam folder free card. This is not something offered to senders who have a poor history. It’s primarily intended for senders who are normally good senders but had some rough sends. The intention behind mitigation is to give senders a way to get out of the spam folder after they’ve fixed their problems. The infamous MS response “we see no problem with your delivery” in respect to spam foldering means exactly that according to their numbers, mail should be delivered to the spam folder.
Mitigation is not automatic.
In most cases mitigation is handled by a human being, that is following policy established by their employer. Real people review the internal data and dashboards and make a decision based on that review. Senders who have a long history of marginal mail are less likely to receive mitigation. The corollary is that senders who have a history of decent mail but a few bad sends are very likely to receive mitigation.
Mitigation requires plausibility.
Email delivery requires cooperation among senders and receivers. Mitigation requires trust on the part of the ISP, and every ISP rep has multiple stories of spammers who abused that trust. Senders who demonstrate they’re acting in good faith, by making receiver visible changes before requesting mitigation, are much more likely to receive mitigation. Repeatedly asking for mitigation decreases the chances of it being granted. Remember, mitigation doesn’t erase data, it simply resets the start time for analysis. The person handling mitigation can see that it was granted and nothing changed. It’s not a plausible request the second, or third, or fourth time.
Mitigation is an exception.
Asking for mitigation is a normalized pathway, but it’s not normal. Senders make the mistake of thinking if they got mitigation once, they just have to ask again. That mistake leads them to ask for mitigation without changing anything about their sends before or after the mitigation. As a result, they discover their mail is back in the spam folder. This also leads to mitigation not being granted a second or third time.
Don’t rely on mitigation.
No sender should rely on mitigation to get to the inbox. Instead, senders should focus on the fundamentals of good delivery: sending mail people ask for and expect. Everything else is rearranging deck chairs on the Titanic.
A bit of advice from the folks at the CBL, posted with permission and some light editing. I’ve been seeing some folks report longer connection times at some places, and this might explain some of it. It’s certainly possible, even likely, that the large ISPs are getting a lot of this kind of traffic.
A botnet, likely a variant of cutwail, has been for the past several years been specializing in using stolen credentials, doing port 25/587 SMTP AUTH connections to the spoof’d users server, and attempting to relay thru the connection to elsewhere. They will also, in some cases, attempt to log into the MX IP using a brute force attack against the email address. Other miscreants try the same thing with IMAP or POP or even SMTPS.
If they manage to compromise an email account, they use the account to send spam. For corporate accounts they can steal employee identities, request wire transfers, and send out corporately authenticated spam. If they get it, game over, the whole account is compromised and they can and do wreak havoc.
This has been going on for a couple of years, and now is the largest volume of spam from botnets. Cutwail is not the only botnet doing AUTH attacks, but appears to be the most prolific. Attacking POP and IMAP appears to be more recent, and is more related to spear-phishing (spamming executives) and other bad things.
In the last month or two, the behavior has changed a bit. The infections are trying to establish as many connections simultaneously as it can get away with. This is similar behavior to ancient or unpatched versions of qmail. This is swamping some servers by tying up a significant number (or even all) of the TCP sockets available.
The CBL is recommending that folks check their mail servers. If the mail server has a “simultaneous connection per IP limit”, it should be set to some limited number. If it’s not set then set it. Otherwise, your server is at risk for being unable to handle real mail. Make sure your IMAP and POP are secured as well as they are being targeted, too.
The XBL can also help with this. But securing your server is the first step.
The SpamCannibal blacklist – one that didn’t affect your email too much but which would panic users who found it on one of the “check all the blacklists!” websites – has gone away.
It was silently abandoned by the operator at some point in the past year and the domain registration has finally expired. It’s been picked up by domain squatters who, as usual, put a wildcard DNS record in for the domain causing it to list the entire internet.
Al has more details over at dnsbl.com.
If you run a blacklist, please don’t shut it down this way. Read up on the suggested practice in RFC 6471. If you just can’t cope with that consider asking people you know in the industry for help gracefully shutting it down.
Blacklist health checks
If you develop software that uses blacklists, include “health check” functionality. All relevant blacklists publish records that show they’re operating correctly. For IP based blacklists that means that they will always publish “127.0.0.2” as listed and “127.0.0.1” as not listed. You should regularly check those two IP addresses for each blacklist and if 127.0.0.1 is listed or 127.0.0.2 isn’t listed immediately disable use of that list (and notify whoever should know about it).
For IPv6 blacklists the always listed address is “::FFFF:7F00:2” and the never listed address is “::FFFF:7F00:1”. For domain-based blacklists the always listed hostname is “TEST” and the never listed hostname is “INVALID”. See RFC 5782 for more details. (And, obviously, check that the blacklists your software supports out of the box actually do implement this before turning it on).
If you use someone else’s blacklist code, ask them about their support for health checks. If your mail filter doesn’t use them you risk either suddenly having all your mail go missing (for naive blacklist based blocking) or having some fraction of wanted mail being delivered to your spam folder (for scoring based filters).
First thing this morning I got an email from a client that they were listed on the UCEProtect Level 3 blacklist. Mid-morning I got a message from a different client telling me the same thing. Both clients shared their bounce messages with me:
550 Conexion rechazada por estar o167890x0.outbound-mail.sendgrid.net[126.96.36.199]:56628 en la DNSBL dnsbl-3.uceprotect.net (ver Your ISP LATINET – TELPAN COMMUNICATIONS/AS11377 is UCEPROTECT-Level3 listed for hosting a total of 193 abusers. See: http://www.uceprotect.net/rblcheck.php?ipr=188.8.131.52)
(Note: the IP is not my client’s IP, it’s the start of the /17 assigned to SendGrid.)
Basically, UCEProtect listed half of SendGrid’s IP space (184.108.40.206/18). Looking at the publicly available data, it appears that in the last 48 hours, there was a lot of mail to UCEProtect’s spamtraps from part of SendGrid’s IP space. If I had to guess, I’d say this was GDPR related, particularly given that UCEProtect is run out of Europe. In fact, if we look at the listing graph from UCEProtect’s own website this is really clear.
As of 4 PM PDT they’re up to 263 IPs listed.
This is, really, no big deal. UCEProtect is not very widely used. Of my two clients, one had 5 emails bounce and one had 150, well under 0.0001% of their sends. Unfortunately, a lot of folks worry about any blacklisting, without really understanding that the vast majority of blacklists have almost no effect on mail delivery. The only way a listing can hurt is if you’re trying to send to a domain that uses a blacklist.
UCEProtect is not widely used and most folks will see little to no effect on email delivery due to this escalation. With that being said, it’s probably time to talk a little bit about UCEProtect as a list.
What they say about their list.
The UCEProtect lists are primarily spamtrap driven, although there are people who can manually add IPs. They have automated escalations, where if there is a specific number of listings over a certain period of time, surrounding space is listed. There are 3 levels.
- Level 1 is a single IP listing. These are the IPs that are sending mail to the UCEProtect spamtraps. These listings are both automated (more than 50 emails from a single IP to the spamtrap network) and manual.
- Level 2 is per allocation. They’re not completely transparent about how they determine allocation (and as I’ll talk about a little later, there is evidence some of the data they’re using is out of date). Basically, if multiple IP addresses in a range are on the list within a 7 day period, then they list more than a single IP.
- Level 3 lists every email in a particular ASN if there are more than 100 IPs and >.2% of all IPs in that ASN on Level 1. This is, in UCEProtect’s own words, a list that will cause collateral damage to innocent users.
Listings expire automatically 7 days after the mail stops. Listees can pay a fee to get delisted faster.
What’s this got to do with GDPR?
UCEProtect’s own listing graph shows a spike in listings starting mid-day Friday. (CEST is 2 hours ahead of UTC).
What happened overnight?
Because of the automated escalation scheme, over 75,000 IP addresses belonging to SendGrid were listed on the UCEProtect Level 3 list overnight. The listing encompassed all IPs announced by AS11377. UCEProtect states this ASN belongs to LATINET – TELPAN COMMUNICATIONS. The ASN was officially registered to SendGrid in June of 2012. Best we can tell, there was a list circulated around in 2007 listing current ASN assignments. I have no idea why UCEProtect is using a list more than a decade old, where they can directly query ARIN for current data through a website, FTP or whois (whois -a ‘a 11377’). Whatever the reason, it doesn’t fill me with confidence in the accuracy of the list.
Now that we’re (almost?) done with GDPR notifications, I expect these listings to age off and go away in the next week.
The good news
UCEProtect listings are unlikely to have any real impact on email delivery. These lists are just not that widely used. I also know SendGrid is aware of the issue and are working with clients who write into support.
My advice for anyone who is worried about blacklists that don’t affect email.
- Note: I chose this IP because it’s the first IP in the range assigned to the ASN and these IPs are generally never used to send mail for technical reasons.
Twitter has some opinions on #GDPR.
— @rianjohnson (Yes, the director of The Last Jedi)
Tomorrow is GDPR day. That’s the day when the new Global Data Protection Regulations take effect in the EU. I’m sure everyone reading this blog has seen dozens, if not hundreds, of blog posts, articles, webinars, and guidance docs about how to comply. I’m not going to rehash it because, other folks know this better than me.
There are a some things I’m finding fascinating watching this whole GDPR thing.
First, the number of companies who have my addresses and I don’t know why. Take Newsweek (yes, the magazine people). They’re sending GDPR notifications to my LinkedIn address. I can’t figure out why they’re harvesting / buying addresses from LinkedIn. Then there’s SALESmango who are some company that started spamming me a few years ago and refuses to accept unsubscribe request. They’re sending me opt-in requests. Yeah, no, go away. I told you to stop, but wow, you won’t.
Another interesting piece is just how much I’ve signed up for over the last 18 – 20 years I’ve been using this set of addresses. Wow. So much mail. And, generally, I thought of myself as relatively careful in who I gave email addresses to. I don’t normally go around dropping addresses into forms but even a couple a month adds up over 20 years.
Then there are the companies violating CAN SPAM in one way or another. Sending mail to unsubscribed addresses and refusing to include an opt-out link are the two things I’ve seen regularly. Yeah, no. I think it’s safe to say that if I’ve opted out from receiving your mail, you should probably put my data away in a dark closet and not touch it again. But.. but.. but… But nothing. Go away. As for the lack of an unsubscribe link, get over yourself. You’re not that special. I don’t think that this really is something that counts for exemption.
Also, is there an official template? So many of these emails look identical. I have to give credit to whomever did it first. Because if plagiarism is the sincerest form of praise, you have an entire industry praising you.
Finally, it’s been amusing to watch the general frustration with all the GDPR mail. It seems many people are getting tired of the deluge. That’s OK, though, it should end by Saturday. Or so we can only hope.
I’ve seen multiple people asking questions about what’s going to happen with the Yahoo and AOL FBLs after the transition to the new Oath infrastructure. The most current information we have says that the AOL FBL (IP based) is going away. This FBL is handled by the AOL infrastructure. As AOL users are moved to the new infrastructure any complaints based on their actions will come through the Yahoo complaint feedback loop (CFL). The Yahoo CFL is domain based. Anyone who has not signed up for the Yahoo CFL should do so.
When registering you will need each domain and the selectors you’re planning on using. Yahoo will send an email with a confirmation link that needs to be clicked on within a short period of time in order to activate the FBL.
Microsoft’s SNDS program had an outage at the end of last week. That’s been fixed, but the missing data will not be back populated into the system. This has happened a couple times in the past. It seems the system gets a live feed of data. If, for some reason, the data is interrupted, then it’s gone and doesn’t get populated.
Because that’s what the spam folder is for unsolicited email.
Sending cold email, particularly in bulk (and let’s be honest, if it weren’t sent in bulk, no one would know or care about it going to spam) is spamming. This is exactly the kind of mail that the bulk folder is designed to catch. Senders that don’t have permission have no path out of the bulk folder except trying to get some permission for their email.
Recently I’ve had an uptick in request for help getting cold emails out of the bulk folder. Some have found me through search engines and this blog. Others have been referred by someone. Whatever the reason, they come to me with a purchased list that isn’t being delivered to the inbox and they want me to help them.
The problem is I can’t help them. They are sending unsolicited email and their mail is being delivered exactly where it should be – in the bulk folder. In the past I’d try to help. I’d pull out my bag of tools and walk them through the steps to fix their delivery. But it often wouldn’t work. They weren’t looking for the kind of help I provide. They were looking for one quick trick to fool the filters into putting the mail in the inbox.
These engagements were frustrating for me, too. I know exactly why their mail isn’t going to the inbox, they’re sending spam. But no company wants to hear they’re spamming. I’d try and explain, using terms like unengaged recipients and unwanted email. I’d offer suggestions on how to create that engagement, how to find their audience, how to be better marketers. That wasn’t what they wanted, they wanted a quick fix that would let them invest pennies into purchased lists that dropped right into the inbox.
It’s not the techniques that are the problem. I regularly use the same techniques with clients who have data containing a mix of opt-in and non-opt-in data. These clients have been collecting data and email addresses through many different channels over many years without an audit trail. We can sort the list out, retain the good data and get rid of the old data.
The problem is that purchased lists are unwanted by recipients and the filters applied to their mail. That doesn’t mean opt-in lists never have delivery issues, they do. And we can fix those delivery issues because, fundamentally, the recipients want that mail. The recipients asked for that mail. There is no such assurance that recipients want mail if the list has been purchased.
Are some purchased lists opt-in? Yeah, probably. But the purchaser has no way of knowing what the address owner originally opted in to receive. All the purchaser has is the assurance of the seller. The seller who makes money even if the list isn’t opt-in.
Why does cold email go to the bulk folder? Because ‘cold email’ is just the most recent euphemism for spam. And the bulk folder is where spam is supposed to be delivered.