Snowshoe
Google and Amazon and B2B spam
Many of the operational goals of a commercial spammer aren’t related to email delivery at all, rather they revolve around optimizing ROI and minimizing costs. That’s even more true when the spammer isn’t trying to sell their own product, rather they’re making money by sending spam for other companies.
Most legitimate network providers pay at least lip service to not allowing abusive behaviour such as spam from their networks, so a spammer has to make a few choices about what infrastructure to use to optimize their costs.
They can be open about who they are and what they do, and host with a reputable network provider, and build out mailservers much as any legitimate ESP would do. But eventually they’ll get blacklisted by one of the more reputable reputation providers – leading to little of their mail being delivered, and increasing the pressure on their provider to terminate them. They social engineer their provider’s abuse desk, and drag their feet, and make small changes, but eventually they’ll need to move to another provider. Both the delaying tactics and the finally moving are expensive.
Or they can host with a network provider who doesn’t care about abuse from their network, and do the same thing. But they’ll still get blacklisted and, unlike on a more reputable network, they’re much less likely to get any benefit of the doubt from any reputation providers.
Every time they get blacklisted they can move to a new network provider. That’s easy to do if your infrastructure is virtual machine based and moving providers just involves buying a new hosting account. But as anyone who’s heard the phrase “ramping-up” knows mail from new network space is treated with suspicion, and as they’re continually moving their mail won’t reach the inbox much.
Preemptively spreading the sources of your spam across many different IP addresses on different providers, and sending spam out at low enough levels from each address that you’re less likely to be noticed is another approach. This is snowshoe spam and spam filters are getting better at detecting it.
What to do? In order to get mail delivered to the inbox the spammer needs to be sending from somewhere with a good reputation, ideally intermingled with lots of legitimate email, so that the false-positive induced pain of blocking the mailstream would be worse than their spam. That’s one reason a lot of spammers send through legitimate ESPs. They’re still having to jump from provider to provider as they’re terminated, but now they’re relying on the delivery reputation of the shared IP pools at each new ESP they jump to. But that still takes work to move between ESPs. And ESP policy enforcement people talk to each other…
As a spammer you want your mail to be sent from somewhere with good reputation, somewhere you can use many different accounts, so your spam is spread across many of them, flying below the radar. Ideally you wouldn’t have any documented connection to those accounts, so your activity won’t show up on any aggregated monitoring or reporting.
If nothing in the mail sent out identifies you there is nowhere for recipients to focus their ire. And if recipients can’t tell that the hundreds of pieces of spam in their inbox came from a single spammer, they’re much less likely to focus efforts on blocking that mail stream.
Over the past couple of years I’ve seen a new approach from dedicated B2B spammers, the sort who sell “buy and upload a list, blast out something advertising your company, track responses, send multiple mails over a series of weeks” services to salespeople. They’re the ones who tend to have glossy, legitimate websites, talking about “lead nurturing”, “automated drip campaigns” or “outreach automation”.
They have each of their customers sign up for gmail or google apps accounts, or use their existing google apps accounts, and then the spammer funnels the spam sent on behalf of that customer through that google account. There’s no obvious connection between the spammer and the google account so there’s no risk to the spammer. Google is fairly unresponsive to spam complaints, so as long as the volume sent by each customer isn’t spectacularly high it’s going to be well below Google automation’s threshold of notice.
Google do record where mail that’s injected into their infrastructure in this way comes from, in the Received headers. But the spammers run their sending infrastructure – list management, message composition, tracking and so on – on anonymous, throwaway virtual machines hosted on Amazon’s EC2 cloud, so there’s nothing in the email that leads back to the spammer.
And, for recipients, that’s a problem. Spam filters aren’t going to block this sort of mail, as they can’t easily tell it is this sort of mail. It’s coming from Google MTAs, just like a lot of legitimate mail does. In terms of sheer volume it’s dwarfed by botnet sourced mail or dubious B2B manufacturing spam out of Shenzhen. But, unlike most of that, it’s in your inbox, in front of your eyeballs and costing you time and focus. And that’s much more expensive than network infrastructure or mailbox storage space.
I’m not sure what, if anything, Google or Amazon can do about it at scale, but it’s something that’s going to need to be dealt with eventually.
Meanwhile, if you receive some marginally personalized mail from a sales rep, one attempting to look like 1:1 mail, look at the headers. If you see something like this …
Bad SPF can hurt your reputation
Can a bad SPF record ruin your delivery, even though all your mail still passes SPF?
Yes, it can.
One of our clients had issues with poor delivery rates to the inbox at gmail and came to us with the theory that it was due to other people using their domain to send spam to gmail. This theory was based on ReturnPath instrumentation showing mail “from” their domain coming from other IP addresses, and a plausible looking correlation between that mail being sent and their problems at gmail.
Checking their bounce handler, we see a lot of bounces coming in suggesting that someone is sending poor quality mail using their bounce domain from quite a few IP addresses, including a suspicious number scattered in small blocks across 69.64.0.0/8.
Their question they had was whether they should publish DMARC records to fix the problem, and whether they should use a DMARC policy of p=reject or p=none. They’re a good candidate for DMARC – their domains are used purely for bulk or transactional mail, they have a tightly controlled mail infrastructure for their marketing domains, and they’re already publishing SPF records and signing all the mail they send with DKIM.
I was half way through writing up my normal answer about DMARC deployment for customers with this sort of mail infrastructure – “It won’t help with delivery problems directly, but publishing with p=none and analyzing the reports you get back will give you insight into your mail flows, and provide the data you need to decide whether using DMARC p=reject is appropriate for your business model and mail flows.” – when I realized that something just didn’t make sense.
Gmail, perhaps more than most other mailbox providers, base their delivery decisions on data they gather mechanically from all their mailboxes. And they really understand domain-based reputation and the difference between authenticated and non-authenticated email. Why on earth would non-authenticated email from an unrelated IP address be damaging the domain reputation, and hence the delivery of authenticated legitimate email? That makes no sense.
Meanwhile, over in our slack channel, Josh was double-checking their infrastructure…
Oops. They have a small block of 8 IP addresses from which they send most of their email. When setting up their SPF records they inadvertently used ip4:69.20.20.48/8 instead of ip4:69.20.20.48/29 for that block of addresses. A /8 isn’t eight IP addresses – it’s every one of the 16,777,216 IP addresses that begins with “69.”.
Suddenly everything makes sense.
The SPF thinko means that all mail claiming to be from the client domain that’s sent from any IP address beginning with “69.” passes SPF – including the deluge of spam coming from the snowshoe spammers in 69.64.*.
Gmail (and other ISPs) don’t see a difference between the legitimate email and the SPF authenticated spam – they’re just seeing a high volume of authenticated email from the client domain, a large fraction of which is spam. That’s damaged the reputation of the client domain, causing their legitimate email to end up in the spam folder.
(The reality of filtering is more than just domain reputation, of course, but a terrible domain reputation is definitely going to cause you problems.)
The immediate action to take is simple – fix the SPF record so only legitimate mail will be authenticated. That’ll take effect within a couple of hours, as the old SPF record has a short TTL, and ISPs will start seeing the correct SPF record and begin rejigging their reputation.
We’ll keep monitoring delivery rates, check how long ISPs take to notice reputation changes, potentially reach out to some ISPs to see if it’s appropriate for them to do a one-time reputation reset for the affected domains, but we’re hoping things will begin to improve in the next few days.
What can you do to avoid or mitigate this sort of problem?
What causes Spamhaus CSS listings
Today’s Wednesday Question comes from Zaib F.
What causes the Spamhaus CSS listing in your experience other than Sender using multiple sets of IPs, to look as if they are a valid sender. Do you think a Spamtrap plays a role?
Read More