BLOG

Industry News & Analysis

ESP being phished is a Black Friday cataclysm

There is currently a phishing attack against a major ESP. The mail came through what I presume was a compromised account hosted at one of the providers. It’s just as possible this was a domain set up for the sole purpose of phishing, though.

Icon of an eye looking around

The underlying attack is pretty good. They took the ESP compliance notification email and changed a couple of the links to point to their phishing page (which is down now). I’m pretty sure a message “your account has been limited due to poor reputation” caused a whole lot of folks to freak out and click the links.

If it were me coordinating the attack, I’d be quietly logging into the compromised accounts over the next 10 days and creating new API keys. I’d set up my spam cannons to use those API keys and then wait for Black Friday. A single button and I can send out … millions and millions of authenticated emails through hundreds of accounts with solid reputations.

Steve and I were talking about this last night and were discussing tracking logins, 2FA and other ways the ESP could mitigate the problem and protect their users. It wasn’t until I woke up this morning that I remembered that the ESP has a full API. Yeah, that makes it even harder. Sure, the spammers need to log in and create new API keys. But individual logins that simply create API keys are harder to detect than a log in that doesn’t do anything but create a key.

This is not something the ESP can easily mitigate in 10 days. They will have had to have infrastructure in place to track creation of API keys and confirm these keys are being used by their customer. I know this ESP and I am hopeful that their security folks have thought about this attack vector.

If you are a Sendgrid customer, it may be worthwhile to revisit your infrastructure today. Identify what needs API keys and regenerate them. Then, nuke all the keys in your account. Change all your passwords. Lock down your account.

I feel for both the ESP and their customers. This was a carefully planned attack. I have zero doubt this is in preparation for sending out a massive spam campaign from the ESP at the height of the holiday email season. Don’t assume your account is safe. Make sure it is.

Otherwise, you may find more than the normal level of delivery problems for your holiday mail.

No Comments

CAN SPAM says I can!

the word spam with a checkmark next to it.

Saw a new disclaimer on mail sent to an address harvested off our website today:

disclaimer: This is an advertisement and a promotional mail-in adherence to the guidelines of CAN-SPAM act 2003. We have clearly mentioned the source id of this mail, also clearly mentioned the subject line, and they are in no way misleading in any form. We have found your email address through our own efforts on the web search and not through any other way. If you find this email unsolicited, please reply with “REMOVE” in the subject line and we will take care that you don’t receive any further promotional mail.

Of course, the “we will take care you don’t receive more” is a blatant lie. They always send more. I mean, I guess they adhere to the absolute letter of the law, they never send to the same address. They just harvest new ones.

Then there is the ‘there is nothing deceptive here!’ comment. If that’s true, why is the from address Kimcarter but the message signed by a Matt somebody, Marketing Director? Of course, just 2 weeks ago ago Matt was their Senior Frontend Developer and used the name Glenn Taylor. Before that it was Glenn Taylor, Jim Whitehead, Andrea Sharp, Ryan Heilman.

The Ryan Heilman message is fun. Apparently, Matt’s scraping software had a bit of a burp and was notifying me of problems with my website williamsnickl.com. Yeah. OK. Ryan, er… Matt, er… spammer. Let’s go with spammer.

It did make me smile today when I noticed this particular group of spammers have had to change their sending domain again. Who is going to tell them that having their actual website in the body of the message means changing the sending domain is useless? (not it!)

No Comments

Identifying domains that don’t accept or send email

A couple folks have asked me recently about MX records that they don’t understand. These records consist of a single . or they contain localhost or they are 127.0.0.1.

In all cases, the domain owners use these records to signal that the domains don’t accept email. What do these records look like?

screenshot of a terminal session that says: 
laura@pazu:~$ host yahooo.com
yahooo.com has address 98.136.103.23
yahooo.com has address 212.82.100.150
yahooo.com has address 74.6.136.150
yahooo.com mail is handled by 0 .

Why do domains do this? In all cases it’s because the domain owners want to signal they don’t accept email. But there are a number of different reasons to do this.

In the yahooo.com example, this is a domain actually owned by Yahoo. The website redirects to the primary yahoo.com site. But, it’s not a domain that accepts mail. They notify us of this by using a dot mx.

Screenshot of a terminal session that says: 
laura@pazu:~$ host collectors.org
collectors.org has address 91.195.240.126
collectors.org mail is handled by 0 localhost.

In the collectors.org example, they list the MX as localhost. This is a convention I’ve seen from a lot of for-sale domains. (As an aside, some of the for sale domains do accept email. The ones I’ve identified use a handful of common MXs that I suspect belong to the companies that sell access to spamtraps.)

I’ve also seen some domains use 127.0.0.1 as a MX record. Again, this is signalling that they really, really don’t want to accept email.

There’s also a way to signal a domain doesn’t send mail. This is accomplished by using a SPF -all record.

Screenshot of a terminal session that says:
laura@pazu:~$ host -t txt collectors.org
collectors.org descriptive text "v=spf1 -all"

There you go. Multiple ways to signal a domain doesn’t accept email and one way to signal the domain never sends email.

1 Comment

Mentally modelling filters

When we talk about filters, we often think there is one filter. But, in many cases there are multiple stages of filters, each examining mail in a different way.

Simple model of an email filter that takes mail and puts it in the inbox or spam folder

In deliverability terms the easiest filters to ignore are the individual user filters. Mostly because there’s nothing we can do about those. These are the baysean style filters built into a lot of email clients as well as specific filters users create to handle their own mail. As bulk senders, there’s not much we can do here. Senders have to accept users will do whatever they want with mail. Sometimes it benefits senders like when a user writes a rule to mark a particular message as important. Other times it doesn’t benefit senders, like when a user decides to trash a message without reading it. In both cases, senders don’t get a say.

It’s these user filters, and individual user actions on messages, that feed back into what we generally describe as “machine learning” filters. These are the black box style filters that measure thousands of different things about an email and make decisions about the whole mailstream. Many email delivery folks understand how SpamAssassin works. I think of SA as the precursor to a lot of the machine learning filters. While ML is much more complicated, the filters basically look at everything about an email and work out a score. That score determines where an email is delivered to the “average” user that doesn’t have any specific filters for that sender.

Machine learning filters are extremely conditional and will deliver mail to different places for different recipients. They’re adaptive and they learn. They’re under constant development and refinement to catch types of bad mail they missed and to let through types of good mail that they caught.

There’s another level of filter here, the SMTP level filters. These are very non-conditional filters. They’re basically hard and fast rules that are pushed out to the MX by the machine learning filter. The questions this filter asks are almost all yes or no questions. Examples of these kinds of questions

  • Is this IP or domain is on a blocklist? If yes, reject. If no, pass it on.
  • Does this email mentions a URL we’ve seen in phishing mail? If yes, reject the message.
  • Is this email is part of a stream we like? If yes, let it in and let it in fast.

There are other parts to these filters as well, but again the MX filters really ask simple yes or no questions.

  • Does this email address exist?
  • Is this message authenticated?
  • Is there a DMARC record and does the message pass DMARC?”

This isn’t a model that encompasses all the complexity of email filters. But it does help drive what we can and should do to troubleshoot delivery problems.

No Comments

Purging to prevent spamtraps

Someone recently asked when they should purge addresses to remove spamtraps. To my mind this is actually the wrong question. Purging addresses that don’t engage is rarely about spamtraps, it’s about your overall communication processes.

Outline of a head with a gear inside it.

Well maintained traps will actively bounce mail for 6 – 12 months before turning the address into a trap. In those cases it’s mostly the whole domain being turned into a trap, not just a single address. The common case where folks start hitting the recycled traps is that they have, for some reason, not regularly sent email to an address.

My general rule is if you’re actively bounce handling your mailings and you’re not avoiding mailing for more than a year then you shouldn’t have to worry about addresses turning into traps.

But you don’t just want to worry about spamtraps. You also want to be concerned about your overall reputation. For instance, an email address that never opens might have been abandoned by its owner (they forgot the password, moved to another account, whatever) and their failure to log into the address and your continuing to send mail to it turns it into a signal for the machine learning filters.

Alternatively, an email address that isn’t opening mail may never see the mail because it’s being delivered to spam and they don’t care enough to correct that. Every email delivered to the spam folder hurts your reputation, it’s less of a negative than if the user put the message there, but it still affects your reputation. Removing addresses that don’t engage removes negative hits to your reputation.

In both of those cases I tend to go reasonably long periods of time 12 – 24 months. But, there are arguments for longer or shorter, depending on your specific business model.

There are many good reasons to stop emailing addresses that don’t engage. Few of those reasons are specific to spamtraps.

No Comments

Microsoft and SmartScreen

There was another thread on mailop today about email filtering. This one was about Microsoft and SmartScreen. After watching a bunch of folks make lots of comments about what SmartScreen was, and get it wrong, I waded in.

Outlook Logo

One thing that I always thought was common knowledge, but apparently isn’t, is that SmartScreen is primarily a content filter. Microsoft does use IP and domain reputation in their filtering but SmartScreen is somewhat separate from those filters.

There are other things I’ve deduced about SmartScreen over the years, through discussions with other delivery folks, marketers, and Microsoft employees as well as following press releases, public statements and reading Microsoft’s extensive help pages. Why the Microsoft help pages? Many years ago SmartScreen filters were incorporated into Exchange installations. This hasn’t been true for quite a while, but I still find the user docs a useful source of insight into Microsoft’s filters.

What have I learned?

One of the major factors in SmartScreen is specifically how users are interacting with mail.

  • Recipients are acting in ways that tell Microsoft that they actively don’t want the mail in some or all of the following ways:
    • marking mail a spam
    • answering “yes” when Microsoft asks “did we classify this mail correctly as spam”
    • answering “no” when Microsoft asks “did we classify this mail correctly as not spam.
  • Recipients are acting in ways that tell Microsoft that they don’t really care about the mail in some or all of the following ways:
    • never opening the mail
    • deleting the mail
    • never looking for the mail when delivered to spam

My professional experience is that Microsoft has the most sensitive and aggressive filters in the top 3 free mailbox providers. I’m not convinced that this is intentional. But whether to not it’s intentional doesn’t matter. It’s their system and, in general, we senders have to deal with it. The good news is that there are MS employees willing to listen and talk to senders. It’s unclear to me how much of those discussions are shared with the development teams and are influencing how the filters are evolving. But I know there are discussions.

While it sounds like I’m dissing Microsoft here, I don’t think the delivery issues are wholly their fault. Many senders with Microsoft delivery problems have underlying issues that need to be addressed to get into Microsoft. But they see Microsoft’s sensitive and aggressive filters as “unfair” and “broken” because they aren’t having problems with other free mailbox providers. They refuse to change what they’re doing and thus see no change in their delivery.

I do think inboxing is a moving target and it’s hard for some types of senders to crack.

I do think machine learning filters are diverging at the different ISPs.

I do think that responsible senders who actually pay attention to their data collection process mostly have fewer delivery problems.

I do think that sometimes the filters are a little bit out of line and do things even the developers don’t expect.

I do think it’s all really complex.

I do think most mail gets delivered correctly, even if there are some spectacular mis-deliveries.

No Comments

Tulsi v. Google response

On Friday Google’s lawyers filed their response to the Gabbard Campaign’s first amended complaint. They asked for the case to be moved to the Northern District of CA as per the contractual agreement that the campaign signed. They also asked for a dismissal as they are not a government entity nor acting in place of a government entity and thus are not covered under either the 1st or the 14th amendments.

Image of a courthouse.

I pulled this case initially because it looked like there was going to be an email component to it. The first amended complaint reduced all the email content down to 1 paragraph.

117. Additionally, Gabbard has learned that email communications sent by the Campaign are classified as Spam by Google’s Gmail product at disproportionately high rates. Few Gmail users regularly check their spam folders. Many never do. Gmail’s Spam filter—which relies on secret algorithms designed and controlled entirely by Google—appear to go out of their way to silence messages from the Campaign, further hindering Tulsi’s ability to convey her message to the American people.

Google’s response to that paragraph was pretty straightforward.

D. Plaintiff’s Allegations Regarding Gmail Spam Filtering
The only other allegations Plaintiff offers about Google’s actions in regard to the campaign is the passing suggestion that “[Ms.] Gabbard has learned that email communications sent by [Plaintiff] are classified as Spam by Google’s Gmail product at disproportionately high rates.” FAC ¶ 117. This allegation is unadorned and unexplained. The FAC does not explain what “disproportionally high rates” is supposed to mean, what comparisons were done with other political campaigns or advertisers, or what basis Ms. Gabbard has for alleging this supposedly disproportionate spam classification.2

2 “Spam” is defined generally as unsolicited bulk email messages. Google maintains detailed Sender Guidelines that explain how to avoid having emails classified as “spam.” See White Decl., Exhibit 3 [pdf link]. Plaintiff does not allege whether any of the emails that Google’s system allegedly classified as “spam” were, in fact, “spam” under Google’s policies. 

At this point, there’s no reason for an email blog to follow this case. Email is a single, unsubstantiated paragraph alleging delivery problems and Google’s response is to point out their publicly available sender guidelines page. Nothing to see here.

No Comments

Forget about engagement, think inboxing

While answering a question about how to improve IP reputation at Gmail I realized that I no longer treat Gmail opens as anything about how a user is interacting with email. There are so many cases and ways that a pixel load can be triggered, without the user actually caring about the mail that it’s not a measure of the user at all.

image of a head with gears and ideas floating around it

That doesn’t mean opens are useless. In fact, they’re very useful. But only if you have the full picture.

  1. Gmail, and other consumer mailbox providers, do not allow images to load for messages in the bulk folder.
  2. Gmail, and other consumer mailbox providers, do some level of individualised delivery. Even if most of a particular mailing is going to the bulk folder, individual users may still get that email in their inbox.
  3. Every message delivered to the spam folder, whether marked as spam by the user or delivered there by the mailbox provider, hurts your reputation.

Every email in the spam folder hurts your overall reputation. Continuing to send mail ending up in spam will decrease your overall delivery in the long term.

One of the ways to improve reputation is to remove anything that is hurting your reputation. This means, removing any emails going to the bulk folder. How do we know which emails are going to the bulk folder? One piece of data is an image was loaded, i.e. an open was recorded. That open won’t happen if mail is in bulk.

How far back we go to remove addresses is an interesting question. I can argue all sorts of timelines. it doesn’t really matter. I’ve seen reputation improvement using just a few thousand emails that we knew were going to the inbox.

The real signal is not that you perfectly remove every address receiving mail in the bulk folder, but that you remove the majority of addresses receiving mail in the bulk folder. Want to go back a year? Sure. 18 months? yeah, probably will work. Longer, well, what’s the likelihood those addresses have been abandoned and no longer have an active user logging in and looking at data?

Once reputation is repaired, you can start to mail some of the suspended folks on your lists. But, stopping mail that is actively hurting your reputation is always the first step. Think about it, if you could remove spamtraps from your lists, wouldn’t you? Mail going to the spam folder can damage your delivery just as much as spamtraps.

No Comments

Opting out of “service” messages

A frequent question in a number of deliverability spaces is how to tell if a message is transactional or marketing. In most cases the decision is related to whether or not to respect an unsubscribe request. All too often companies decide that their messages are too important to allow someone to opt-out of. The problem is, in some cases, there is no longer a customer relationship to send notices about.

Picture of a stream in Dublin, Ireland

This came up because it’s been just about a year now since I unsubscribed from most of my US based commercial lists. (Yes, we’ve been in Dublin more than a year now!). Because it’s been a year I’m getting a lot of “transactional” messages. Many of them are reminding me to log into my account to see my current rewards level. Others are offering me coupons if I come back.

These aren’t transactional messages, they have nothing to do with any transactions I’ve made. They’re spam, plain and simple. They violate CAN SPAM, as I have opted out from mail from those companies. I’ve not made any purchases from those companies in more than a year.

But, I’ve seen first hand how marketing departments justify emails like this as ‘transactional.’ This is also the time where they start talking about the 80/20 rule that some spammer made up to justify calling their marketing mail spam. Then they don’t let you actually opt-out of this message because they’re “account messages.”

I can’t actually affect delivery of this kind of mail as I’m not at one of the commercial providers. But, if I were, you can bet I’d be reporting each and every one of these messages as spam. Even if it only taught the provider to put this mail in my spam folder, it’s still better than getting mail that I have asked to no longer receive but am still receiving.

Seriously, marketers, at some point you’re going to have to stop biting the apple. When recipients tell you to stop mailing them, take that as an instruction that you should stop mailing them. Continuing to mail people who’ve opted out only injures your reputation and your overall delivery.

1 Comment

Details matter

I field a lot of delivery questions on various online fora. Often people try and anonymise what they’re asking about by abstracting out the question. The problem is that there are very few answers we can give in the abstract.

Outline of a head with a gear inside it.

What are some examples of these types of questions?

  • Should you always remove an address that hard bounces? Well, in general, yes. But there are a small number of cases where the hard bounce is a mistake on the part of the receiver and you shouldn’t remove that address.
  • Should you send email to recipients who haven’t engaged in 3 years. Well, in general, no. But I’ve seen and managed campaigns to recipients much older than that. What are you really trying to do?
  • If we limit our sending to people who’ve opted in to email, we’ll solve our spamtrap problem, right? Well, first, why do you think you have a spamtrap problem? If you’re Spamhaus listed, there’s a lot more you need to do. If you’re seeing one or two traps at the commercial sensor networks, then what’s your overall deliverability look like?
  • Why would our mail suddenly start to go to bulk? Overall, it wouldn’t. What did you change? Did your website get compromised? Have you linked to a new image server? Did you publish a DMARC record? Did you mention a domain with a bad reputation?
  • If we change the from address of our mail will it affect our deliverability? It can, but what from domain you’re talking about, what you’re changing it from and what you’re changing it to all matter before anyone can actually answer the question.

Deliverability is not a science. There are no hard and fast rules. Even the rules I wish were true, like only send opt-in mail, aren’t really hard and fast. A lot of folks get decent delivery using purchased or otherwise non-opt-in lists. I don’t like it, but I acknowledge it.

In order to get good deliverability advice for a situation the full situation needs to be described. History, specifics, IPs, and domains all matter. Where your email addresses came from and how you’ve maintained your database matters. It all matters. Abstracting out a question just means you get an abstract and generic answer, and that doesn’t help anyone.

No Comments