Sendgrid announced their volumes for Black Friday and Cyber Monday:
To provide a clearer understanding of our scalable systems, on Black Friday 2019 we processed 4.1 billion emails and this Cyber Monday we processed 4.2 billion emails (46% more than 2018) and processed up to 315 million emails/hour (burst rate) into inboxes all around the world! To give you a better understanding of this scale, you can get to the moon and back 4 times in fewer yards than the number of emails we sent on Black Friday.
That’s more than double the volume they sent in 2017.
I expect other large senders did similar volumes, which makes the total amount of email over the last week mind blowingly huuuugee.
I’m basically waiting for the various ESPs to announce Just How Much Mail they’ve sent over the last 4 days. Early information from one ESP shows a hefty percentage over the amount they sent last year, and that amount had many, many, many zeros in it.
In terms of best practices and ongoing advice: you can never put too many lights on a Christmas tree.
I’ve shared a version of this image repeatedly. I think it was only my Facebook friends that got the stick figure screaming in frustration, though.
The reality is bounce handling is one of the most frustrating pieces of email delivery. Not only that, many people in the email space treat it as a simple process. It’s really not as simple as we’d like it to be.
The above image was created based on docs from 3 different ESPs a client was using. They wanted to normalise their bounce handling across ESPs, and asked me for policy recommendations. I ended up digging through a bunch of docs from their 3 ESPs. I recorded the reasons as reported in the docs in a colored block corresponding to the ESP, then dropped them in the appropriate circle: soft, block or hard.
The shaded circles are based on my interpretations of why these bounces happen.
The big grey circle surrounds bounces due to reputation issues.
The green circle is primarily networking and technical issues.
The top purple circle is non existent or bad addresses
Note, nothing here indicates how we should react to the bounces, this is just a categorisation activity. This classification also has nothing to do with what the actual SMTP response is.
Just remember, next time someone says bounce handling is simple: they’re wrong.
The phishing attack against Sendgrid is still going on. Most of the mail and the websites are being hosted on Linode. I’ve still not gotten to see what one of the sites looks like, as Linode is getting the sites down before I click on the links.
Everyone here is doing the Right Things(tm) in order to address the problem. Sendgrid has a p=reject message in their DMARC record, Linode seems to be reasonably competent at getting the phishing sites down quickly enough.
I did notice in today’s round of emails the phishers have evolved. Sometime between midnight GMT and 4pm GMT they stopped forging @sendgrid.com in the from address. Now they’re just forging random domains. Any protection SendGrid was getting from their DMARC record is now gone.
This is a prime example of why I roll my eyes whenever anyone tells me DMARC stops phishing. DMARC stops one very specific kind of phishing that is trivial to work around. Even if every company on the planet went p=reject there is absolutely nothing to stop the phishers from registering their own domains and publishing their own DMARC records.
I’m pretty sure that many of these emails are being blocked and filtered, but if even a few get through and are clicked on it can cause major pain for the victim companies.
For a long time one of the “best practices” for links in html content has been to avoid having anything that looks like a URL or hostname in the visible content of the link, as ISP phishing filters are very, very suspicious of links that seem to mislead recipients about where the link goes to. They’re a very common pattern in phishing emails.
(The code block is mangled, because WordPress is just terrible software, but I hope you get the idea.)
Why is that last one risky? It’s OK, and not misleading as you write it but if your ESP uses click-tracking then they’ll rewrite the link as they send it, to redirect through their systems. And that looks very suspicous.
I hadn’t really thought about the implications of this when it came to images, though. An image doesn’t really have any text associated with it, at least not in a way that a phishing filter has easy access to, so shouldn’t be a problem.
Except they do, of course. The alt text that you add to the image to make it accessible to screen readers, and to provide some visible content when the recipient isn’t loading images.
I signed up for an account today, and the address confirmation email had a call to action button that looked like this:
One of the interesting things about moving to the EU is experiencing the internet where GDPR is a thing. We get asked permission for everything. Including if we want shopping cart updates.
On the email space, though, we’ve been visiting various home shows as we look at options and furniture for our new house. Part of it has involved giving email addresses to various groups in exchange for tickets. I’ve been pleasantly surprised at how I get the mail I expect related to those events and that address hasn’t leaked to all and sundry.
Contrast this with addresses I’ve shared with US companies, and then get bombarded with mail from all of them and their affiliates and anyone who could beg, borrow or steal my email address. It is phenomenally different. It also means I am much, much more likely to give local companies my email address. They don’t misuse it! What magic is this?
One of the very frustrating pieces is how many American News outlets refuse to even show anything to us.
A friend even commented something similar over on FB. Just how many major US news outlets can’t be bothered to do anything but block anyone coming in from the EU. I figure it’s mostly laziness, but it’s always possible they’re doing nefarious things with tracking and just are that afraid they can’t comply with the law.
Not that GDPR is that enforced. In fact, we were at a digital summit a few weeks ago talking about how digital attacks are rocking the very foundations of democracy. One of the panelists even said, “GDPR. It’s a great idea, someone should really try it someday.”
There is currently a phishing attack against a major ESP. The mail came through what I presume was a compromised account hosted at one of the providers. It’s just as possible this was a domain set up for the sole purpose of phishing, though.
The underlying attack is pretty good. They took the ESP compliance notification email and changed a couple of the links to point to their phishing page (which is down now). I’m pretty sure a message “your account has been limited due to poor reputation” caused a whole lot of folks to freak out and click the links.
If it were me coordinating the attack, I’d be quietly logging into the compromised accounts over the next 10 days and creating new API keys. I’d set up my spam cannons to use those API keys and then wait for Black Friday. A single button and I can send out … millions and millions of authenticated emails through hundreds of accounts with solid reputations.
Steve and I were talking about this last night and were discussing tracking logins, 2FA and other ways the ESP could mitigate the problem and protect their users. It wasn’t until I woke up this morning that I remembered that the ESP has a full API. Yeah, that makes it even harder. Sure, the spammers need to log in and create new API keys. But individual logins that simply create API keys are harder to detect than a log in that doesn’t do anything but create a key.
This is not something the ESP can easily mitigate in 10 days. They will have had to have infrastructure in place to track creation of API keys and confirm these keys are being used by their customer. I know this ESP and I am hopeful that their security folks have thought about this attack vector.
If you are a Sendgrid customer, it may be worthwhile to revisit your infrastructure today. Identify what needs API keys and regenerate them. Then, nuke all the keys in your account. Change all your passwords. Lock down your account.
I feel for both the ESP and their customers. This was a carefully planned attack. I have zero doubt this is in preparation for sending out a massive spam campaign from the ESP at the height of the holiday email season. Don’t assume your account is safe. Make sure it is.
Otherwise, you may find more than the normal level of delivery problems for your holiday mail.
Saw a new disclaimer on mail sent to an address harvested off our website today:
disclaimer: This is an advertisement and a promotional mail-in adherence to the guidelines of CAN-SPAM act 2003. We have clearly mentioned the source id of this mail, also clearly mentioned the subject line, and they are in no way misleading in any form. We have found your email address through our own efforts on the web search and not through any other way. If you find this email unsolicited, please reply with “REMOVE” in the subject line and we will take care that you don’t receive any further promotional mail.
Of course, the “we will take care you don’t receive more” is a blatant lie. They always send more. I mean, I guess they adhere to the absolute letter of the law, they never send to the same address. They just harvest new ones.
Then there is the ‘there is nothing deceptive here!’ comment. If that’s true, why is the from address Kimcarter but the message signed by a Matt somebody, Marketing Director? Of course, just 2 weeks ago ago Matt was their Senior Frontend Developer and used the name Glenn Taylor. Before that it was Glenn Taylor, Jim Whitehead, Andrea Sharp, Ryan Heilman.
The Ryan Heilman message is fun. Apparently, Matt’s scraping software had a bit of a burp and was notifying me of problems with my website williamsnickl.com. Yeah. OK. Ryan, er… Matt, er… spammer. Let’s go with spammer.
It did make me smile today when I noticed this particular group of spammers have had to change their sending domain again. Who is going to tell them that having their actual website in the body of the message means changing the sending domain is useless? (not it!)
A couple folks have asked me recently about MX records that they don’t understand. These records consist of a single . or they contain localhost or they are 127.0.0.1.
In all cases, the domain owners use these records to signal that the domains don’t accept email. What do these records look like?
Why do domains do this? In all cases it’s because the domain owners want to signal they don’t accept email. But there are a number of different reasons to do this.
In the yahooo.com example, this is a domain actually owned by Yahoo. The website redirects to the primary yahoo.com site. But, it’s not a domain that accepts mail. They notify us of this by using a dot mx.
In the collectors.org example, they list the MX as localhost. This is a convention I’ve seen from a lot of for-sale domains. (As an aside, some of the for sale domains do accept email. The ones I’ve identified use a handful of common MXs that I suspect belong to the companies that sell access to spamtraps.)
I’ve also seen some domains use 127.0.0.1 as a MX record. Again, this is signalling that they really, really don’t want to accept email.
There’s also a way to signal a domain doesn’t send mail. This is accomplished by using a SPF -all record.
There you go. Multiple ways to signal a domain doesn’t accept email and one way to signal the domain never sends email.