Many years ago, we bought a VMWare license to manage the various virtual machines running our business infrastructure. As part of our move to Dublin, we decommissioned our cabinet and moved all of services into various bits of the cloud. This meant that when our VMWare support contract came up for renewal we declined the renewal.
Despite no longer being customers and unsubscribing from email, I’ve had ongoing problems getting VMWare to leave me alone. They keep sending me email, despite unsubscribing through various channels. I’ve clicked on links, I’ve responded to emails. This is what my outgoing mailbox looks like when I search for @vmware.com:
What prompted the search was yet another email from Dell this weekend:
The documentation of our refusal to renew our VMWare support contract was completed over the summer. VMWare was extremely persistent in their emails then, too. We were asked to fill out multiple pages of documents, to ‘prove’ we were no longer using VMWare software. Even after we provided written assurances we were not using VMWare software they continued to send mail, telling us that the mail was automated and there was no way for them to stop us from getting it. That’s OK, we can stop the mail, and the address doing the harassment was the first email address fully blocked on our mail server.
This mail from VMWare is no longer marketing. This is, in fact, harassment. I’ve asked repeatedly for VMWare to stop contacting me. I even have a note from June of 2013 marking I requested unsubscription from their lists on 6/24. I don’t record all of my unsubscribes, only the ones where it seems I’ve asked before and they won’t stop.
CAN SPAM, and many other laws, say that a company has to stop sending email when the recipient opts-out. I’ve opted-out, but continue to receive email with little recourse other than expanding the block to all mail from VMWare. I really hate doing that, but as they appear to be unable to accept an opt-out, we may have no other choice.
Trust me, if we decide to use VMWare for virtualisation I know how to reach them.
Sendgrid announced their volumes for Black Friday and Cyber Monday:
To provide a clearer understanding of our scalable systems, on Black Friday 2019 we processed 4.1 billion emails and this Cyber Monday we processed 4.2 billion emails (46% more than 2018) and processed up to 315 million emails/hour (burst rate) into inboxes all around the world! To give you a better understanding of this scale, you can get to the moon and back 4 times in fewer yards than the number of emails we sent on Black Friday.
That’s more than double the volume they sent in 2017.
I expect other large senders did similar volumes, which makes the total amount of email over the last week mind blowingly huuuugee.
I’m basically waiting for the various ESPs to announce Just How Much Mail they’ve sent over the last 4 days. Early information from one ESP shows a hefty percentage over the amount they sent last year, and that amount had many, many, many zeros in it.
In terms of best practices and ongoing advice: you can never put too many lights on a Christmas tree.
I’ve shared a version of this image repeatedly. I think it was only my Facebook friends that got the stick figure screaming in frustration, though.
The reality is bounce handling is one of the most frustrating pieces of email delivery. Not only that, many people in the email space treat it as a simple process. It’s really not as simple as we’d like it to be.
The above image was created based on docs from 3 different ESPs a client was using. They wanted to normalise their bounce handling across ESPs, and asked me for policy recommendations. I ended up digging through a bunch of docs from their 3 ESPs. I recorded the reasons as reported in the docs in a colored block corresponding to the ESP, then dropped them in the appropriate circle: soft, block or hard.
The shaded circles are based on my interpretations of why these bounces happen.
The big grey circle surrounds bounces due to reputation issues.
The green circle is primarily networking and technical issues.
The top purple circle is non existent or bad addresses
Note, nothing here indicates how we should react to the bounces, this is just a categorisation activity. This classification also has nothing to do with what the actual SMTP response is.
Just remember, next time someone says bounce handling is simple: they’re wrong.
The phishing attack against Sendgrid is still going on. Most of the mail and the websites are being hosted on Linode. I’ve still not gotten to see what one of the sites looks like, as Linode is getting the sites down before I click on the links.
Everyone here is doing the Right Things(tm) in order to address the problem. Sendgrid has a p=reject message in their DMARC record, Linode seems to be reasonably competent at getting the phishing sites down quickly enough.
I did notice in today’s round of emails the phishers have evolved. Sometime between midnight GMT and 4pm GMT they stopped forging @sendgrid.com in the from address. Now they’re just forging random domains. Any protection SendGrid was getting from their DMARC record is now gone.
This is a prime example of why I roll my eyes whenever anyone tells me DMARC stops phishing. DMARC stops one very specific kind of phishing that is trivial to work around. Even if every company on the planet went p=reject there is absolutely nothing to stop the phishers from registering their own domains and publishing their own DMARC records.
I’m pretty sure that many of these emails are being blocked and filtered, but if even a few get through and are clicked on it can cause major pain for the victim companies.
For a long time one of the “best practices” for links in html content has been to avoid having anything that looks like a URL or hostname in the visible content of the link, as ISP phishing filters are very, very suspicious of links that seem to mislead recipients about where the link goes to. They’re a very common pattern in phishing emails.
(The code block is mangled, because WordPress is just terrible software, but I hope you get the idea.)
Why is that last one risky? It’s OK, and not misleading as you write it but if your ESP uses click-tracking then they’ll rewrite the link as they send it, to redirect through their systems. And that looks very suspicous.
I hadn’t really thought about the implications of this when it came to images, though. An image doesn’t really have any text associated with it, at least not in a way that a phishing filter has easy access to, so shouldn’t be a problem.
Except they do, of course. The alt text that you add to the image to make it accessible to screen readers, and to provide some visible content when the recipient isn’t loading images.
I signed up for an account today, and the address confirmation email had a call to action button that looked like this:
One of the interesting things about moving to the EU is experiencing the internet where GDPR is a thing. We get asked permission for everything. Including if we want shopping cart updates.
On the email space, though, we’ve been visiting various home shows as we look at options and furniture for our new house. Part of it has involved giving email addresses to various groups in exchange for tickets. I’ve been pleasantly surprised at how I get the mail I expect related to those events and that address hasn’t leaked to all and sundry.
Contrast this with addresses I’ve shared with US companies, and then get bombarded with mail from all of them and their affiliates and anyone who could beg, borrow or steal my email address. It is phenomenally different. It also means I am much, much more likely to give local companies my email address. They don’t misuse it! What magic is this?
One of the very frustrating pieces is how many American News outlets refuse to even show anything to us.
A friend even commented something similar over on FB. Just how many major US news outlets can’t be bothered to do anything but block anyone coming in from the EU. I figure it’s mostly laziness, but it’s always possible they’re doing nefarious things with tracking and just are that afraid they can’t comply with the law.
Not that GDPR is that enforced. In fact, we were at a digital summit a few weeks ago talking about how digital attacks are rocking the very foundations of democracy. One of the panelists even said, “GDPR. It’s a great idea, someone should really try it someday.”
There is currently a phishing attack against a major ESP. The mail came through what I presume was a compromised account hosted at one of the providers. It’s just as possible this was a domain set up for the sole purpose of phishing, though.
The underlying attack is pretty good. They took the ESP compliance notification email and changed a couple of the links to point to their phishing page (which is down now). I’m pretty sure a message “your account has been limited due to poor reputation” caused a whole lot of folks to freak out and click the links.
If it were me coordinating the attack, I’d be quietly logging into the compromised accounts over the next 10 days and creating new API keys. I’d set up my spam cannons to use those API keys and then wait for Black Friday. A single button and I can send out … millions and millions of authenticated emails through hundreds of accounts with solid reputations.
Steve and I were talking about this last night and were discussing tracking logins, 2FA and other ways the ESP could mitigate the problem and protect their users. It wasn’t until I woke up this morning that I remembered that the ESP has a full API. Yeah, that makes it even harder. Sure, the spammers need to log in and create new API keys. But individual logins that simply create API keys are harder to detect than a log in that doesn’t do anything but create a key.
This is not something the ESP can easily mitigate in 10 days. They will have had to have infrastructure in place to track creation of API keys and confirm these keys are being used by their customer. I know this ESP and I am hopeful that their security folks have thought about this attack vector.
If you are a Sendgrid customer, it may be worthwhile to revisit your infrastructure today. Identify what needs API keys and regenerate them. Then, nuke all the keys in your account. Change all your passwords. Lock down your account.
I feel for both the ESP and their customers. This was a carefully planned attack. I have zero doubt this is in preparation for sending out a massive spam campaign from the ESP at the height of the holiday email season. Don’t assume your account is safe. Make sure it is.
Otherwise, you may find more than the normal level of delivery problems for your holiday mail.