I’ve been blogging regularly for over a decade now, and for much of that time I’ve posted 5 days a week. For a lot of reasons I’m finding that schedule harder and harder to keep up with. Part of it is that this spring I took on more, and bigger, clients than I have in the past. This means a larger portion of my time is scheduled and committed than in the past. I also find myself wanting to write about bigger, more complex issues; stuff that takes longer than the 45 minutes – 2 hours I regularly spend on blog posts.
The last few months, I’ve been considering what to do about blogging. I could simply cut back the amount I write here. Except that regularly blogging forces me to think about what’s going on in the broader industry, and that’s important to me and I think makes me a better consultant. I could write a few short posts a week, and a bigger meatier post once or twice a month, but I’ve been me long enough to know that’s not the best solution. I could just keep going as I have been most of this year and just post when I have something to say and not worry about frequency.
I still don’t have the answer. Of course, there’s not a right answer, there’s just a move forward and do what works. I have a lot of travel coming up next month (including speaking at Activate: The ActiveCampaign Conference) so things might get wonky for a while. But, I’m not planning on giving up blogging.
One of the consequences of my time constraints is that I have handed comment moderation off to other folks. Comments might sit for longer than they used to before approval. They’re being processed, just a little more slowly than they have in the past. I don’t think it’s a big deal, it’s not like there’s a significant horde of commenters here. When I was moderating comments basically anything that contributed to the discussion and didn’t come from a forged email address was approved. The current policy is similar.
I am around on the email geeks slack channel, and am often talking about stuff on the deliverability channel.
Thus ends the housekeeping.
I was doing some research today for an article I’m working on. The research led me to a San Francisco Law Review article from 2001 written by David E. Sorkin. Technical and Legal Approaches to Unsolicited Electronic Mail (.pdf link). The text itself is a little outdated, although not as much as I expected. There’s quite a good discussion of various ways to control spam, most of which are still true and even relevant.
From a historical perspective, the footnotes are the real meat of the document. Professor Sorkin discusses many different cases that together establish the rights of ISPs to filter mail, some of which I wasn’t aware of. He also includes links to then-current news articles about filtering and spam. He also mentions different websites and articles written by colleagues and friends from ‘back in the day’ discussing spam on a more theoretical level.
CNET articles on spam and filtering was heavily referenced by Professor Sorkin. One describes the first Yahoo spam folder. Some things never change, such as Yahoo representatives refusing to discuss how their system works. There were other articles discussing Hotmail deploying the MAPS RBL (now a part of Trend Micro) and then adding additional filters into the mix a few weeks later.
We were all a little naive back then. We thought the volumes of email and spam were out of control. One article investigated the effectiveness of filters at Yahoo and Hotmail, and quoted a user who said the filters were working well.
“It’s really awesome because I get maybe 20 emails a day, and [it’s] mostly junk mail,” said longtime Yahoo Mail user Daniel Nikaiyn. “It’s saved me a lot of time splitting up junk mail and my email. Now I don’t have to sift through them.”
I think I got 20 emails yesterday just trying to register at one new site and do the password reset dance with another.
In addition to the news articles, I saw a bunch of documents and websites I’d nearly forgotten about. There were a group of people, and I include myself among them, that spent a lot of time trying to figure out how to fix spam. When it was 20 emails in my inbox it did seem somewhat silly. Yes, I can delete them. But the bigger issue was the lack of external economic constraints on the amount of mail senders could send. Sure, that day was 20 emails, but there was nothing stopping it being 100 in 6 months and 500 6 months after that.
In fact when I gave up the email address I was using in the late 90s there were days it was receiving hundreds of spams a month, and that was behind commercial grade filters run by my ISP which caught most botnet and snowshoeing spam. And that was just last year, when the overall volume of spam traffic had dropped from over 95% of email traffic down to under 85%.
The whole document is long, but Professor Sorkin did get one thing right.
Coordination of technical and legal mechanisms seems to be the most promising approach to the spam problem. The first step must be to agree upon the ultimate objective: it is quite easy to declare “get rid of spam,” but the definition of spam is sufficiently controversial that this first step may be the most difficult. Technical and legal measures can then be used in a complementary fashion—for example, technical measures can be designed so that one must break the law (or subject oneself to liability) in order to circumvent them, while those who evade or ignore legal controls could be subjected to blackholing and other technical responses.
Yet it is probably unrealistic to expect that the consensus required for such coordination can be achieved. More likely, the technical arms race between spammers and anti-spammers will escalate, and more and more innocent bystanders will be caught in the crossfire. States and countries will continue enacting an increasingly diverse set of spam-related statutes, and traditional legal theories will be stretched and distorted even further in efforts to address spam and other forms of “network abuse.” The news is not all bad; there have been advances in collaborative filtering by companies such as Brightmail, and some recent legislation seems to incorporate at least a rough comprehension of the underlying technology. Nonetheless, a coordinated solution to the problem of spam remains elusive at best. (footnotes removed)
Spam affects endusers less now than it did in in 2002 when the article was written. I don’t think Professor Sorkin envisioned a multi-billion dollar industry spam filter industry, but that is a major reason our inboxes are still useable. I don’t think the laws have necessarily caught up. In fact, my research this afternoon was started as I was thinking about how CAN SPAM is antiquated and doesn’t provide sufficient tools to effectively address spam as it is now. Despite how far we’ve come and how much has changed, spam is still here and will likely be here for the foreseeable future.
There’s going to be a lot of hype today about something the security researchers who found it are calling “EFAIL”. Interviews, commemorative T-Shirts, press tours, hype.
The technical details are interesting, but the un-hyped end-user advice would probably be “If you’re using a mail client that’s got bugs in it’s MIME handling, and you’ve configured it to load remote content automatically, and you’re using a less secure encryption tool or protocol, and you’ve configured it to decrypt things automatically, and security of your email is so important to you that you’re defending against skilled attackers who have already acquired the encrypted emails you’re concerned about (by compromising your ISP? Sniffing non-TLS traffic?) then you may have a problem.”
I can’t imagine anyone for whom email security is a critical issue would make all those mistakes, so this mostly merits a heads-up to the MUA developers (which has happened) and maybe a “Do people rely on S/MIME? Why?” retrospective. But as someone on twitter described it “The Vulnerability Hype Train has begin, choo choo.”
There are several different issues all mixed together by the efail folks. All of them require an attacker to already have access to (encrypted) sensitive emails, and to send copies of those to you wrapped up in another message and to have you decrypt that incoming mail.
- “Direct exfiltration”. Some mail clients with badly broken mime handling can apparently be convinced – if remote images are loaded automatically and email is decrypted automatically – to send decrypted plaintext to the attacker. This is bad, but mostly only applies to Apple Mail and Thunderbird. Apple have already fixed the issue on March 29th. And ErrataRob couldn’t replicate the claimed attack in current Thunderbird releases.
- “Indirect exfiltration of PGP encrypted messages”. It’s possible to inject plain text into an encrypted message in some cases. This isn’t particularly reliable. GPG and other OpenPGP implementations have mitigated this for many years, so it’ll only be possible to attack PGP if the implementation a mail client uses is faulty. That seems to be the case only for Thunderbird and Apple and a few niche clients. That can then be used to leak information about the decrypted text to the attackers server (via CSS, image loads, TLS OCSP requests, etc.).
- “Indirect exfiltration of S/MIME messages”. This is similar to the PGP approach but much more of an issue, both because S/MIME implementations seem to be far less robust against this sort of attack and because in the corporate environments where S/MIME is used it’s more likely to be installed by default with automatic decryption of messages (either on the desktop or on a corporate mail gateway). Attempts to exploit this will be really obvious, though, so in addition to MUAs fixing the issue (Apple have already fixed it, and I’d be surprised if Microsoft haven’t, though I pay less attention to their ecosystem) it’d be easy to block at the edge by pushing spam or malware filter updates.
The analysis on this seems good, technically. The cryptographic work appears solid – even if it’s pretty much just a rehash of decade old known issues. And the work done surveying clients for vulnerabilities is useful. MUA developers and people developing border mail filters should definitely read the paper.
But it follows the recent history of such disclosures, having a memorable name, it’s own website, a logo and announcement headlines chosen to get a response rather than top be accurate. This is primarily an issue with corporate encrypted email using S/MIME, but the announcements mostly describe it as being a bug in PGP (it isn’t) and stress it being used to attack those who use PGP – “journalists, political activists, whistleblowers, …” – which it probably isn’t, at least not successfully.
Keep your mail client updated. If privacy is important to you at all, don’t have it load remote content automatically.
And if you deal with sensitive emails that have to be encrypted and where someone may specifically be targeting you, you really have to try and understand what the threats against you are, as most of them aren’t as simple as this one. There’s no “do these five things and you’ll be secure” listicle.
We sent out some W-9s this week. For non-Americans and those lucky enough not to have to deal with IRS paperwork those are tax forms.
They’re simple single page forms with the company name, address and tax ID numbers on them. Because this is the 21st Century we don’t fill them in with typewriters and snail mail them out, we fill in a form online at the IRS website which gives us PDFs to download that we then send out via email.
We started to get replies from people we’d sent them to that we hadn’t included the tax ID number. Which was odd, because it was definitely there in the PDFs we’d sent.
The reports of missing numbers came from Google Apps users, so we sent a copy to one of our Gmail addresses to see. Sure enough, when you click on the attachment it’s mostly there, but some of the digits of the tax ID number are missing.
And all the spaces have been stripped from our address.
The rest of the form looked fine, but the information we’d entered was scrambled. Downloading the PDF from Gmail and displaying it – everything is there, and in the right place.
Weird. After a brief “Are gmail hiding things that look like social security numbers?” detour I realized that the IRS website was probably generating the customized forms using PDF annotations.
PDF is a very powerful, but very complex, file format. It’s not just an image, it’s a combination of different elements – images, lines, vector artwork, text, interactive forms, all sorts of things – bundled together into a single file. And you can add elements to an existing PDF file to, for example, overlay text on to it. These “annotations” are a common way to fill in a PDF form, by adding text in the right place over the top of an existing template PDF.
I cracked the PDF open with some forensics tools and sure enough, the IRS had generated the PDF form using annotations.
<< /Type /Annot /DV (Palo Alto, CA) /T (topmostSubform.Page1.Address.f1_8)
/Rect [ 57.6 539.968 388.8 553.969 ] /AP 81 0 R /FT /Tx /DA (/Helvetica-Bold 9 Tf 0 g)
And the Gmail PDF viewer isn’t rendering that annotated text correctly.
filed a bug sent feedback to Google, so hopefully it’ll be fixed. Meanwhile, if you’re sending customized content to recipients using PDF you should probably check that it renders correctly when previewed in Gmail.
I sometimes get in arguments with clients where I say, “your open rate is 3%, you need to do some list pruning” and they say, “my recipient list is 100% b2b, and b2b filters don’t care about engagement, so it doesn’t matter if my list is really old and unengaged.” This is wrong in cases where the business is using Outlook or Gsuite, both of which are going to care if they see no-one opening your messages, but what about all the corporate domains using barracuda, mimecast, or even proofpoint? Is engagement a factor with these filters? Do they care if you are sending to a list that is not opening your messages?
Not a Fighter
Dear Not a Fighter,
As with most answers related to deliverability, details matter. You are correct that many businesses are now hosting their email at Office365 and Gmail. Both of these organizations have filters that do measure engagement and use that information to make decisions on where to deliver email. From what I’ve observed, though, mail is still handled differently
The clearest example of this for the same filter using different inputs is is the SCL (spam confidence level) and BCL (bulk confidence level) used by Microsoft. Every message coming into Microsoft’s MXs gets a SCL and a BCL score. However, one score is used solely for filtering of Office365 mail and the other is used for filtering of consumer webmail.
Your clients are right, though, about other business filters like mimecast, barracuda and proofpoint not monitoring engagement. They don’t. There are multiple reasons they don’t from they technically don’t have the access, through to engagement is irrelevant to businesses. If a company provides an appliance or even a hosted service, it’s likely their setup doesn’t allow access to user specific data on a regular basis.
This doesn’t mean, however, that addresses should never be purged from B2B lists. Hygiene is critical for delivery no matter where you’re sending to. People move jobs and change companies all the time. Sometimes those addresses are turned off, sometimes they’re forwarded to a mailbox no one looks at, sometimes they’re forwarded to another person inside the company, sometimes they’re handed over to a reputation company to be used as a spamtrap. The sender has no way of knowing which thing will happen. The other issue is that companies have a lot of direct control over their spam filters. Employees can ask for certain sources of email to be blocked and technical staff are often empowered and entrusted with the ability to act on those requests.
Being blocked at a company, not a filter provider, is nearly impossible to reverse. Senders trying to get blocks lifted, when that block is imposed by the company itself, discover very quickly that unless the company finds value in the mail, they don’t care if it gets delivered. That’s why engagement matters, sending mail that annoys the individual employees is a fast path to never being able to contact any employee at the company in the future.
Hygiene is an overall good. Companies that value contacting people, rather than just sending as much email as possible, understand this.
Confused about delivery in general? Trying to keep up on changing policies and terminology? Need some Email 101 basics? This is the place to ask. We can’t answer specific questions about your server configuration or look at your message structure for the column (please get in touch if you’d like our help with more technical or forensic investigations!), but we’d love to answer your questions about how email works, trends in the industry, or the joys and challenges of cohabiting with felines.
Countless questions about email troubleshooting start with “does anyone know why.” Unfortunately, most of these questions don’t contain enough detail to get a useful answer.
In the case of email, even the smallest redactions, like the IP address and the domain in question, can make it difficult for anyone to provide help. Details matter.
Every detail matters, sending IP and domain are just the beginning. Who’s doing the sending? What is their authentication setup? What IP are they using? How were the addresses collected? What is their frequency? What MTA is used? Are they linking to outside sites? Are they linking to outside services? Where are images hosted? Is the mail going to the bulk folder or being rejected? What ISPs or filters are involved?
The relevant questions go on and on and on.
We send fairly detailed question lists to clients. I regularly look at them to try and make them shorter. But the reality is these are questions that are relevant. Without enough information we simply cannot troubleshoot delivery problems.
Yesterday I talked about all the reasons that using affiliate email can hurt overall delivery. In some cases, though, marketing departments and the savvy email marketer don’t have a choice in the matter. Someone in management makes a decision and employees are expected to implement it.
If you’re stuck in a place where you have to hire an affiliate, how can you protect the opt-in marketing program you’ve so painstakingly built? Nothing is foolproof, but there are some ways you can screen affiliates.
Who are they?
First step is to ask them for a bunch of information about their company.
- What is their full corporate information: company name, address, phone number and online URL.
- Where do people sign up for mail?
- What domains and IPs they use to send email?
- Do they use ESPs or manage their own servers?
- Will they contract out your send to other parties?
Trust but verify
Next step is to visit the websites they shared with you.
- Does their corporate site have any person’s name on it anywhere?
- Does the corporate site mention any of their brands? Again, if they’re hiding something why are they hiding it?
- Does the signup site link back to the parent company?
- Is there any information about the corporate structure on the signup site?
🚩When should you worry?
Signs that all may not be as it seems.
- When the vendor can’t or won’t tell you the websites where they collect email addresses.
- When you visit the website they told you about, but there isn’t a clear way to opt-in to any mail.
- When they won’t tell you what domains they use in email.
Any one of these things signals something might not be right. But any combination of them should set off alarm bells.
Other investigative routes
Check the company and your contacts through LinkedIn. Do they have a profile and if so, how does it match with what they’ve told you? And, really, what sales person doesn’t have a LinkedIn page?
Sign up for their mail. I suggest you don’t do it through your regular mailbox, setup a freemail account on each of the major services and use that. See what happens. Monitor them for a while. The mailbox I shared in my earlier affiliate post was almost 2 years after I first signed up at a job site. It took about 6 weeks to start getting stuff that wasn’t job offers. Then it took another few months before I started getting actual spam. For that mailbox I initially signed up June 6; the first unauthetnicated and non-job email showed up September 16 (Quick Loans eLoanPersonal). The address got a mix of requested mail and spam through October 6 and then the spam floodgates opened.
One of the biggest red flags is not telling you what domains and IPs they send from. If you sign up for their mail you’ll get it. I once had a customer tell me their brands, domains and IPs were proprietary information. That’s just silly. And it reeks of the sender being a spammer and not wanting you to know they are using botnets.
Ask them how they monitor for and deal with delivery problems.
These questions and investigative techniques aren’t fool proof. But they’ll open up a discussion with the vendor. I pointed out some of the red flags here, but the crux of the matter is this is a company you are hiring to do work for you. If they do it badly you’re not just wasting money, you’re risking having to clean up a deliverability mess. Can you trust this company to value your mail and your company reputation the same way you do? If the answer is no, maybe this isn’t the vendor for you.
Most retailers have realized that sending unsolicited email is bad for their overall deliverability. Still, the idea they can send mail to people who never heard of them is seductive.
Enter affiliate email. That magical place where companies hire an agency, or a contractor, or some other third party to send email advertising their new product. Their mail and company reputation is protected because they aren’t sending the messages. Even better, affiliates assure their customers that the mail is opt-in. I’m sure some of them even believe it.
The reality is a little different from what affiliates and their customers want to believe.
Affiliate marketing is sold as opt-in
It’s been a while since I’ve taken on affiliate mailers as clients, and I routinely turn down clients who tell me ahead of time they use affiliates. Sometimes, though I’ll take on a client who is having problems with their mail and discover that they use affiliates. “Oh, we probably should have mentioned we also have this affiliate program way over there, but that shouldn’t be why our opt-in and transactional mail is failing at Gmail.”
That’s when I pull out the Google Bulk Mail Senders Guidelines and point at the very bottom of the page.
In reality, using affiliates can affect all mail from a company. I’m not sure how Google does it, but their ability to draw connections between a company’s affiliate mail and their opt-in mail is pretty good. Senders using affiliates in the hopes of prospecting without affecting their “regular” mail discover this, eventually.
Affiliate marketing is kinda opt-in
In my experience most affiliate websites are not very user friendly. Going through signups seems designed to distract and confuse visitors into clicking on agreements. This isn’t just evident in the flashy website design, but the wording on many pages seems designed to confuse.
About a decade ago, one of the MTA vendors hired me to be their in house deliverability expert for some of their major clients. One of the clients they asked me to work with was an affiliate marketing company. They were attempting to “do things right.” And, in fact, they were confirming email addresses before mailing.
However, this company was also sharing data with third parties. One of those parties started sending email to me before the actual client sent me the opt-in request. When I mentioned this to the client, they explained that the company spamming was supposed to only send direct mail, not email. They couldn’t explain why they were passing on email addresses if their partner wasn’t supposed to mail them. 🤔
Affiliate marketing is overwhelming
In June 2016, one of my clients revealed they were collecting addresses through affiliates. They sent me to a few different websites to sign up for mail. I did. In the 22 months since I signed up, I’ve received a lot of mail.
A lot of mail.
Yes, those are actual email counts. I’m most intrigued by the addresses with only a couple emails, they appear to be truncated versions of some of the addresses I actually used to sign up. I’m not sure what kind of horrible data processing does that, but clearly there’s something truly broken out there mangling email addresses.
Not only did the sites mangle the addresses I gave them, most of the current messages aren’t even job related. Phishing, male enhancement drugs, dating scams they’re all in there. Even the one message offering job vacancies is a work from home scam.
Want to see what one of the emails looks like? I picked CVS/Drug Mall ! Expect Something Extra, Jane Doe
Not all affiliates…
I’m sure it’s not all affiliates. But 95% of affiliate marketers give the other 5% a bad name.
A bunch of folks reported problems with Microsoft’s SNDS page earlier today. This afternoon, our friendly Microsoft rep told the mailop mailing list that it should be fixed. If you see problems again, you can report it to mailop or your ESP and the message will get shared to the folks who can fix it.
The other big thing that happened today was Gmail rolled out their new inbox layout.
It’s… nice. I’ll be honest, I am not a big gmail user and have never been a huge fan. I got my first account way-back-during-the-beta. I used it to handle some of my mailing list mail. I could never work out how to get it to stop breaking threads by deciding to put some mail into the junk folder. I just gave up and went back to my shell with procmail (now sieve) scripts. I still have a couple lists routed to my gmail account, and the filtering is much improved – I can at least tell it to never bulk folder certain email.
The feature I’m really interested in is the confidential, expiring email. I’m interested in how that’s going to work with non-Gmail accounts. Within Gmail makes perfect sense, but I don’t think Gmail can control mail once it’s off their system.
My best guess is that Gmail will end up sending some type of secure link to recipients using non-Gmail mail servers. The message itself will stay inside Google and recipients will only be able to view mail through the web. That’s how the vast majority of secure mail systems work.
If anyone has the secure message already, feel free to send me a secure message. I’ll report back as to how it works.