One challenge when implementing DMARC is to ensure that all mail, and I do mean ALL mail is authenticated correctly, before switching to a p=reject notice. The easiest way to do this is to set up a p=none record and check reports to see what mail isn’t authenticated. At least some of this mail is actually going to be valid but unauthenticated email.
I regularly recommend monitoring for 6 – 12 months in order to catch some irregular emails. Even then, someone should regularly monitor DMARC reports in order to identify systems that need authentication added.
One of the cases I worry about is system monitoring emails. These are emails intended to notify sys admins about problems and errors. They often don’t go through the main SMTP server. They usually don’t have an external facing IP and there are security arguments against putting internal IPs into external SPF records. These emails are important and are, usually, not authenticated.
Overall, I could imagine cases where a DMARC record would lead to some problems. And, well, it can. Reading through the postmortem of a significant system failure, one of the problems was no one knew backups weren’t running because notification emails were failing DMARC.
While notifications are enabled for any cronjobs that error, these notifications are sent by email. For GitLab.com we use DMARC. Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late. GPostmortem of database outage of January 31
Backup failures happen. System outages happen. Crashes happen. (We once had a system crash and take out a disk while during the installation of backup software. #badthing). DMARC failures of this type are preventable.
Part of the challenge here is that gitlab is using google to host their domain and email. That give gitlab a little less control over handling DMARC rejections. Gmail applies the public policy to email. Internal systems sending mail to a SMTP server you control can be whitelisted and exempted from DMARC policy. But using a hosted system means giving up some overhead, and some flexibility.
Of course, DMARC reporting should show internal systems trying to send mail. But I don’t think most places are actually reading their DMARC reports. If anyone at Gitlab was, they didn’t understand the significance of these emails.
DMARC is, overall, a useful protocol and gives senders a little more control over their email. Sometimes, though, there are unexpected consequences. Like most things in email, you can’t set and forget. You have to monitor your DMARC reports and pay attention to systems sending legit but unauthenticated email.
Earlier this month I wrote about how we can’t trust Equifax with our personal data. I’m not sure we can trust them with a cotton ball. Today, we discover Equifax has been sending consumers worried about their personal information leaking to the wrong site.
[O]n multiple occasions over the span of weeks, the company’s official Twitter account responded to customer inquiries by apparently directing them to a fake phishing site called www.securityequifax2017.com.
Luckily, the fake site — blocked or flagged by many Internet browsers, then taken down Wednesday afternoon — was set up by software engineer Nick Sweeting to educate people rather than steal their information. A banner on the top read: “Cybersecurity Incident & Important Consumer Information Which Is Totally Fake, Why Did Equifax Use A Domain That’s So Easily Impersonated By Phishing Sites?” NPR
There’s been quite a bit of breakage and delivery failure to various Microsoft domains this month. It started with them changing the MX for hotmail.co.uk, then the MX for hotmail.fr… and both these things seem to have broken mail. I also saw a report this morning that some of the new MXs have TLS certificates that don’t match the hostnames.
What’s going on?
Historically, Microsoft had two completely separate commercial mail systems, their free webmail and a hosted enterprise solution. The free mail systems started when MS purchased Hotmail, evolved into Live and then eventually Outlook. The enterprise solution is currently Office365. It, too, started off when MS purchased a filtering company and then evolved it into Office365.
A few years ago Microsoft decided they no longer wanted two email systems. So they were going to migrate both to the same back end.
That was 2 years ago…
Yes. Much of the backend stuff has been moving around behind the scenes over the last 2 years. Recently, though, they’ve been deploying new MX records for some of the regional Hotmail domains.
Why should we care?
Well, all things being equal we shouldn’t. But, y’know, nothing about email can ever be easy. On Sept. 1, 2017 a number of people started reporting that all their mail to hotmail.co.uk was bouncing with user unknown messages. Even addresses that were known live. That’s when some folks noticed that the MX records had updated from mx[1-4].hotmail.com to *.olc.protection.outlook.com.
Based on some of the discussions surrounding the change, Hotmail intended to make this change quietly and let the old and new MXs run in parallel. But, spammers had to ruin it. There were so many senders that didn’t update their MX records MS decided to set the old mail servers to reject all mail to hotmail.co.uk. This broke a bunch of stuff with senders, particularly for those who cache MX records for a very long time.
They fixed it.
It appears most senders / ESPs got the message and updated their MX records. MS, to their credit, realized this wasn’t an ideal situation and adapted their transition process. It appears hotmail.fr is the next regional domain to move to the new MX. Instead of swapping out all of the records, MS has added *.olc.protection.outlook.com to the MX records for hotmail.fr. Mail to hotmail.fr is being handled by both the old mx[1-4].hotmail.com and the new *.olc.protection.outlook.com servers. I expect that they’ll phase out the old ones at some point.
Of course, the only reason I know the changes have happened at hotmail.fr is because I’ve seen some folks mentioning problems with sending to those domains. So not everything is wonderful here. I don’t have many details on what the symptoms are, but if you’re seeing some level of problems to hotmail.fr it’s possible it’s them and not you.
There was also a report on the mailop list today that there is an issue with the TLS certificates on the new MX domains – the server names don’t match. This is minor, but may cause TLS to fail until they fix it.
Microsoft is undertaking a major infrastructure upgrade, without taking down any services. It’s not going to happen as smoothly or as transparently as anyone wants it to. The issues we’ve seen so far are all pretty minor given the scope of the project. If we can get through the rest of this changeover with this level of hiccups we should all be thankful.
Do be aware that things might change and delivery might be erratic until the changeover is finished. Just keep sending good mail and give them the space to work things out on their end. Even though in the short term this might drive up the consumption of antacids, I think long term, this is going to be a win.
I was chatting with folks over on one of the email slack channels today. The discussion was about an ESP not wanting to implement a particular change as it would hurt deliverability. It led me down a path of thinking about how we think of deliverability and how that informs how we approach email.
The biggest problem I see is the black and white thinking.
There’s an underlying belief in the deliverability, receiving, and filtering communities that the only way to affect sending behavior is to block (or threaten to block) mail.
This was true back in the ancient times (the late 90’s). We didn’t have sophisticated tools and fast CPUs. There weren’t a lot of ways to handle bad mail other than to block. Now the landscape is different. We have many more tools and the computing capacity to quickly sort large streams of data.
At most places these days, blocking is an escalation, not a warning shot. Many places rate limit and bulk folder questionable mail as a first strike against problem mail. Sometimes the mail is bad enough to result in a block. Other times, it’s not bad enough to block, so it disappears into the bulk folder.
There’s a corresponding belief in the sending community that if their behavior doesn’t result in blocking then they’re acting acceptably. This isn’t true either. There are a lot of things you can do (or not do) that don’t help delivery, but will actively harm delivery. Likewise, there are things you can do that don’t actively harm delivery, but will help. All of these things add up to reaching the inbox.
It’s all shades of grey
Things don’t happen on a short timeline. Bad email behavior might take days, weeks or even months to affect things.
Take the situation where a company buys a list and starts sending spam. On one side we have anti-spammers on the other we have marketers.
Anti-spammers often react to the spam to a purchased list with bluster and threats. This behavior is illegal! It’s against CAN SPAM! All your mail will be blocked! Your ESP will throw you off! Yeah, no. That’s not how it works back here in the real world. Purchasing lists is not illegal under CAN SPAM. Blocking might happen, but unless it’s a filthy list it’s not likely to happen the first time. Many ESPs will work with senders, requiring them to remove the purchased list but allowing them to continue mailing their regular lists. Eventually, people start ignoring the boy who cried spam.
It’s not like senders are all honest about the effects of purchased lists, either. Talk to a few marketers and they will swear blind purchased lists are great. They’re kinda right, in the short term under very specific conditions. If the sender starts out with a good reputation, makes smart purchases, and pays attention they’ll often get away with purchasing lists for a while. Eventually, though, delivery problems show up. But because the problems take so long to show up, they refuse to believe purchasing lists led to their mail going to bulk.
In both cases we have folks who aren’t looking at the whole picture. Thus, you end up with delivery folks who simplify their message to “it will hurt deliverability” when the reality is a lot more complex. Sometimes senders have a bad ideas that shouldn’t be done, even though they won’t hurt deliverability. But they won’t listen to any advice containing more realistic consequences.
There is a meme going around related to the Equifax hack that points out an executive in charge of security doesn’t have a degree related to security.
Surprise! A lot of the folks who currently keep us safe on the internet don’t have degrees in security. They just didn’t exist when we were in school. I think Paul summed it up best:
[T]alking about Susan Mauldin’s music degree is a socially acceptable way for men (and they’re almost all men) to vent about a woman who they don’t feel belongs in their workplace – especially not in a senior role. That truth is simply unavoidable.
Paul’s article over on Security Ledger is well worth a read looking at security professionals and what their credentials are. Also, a summary of the discussions happening in various online fora about her and the breach.
On my Facebook feed, there have been a lot of discussions. It’s interesting because many of my friends are experts in security and/or internet technology. Some have degrees in relevant studies, but a lot are self taught. They are the embodiment of Chris Robert’s quote in Paul’s article.
“So many of us in security have worked our way in and clawed our way up and we stand on the experience that we have and build on the experience of others,” noted security expert Chris Roberts (@sidragon1) told [Paul]. “This realm we’ve created over the last 20+ years has only recently lent itself to certification and most of us have the scars and bruises from so many years of experience which arguably counts for as much if not more in some cases.”
Anti-abuse and deliverability are even newer field than security and they don’t have much in the way of certification, either. But most of us working in the field do have the scars and bruises from experience.
We are living in the future. Those of us who are creating the future are doing the best we can. Sometimes that means we have a degree in music. This doesn’t make us unqualified.
A friend posted a link in IRC pointing at a couch at Wayfair.com. Now I have Wayfair.com ads following me around the internet.
ProPublica wrote an article about how Facebook lets advertisers micro target “jew haters” and other hate groups.
I received this postcard in the mail.
Hello! Hope all are keeping safe through Harvey, Irma, Katia and the aftermath. I know many people that have been affected and are currently out of their homes. I am proud to see so many of my fellow deliverability folks are helping our displaced colleagues with resources, places to stay and money to replace damaged property.
Here’s a mid-month late wrapup of our August blog posts. Our favorite part of August? The total eclipse, which was absolutely amazing. Let me show you some pictures.
Ok, back to email.
We’re proud of the enormous milestone we marked this month: ten years of near-daily posts to our Word to the Wise blog. Thanks for all of your attention and feedback over the past decade!
In other industry news, I pointed to some interesting findings from the Litmus report on the State of Email Deliverability, which is always a terrific resource.
I also wrote about the evolution of filters at web-based email providers, and noted that Gmail’s different approach may well be because it entered the market later than other providers.
In spam, spoofing, and other abuse-related news, I posted about how easy it is for someone to spoof a sender’s identity, even without any technical hacks. This recent incident with several members of the US presidential administration should remind us all to be more careful with making sure we pay attention to where messages come from. How else can you tell that someone might not be wholly legitimate and above-board? I talked about some of what I look at when I get a call from a prospective customer as well as some of the delightful conversations I’ve had with spammers over the years.
In the security arena, Steve noted the ongoing shift to TLS and Google’s announcement that they will label text and email form fields on pages without TLS as “NOT SECURE”. What is TLS, you ask? Steve answers all your questions in a comprehensive post about Transport Layer Security and Certificate Authority Authorization records.
Also worth reading, and not just for the picture of Paddington Bear: Steve’s extremely detailed post about local-part semantics, the chunk of information before the at sign in an email address. How do you choose your email addresses (assuming they are not assigned to you at work or school…)? An email address is an identity, both culturally and for security purposes.
In subscription best practices — or the lack thereof — Steve talked about what happens when someone doesn’t quite complete a user registration. Should you send them a reminder to finish their registration? Of course! Should you keep sending those reminders for 16 months after they’ve stopped engaging with you? THE SURPRISING ANSWER! (Ok, you know us. It wasn’t that surprising.)
There are widespread reports this morning (9/11/17) that Google postmaster tools is showing bad IP reputation for IPs starting on 9/9. This issue is affecting just about everyone. Looking through my client’s postmaster pages, I’m seeing red for IP reputation on every client. Even my clients with generally good reputation are seeing bad reputation since 9/9.
This looks like a reporting or a display error on the part of Google. Many people who are reporting the bad IP reputation are not seeing any significant change in Gmail deliverability.
Looking through client data it appears that domain reputation reporting stopped on 9/8. I am seeing FBL reports for 9/9 and 9/10, for some but not all clients.
My current read on the situation is that something broke internally with the Gmail postmaster reporting. This does not currently appear to be affecting delivery of mail. (If anyone sees differently, drop me an email or tweet me @wise_laura).
I know folks are making sure Google knows. I know that some Gmail folks were directly notified and another Google person is active on Mailop. And we have confirmation that they are aware and are working on fixing it. I will let you know if I hear of a fix timeline.
EDIT: It’s been fixed. Google even fixed the older data. Same client, screenshot from this morning.
This popped up on my Facebook memories this morning. I don’t post about client events very often, but given I can’t remember even what client this is, I don’t think I’m revealing too much info.
FB memory from a few years ago.
I’m dealing with a client who has a pretty big SBL listing. They’re going through a lot of contortions to fix the problems the SBL is pointing out, but I’m concerned that the SBL listings aren’t the complete story.
I sent them this in an email this morning. I’ve tried to tell them during our calls, but they keep mis-hearing or mis-interpreting what I’m saying. So I write down some speculations and send them by email.
Then a friendly, neighborhood SBL person contacts me and asks me a question about something totally unrelated. So I get about 4 sentences into my question and he says “Imma gonna let you finish, but let me tell you <exactly what I told the client might be going on behind the scenes>.
Yeah. Sometimes I get it right.
Today it was announced that someone infiltrated Equifax earlier this year and stole 143,000,000 identities. These identities include names, birthdates, and addresses, at a minimum. Details are available at your favorite news site.
What I want to talk about is the website they’ve put up to address the issue. This website is Yet Another Example of how the financial services industry trains users to be phishing victims.
Equifax set up a website for people concerned about the possibility of identity theft after this major data leak. The URL, as distributed by the press and linked to from Equifax’s own website is https://www.equifaxsecurity2017.com.
When I was first sent to the site, I thought it was a phishing site because there is absolutely no way to confirm this site is owned and managed by Equifax. Zero. In fact, there’s a lot of evidence that the site isn’t owned by Equifax. And most of the rest of the evidence relies on trusting that the hackers still don’t have some level of access to Equifax systems.
Visiting the Website
The site does have SSL, but the certificate is issued to “ssl511860.cloudflaressl.com”. Cloudlfare has a somewhat sketchy history and I would not put it past them to host and protect a phishing site. So the SSL certificate doesn’t give me any faith this is Equifax.
There’s a link back to Equifax’s pages, but that’s SOP for phishing, so no help there. Yes, I can get to the website from Equifax’s home page, but this URL is being given out in various press articles.
When we check the whois. The registrar is MarkMonitor but the registrant is some random company called DNStinations.
Registrant Name: Domain Administrator
Registrant Organization: DNStination Inc.
Registrant Street: 3450 Sacramento Street, Suite 405
Registrant City: San Francisco
Registrant State/Province: CA
Registrant Postal Code: 94118
Registrant Country: US
Registrant Phone: +1.4155319335
Registrant Phone Ext:
Registrant Fax: +1.4155319336
Registrant Fax Ext:
Registrant Email: firstname.lastname@example.org
Registry Admin ID:
When I visit http://dnstinations.com/ I get an error message:
The domain owner has no website. That doesn’t reassure me.
Now, when I drop the physical address into Google I get an address that’s listed as a Proxy Partner on the MarkMonitor website. At this point, I start to wonder if my faith in MarkMonitor is misplaced.
Of the things we can check on the website, the best assurance we have here is that MarkMonitor is a service that big companies use to protect their domains. Not that reassuring overall, but we’ll go with it.
ID Protection Services
We’ll assume the press has done some level of fact checking and this website is valid. We click on the link that says “enroll in ID protection” which takes us to a different website: https://trustedidpremier.com. The first thing it asks me for is my name and the last 6 digits of my social security number.
What evidence do I have that this website is valid? It has a SSL cert issued by Amazon. It’s hosted on EC2. The whois data is all behind privacy protection. The bare domain serves up an error message.
This is the website that they want me to trust to protect my identity?
It will work
The thing that really gets me is that most people are going to drop their information into this website. They’re going to go through whatever process is going on.
This is training for people to be phished. I’ve actually seen better phishing websites than this one. In fact, I’m more inclined to believe this is legitimate because phishing sites are done better!
Now, I’ve long believed that all my information is compromised and is out there for sale. I know some of it is. Two universities that had my info have lost it. I’ve had my credit card company send me new cards “for security purposes”. I shopped at Home Depot during their compromise. None of that is any of my fault, which is why I’m careful but not obsessively careful with personal info. I can’t control if it stays private. That’s clear.
But acknowledging that I can’t control my personal info doesn’t mean I am going to go around and drop my name and most digits of my SS number into some website as horribly set up as this one.
Equifax dropped the ball here, not just by their lax security but also by their insecure response to the compromise. (And that’s before we get to the fact that 3 senior Equifax executives sold 1.8 million dollars worth of stock after they found the compromise but before they announced it.) This isn’t unique to Equifax, Target dropped the ball, too, when they were compromised.
I was on a call today discussing DMARC. It’s great that email folks are stepping up to make email more trustworthy. But when a company pulls such a boneheaded move, in response to a major leak of data, I know that DMARC and email isn’t enough. We need to work towards a safer internet and that means companies have to step up and stop being stupid about cybersecurity.