One of the things I discovered yesterday while looking at Krebs on Security was that Google Alphabet has a program to provide hosting and dDOS protection for journalists. Project Shield, as it’s called, is a free service for approved applicants that keeps up websites that might be taken down otherwise. Eligible organizations include those providing news, information on human rights and monitoring elections.
This is something I hadn’t heard of before and my only reaction is good for Google.
Look, we’ve gotten to the point where attackers have resources beyond the scope that most of us can imagine. It’s expensive even for large organizations to manage and pay for the level of protection they need.
Even more importantly a lot of very important work is done by individuals or small organizations. Brian is a prime example of that. He does an incredible job investigating online crime on his own time. His site and his information is an invaluable resource for many. Losing his site, and losing his information would leave a huge hole in the security community. There are other folks in other spaces who, like Brian, don’t have the resources to protect themselves but do have important things to say and share.
I’m glad to see Google committing their resources and skills to help organizations protect themselves. It’s so important that this work is done and we don’t lose voices just because they can’t afford hundreds of thousands of dollars a year.
There has been abuse and harassment online for as long as I’ve been here. But it seems recently the size and severity of attacks have increased. And a lot of service providers are struggling with how to manage it and what their responsibilities are.
A few weeks ago Facebook deleted an iconic photo from the Vietnam era due to child nudity in the photo. That decision was reversed and discussed in many, many different places. One of the most interesting discussion happened on a friend’s Facebook feed. Many of the participants work at various online providers. They have to make these kinds of decisions and create policy to do the right thing – whatever the right thing is. It was very interesting to be able to follow the discussion and see how many different issues FB and other online providers have to consider when creating these types of policies.
I thing the thing I have to confront the most about the internet is how big it is. And how crucial it’s become to all sorts of issues. Social media can be a cesspool of abuse, there’s no question. But it can also be a force for good. I’m glad companies like Google are stepping up to preserve the good parts of the internet.
Cybersecurity has been on my mind lately. There is a lot of bad stuff going on, from giant dDOS attacks, to subscription bombing, to the ongoing low level harassment that some people have to deal with on a daily basis. I’ve written a lot about how I think marketers are going to have to step up and stop being a conduit for abuse. I do believe this. There are a lot of different issues to discuss but there are also many, many different stake holders in the issue of cybersecurity.
I’ve been on multiple calls with different groups over the last few weeks discussing the implications of the subscription attack and how it was carried out. The majority of my focus is email and how to protect senders from becoming a conduit for abuse. Other folks participating on the call are looking at what abuse is out there and how to stop it or minimize it.
One thing that came up on a recent call is that the bulk of dDOS traffic that took Brian Krebs’ website down was from various Internet of Things devices. Security cameras, DVD players, televisions, lightbulbs and other connected devices were part of the problem. It’s a huge issue, and one that cannot simply be mitigated by just ISPs and providers. But convincing individuals to secure their lightbulbs can be a challenge, we can’t even protect their computers completely. Convincing companies to stop providing default usernames and passwords or using the same keys for every device is another challenge.
These are big issues that we’re going to have to deal with.
Last night, with 100 million of my virtual friends and a small group of local ones, I watched the first Presidential debate. Part of the debate was about cyber security. To misquote Vice President Biden, “Cybersecurity is a big freaking deal.” We have nation states, and groups with the resources of nation states, conducting covert operations online. We have hacking, compromises, bonnets and other malicious activity occurring every, single day. And, the more complex the site and the more users it has the more likely it is to be compromised. Cybersecurity is a critical part of national security and our own individual security. We must take it seriously and we must address it.
Now, I’ll be honestI don’t think there is a solution to the problem. I think, though, that there are hundreds of things we can do as individuals, as companies, as nations, as volunteer organizations, as NGOs and as coalitions to solve different parts of the problem. We all need to think about what it is and who’s doing the bad stuff.
It’s common to think of hackers as lonely boys in basements who have too much time and too little to do. Back in the ancient days of the spam wars some folks referred to them as “chickenboners“: beer drinking rednecks who ate fried chicken and threw the bones on the floors of their trailers. The reality even then, though, was that many spammers ran businesses and made a lot of money. Admittedly, the descriptions of how the business was run are cringe inducing and full of illegal activity.
Now, much of the hacking is actually organized crime outside the US. This makes it hard to address successfully through legal channels.
It’s all very complicated. But I think we can agree security is a big deal. We are all part of the solution, by securing our sites and our personal devices. We’re also part of the solution by paying attention to the larger issues and events going on around us.
Al did a great post over on Spamresource about the how the new list unsubscribe function in the default mail client from iOS10. What’s been interesting to me is how much I’m hearing from ESP folks about how their customers want it gone.
If you don’t know what we’re talking about, in the default mail client on iOS10, Apple is now offering a way to unsubscribe from list mail by placing an unsubscribe link at the top of the message.
As you can see, this isn’t just for commercial mail, it’s in place for every mailing list that has a List-Unsubscribe header. (This is a screenshot from something I posted to OI this morning). For me, it’s somewhat intrusive. I’m on a lot of discussion lists – technical, marketing, business and even a couple social ones. Reading them on my phone has become a challenge, as every email in a thread contains the “unsubscribe” button now.
Luckily, you can dismiss the message for all posts to that mailing list by hitting the ⮾⮾⮾⮾x. Interestingly, once you’ve turned it off there seems to be no way to turn it back on for that list.
Senders have different complaints, however, they do not have to do with intrusiveness or usability issues.
I’ve heard complaints about placement and about how easy it makes it to unsubscribe. One person even stated that everyone knows the place for an unsubscribe is at the bottom of a message and it should never be at the top of a message. I find these arguments unpersuasive. Unsubscribing should be easy. Unsubscribing should be trivial. People should be able to stop getting mail on a whim. Particularly here in the US, where unsolicited mail is legal, being able to quickly opt-out is the only thing keeping some of our mailboxes useful.
I’ve also heard some concerns that are a little more understandable. One company was concerned that unsubscribes go directly to their ESP rather than directly to them. This is a somewhat more understandable concern. Good senders use unsubscribes as part of their KPIs and as part of their campaign metrics. They know how much an unsubscribe costs them and will use that as part of their metrics for defining a successful campaign. Still, though, it’s not that big a concern. ESPs are already handling these kinds of unsubscribes from providers like gmail and hotmail.
Almost 7 years ago I blogged about a sender who wanted an unsubscribe link in the email client. It was a bit of snark on my part. The interesting part, though, is that some senders want unsubscribe mediated in the client and others things it’s horrible. I think this tells me that there’s no universal right answer. It Depends might be the most hated statement in deliverability, but it is the absolutely the reality of the situation.
Global Suppression List.
Whatever you call it, it’s the list of email addresses you suppress from every mailing.
If you’re an ESP, this is the list of people who you never, ever want to send email to – and I’m talking about ESP-wide global suppression lists here, not the suppression lists maintained per-customer.
Global suppression lists are a vital tool to have, as it’s the only way you can comply with requests like “Never mail me again.” – and failing to comply with those will lead to, at best, irritation, yelling and blocking, and at worst legal action.
But it’s only the right tool for suppressing mail in a few cases. One obvious one is when someone specifically requests no more mail, ever, through your system. Another is when there’s a technical reason (you never want to send mail to autoresponders, for instance), or a legal reason (pending litigation, or an incompatibility between the mail you send and a specific jurisdiction).
And there are a very few people who just cause way too much support overhead when you send them email – that’s the origin of the term screamer list, I’m sure.
But it’s not what you should be reaching for in response to spam complaints, even heated ones, or feedback loop hits. A spam complaint is a sign that your customer is probably doing something wrong, and that this recipient doesn’t want that customers mail. A feedback loop hit says that this recipient doesn’t want that customers mail (and, statistically may indicate that your customer has a problem).
Neither of them is a sign that the recipient doesn’t want mail from any of your customers. You definitely wouldn’t want one of your customers sending spam to cause mail from all of your customers to be blocked – so why would you let a complaint about one of your customers block mail to that recipient from all your customers?
(We’ve occasionally come across ESPs who have preemptively blocked all mail to addresses @wordtothewise.com, for no clear reason. When our clients discover that their ESPs are silently discarding our attempts to subscribe to their mailing lists it doesn’t do much for that ESPs reputation in our clients’ eyes.)
And whatever you do, don’t respond to a spam complaint telling them you’ve added them to a global suppression list. That says several things, to an already annoyed person. It tells them that you’ve just broken their subscriptions, past or future, to your other customers. And by “fixing” the spam problem for this one recipent in this way it suggests that you’re not actually going to do anything to deal with the customer they’re complaining about. Nothing about this can end well.
Instead, tell them that you’ll make sure they don’t receive any further mail from that customer, and that you’ll talk with the customer and take action that you deem appropriate. (And then do that).
P.S. Does anyone know the origin or etymology of the term “pander file”?
I’ve been talking about security more on the blog. A lot of that is because the security issues are directly affecting many senders. The biggest effect recently has been on companies ending up on the SBL because their signup forms were the target of a subscription attack. But there are other things affecting online spaces that are security related. Right now not much of it is affecting email senders, but it’s good to be aware of.
There has been an increase in DDOS attacks against different companies and network. Some of the online game sites have been targeted including EA, Blizzard and others. A group called PoodleCorp is claiming responsibility for those attacks.
Another set of DDOS attacks hit Brian Krebs’ website this week. The site stayed up, but Akamai has told Brian they can no longer host his website. His website is down for now and the foreseeable future.
While this activity doesn’t affect marketers directly, it does tell us that there is active development happening on the less legal side of the internet. The volumes of the recent attacks have sent records. They’re also changing in scope and including new kinds of traffic in an effort to knock sites offline. Even more concerning, they appear to be systematically attempting to discover defenses in order to attack the internet as a whole.
Increase in Spam
Spam has been on the decrease over the last few years. Many of us were treating it as a mostly-solved problem. But a new report from Cisco Talos shows that trend is reversing and spam levels are increasing. Current levels are approaching those last seen more than 5 years ago. Cisco Talos has used a number of different sources of data, all showing an increase in spam directly and indirectly.
CBL Volumes over past 10 years:
Cisco Talos also looks at the number of IP addresses in the Spamcop blocklist as a proxy for the amount of spam sent. Average numbers of listed IPs have doubled over pre-2106 levels.
According to the author, this rise is mostly attributed to the Necurs botnet. This botnet is a little different than most, in that it only uses a small subset of infected machines for each spam run. It sends some mail, and then the bot goes quiet.
While this doesn’t affect marketers directly, it does mean that spam filters will be under even more active development. I’ve actually seen some of this increase in activity myself. For me, the addresses hit hardest are the ones stolen from ESPs and retailers over the years.
ISPs being compromised
This week Yahoo announced that over 500 million accounts were compromised. Account owners are being alerted to update their passwords when they log in. Yahoo also cautions that actual Yahoo mail will have a special badge when viewed in the Yahoo web client and the smartphone applications.
The icon is a small purple Y next to the from address in the inbox:
And in the message itself:
Of concern is that Yahoo has attributed the hack to state sponsored actors. On the surface it’s hard to believe that a government would care about getting into people’s Yahoo mail. But, as Yahoo and other mail providers are used worldwide, they may be looking for access to certain accounts and it’s easier to take all of them or some of them. Yahoo has set up a website for customers concerned about the compromise and to answer common questions.
For marketers this isn’t necessarily a direct concern. However, companies that tie account access to email addresses need to address the security of those accounts. What happens when the email address is compromised? How easy is it for someone to get into your system if they own someone else’s email address? Can they find credit card numbers and other PII?
Well, we don’t really have a what’s next. But security is a major issue online and with the active development of new tools everyone online needs to start prioritizing security. What are your defenses? What happens when you’re compromised? What can you do? Who do you call? These discussions need to happen and they need to happen sooner or later.
A lot of senders get frustrated with the time it can take to get a response from some ISPs. It’s totally understandable, for a lot of companies delivery problems are all hands on deck level problems. They want them fixed and they want them fixed IMMEDIATELY. They want feedback that their issue is being addressed. They want to know someone at the ISP knows there is a problem.
I’ve talked before about visiting my friend Anna and watching her laptop screen explode with IMs from senders who wanted help with an AOL issue. She’s awesome and conscientious and tried to address all of those issues as fast as she could. She did want senders to feel like their issues were important and that someone inside AOL cared about the mail blocks.
I was always a strong advocate for following the official pathways for addressing problems. That was the whole point of the 2009 blog post. These days it’s easier to do than it ever was. Many ISPs have forms and process around handling delivery issues. This is good! In the past getting an answer to “why is my mail blocked” required knowing the right people. Now, it’s not about who you know. The ISPs and filtering companies who are open to senders have postmaster pages, unblock forms and official request channels. Those that don’t have those channels have made certain business decisions to not provide support for senders.
Despite the availability of webforms and knowledge bases and detailed information, a lot of people still think that the only way to get attention or get an issue addressed is to get someone on the phone. It’s not, though.
ISPs have their processes. If you want things handled quickly use those processes. Even in the places where very helpful reps are, they can’t (on order of lawyers and executives) help people unless there is a ticket already open.
Always, always use the recommended processes before trying to find “a real person.” Most of the time your issue can be solved faster if you fill out the form than if you hunt around for a person. In the worst case, all that time will be wasted as the person in question will tell you to fill out the form.
Next month I’ll be in London for the Email Innovations Summit. This will be an updated version of what you need to know to talk with technical folks.
In early December I’ll be doing a DMA webinar discussing the subscription bombings. That’s still in the works.
I’m looking at some events for next year. I am planning on being at M3AAWG in San Francisco in February.
I’m looking at others, too. What are your favorite events?
Last week Spamhaus posted information on the ongoing subscription attacks. They provided a more information about them that was not make public previously, including some information about the volume of mail some targets received.
Today SendGrid also blogged about this, going into a little more detail about why senders should care about this. They also provided a number of suggestions for how to mitigate the risk of being part of an attack.
There are a couple of things I think it’s important for folks to realize.
This is the new normal
As Spamhaus states, there is some evidence that this may have been a test run for a new product selling mailbombing as a service. Even if it’s not, although I do agree with their assessment, this is something we need to address. Many online companies are struggling with how to stop being a conduit for abuse and harassment. These issues aren’t easy, but they’re there and we have to address them.
Spamhaus saw a direct attack yesterday and a number of ESPs woke up to new SBL listings this morning.
The damage is ongoing
ESPs and other relevant parties have stepped up to the plate to minimize the effect on victims. Despite this there are many addresses still receiving email at significant volumes. Certainly it’s not the hundreds per minute but addresses are permanently affected by this kind of abuse. Because of the targets, including WordPress installations, much of the mail isn’t coming through traditional ESPs.
This diverse sources make it difficult to block the mail, in the short term and the long term.
This is not about spam
This isn’t just about marketing mail. Again, a lot of the conduits for abuse are WordPress forms. Some of the conduits are online alert services. This is about online services being used as tools for harassment.
We need new tools
The problem with spam is a lot of people suffer a little bit of damage. This means most tools use volume of complaints as a primary metric. But with direct harassment like this, it’s a lot of damage for a small number of people. Until Spamhaus started listing ESPs, no one knew it was happening. This includes the ESP that sent 81,000 confirmation emails to 9 email addresses over the course of 2 weeks.
We need new strategies
COI isn’t a great solution for this. In fact, the 81,000 emails were all COI requests. Captchas are not idea for a number of reasons, including discouraging signups from actual customers. We, as an industry, are going to have to think of ways to fix this. Yes, right now COI and captcha are the only solutions we have. But that doesn’t mean they are the only solutions, they’re just the stop gap. I don’t think it’s a huge secret that I don’t like the subscription validation companies very much, but they have the opportunity here to really stop this kind of abuse. No, their current SMTP tickling and delivery testing isn’t going to catch this (and, in fact, will cause problems for smaller targets), but there are other strategies they can create to address this.
Overall, this is something that needs to be addressed to prevent significant damage to individuals. Subscription forms need to be secured better and high volume senders need to pay attention to their address lists. One thing that was discovered is that this is not new. Some ESPs found a single address on thousands of their lists added over months. Low level abuse was happening, we didn’t see it because we weren’t looking. Now, we know it’s there and we must act to fix it.
Spamhaus released a blog post today discussing the recent subscription bombing: Subscription bombing COI captcha and the next generation of mail bombs.
As I mentioned in my initial posts, this abusive behavior goes beyond spamming. This is using email to harass individuals. Spamhaus even mentions a potential service that can be used to do these kinds of mailbombing.
Things folks need to know is that this is not just about ESPs and commercial mail. One of the big targets was WordPress admin forms. As Spamhaus says:
[T]he onus of stopping this kind of attack is not only on ESPs or mailing list owners. It is on everyone that has any sort of web-based signup that results in an email being sent: somebody clearly spent a great deal of time assembling URLs of mailing lists, and of account sign up pages, and has written a script to submit addresses to them at speed. We suspect that this was a test run for a tool that will will soon be offered for sale in the ‘Underground Economy’: Mail-bombing as a Service – MaaS.
With more and more abuse happening, every one who runs a service online needs to be cognizant of the abuse potential. Moreover even paths that have been around and haven’t been exploited may be exploited in the future.
We need to protect ourselves by making services that are difficult, if not impossible, to use as abuse vectors.
There’s been extensive and ongoing development of email through the years, but much of it has been behind the scenes. We were focused on the technology and safety and robustness of the channel. We’re not done yet, but things are much better than they were.
The good part of that is there is some space to make improvements to the inbox as well. Over the last few months there have been a number of announcements from different mail client providers about how they’re updating their mail client.
Unsubscribes handled by the email client
Apple announced they were adding a link to unsubscribe into the mail client for iOS 10. It works much like the links in the Gmail and Hotmail clients, by looking for the list unsubscribe header and then sending a message to that address. Al did a bunch of testing and has a full blog post on how the list unsub link works in iOS, so go check out his post.
The important bits are they’re only using the mailto: link, they are not following any URLs. For those of you who want to support this, you’ll need to provide an address for unsubscribes. One of the absolute easiest ways to do this is use an encoded left hand side of the address so each unsubscribe can be processed based on the email address. Think of it like a VERP string.
It’s worth noting that two large players, Apple and Google, in the email client space have focused on the mailto: link for unsubscribes. There are issues that come up with an email client mediated unsubscribes, but a number of them go away using a http:// level unsubscribe.
It’s also interesting that this innovation has created a discussion among some marketers about where the unsubscribe link should be in an email. Some people feel very strongly that the only right place to put an unsubscribe link is at the bottom of an email. That choice is being somewhat removed from their hands with these changes to the mail client.
Better CSS and HTML support
Last month Microsoft went to the Litmus design conference and announced they were going to be working with Litmus and email senders to improve mail display in Outlook. This week Gmail announced they were supporting more CSS to make responsive design easier.
Displaying security information to end users
Earlier this year, Gmail started showing their users if mail came in over an encrypted connection. Mail sent without using TLS received an open red padlock next to the sender’s name.
This week folks noticed Gmail had quietly rolled out another feature to communicate security status to end users. Now, if you click on “show original” Gmail doesn’t simply show you a raw text version of the file, they show you specific authentication information about that message.
I have a screen shot of what that information looks like.
There are multiple features here that make it easier to see what’s going on with email.
- How long the delivery took! This is great, because there are so many places email can get caught up. This will tell senders wether the problem is on the sending side or the receiving side. Looking at the headers of this particular message, the time is looking at how long the message took to get from the Gmail MX to the user’s inbox (or, in this case, spam folder).
- SPF pass. The learn more link is a little disappointing, as it mostly talks about how you can implement SPF, not about what it means for recipients. It also says it helps recipients distinguish spam, except this particular message is a classic 419 spam. But it’s a good start.
- DMARC pass. Again, there isn’t much information about why a user should care about DMARC passing in the learn more link, but it is a good start.
Overall, these are exciting developments for recipients and senders. It’s really nice to see some work being done at making mail clients more descriptive. Because so much online security revolves around email, it’s a critical security step to show authentication results to end users. I expect some of these changes will be pushed out to the inbox over time, as Gmail wrestles with providing enough but not too much information.
All in all, these are more meaningful changes to email clients than I’ve seen in years.