BLOG

Industry News & Analysis

Hitting the ground running

We’ve landed in Dublin and are back at work. Blogging will pick up as I get back into the swing of things.

I’ll be speaking on a panel at the Selligent user conference in Amsterdam tomorrow and in London on Thursday. If you’re a Selligent customer, introduce yourself and say hi!

Speaking of being on panels, I heard recently that some folks were adding conference speakers to newsletter and marketing lists. The scenario was something like this. Person goes to a conference and sees speakers talking about things they’re very interested in. They return to the office and dig around to find email addresses of the speakers. Once they find the email addresses, they add them to the company mailing list.

As a speaker, let me tell you, this is a bad idea. All of us are thrilled you found our talk inspiring, interesting and worth following up on. Follow up with us, absolutely. Don’t add us to your mailing list. Send us an email, introduce yourself, tell us who you and your company are. All of that is great. Love to hear from you, love to hear about the interesting things your doing and how our talks have made you think about your program. Don’t just throw us on your newsletter list.

First off, think of the numbers. Small venues might be a dozen or so people. Larger venues can be in the hundreds. If this catches on, we’ll be swamped with mail.

Secondly, we don’t always know who is in the audience. Adding us to a mailing list without permission just looks like another bit of spam. We don’t know that you added us because you were in the audience. All we know is that your company has started mailing without permission.

Send us a direct email telling us about your company and how our talk impacted you and the company? Absolutely.

Invite us to opt-in to your newsletter so we can follow your company development? Sure. We might not actually do it, but the invite is OK.

Add us to your company newsletter with no warning? Don’t do that. It’s not just rude, it’s spamming.

No Comments

Changes are coming…

We’ve been blogging here about email for 11 years now. My first post was published August 29, 2007. In that time, we’ve published more than 2300 posts, and written probably millions of words. For years we have blogged multiple times a week.

This summer we’ve not kept up our normal posting schedule. We’ve been a little busy with non-email stuff. We’ve spent this summer planning, purging, and packing for a big move. We’re almost finished, and early next month we’re getting on a plane with our cats and moving to Dublin, IE.

Picture of a church on a hill in Dublin, Ireland

We’re both excited about the move, and looking forward to living in a new place and meeting new friends.

Most of the changes are personal, but we’ve also been thinking about what business changes we’d like to implement. I have some exciting thoughts about the next phase of Word to the Wise, too. I am looking forward to sharing them with you in the coming months.

One of the big business changes is that we decommissioned our cabinet and have moved all our services into the cloud. We’ve run services on our own hardware since before we were Word to the Wise. Starting with a server sitting in the closet in a Boston apartment, then moving up to donated colo for SamSpade.org/. We’ve had a full cabinet for close to 15 years, first at Integra, then later at Hurricane Electric.

A few weekends ago we pulled the last of our hardware from HE. After we left the support folks sent us this picture of our servers acting as cat bed for the colo cat. 

Cat sitting on servers

Happily, this is all the e-waste we generated from all of the hardware gear we were using. A small amount is going with us to Dublin. But the bulk of our hardware, including some from ‘inside the firewall’ (i.e., the house) was donated to the Network Time Foundation.

 

4 Comments

Check your abuse addresses

Even if you have excellent policies and an effective, empowered enforcement team you can still have technical problems that can cause you to drop abuse mail, and so lose the opportunity to get a bad actor off your network before they damage your reputation further.

It’s not quite as simple as “We’re seeing email in our abuse ticketing system, so everything must be fine.”

How many abuse addresses have you published? Do all of them work? Them being deliverable but not routing to your compliance desk may be worse than them bouncing.

What abuse addresses might people expect to exist? Do they exist and deliver to the right place?

And do any of your abuse addresses discard mail about spam? It seems ridiculous that they would, but a surprising number of abuse addresses are behind the same content filters as the other email addresses in a domain, meaning that reports of spam get content-filtered.

What abuse@ addresses might a reasonable person expect to work?

  • abuse@ your main domain; the one you advertise and where your website is
  • Any abuse address listed in the OrgAbuseEmail (or the relevant point of contact on the web search form) field when you look up any of your network space at ARIN, or the abuse-mailbox field when you look them up at RIPE, or the equivalent at whichever regional registry you get your IP addresses from. (“whois -h whois.arin.net 10.11.12.13” will let you query american addresses at ARIN from a non-Windows commandline; use whois.ripe.net for europe, whois.apnic.net for asia-pacific, whois.afrinic.net for africa and whois.lacnic.net for latin america)
  • If you’re a RIPE customer, then the result of “whois -h whois.ripe.net 10.11.12.13” might include a line that looks like “% Abuse contact for ‘10.216.128.0 – 10.216.159.255’ is ‘abuse@example.uk'” – that email address should work (and if it’s not the same as your abuse-mailbox field you might ask why).
  • Many ESPs use domains for their infrastructure other than their main domain. If a hostname of yours appears in the Received headers of email you send (“mta24.lhr.example.co.uk”) then abuse@ the base domain (“abuse@example.co.uk”) should work. Making  abuse@mta24.lhr.example.co.uk deliverable (by creative use of a wildcard MX record at *.lhr.example.co.uk, perhaps) I’d be impressed by your infrastructure chops, but it’d be pretty pointless – anyone sending you a useful report will understand the concept of an organizational domain.
  • The base domain of the right hand side of your bounce address should provide a working abuse address – e.g. “Return-Path: bounces@bounce.example.de” means that abuse@example.de should work
  • abuse@ the base domain of your clickthrough domain should work
  • abuse@ the base domain of your image hosting domain should work
  • If you’re using custom domains for customers, then you should at least know where abuse@ those domains delivers to for those domains
  • If you mention a contact address in your acceptable use policy, privacy policy, affiliates policy or anything similar then not only should that email address (“privacy@example.ie”) work but so should abuse@ the same domain (“abuse@example.ie”)
  • The email addresses listed for your domains at abuse.net (there’s a web lookup form and you can also query it via whois or dns)
  • Any email address you use in the From field of mail you send as replies from your abuse desk should work too.

Some of these may seem a bit obscure, but some of the most immediately useful abuse reports sometimes come in via routes that aren’t easily connected to your main corporate domain.

As one example, it’s not unheard of for a spammer to use a demo account to send themselves a single copy of their spam, created in your message composition tool and using your image and clickthrough domains, and then blast that out via compromised machines or a throwaway server. This damages the domain reputation of domains that are common across most of the mail you send, damaging delivery for all your customers. If you spot it quickly and kill the images and redirectors they’ll go elsewhere next time. And you’re much more likely to be able to do that if a civic-minded recipient wants to tell you it’s happening – and can do so based on the links in the body of the message.

And it’s a contractual requirement to provide valid contact information when registering IP addresses, so any abuse aliases listed with ARIN or RIPE should be deliverable or the registration may be flagged as having invalid contact information. That doesn’t mean any action is likely to be taken, other than adding a note to your address registration saying something like “ARIN has attempted to validate the data for this POC, but has received no response from the POC since 2018-02-17”, but it might make for an awkward conversation next time you’re trying to acquire new network space, I guess?

What does “work” mean, when it comes to an abuse@ alias? Obviously, it shouldn’t bounce – either immediately or a week later when some internal forwarding fails. It should reliably deliver any mail sent to it to someone who can handle it in a timely fashion. That means that if the email contains content that’s “spammy” it shouldn’t be discarded, bounced, sent to a junk folder or replied to with a snarky autoresponse – as any useful spam report will contain a copy of the spam, and so will contain spam-like content. And if it delivers to your NOC, your postmaster team, your tier one support desk or your sales team (don’t laugh, I’ve seen that happen) they should make sure it’s forwarded appropriately (though if they do that all or even most of the time, maybe it’s being delivered to the wrong place). If your policy is to send automatic responses for abuse complaints it’s probably a good idea to send them consistently, whichever route they arrive via.

The only real way to tell whether all your abuse aliases work is to occasionally try and deliver mail to them, and see if it shows up in your ticketing system. Send it from somewhere not on your corporate network – in case the internal email address is working fine, but bitrot has set in somewhere on your external MX. Try and include some sort of spammy content in it – either something you pull out of your spam folder, or perhaps an artificial GTUBE spam test string. Include the email address you sent it to in the Subject line, so you can easily track which email addresses work and which ones bounce (and which ones just silently swallow mail).

If you don’t have a separate security group you might want to check that security@ aliases deliver to you too.

Aaaand while writing this I discovered that two of the listed abuse contacts for my personal domain bounce. Even if you’re trying to pay attention it’s really easy for bitrot to set in as your infrastructure ages, leaving old, non-working email addresses still published somewhere. Check they work everyso often and after any significant infrastructure change.

No Comments

Can I get access to Google Postmaster tools if I’m using an ESP?

The answer is almost certainly yes, but there are definitely cases where it the answer is no.

If you’re using your own domains for the return path and/or the d= value then you can set up postmaster tools for those domains. If you’re using a domain managed by the ESP, or a subdomain where the ESP manages the DNS, you may need your ESP to publish the correct key in DNS to authenticate the domain to Google.

If you, and you alone, are using a custom return path and/or d= value (even if it’s you.espdomain.example) then ask your ESP to authenticate the domain or grant you access. It’s not a big deal, just tell them what email address you’re using and have them authorize it.

If you’re using a shared return path and/or d= value you likely can’t have access for privacy reasons. No ESP wants customers to see other customers’ data.

ESPs should not prevent access even in the case where they control the domain if they are assigning subdomains per customer. Every customer with dedicated domains, even if those domains are owned or managed by the

I actually have access to GPT for many of my clients – I just add the domain to my dashboard and tell them to authorize access for my gmail address. Even better, when the contract is over they can remove my authorization and their new data is still private.

1 Comment

Your idea will not work. Here is why it won’t work.

Matthew Green reminded me of an old bit of spam lore.

It’s a canned response to someone’s New and Awesome and entirely unoriginal Final Ultimate Solution to the Spam Problem. It originated on the news.admin.net-abuse.email newsgroup, I think, maybe twenty years ago? While one or two details have changed it’s still applicable to most of the current generation of under-researched proposals.

 

Your post advocates a

( ) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won’t work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we’ll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don’t care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else’s career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( ) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don’t want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

( ) Sorry dude, but I don’t think it would work.
( ) This is a stupid idea, and you’re a stupid person for suggesting it.
( ) Nice try, assh0le! I’m going to find out where you live and burn your house down!

It still very much applies. It just needs some mention of ( ) blockchain

2 Comments

The Problem With Affiliates (2)

On Friday I mentioned spam coming from a BarkBox affiliate programme.

The original email is here. It’s not terribly exciting, it’s rather typical spam of the sort sent by professional spammers.

It’s validly DKIM and SPF authenticated, and DMARC-aligned. It includes invisible white-on-white padding text so that it doesn’t look like image-only spam to naive filters (using text from the BarkBox wikipedia page). It’s sent from a throwaway CentOS host on a second tier VPS provider. That is a single (templated) machine that handles sending the spam and serves the images and does the clickthrough handling – meaning that there is no reference in the spam to anything other than the throwaway host. And so there’s not really any way to track who the spammer is without subpoena power other than by following the money.

Let’s follow the money.

I’m assuming that the spammer and his associates aren’t using any particularly clever defensive measures against tracking (and if they are, meh, all I’ll have lost is some blog content). So I’m using a simpler approach than I’d use in an actual case, one that’s easier to demonstrate. I’m fetching each link manually with the curl command, and looking at the result manually.

“curl -D- http://example.com” is a really useful curl idiom. By default curl will fetch the page you give it and print out the content. If you add the flag “-D-” it will print the headers of the response as well as the body. That’s particularly important for looking at clickthrough links as in a normal “302 redirect” the destination URL is provided in the Location: header.

The full trace of me checking the chain of redirection links is here.

Hitting the long http://mindpowrr.bid/… link in the email it 302 redirects to https://seenior.bid/aa/bbp3/. seenior.bid feels like it’s probably owned by the same spammer as mindpowrr.bid, just from the misspelling and choice of TLD. It’s hosted on cloudflare, so the hosting is bulletproof and the TLS certificate is useless. The “bbp3” at the end of the URL is suggestive of it being a manually assigned tag for BarkBox promotions.

Fetching the seenior.bid URL shows a different way to redirect, a rather creative one. It uses a bootstrap modal component to display a “fake” dialog box with an html form containing a google recaptcha. Clever. People are used to having to click on those to continue, but any automated analysis isn’t going to click on the form submit button.

Clever. (Note those unsubscription links – I’ll come back to them later).

The spammer didn’t bother checking the response from the recaptcha when you submit the form, that’s not what it’s there for. In fact, they don’t even see the form results as the form posts to https://tinyurl.com/yalfp3x9 – and tinyurl is a commercial link redirector, not something that handles forms.

I know tinyurl don’t care whether I send it a GET or a POST, so I just use the same curl command , and tinyurl 302 redirect me to http://trkpaper.com/?a=21966&c=62394&s1=5702BBP3.

trkpaper.com is abovealloffers – an affiliate network. Marketers pay abovealloffers for clicks, abovealloffers pay anyone who provides them with clicks. Because the core of their business is distributing money for clicks it’s critical that abovealloffers keep track of who is sending them traffic. Looking at the URL it’s nicely transparent. “a=21966” is probably the identity of the affiliate (spammer). “s1=5702BBP3” has the same “bbp3” identifier in it, and is probably how abovealloffers are describing the particular campaign BarkBox is paying for. “c=62394”? Not sure, I’d guess it’s a client identifier.

Clicking on the trkpaper.com URL 302 redirects to a URL at https://aboveallurl.com/…, including all the parameters that were passed above.

Following the https://aboveallurl.com/… link does a simple 302 redirect to http://superchewer.barkbox.com/aao/?AFID=21966 – and that’s the advertiser. The AFID shows me that it’s an affiliate programme run by BarkBox rather than anything custom to abovealloffers.

Those unsubscription links are both using bit.ly as a link shortener. I love when people do that, as you can add a “+” on the end of any bit.ly URL to get the tracking metrics. The first one isn’t terribly interesting – it goes to a bland, EC2 hosted generic unsubscription form at “optout-pljz.net”, and has been clicked on three times.

The second one is their global unsub request. It’s been used a bunch of times, and it redirects to a page hosted on apexpoint.co.in. The path of the page is /unsub.php, and it only shows an unsubscription form if passed the right parameters. That tells me that this server is under the control of the spammer, and may belong to the spammer. That support@apexpoint.co.in is used as an alternate unsubscription route means it’s the latter. (And, to sum up just how the spammer approaches things, the unsubscription form doesn’t actually do anything at all if you try and use it).

So … apexpoint.co.in … The whoispocalypse has apparently affected Indian domains, so no help from whois. The website has been locked down, so the root page just throws a (broken) error. It’s hosted on a lower end US provider who’d be responsive to a subpoena but won’t be a source of information without one.

But it’s running a TLS webserver with a self-signed certificate:

Naman Kathal – naman627 – out of Bhopal, Madhya Pradesh province, India.

Searching for naman627 finds Naman’s twitter account and several member profiles on affiliate / spammer bulletin boards:

Dedicated servers 97% to 99% network uptime guarantee…

Hi,

If anyone need server’s for email marketing or any other stuff. Kindly contact me, I can provide server with Best Quality and Best Price with 24/7 quality support. You can also contact me on skype ID :- naman.kathal1 for more details.

Thanks

The company mentioned in Naman’s bio is the same one the only Naman Kathal on LinkedIn claims to own. And it’s in Bhopal. And it lists apexpoint.co.in as a project. And it’s website is on the same IP address as apexpoint.co.in. They’re one and the same.

Namal claims to be a CPA affiliate network (what abovealloffers actually is) but I suspect that’s aspirational and the spammer is either Namal himself, or maybe someone he sells his “server’s for email marketing” to.

So what most likely happened is that AboveAllOffers signs up for the BarkBox affiliate programme. BarkBox pay AboveAllOffers for clicks. AboveAllOffers pay anyone who’ll drive traffic to them, including spammers. Neither of them ask too many questions about where the traffic comes from.

BarkBox don’t really advertise their affiliate programme, rather they’ve mentioned it on social media and asked people to contact them via email to sign up for it, but I’ve seen affiliate networks talking about BarkBox promotions offering rates of $16 or 27% of the purchase. Definitely enough to be worthwhile if you’re sending enough email or you’re based out of India. Or both.

And that’s why Indian spammers are pumping out spam advertising BarkBox.

1 Comment

The Problem With Affiliates

If I see BarkBox I think Spam.

That’s because, despite their marketing team effort, facebook and banner ad budget, the main place I see them advertised is via spam in my mailbox.

It’s not even good spam.

There’s quite a lot of it. Most of it looks much the same, other than the spammer randomizing colours. This one looks better than the black on cyan version, or any of the other geocities-esque variants. Loading images actually makes them look worse.

I’m also seeing a lot of spam that, from the design and structure of the mail, looks like it’s from the same spammer advertising printer ink and conspiracy theories coming from contact@barkboxdog.us.

This has been going on consistently for well over a year. While it isn’t being sent by BarkBox directly it is advertising BarkBox products and it is being paid for by BarkBox.

Badly run affiliate programmes like this one damage the brand. I wouldn’t be surprised if they damage delivery of legitimate mail sent by the brand owner – I know that my bayesian spam filters seem to put any mail mentioning barkbox into my spam folder, and they’re far less sophisticated than the ones used by large consumer ISPs.

I’ll do a tear-down of the spam on Monday and see if there’s any insight about how it all works and where it’s going wrong.

2 Comments

Reading RFCs

We mention RFCs quite a lot, both explicitly (RFC 6376 is the specification for DKIM) and implicitly (the 822.From aka bounce address aka return path).

And we have local copies of a bunch of them to make them easy to refer to (SMTP, MIME, Carrier Pigeons ).

They use quite a lot of jargon and implicit information and metadata that’s not really explained terribly clearly anywhere, rather picked up from experience using them.

But now Mark Nottingham has written a great intro “How to Read an RFC“.

It’s good. If you ever need to refer to RFCs you should go and read it.

No Comments

Wildfires and deliverability

A few weeks ago we took a drive down I5 to attend a service at Bakersfield National Cemetery. Amid the acres and acres of almond farms there were patches of black from recent grassfires. Typical but boring California landscape. Wildfires are a hugely destructive but continual threat in California. Growing up on the east coast, I never really understood wildfires. How can acres and acres and square miles just burn?
Having lived in California for almost as long as I lived on the east coast, I understand a bit better. In some ways, I have to. Even living right on the bay, there’s still some risk of fire. Like the grass fire a few miles from here across the street from the FB headquarters a few years ago. Further up the hills, there’s an even bigger risk of fire. Every driver can see the signs and precautions. Fields have plowed firebreaks around the edges. CAL FIRE posts signs alerting the public to the current fire risk status.
Fire Danger
What do wildfires have to do with deliverability?
I associate wildfires and deliverability together because of a radio show I did a few years ago. It was pitched as a “showdown” between marketers and deliverability. I was the representative of deliverability. During the conversation, one of the marketers mentioned that deliverability people were too focused on the worst case scenario. That we spoke like we expected a fire to break out at any moment. His point was that deliverability spent too much time focused on what could happen and not enough time actually just letting marketers send mail.
His overall point was deliverability people should put out the fires, rather than trying to prevent them in the first place.
I thought about that conversation during the long drive down I5 the other day. I saw the firebreaks plowed into fields at the side of the road. And I saw the patches of blackness from fires reach along the highway where there were no firebreaks.
There are a group of marketers who really hate the entire concept of deliverability. Their point of view is that deliverability is hampering their ability to make money. I’ve even heard some of them assert they don’t care if 70% of their mail goes to the bulk folder. They should be allowed to send blasts of mail and deliverability shouldn’t tell them what they can do. Deliverability, so the complaint goes, is simply out to hurt marketers.
The only good deliverability is that which gets them unblocked when their behavior triggers IP based blocks. When the field is burning down, they’d like us to come spray water on it. And then go away and let them keep throwing lit cigarettes out their car windows.
But that’s not all that firefighting is about. Much of the work is preventing fires in the first place.  In the US, a lot of that work is done through building codes. There are mandates like smoke detectors, fuel free spaces around dwellings, and sprinklers for some buildings. Monitoring local conditions and enforcing burn bans are also a large part of what the fire service does.
I like the fire fighter motif a lot. Much of what deliverability does is actually about preventing the block. ESPs have building code like standards for what mail is good and what is bad and what can be sent on their networks. Many of us publicly speak and educate about good practices and preventing blocks in the first place.
Fire prevention is about risk management and understanding how little things add up. Deliverability is similar. All the little things senders do to improve their deliverability adds up to a lower risk of fire. Yes, things like listbombing happen where even the best deliverability advice wouldn’t have prevented it. But, overall, deliverability wants to help senders get their mail in front of the people who can act on it. Some of that advice, though, takes the form of risk management and saying no.

1 Comment

Microsoft using Spamhaus Lists

An on the ball reader sent me a note today showing a bounce message indicating microsoft was rejecting mail due to a Spamhaus Blocklist Listing.
5.7.1 Client host [10.10.10.10] blocked using Spamhaus. To request removal from this list see http://www.spamhaus.org/lookup.lasso (S3130). [VE1EUR03FT043.eop-EUR03.prod.protection.outlook.com]
The IP in question is listed on the CSS, which means at a minimum Microsoft is using the SBL. I expect they’re actually using the ZEN list. ZEN provides a single lookup for 3 different lists: the SBL, XBL and PBL. The XBL is a list of virus infected machines and the PBL is a list of IPs that the IP owners state shouldn’t be sending email. Both of these lists are generally safe to use. If MS is using the SBL, it’s very likely they’re using the other two as well.
 

No Comments