Is your website up? Are you sure?

“What would you do for 25% more sales?”
It’s panicked gift-buying season, and I got mail this morning from Boutique Academia, part of their final push before Christmas.
Inbox__18_975_messages__26_unread_
They’re hoping for some Christmas sales in the next three days. They do make some lovely jewelry – ask Laura about her necklace some time – so I clicked on their mail.
Failed_to_open_page
That’s not good. I like Boutique Academia, and fixing email and dns problems is What We Do, so I took a look.
Safari isn’t quite as bad with not-exactly-truthful error messages as Internet Explorer, but I still don’t really trust it. Perhaps the problem is with the click-tracking domain in the email, rather than with boutiqueacademia.com? So I open the base page at http://boutiqueacademia.com, get redirected immediately to https://www.boutiqueacademia.com – which fails to load.
15542402_1501169219896451_6901276936993410491_n
OK, start with the basics. DNS.

~ ∙ dig www.boutiqueacaedmia.com
; <<>> DiG 9.8.3-P1 <<>> www.boutiqueacademia.com
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 19459
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.boutiqueacademia.com. IN A

That’s pretty authoritative (DNS joke … never mind). The response says “NXDOMAIN”, which is DNS-speak for “no such domain”. It means that there isn’t any DNS record for www.boutiqueacademia.com – no A record, nor any DNS record of any other type for that hostname. And it isn’t something that can be caused by a temporary network glitch – it means that the DNS server for the domain says authoritatively “there’s no DNS record”.
This is beginning to look familiar. Lets see what the authoritative DNS servers for boutiqueacademia.com have to say.

~$ dig +short boutiqueacademia.com ns
ns2.hostdns4u.com.
ns1.hostdns4u.com.
~$ dig @ns1.hostdns4u.com www.boutiqueacademia.com
; <<>> DiG 9.9.5-3ubuntu0.7-Ubuntu <<>> @ns1.hostdns4u.com www.boutiqueacademia.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4587
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
;; QUESTION SECTION:
;www.boutiqueacademia.com. IN A
;; ANSWER SECTION:
www.boutiqueacademia.com. 30 IN CNAME boutique-academia.myshopify.com.

There’s our problem. Boutique Academia are using hostdns4u.com to host the DNS for their domain – and hostdns4u.com are sending broken DNS responses. They’re responding with a CNAME to myshopify, the ecommerce site, but they’re also responding with the status “NXDOMAIN” – there’s no DNS record for this hostname.
That response violates the way DNS works. If it’s returning any matching DNS record – such as the CNAME – it should return a “NOERROR” status, not “NXDOMAIN”.
Where this gets interesting is when you ask what a DNS resolver will do with this invalid response. That’s not really defined, and different recursive resolvers will treat this response in different ways. Some will ignore the NXDOMAIN and return the CNAME record. Some will ignore the CNAME record and return NXDOMAIN.
That means that depending on which sort of DNS server your recipient is using everything may work perfectly, or any attempt to visit the website may give an error. If the DNS server you’re using is sloppy about NXDOMAINs you may never see the problem in your own web browser.
As far as I can tell this bug is specific to CNAME records, and exists at several DNS hosting companies.
The end result is that if you’re using an ecommerce or blog-hosting company that uses CNAMEs in your DNS zone to point to their servers (such as shopify or wordpress) and you’re  hosting your DNS with a company that has this bug then a significant fraction of the Internet will not be able to reach your website.
You’ll see their opens and clicks on your email. If you’re using pay-per-click advertising you’re paying for each customer. But they’ll never see your site.
I’m not sure what fraction of the Internet is using recursive resolvers that are strict about invalid CNAME responses, but I’d guess it’s at least 20%. If your website suffers from this issue then you could get 25% more sales by moving your DNS hosting to a company that doesn’t suffer from this issue.
(I have told Boutique Academia about this issue. Hostdns4u.com have a cPanel error page for their website, so I didn’t try to contact them.)

Related Posts

The Internet is hard.

There are so many things that need to happen to make the Internet work. DNS entries need to be right. MXs need to be set up. Web servers need to be configured. And, let’s be honest, anyone who has ever run their own services on the Internet has flubbed a configuration.
We don’t think about it, because most of the time the configurations are handled by scripts and they do things right. But at some point someone needs to type in something and there’s a risk it will go horribly wrong. I’ve been digging into domain data for a client of mine today. I think I’m going cross-eyed over it. But I have found so many weird things that just mean someone isn’t paying attention to what they’re doing.
Like the domain that has a MX record that says:
nullmx
 
I’m pretty sure the intention of the domain owner is to publish a null MX. But they added an extraneous “0” in there and ended up publishing something really weird. Even worse, the MTA that this client is using is listing this as a “delivered” email. I’m pretty sure that mail to that domain never left the MTA.
I’ve found horribly typoed MX domains for popular spam filters. I’ve found domains that have invalid characters in them. I’ve found domains that are totally a mess.
The vast majority of us have some story or other of the time we really broke things by accident. Like the time a very large ISP deleted their MX records. Or when a different ISP changed their internal forwarding and broke SPF authentication for everyone mailing that domain. Or when another ISP accidentally blocked every IP beginning with 6.
Sometimes I’m amazed that the Internet ever works. No matter how big it gets, there are actual people writing actual code and configurations. The number of things that have to happen to get packets from A to B is pretty impressive. We rarely ever notice the breakages, the people who run things are really good at their jobs. But sometimes poking in the grotty corners reminds me how easy it is to break things. It’s sometimes a wonder things actually work.
 

Read More

SPF: The rule of ten

Some mechanisms and modifiers (collectively, “terms”) cause DNS queries at the time of evaluation, and some do not. The following terms cause DNS queries: the “include”, “a”, “mx”, “ptr”, and “exists” mechanisms, and the “redirect” modifier. SPF implementations MUST limit the total number of those terms to 10 during SPF evaluation, to avoid unreasonable load on the DNS. If this limit is exceeded, the implementation MUST return “permerror”.

Read More

4 things spammers do legitimate marketers don't

I’ve never met a spammer that claims to be a spammer. Most that I’ve met claim to be legitimate marketers (or high volume email deployers). But there are things spammers do that I never expect to see a legitimate marketer doing.
I’ve written about these things throughout the blog (tag: TWSD), but it’s probably time to actually pull them together into a single post.

Read More