Do system administrators have too much power?

Yesterday, Laura brought a thread from last week to my attention, and the old-school ISP admin and mail geek in me felt the need to jump up and say something in response to Paul’s comment. My text here is all my own, and is based upon personal experience as well as those of my friends. That said, I’m not speaking on their behalf, either. 🙂
I found Paul’s use of the word ‘SysAdmin’ to be a mighty wide (and — in my experience — probably incorrect) brush to be painting with, particularly when referring to operations at ISPs with any significant number of mailboxes. My fundamental opposition to use of the term comes down to this: It’s no longer 1998.
The sort of rogue (or perhaps ‘maverick’) behavior to which you refer absolutely used to be a thing, back when a clean 56k dial-up connection was the stuff of dreams and any ISP that had gone through the trouble to figure out how to get past the 64k user limit in the UNIX password file was considered both large and technically competent. Outside of a few edge cases, I don’t know many system administrators these days who are able to (whether by policy or by access controls) — much less want to — make such unilateral deliverability decisions.
While specialization may be for insects, it’s also inevitable whenever a system grows past a certain point. When I started in the field, there were entire ISPs that were one-man shows (at least on the technical side). This simply doesn’t scale. Eventually, you start breaking things up into departments, then into services, then teams assigned to services, then parts of services assigned to teams, and back up the other side of the mountain, until you end up with a whole department whose job it is to run one component of one service.
For instance, let’s take inbound (just inbound) email. It’s not uncommon for a large ISP to have several technical teams responsible for the processing of mail being sent to their users:

  • the aforementioned system administrators (who are responsible for running the operating system and base-level support applications — but not the hardware; that’s handled by the hardware team),
  • the application administrators (who manage the process(es) that handle the actual SMTP transactions),
  • and the database administrators (this volume of mail most likely not happening strictly using flat files.) (Oh, and the database hosts have separate hardware administrators and system administrators, too.)

You’ll note that none of these groups have taken responsibility for saying what actually gets to be delivered. In most cases, at most large ISPs, it’s simply not their job. Their jobs are simple: Keep their piece of the machine up and performing within acceptable technical parameters.
The decision to block (or unblock) messages (whether by IP, domain, content, or any other criterion) is made by an entirely different team, with a different set of marching orders. At many ISPs, these people don’t even report into the same organization as the guys listed above. These folks go by many names, but I’ll use the term ‘Postmaster’ here. Postmaster teams have a weird job that’s both science and art.
The science part is pretty easy, as it’s supported by data provided by users. For instance: how many complaints are we getting about a sender and what percentage of messages we receive from the sender results in complaints? (It’s important to remember that those are two very different things.) Since this part of the job is is so data-driven, it is often at least partially automated, or at least streamlined via tools. As dashboard displaying messages from suspected senders or those containing suspected content, with links to or copies of the supporting data would allow the Postmaster team to proactively review inbound messages that may not have yet triggered automated systems, but that are likely to in the near future. (Oh, and this dashboard was probably written by a tools team (still not the system administrators!) who have access to the hosts where the logs are kept (… need I even say it?))
The ‘art’ part is where things get tricky, and is one reason actual system administrators shy away (or run screaming) from this type of work. There are potential legal ramifications to this sort of thing, particularly when blocking on message content (as opposed to message source). What percentage of messages that generate complaints is acceptable? What if nobody (for whatever reason) ever complains, but the message contains illegal content? What happens if a small, but vocal, number of people complain? Do we have a partnership (it happens) with the sender that in any way changes our response to complaints about their messages? Do we attempt to work with the sender prior imposing a block, or simply block the incoming stream without warning? What is the best way to block this sender? Should we attempt to block similar messages? If so, how do we identify these messages?
These are all squishy, unquantifiable things, many of which can be (more or less) legally defined, but not necessarily implemented via a particularly elegant piece of code. Human intervention is required. Actual people actually looking at messages and complaints, taking them in context, comparing that context against the minimal guidelines provided by the various technical (host, system, database) admins, as well as the more nebulous legal and policy guidelines provided by any number of groups: sales, marketing, legal, etc. Calls (like, actual phone calls, not decisions) need to be made.
I have a number of years as a system administrator under my belt, most of them spent at large ISPs, and I speak from experience when I say I know of very very few people who sign up to be a system administrator who want anything to do with this kind of work.
(All of that was my long way of saying that any ISP with a significantly large number of users is long past the point where such unilateral decisions can be made. Or, if they can be made, systems are assuredly in place that they cannot be made without being noticed. Any single player who decides to block all mail from ‘Opposition Candidate’ had best be ready to have technical- and policy- based reasoning at the ready, because actions will need to be justified, and in short order.)

Related Posts

The long tail of domains

I frequently get clients telling me that they have about 15 (20, 30) major domains on their list, and then a long tail of domains with only a couple of recipients. If you sort simply by the left hand side of the @, that’s true.
When you’re sending email, it’s not just the domain in the email address that is important. Of equal importance is the MX. The MX is what actually handles the mail and where many filters are applied. Sorting by MX, instead of simply recipient domain, can identify that most of your small business clients are hosted at a particular provider. The number of subscribers behind that filter may be enough to push that filter into your top 10 or even top 5 recipient domains.
There’s a much smaller tail when grouping recipients by MX domain. It makes it much easier to understand where blocks are happening. I have even seen cases where clients didn’t realize they were blocked at a commercial provider because they only saw the “onesie twosie” domains as undeliverable. They missed a real problem with blocking because they were looking at the wrong data.
I sometimes get the side eye from some ISP folks if I use the term receiver (because, well, they’re senders as much as they are receivers). But I use receiver to help distinguish between the recipient domain and the actual domain handling the email.
When was the last time you looked at your delivery by filter or MX rather than by recipient domain? What did you find?

Read More

Don't wait to address delivery problems

One of the worst ways to deal with blocking issues is to ignore them and hope your mail magically moves from the bulk folder back into the inbox. While this does happen as ISPs and filter companies update their filters, it’s not that common and it’s usually the result of a sender actually cleaning up their sending processes and improving the quality of the mail they send.
Do not ignore blocks. What I generally tell people is that it takes at least as long to repair a bad reputation as it took to get that bad reputation in the first place. If you wait months before actually addressing delivery problems, you’re not going to make a change and have the filters react in hours.
This doesn’t mean that every block is a business crisis. Blocks happen and they do go up and down based on thresholds and automatic monitoring scripts and content. But if a block happens consistently for 4 or 5 days in a row it is time to look at what you’re doing. Don’t just focus on the sidelines and little stuff, either. Look at your marketing program and the mail you’re sending.

Read More

Politics and Delivery

Last week I posted some deliverability advice for the DNC based on their acquisition of President Obama’s 2012 campaign database. Paul asked a question on that post that I think is worth some attention.

Read More