… or Find the False Positive.
Anyone sending a lot of email has complained about spam filters and false positives at some point. But most people haven’t run a mailbox with no spam filters in front of it in recent years, so don’t have much of a feel for what an unfiltered mailbox looks like, how important filters are and how difficult their job is.
I run no transaction level filters in front of my mailbox, just content filters that route mail to one of several inboxes or a junk folder, so if I want to I can look at what unfiltered email looks like. I took data from all mail that was sent to me yesterday, and put it in a format that really shows the problem filters face and especially the difficulty of spotting which mail in the junk folder is a false positive.
An inbox with no filters looks like this.
Running a spam filter against it, simply categorizing each email as spam (pink) or not-spam (green) looks like this.
Even with the messages categorized as spam vs not-spam it’s hard to work out which messages are important and which aren’t, let alone where the false positives might be.
If I sort the categories by hand you get this – where you can see that out of 1200 or so mails about three quarters were spam. Of the three false positives two were bulk email that I didn’t care that I didn’t receive and only one was email that I considered important.
Email without filters
E
RSS - Posts
In the words of the creator of an antti-spam filter being currently used by a top ISP……..If 1% of a billion messages get through mistakenly, then is a filter really working? He touts his product exceeds 99% accuracy….Conversely, think about the other side, 10 million false positives (missed by the filter)…..
My comments seem to be censored now.
I don’t think there’s a major anti-spam provider that only has 99% accuracy, so that question seems rhetorical.
And no, your comments are not being censored, because this is not Speakers Corner and the government does not guarantee your free speech rights here; this is the professional blog of the email consultancy Word to the Wise LLC, and they alone have the final say as to what comments (if any) appear here. If you take issue with this policy, my advice is for you to get your own blog.
Those graphics make me queasy – not because of all the spam (!) so much so that if you stare at them long enough they are kind of dizzying.
Huey,
Abaca claims to have a 99.9% accuracy with a money back guarantee. The creator actually challenges anyone to beat it. He’s serious about this.
Several years ago, AT&T was the first ISP to implement spam filtration on all users’ account without their consent. I’ll never forget the day when my 300 spam messages per day was reduced to 5. I knew something was wrong, and sure enough, I lost at least 3 or 4 legitimate e-mails every month. Losing 1 per month is unacceptable to me. So I switched to Charter. Shortly after, I think all ISPs followed suit. But then they must have experienced quite a backlash from people like me, because a few months later, my spam returned.
Does anyone know if (as of 8/5/11) Charter Communications does spam filtering without users’ consent? I still get a lot of spam, but over the past month I’ve suddenly had 3 or 4 people ask me if I got their e-mail, and they said no. I do use my own domain name for e-mail forwarding to my @charter.net account, but I can’t imagine the auto forwarding being the culprit (I’ve used it with bluehost.com for many years with no problems). I get hundreds of spam messages every day, and I’m more than happy to sift through it to avoid false positives.
Charter does filter, but I don’t know anything about how they communicate their filtering to customers. There is quite a bit of information on their support pages about their filtering, and they do state they use blocklists to stop mail at the SMTP transaction.
Yes, they are, with no re-course I can find, short of calling.