In a recent discussion about spamtraps and address lists and data collection a participant commented, “[E]very site should be utilizing a real-time email address hygiene and correction service on the front end.” He went on to explain that real time hygiene prevents undeliverable addresses and spamtraps and all sorts of list problems. I was skeptical to say the least.
Yes, there are APIs that can be queried at some of the larger ISPs to identify if an account name is taken, but this doesn’t mean that there is an associated email address. Yes, senders can do a real time SMTP transaction, but ISPs are quick to block SMTP transactions that quit before DATA.
I decided to check out one service to see how accurate it was. I’m somewhat lucky in that I created a username at Yahoo Groups over a dozen years ago but never activated the associated email address. This means that the account is shown as taken and no one else can register that address at Yahoo. But the address doesn’t accept any mail.
I next sent an email to both Yahoo accounts. Yahoo accepted mail to my working account but bounced mail to the Yahoo Groups only account.
Final-Recipient: rfc822; biskybabe@yahoo.com Original-Recipient: rfc822;biskybabe@yahoo.com Action: failed Status: 5.0.0 Remote-MTA: dns; mta5.am0.yahoodns.net Diagnostic-Code: smtp; 554 delivery error: dd This user doesn't have a yahoo.com account (biskybabe@yahoo.com) [-5] - mta1289.mail.ac4.yahoo.com
This tells me that for Yahoo addresses, Briteverify is using some sort of API call to identify whether or not an account name is taken. But just because an account name is taken doesn’t specifically mean that an account is a valid email address. It’s probably better than no verification, but usage of all real time verification isn’t going to help in all cases.
What about email accounts that don’t provide an API or a way to check the validity of an account? In that case it appears that they are using an aborted SMTP transaction. we tested
Jan 24 15:20:00 misc postfix/smtpd[28917]: connect from smtpout9.briteverify.com[107.20.232.98] Jan 24 15:20:01 misc postfix/smtpd[28917]: NOQUEUE: reject: RCPT from smtpout9.briteverify.com[107.20.232.98]: 550 5.1.1 <mu/er9w9kmbyg+s5uehqdxqe@blighty.com>: Recipient address rejected: User unknown in virtual alias table; from=<admin@origindata.com> to=<mu/er9w9kmbyg+s5uehqdxqe@blighty.com> proto=SMTP helo=<emailver.briteleads.com> Jan 24 15:20:01 misc postfix/smtpd[28917]: lost connection after RCPT from smtpout9.briteverify.com[107.20.232.98] Jan 24 15:20:01 misc postfix/smtpd[28917]: disconnect from smtpout9.briteverify.com[107.20.232.98] Jan 24 15:20:01 misc postfix/smtpd[28915]: connect from smtpout7.briteverify.com[184.73.155.120] Jan 24 15:20:01 misc postfix/smtpd[28915]: NOQUEUE: reject: RCPT from smtpout7.briteverify.com[184.73.155.120]: 550 5.1.1 <aardvark@blighty.com>: Recipient address rejected: User unknown in virtual alias table; from=<admin@origindata.com> to=<aardvark@blighty.com> proto=SMTP helo=<emailver.briteleads.com> Jan 24 15:20:01 misc postfix/smtpd[28915]: lost connection after RCPT from smtpout7.briteverify.com[184.73.155.120] Jan 24 15:20:01 misc postfix/smtpd[28915]: disconnect from smtpout7.briteverify.com[184.73.155.120]
The verification service did correctly identify both addresses as invalid. However, this is exactly the kind of SMTP behaviour that is blocked by many places.
Real time address verification for 100% of addresses is incredibly difficult. As I demonstrated above, their use of testing APIs makes the assumption that everyone with a login at Yahoo (or google or other places) has an email address, but this isn’t necessarily true.
There are other assumptions that realtime address verification makes.
- No one ever typos the left hand side of their email address into an address of another user at the site. This isn’t true, for instance, I entered a common typo of my email address into the form and the service verified it as accurate. It probably is a valid, deliverable account but that doesn’t mean that it’s a good address.
- Spamtraps are always undeliverable addresses. This is not true and the above form did verify a spamtrap address that a friendly blocklist admin checked for me.
- No one typos the right hand side of an address to a valid domain. This is not true. For instance, I know a number of spamtrap domains used by Trend Micro. The form validates addresses there and tells me I’m good to send.
I’m not trying to knock the real time address verification services, I think what they’re attempting to do is good. I think the glossy marketing, though, will lead senders into a false sense of security. Just because a 3rd party service tells you an address is deliverable, doesn’t mean that the address is deliverable or that the address is safe to mail.
I do think potential verification customers deserve to understand how the services work so that they can make good decisions about purchasing those services.
RSS - Posts
Hmm, somebody was just asking me about this verification service. Glad to see I’m not the only one that figured this kind of thing is not likely to be all that accurate.
Yahoo is not a good example to use for the test of valid account/invalid email address. In the case where an email address at Yahoo is invalid (and there is no account by that name either), Yahoo will happily give you a 250 for anything you enter with the RCPT TO in real-time. If an email address is invalid at Yahoo, and you actually send an email, your SMTP transaction will get a 250 for RCPT TO; you will then get an out-of-stream bounce (as you did in your example) after the fact (not real-time). With real-time verification using aborted SMTP transactions, there is no bounced email to rely on since you aren’t actually sending an email. Did you try a test at Yahoo with an unused account and unused email address?
I don’t understand your statement “…but ISPs are quick to block SMTP transactions that quit before DATA.” An ISP has no way to know you are about to abort an SMTP transaction until you actually do it. By the time you do abort, you have an answer to RCPT TO. An aborted SMTP transaction means you connect, issue a HELO/EHLO, MAIL FROM and RCPT TO, then disconnect, issue a RSET or QUIT, etc., before sending a DATA command. Once you have sent the RCPT TO and the ISP responds, you have the answer from the ISP. The beginning of an aborted SMTP transaction and the beginning of a valid SMTP transaction look identical through the RCPT TO.
The only way the ISP could block you is if they have history that you are aborting SMTP transactions. In that case, the ISP will likely give you a 5xx code on one of the initial commands (HELO/EHLO or MAIL FROM), or simply disconnect when you connect.
The email verification technique is an interesting area. For those who are truly using it to clean lists or validate an email address is deliverable before accepting it (on a web form for example), its extremely valuable. There are many opponents, however, who simply see the technique as abusive and used to “harvest” email addresses. No doubt, like most things, it can be used for good and for evil.
I was using one of the commercial services, which claimed that my Yahoo account was valid based on API calls, but was invalid according to an actual SMTP transaction. The point of my post was that some of the services that claim to verify addresses will tell users that addresses are valid when they are not. I clearly demonstrated that at least one service will claim a Yahoo address is valid when it is not.
My statement about blocking perhaps could have been clearer. ISPs block IP addresses that routinely quit before data. It’s a part of reputation and that kind of behavior will lead to blocking of SMTP transactions from that IP.
I dunno about your spamtraps, but if you try to send mail to my spamtraps, it’ll work. That’s why they’re spamtraps. So anything that tries to verify that they’re deliverable will find that, yes, they’re deliverable. I’m having trouble imagining how any sort of service like this could purport to avoid them.
There are some low level mechanical checks that make sense, e.g., ensure that an address is syntactically valid, check that the domain part can resolve to an MX or A record. Beyond that, the only way to tell if you can send mail to an address is to do so.
While it’s true that the particular technique outlined above will not accurately verify yahoo user accounts, the newer techniques being used by other smaller name services are about 93% accurate in verifying delivery capability. As for spam traps, that is an entirely different issue in which I have no real confidence in the primary players.
This is not always true because if you connect to Mail Relay Frontend server this server does not know if the mailbox exist or not (for example in AD/Exchange environment).
😉
These are some great points that you bring up. Although BriteVerify does a good job of cleaning your list and/or providing realtime checking it is not a silver bullet. I’m sure email marketers that do acquisition mailing already know this and BriteVerify is just one component of their overall bag of tools. Does anyone know of other services that are more accurate?
Does anybody else suspect that the “93% success rate” is a very fluid, very unclear statement? I’m just not buying it.
We use Strikeiron’s API to scrub email addresses (a bit pricey, but we’re scrubbing a lot of addresses, as we are a Tech college) and have found that Exchange servers return a 202 code, verifying that the domain component has a valid MX while POP servers verify that the mailbox name is valid and accepting mail. This is adequate for our purposes.
are there any legal issues in doing a large number of bulk SMTP transactions where you would connect issue say 20 RCPT TO commands to verify the user then send a RSET and QUIT. Immediatley connect again and do another 20 RCPT TO’s etc…
If you had say 250 thousand email addresses all pointing to the same MX record that would be you continually connecting 12500 times to cleanse your historical email list, then numbers would be small on forward flow as you would check them realtime as they came in.
Could you get into any serious trouble doing this?
[…] the comments on my Address Verification in Real Time Ken asks: are there any legal issues in doing a large number of bulk SMTP transactions where you […]
We finally found http://www.email-validator.net, an email address validation service with a real-time API (batch list upload is available as well) that is really able to validate Yahoo and Hotmail addresses. We have been using the service for 6 months now and it works very well for us.
That company just validated a yahoo address I know doesn’t accept email.
[…] looking for a valid domain and username), whereas other services determines user authenticity with real-time verification. Â Unique confirmation links following signups and registrations is a basic and commonly used way […]
Very relevant post, Laura!
Just a quick note:
The behavior you say is “banned” actually isn’t “banned” per se — or at least it shouldn’t be. It’s actually commonplace for MTAs to do these kinds of checks automatically.
http://www.tldp.org/HOWTO/Spam-Filtering-for-MX/smtpchecks.html
It might get you “banned” if you’re checking like there’s no tomorrow. But acceptable, polite use of this technique is actually an important part of the email infrastructure.
I didn’t say any behavior was banned. I said that if an IP is continually quitting after RCPT TO: they will be blocked. And they will. It’s called dictionary attacking and has been a behavior the ISPs haven’t liked for at least a decade now.
Quite bizarre that ReturnPath now partners with BriteVerify – https://blog.returnpath.com/return-paths-partnership-briteverify
Try your Yahoo address with the demo at emailchecker.com – they claim to have Yahoo accuracy that goes further than any other provider.
That tester tells me my invalid yahoo address is valid.