spareknet.org
Customized CPanel Solutions / Server Administration

Home
Sender Validation Published on July 2nd, 2007 Last Modified on: September 8th, 2008

Lately I have been seeing quite a few problems with users not receiving e-mail when a mail server is using sender callouts or sender callbacks. I have been experiencing this problem with the hosting companies that I am associated with. I am also seeing similar reports through various support forums that I frequent, where other users are having this problem. Through my investigations, most of these are a problem caused by the sender that is sending out the message. Larger companies are insisting on sending out newsletter and other announcements from invalid e-mail addresses, which is causing the sender callout to fail.

To understand this problem, lets take a look at how sender callouts work. An e-mail transaction takes place between two computer systems. The computer system that is receiving the message is called the server. Likewise, the computer system that is sending the message is called the client. So another words, if you are sending out a message through your mail server to a gmail.com e-mail address, then your mail server will act as the Client sending a message to the Gmail mail servers, acting as the Server.

Basically an e-mail transaction will look something like this:

  1. Client connects to Server
  2. Server responds with a 220 code, advertising its name
  3. Client sends a HELO or EHLO command with its hostname, introducing itself to the Server
  4. Server accepts this commands with a 250 code
  5. Client tells the server who the message is from, this is the envelope-sender
  6. Server accepts this with a 250 code
  7. Client sends who the message is being sent to, this is the envelope-recipient
  8. Server accepts this with a 250 code

This is the end of the recipient stage of the e-mail transaction. The Client next sends a DATA command, which tells the server that the next lines being sent are the actual e-mail message.

Let’s take a look at a real life example of this order. To help illustrate this, I have color coded the text. Text that is in blue refers to responses from the Server. Text that is in red refers to data that is being sent by the Client.

E-mail Transaction
Connected to gmail-smtp-in.l.google.com (209.85.133.27).
Escape character is ‘^]’.
220 mx.google.com ESMTP d25si10513773and
EHLO myhostname.com
250-mx.google.com at your service, [xx.xx.xx.xx]
250-SIZE 28311552
250-8BITMIME
250 ENHANCEDSTATUSCODES

MAIL FROM: <[email protected]>
250 2.1.0 OK
RCPT TO: <[email protected]>
250 2.1.5 OK

Now lets split this up into their corresponding descriptions. It just follows in order.

1. Client connects to Server
Connected to gmail-smtp-in.l.google.com (209.85.133.27).
Escape character is ‘^]’.

2. Server responds with a 220 code, advertising its name
220 mx.google.com ESMTP d25si10513773and

3. Client sends a HELO or EHLO command with its hostname, introducing itself to the Server
EHLO myhostname.com

4. Server accepts this commands with a 250 code
250-mx.google.com at your service, [xx.xx.xx.xx]
250-SIZE 28311552
250-8BITMIME
250 ENHANCEDSTATUSCODES

5. Client tells the server who the message is from, this is the envelope-sender
MAIL FROM: <[email protected]>

6. Server accepts this with a 250 code
250 2.1.0 OK

7. Client sends who the message is being sent to, this is the envelope-recipient

8. Server accepts this with a 250 code
250 2.1.5 OK

Now lets take a look at an e-mail transaction for a server that uses sender callouts, whenever a bad envelope-sender is used.

E-mail Transaction using sender callout and bad envelope-sender
Trying xx.xx.xx.xx…
Connected to server.mailserver.com (xx.xx.xx.xx).
Escape character is ‘^]’.
220-server.mailserver.com ESMTP Exim 4.66 #1 Sat, 30 Jun 2007 13:56:24 -0400
220-We do not authorize the use of this system to transport unsolicited,
220 and/or bulk e-mail.

EHLO client.mailserver.com
250-server.mailserver.com Hello client.mailserver.com [yy.yy.yy.yy]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250-STARTTLS
250 HELP

MAIL FROM: <[email protected]>
250 OK
RCPT TO: <[email protected]>
550-Verification failed for <[email protected]>
550-Called: zz.zz.zz.zz
550-Sent: RCPT TO:<[email protected]>
550-Response: 550-“The recipient cannot be verified. Please check all recipients of this
550-550 message to verify they are valid.”
550 Sender verify failed

Here you see that the verification of the [email protected] e-mail address failed. Values have been obfuscated for privacy reasons. In this example the Server is using sender callouts to verify the validity of the envelope-senders. After the line RCPT TO: <[email protected]>, the Server in this instance attempts to send a message back to [email protected] which will determine whether or not that address is valid. If its valid, the server will want to accept mail for that address. If it is not valid, the server won’t recognize it. A sender callout transaction attempt looks like:

Sender Callout Attempt
Connected to client.mailserver.net (zz.zz.zz.zz).
Escape character is ‘^]’.
220-client.mailserver.net ESMTP Exim 4.66 #1 Sat, 30 Jun 2007 14:05:25 -0400
220-We do not authorize the use of this system to transport unsolicited,
220 and/or bulk e-mail.

EHLO server.mailserver.net
250-client.mailserver.net Hello server.mailserver.net [xx.xx.xx.xx]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250-STARTTLS
250 HELP

MAIL FROM: <>
250 OK
RCPT TO: <[email protected]>

At this point the mail server will either report a 250 code if the e-mail address is valid. If it is not valid, another, non 200 level return code will be given. This way you know that the e-mail address is either valid and accepting messages or invalid and not accepting any messages.

Response if [email protected] is a valid e-mail address on the server.
250 Accepted
Response if [email protected] is not a valid e-mail address, thus failing the callout
550-“The recipient cannot be verified. Please check all recipients of this
550 message to verify they are valid.”

Basically, think of it this way. If a Server is using sender callouts to verify the instance of a sender, in the E-mail Transaction using sender callout and bad envelope-sender example after the Client sends RCPT TO: <[email protected]> the Server, in the background, performs the sender callout as described in the example labled Sender Callout Attempt. The response from Sender Callout Attempt has a deciding factor in how the Server will respond in E-mail Transaction using sender callout and bad envelope-sender.

Null Sender
In this example of the Sender Callout Attempt notice that the MAIL FROM line, MAIL FROM: <>. The <> refers to the null sender or the empty sender, or sometime called an empty return path. This is because there is no e-mail address listed between the less-than and greater-than symbols.

Not all sender callouts use the null sender, although it is preferred. However, not all servers will accept mail from the null sender. If a mail server is not accepting mail from the null sender, then this can also cause incoming messages to be rejected by your server. Not supporting the null sender is in violation of RFC standards. This is highlighted in the RFC 1123 description from RFC Ignorant.

The syntax shown in RFC-821 for the MAIL FROM: command omits the case of an empty path: “MAIL FROM: <>” (see RFC-821 Page 15). An empty reverse path MUST be supported.

While the null sender does not have to be used for the callout attempt, a special e-mail address needs to be used or otherwise you risk getting mail that is in a loop. Some servers that perform sender callouts will use [email protected]. But then you run into a problem when your server is performing sender callouts, and the mail server your server is connecting to, to verify the envelope-sender uses a sender callout, then you get into a loop. This would not be the case if an address was designated for this purpose, something that the null sender accomplishes.

The scope of this article is to point out that using bad envelope-senders can cause sender callouts to fail, but that is not the only reason sender callouts will fail. I thought it was worth mentioning the null sender and that some mail servers do not properly follow this rule, but discussion in that topic goes beyond the scope of this article.

To tie this in with cPanel accounts, this is why you want to set your default box to :fail:. This way, if a spammer tries to use a random e-mail address at your domain name and they send out a spam message to someone whose server is using sender callouts, then that individual will not receive the spam message, because the random e-mail address at your domain name will go back to the default box, which is set to :fail: and will not accept messages. This prevents someone from spoofing your domain name using random e-mail addresses, when sending out spam e-mails. This is why it is important that you have your default box set to :fail: on cPanel accounts.

A lot of larger companies or corporations are not using a real e-mail address as the envelope-sender. This is exactly what spammers do and by doing this, a legitimate e-mail from a legitimate company is getting misconstrued as spam. Spammer’s don’t have a legitimate message that they are sending out. They are just out trying to get their messages out to as many individuals as possible. Spammer’s will always hide their real identity by using an envelope-sender that is not referring to their e-mail address. Sometimes the envelope-sender that spammers use will be valid, in this case a sender callout system alone won’t prevent you from receiving that spam message. However, a good portion of the time, spammers will simply use any e-mail address that is syntactically correct, regardless of whether or not that e-mail address is really accepting e-mail.

Unfortunately, when a company or large corporation is sending out an e-mail message and they use an envelope-sender that is not also accepting mail, then it gets clustered into being a spam message. One would think that a large corporation, one with a lot of resources at its disposal would care about this and would take as many measures as possible to prevent their messages from being grouped into a system that spammers use.

This is just a situation where you cannot have it both ways. There is no magically algorithm that can be put in place that says “This message is not a spam message, who cares if they are using a bad envelope-sender, accept the message” and then when it receives a spam message “This message is a spam message, if the envelope-sender is bad, don’t accept the message”. It just can’t be done. For one thing, the contents of the e-mail message have not been seen by the time the sender callout is done. If you waited and rejected the message after the contents have already been sent, then there’s really no point in having the sender callout in place. You would have to scan each message to determine whether the message is spam, which is basically what you are trying to avoid, en masse, with sender callouts.

Ask yourself this question, would it be easier to tell companies and large corporations to use legitimate e-mail addresses as their envelope-senders when they send out a message or would it be easier to tell all spammers to quit sending you spam? You might think the decision would be an easy one, but from my experience, I’m not sure. I don’t think spammers will stop sending you spam, just because you tell them to. If the solution were that simple, then we wouldn’t have all of the spam problems on the Internet. So that leaves you with informing companies and large corporations to use legitimate, valid e-mail addresses as their envelope-senders if they have a message that they really want you to receive, which in my opinion just makes the most sense.

Basically when it comes to sender callouts, you have to make one of two decisions. Either you enable it or you don’t. If you choose not to enable sender callouts, then your server will receive messages from these companies and corporations that insist on sending out messages from bad envelope-senders, but you will also receive a lot more spam. I would venture a guess that for every legitimate message you receive through your mail servers that comes from a legitimate company using a bad envelope-sender, you are receiving 100 spam messages from spammers who are also using a bad envelope-sender. Thats a 100 to 1 ratio (again, no scientific data or testing to back up this claim). Is it worth it? Thats 100 spam messages that your e-mail users are receiving, just because some company out there that is not spamming you, but insists on using tactics that identifies their message as spam. If you have SpamAssassin or other anti-spam measures, thats 100 messages that have to be scanned and filtered, taking up CPU time and processing power, that could be used for other purposes. Why should these messages have to be scanned for spam, when there’s a 99% chance that they are spam? If you choose to enable sender callouts, then all of those spam messages will be rejected at the recipient stage, but you also won’t receive e-mail from companies that use bad envelope-senders.

Personally, I am of the thinking that if a legitimate company is wanting to send you a legitimate message, then they should use a valid, legitimate envelope-sender, one that also accepts e-mail, so that messages from that company do not get grouped together with spammers. To me this is common sense. However, as far as I am aware, there is no rule or governing system that says an envelope-sender address must also accept messages. So when these large companies are sending out legitimate messages and using a bad envelope-sender, technically they are not breaking any rules. But just because you are not breaking any rules doesn’t mean that it’s not wrong. Companies that conduct such activities need to understand that their messages will not reach recipients that are using a sender callout system. If those companies will just use a valid envelope-sender, then mail servers that do not use sender callouts and mail servers that do use sender callouts will both be able to accept their message. I think that if a company cares about its reputation and cares about its end-users, then they would want to take action to prevent themselves from being grouped together with spammers.

If anyone is aware of any RFC or other standard that says that an envelope-sender of an e-mail message must also accept mail messages, please let me know. I don’t think anything like that exists, but I can’t be absolutely certain. I’m not sure if there would be any ramifications involved in creating such a standard or rule. There may be a completely legitimate reason where a message should be sent out and accepted even if the envelope-sender is not accepting mail, I am just not aware of a reason. Again, if you know of a reason please let me know.

As for the companies and corporations that insist on using these bad envelope-senders, I can understand where you are coming from. I know that a lot of times you want to send out a newsletter or message and you may specifically say in the message “Do not reply back to this message”, because you don’t want to receieve people’s replies or people’s messages to that address. You also probably don’t want to receive all of the failed delivery notices when an e-mail address on your list goes bad. I understand this, but there are better ways to accomplish this than just flat out not accepting messages for that envelope-sender address.

If you are going to send out mail from an e-mail address then you need to configure that e-mail address to also accept e-mail. If you do not want to bother with collecting mail sent to that e-mail address, then configure that e-mail address to immediately delete any messages sent to it. This can be done by redirecting the messages to /dev/null on Linux systems. I don’t have a lot of experience with different MTAs. I know with exim, specifically cPanel’s rendition of exim, you would just set up an e-mail forwarder for that specific address and forward it to :blackhole: which send messages to that address into an endless oblivion. The valiases file for that domain would like:

Exim valiases file illustrating a noreply address
[email protected]: :blackhole:

This way, I can send out an e-mail message using [email protected] as my envelope-sender. If a server is using sender callouts, this e-mail address will still return as valid. If someone tries to send me a message at [email protected], my mail server accepts the message, but just immediately deletes it. I can’t speak for other MTAs, but I am almost certain that this task can be accomplished with virtually any MTA.

Fighting spam is all about using algorithms and various testing methods to gauge the probability that a message is a spam message. When you find something that is statistically in the vast majority of all spam messages, then you can write algorithms or inact strategies to check for this anamoly. This is what sender callouts and sender verifications does. For the most part, any message that is using an envelope-sender that is an address that cannot also be written back to, statistically speaking, these messages are usually spam. Why should everything else have to suffer because a handful of companies, corporations, or individuals believe they are bigger than this. In most modern senses, majority rules, if the majority of messages that are using bad envelope-senders are spam messages, then it makes sense that those companies, corporations, or individuals that are trying to send legitimate messages using bad envelope-senders should be the ones that change. You can’t fight spam without making changes. I am sorry that sometimes these changes affect you. In my opinion, it basically comes down to you either quit complaining about all of the spam you are receiving or you accept that sometimes changes are necessary in order to cut spam.

True