At cpanel, Technical Analysts will get asked this question many times. Unfortunately the question
is a little vague since there are really hundreds of reasons why an email may not get delivered.
These can be:
- Authentication issues (the most common).
- DNS related issues (is your rDNS set up correctly?).
- SPF, DKIM and/or DMARC settings (misconfigured or missing completely).
- Firewall issues (or proxying of port 25 by the NOC, Datacenter, Host or customers ISP) or other networking issues.
- Blocked IP addresses (Customers IP address may be blacklisted).
- cpHulkd brute force lockouts.
- SpamAssassin settings too strict.
- Spammer (or more likely a compromised email account) has filled up your exim mail queue with spam messages and legit messages are now waiting to be sent.
- Exim misconfiguration.
- Sender Verify and even Greylisting or other temporary soft-errors.
- An incorrect setting within your /etc/resolv.conf file.
- A misconfigured filter.
- Mailbox is over quota (5.2.2. error).
- Virus was attached to message and virus scanner redirected your email to /dev/null.
A very important thing to remember is that email either gets delivered or it bounces back with an error. There is no in-between and mail doesn't just "disappear into limbo" as many people believe.
What are bounce messages?
A bounce error is also called a Non-Delivery Report/Receipt (NDR), a (failed) Delivery Status Notification (DSN) message, a Non-Delivery Notification (NDN) and is an automated electronic mail message from the mail server informing the sender about a delivery problem. There are two types of errors. Temporary errors (which start with a 4.x.x) are known as soft-bounce errors, and permanent delivery errors (which start with 5.x.x) and are known as hard-bounce errors. Temporary errors will usually be re-delivered, while permanent errors will be returned or bounced.
That being said, the most common cause is usually user error in setting up the email on their mail client (Outlook, Thunderbird, Eudora, etc...) So the first thing that should be checked is the configuration settings. In cPanel all user names for both POP (Post Office Protocol) and IMAP (Internet Mail Access Protocol) are the full email address.
The mail server in most cases is mail.DOMAINNAME.TLD (where DOMAINNAME.TLD is the domain name in question). Verifying authentication issues for sending email is fairly trivial and not part of this post.
Below, I've outlined some of the things that should be checked when trying to troubleshoot email delivery. These are just some of the more common error messages we (as technical analysts) have seen.
550 errors
Take a look at the following error...
Code:
[email protected] <[email protected]> R=dkim_lookuphost T=dkim_remote_smtp: SMTP error from remote mail server after initial connection: host somedomain.com [xx.xxx.xx.xxx]: 550 Your host is not allowed to connect to this server. Frozen (delivery error message)
See the "550" Your host is not allowed to connect to this server message? That means point blank that the host you are trying to send to, has blocked your server. This could be do to spam complaints or other abuse issues, or even because that server is misconfigured. You would need to contact the host in question to ask why they have decided to block your server. You should also check to see if your server has been blacklisted. A good site to use to check that is:
MX Toolbox.
521 errors
Another error that might creep up from time to time is this one:
Code:
SMTP error from remote mail server after end of data: host someserver.mx.domain.com [xx.xx.xxx.xxx]: 521 5.2.1 : http://postmaster.someserver.com/errors/521.html
This one actually contains a link (obscured for security) to a site that will explain why the message failed, and usually why it failed.
421 errors
I mentioned earlier about soft-errors vs hard-errors, take a look at the following error message:
Code:
2013-12-29 23:41:45 1VxPzN-3420m0-1p SMTP error from remote mail server after initial connection: host mx.somedomain.com [xxx.xxx.xxx.xxx]: 421 mx.someserver.com Service unavailable - try a gain later
The temporary error could very well be Greylisting. Greylisting is a method of defending e-mail users against spam. A mail transfer agent (MTA) using greylisting will temporarily reject" any email from a sender it does not recognize. If the mail is legitimate the originating server will, after a delay, try again and, if sufficient time has elapsed, the email will be accepted.
It might be some other temporary issue however so you would need to contact the mail service provider of the host in question to find out what that temporary reason is (or just wait to see if it goes through later).
Firewall issues?
We've also seen a few issues with the big mail servers (Gmail, Yahoo, Hotmail/Outlook.com) rejecting mail connections. The majority of them have been firewall related (either your servers firewall or one upstream from your server).
To test if it's a firewall issue, you can try a simple telnet command to port 25 from your server. First you have to obtain the correct MX record, which can be done via a dig (Domain Information Groper) query.
Code:
root@server [~]# dig gmail.com MX +short
5 gmail-smtp-in.l.google.com.
30 alt3.gmail-smtp-in.l.google.com.
40 alt4.gmail-smtp-in.l.google.com.
10 alt1.gmail-smtp-in.l.google.com.
20 alt2.gmail-smtp-in.l.google.com.
Now armed with the MX records, you can try to telnet to port 25 on one or more of them.
Code:
root@server [~]# telnet alt3.gmail-smtp-in.l.google.com 25
Trying 173.194.75.27...
Trying 2a00:1450:4008:c01::1b...
telnet: Unable to connect to remote host: Network is unreachable
root@server [~]# telnet mta6.am0.yahoodns.net 25
Trying 66.196.118.240...
Trying 98.138.112.37...
Trying 98.136.217.203...
Trying 98.138.112.34...
Trying 66.196.118.36...
Trying 98.136.216.25...
Trying 63.250.192.45...
Trying 98.138.112.35...
telnet: Unable to connect to remote host: Connection timed out
root@server [~]# telnet mx4.hotmail.com 25
Trying 65.54.188.126...
Trying 65.55.92.152...
Trying 65.55.92.136...
Trying 65.55.37.72...
Trying 65.54.188.94...
Trying 65.55.37.120...
Trying 65.55.92.184...
Trying 65.55.92.168...
Trying 65.54.188.72...
Trying 65.54.188.110...
Trying 65.55.37.104...
Trying 65.55.37.88...
telnet: Unable to connect to remote host: Connection timed out
As you can see from the above, all of them sit there simply Trying to connect. That's a big indication that a firewall (either yours or one upstream from you is blocking the connection). To test this thoroughly, you should also try to telnet to each one of the above to port 25 from another location that is not your server and not using your servers internet connection. If it works from the other location, then chances are it is your server or your data center, NOC or hosting provider that is blocking (also known as proxying) port 25.
NOTE: The MX records for them can change at any time without notice. So you should verify the servers to use for testing each time.
Here's another one where a server administrator added a DROP rule for port 25 to his server and wondered by email was not being delivered...
First the rule:
Code:
12 14 NNN DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:25
Here's the resulting log file entries:
Code:
2013-12-24 17:19:25 1VvSpO-0006ga-QZ SMTP error from remote mail server after initial connection: host mx.aol.com [xx.xx.xx.xxx]: 421 mx.aol.com Service unavailable - try again later
SPF/dkim/dmarc issues
Let's talk a little about SPF (Sender Policy Framework). Lots of information can be found atwww.openspf.org.
A message in the /var/log/exim_mainlog file might look like this:
Code:
SMTP error from remote mail server after HELO server.someserver.com: host backup.anotherserver.net [xxx.xxx.xx.xx]: 550-SPF HELO check failed (Fail): 550 See http://www.openspf.org/why.html?send...therserver.net
This message indicates that the message was rejected by the mail server because it didn't have a valid SPF record. To check the SPF record of the domain in question we would again use the dig command:
Code:
root@server [~]# dig anotherserver.net TXT +short
If you get nothing back, then you need to have the customer add an SPF record to their DNS. Documentation on how to set up SPF and DKIM records can be found here.
Additional note: There is another form of email authentication that Google/Gmail is using called dmarc.
If you have email that is either being rejected or is landing in the Spam folder at Gmail, look at the full headers (Show Original) and look for "dmarc=fail If you don't yet have a dmarc record, you can learn how to set one up here.
There is a Feature Request for adding dmarc to cPanel's Email Authentication section.
If you do get something back, then using the information at openSPF.org should help you decipher what the record is doing (or not doing). Then armed with that information you can go to the link in the error message (if one is provided) and find out why they are blocking the message.
Not permitted to relay errors
Another common message we see is this one:
Code:
Please turn on SMTP Authentication in your mail client. mx.someserver.net (mx.someserver.net) [xxx.xxx.xxx.xx]:4471 is not permitted to relay through this server without authentication.
This is pretty self explanatory, and usually means that the MUA (Mail User Agent) which is usually Outlook doesn't have the "My SMTP server requires authentication" option checked.
However, it can also mean that you have a script or program/application that is not authenticating properly before attempting to send email.
UNKNOWN email address
Here's a log entry showing a rejection due to an invalid email address:
Code:
2013-12-12 12:44:55 1Vr5dP-0004RG-6a SMTP connection outbound 1386852295 1Vr5dP-0004RG-6a someserver.com [email protected]
2013-12-12 12:44:56 1Vr5dP-0004RG-6a ** [email protected] R=dkim_lookuphost T=dkim_remote_smtp: SMTP error from remote mail server after RCPT TO:<[email protected]>: host mail.someserver.com [xx.xxx.xxx.x]: 553 sorry, unknown email address;
In this case, the person obviously tried sending to an email address that doesn't exist.
No such user here (but with a slight twist)
This one had us scratching our heads at first. See if you can figure out what is going on here:
Code:
2013-12-14 02:01:57 1VrkAf-0007F8-30 <= [email protected] H=localhost (onthisserver.com) [127.0.0.1]:53861 P=esmtpa A=dovecot_login:someuser S=853 id=a460ba02532956bad8e1b98fee9c5b12.squirrel@onthi sserver.com T="test from cPanel - please ignore" for [email protected]
2013-12-14 02:01:57 1VrkAf-0007F8-30 ** [email protected] R=virtual_aliases: No Such User Here
2013-12-14 02:01:58 1VrkAf-0007F8-30 Completed
See the R=virtual_aliases line? That means it is routing to a virtual alias on this server, and that user doesn't exist. But wait, it's going to a @yahoo.com account??? The problem here was that the server administrator added the following line to the /etc/userdomains file:
Code:
# grep yahoo /etc/userdomains
yahoo.com: yahoo
Obviously this server is not answering for Yahoo https://www.rise.company/forum/clear.gif Simply removing that line from /etc/userdomains fixed that issue.
Custom SpamAssassin Rules
Let's look at SpamAssassin and a message that was marked as spam and bounced back to the recipient even though it really isn't spam at all. In the exim_mainlog file, you may see something like this:
X-Spam-Status: No, score=0.8
X-Spam-Score: 8
X-Spam-Bar: /
X-Ham-Report: Spam detection software, running on the system "server.somedomain.com", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email.
This particular message got a score of 0.8 (less than 1) so it wasn't really flagged as spam, but as ham. Now, you may be asking your self, what the heck is ham?? "Ham" is email that is not Spam. In other words, "non-spam", or "good mail".
So why was this message returned? It's not considered spam so shouldn't it have been delivered? Once again, looking at the exim_mainlog file, we see the following:
Code:
spamd[17011]: config: score: requires a symbolic rule name and 1 or 4 scores
spamd[17011]: config: SpamAssassin failed to parse line, no value provided for "score", skipping: score 2
Hmmm. Looks like a SpamAssassin configuration problem. Looking at:
SpamAssassin Configuration Documentation, the correct syntax for using the score option, is "score SYMBOLIC_TEST_NAME n.nn" So a configuration of "score 2" is syntactically incorrect.
Spam detected - Message bounced
So here's one where spam was actually detected and the message was bounced. I've included this just so you can see what it looks like in your /var/log/exim_mainlog file.
Code:
2014-01-15 11:39:48 1W3TVL-003z0J-Lq H=(server.domain.com) [xx.xx.xxx.xxx]:44151 Warning: "SpamAssassin as server detected message as spam (26.2)" 2014-01-15 11:39:48 1W3TVL-003z0J-Lq <= [email protected] H=(server.domain.com) [xx.xx.xxx.xxx]:44151 P=esmtp S=11026 [email protected] T="Update Your Biography" for [email protected] 2014-01-15 11:39:48 1W3TVL-003z0J-Lq => /dev/null <[email protected]> R=central_filter T=**bypassed** 2014-01-15 11:39:48 1W3TVL-003z0J-Lq Completed
Here, you can tell that it was NOT delivered because of the "central_filter" and "bypassed" line.
Filters... Friend or foe?
What exactly is a filter? A filter is a way of processing and email message but with certain criteria. Common uses for mail filters include organizing incoming email and removal of spam or viruses.
Code:
2014-01-02 10:19:53 1Vykza-0005s5-La <= [email protected] U=root P=local-esmtp S=449 T="This is a test message with filters" for [email protected]
2014-01-02 10:19:53 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1Vykza-0005s5-La
2014-01-02 10:19:53 1Vykza-0005s5-La => /dev/null <[email protected]> R=virtual_user_filter T=**bypassed**
2014-01-02 10:19:53 1Vykza-0005s5-La Completed
If you see this: (R=virtual_user_filter and T=**bypassed**), that means that a filter has been set up to discard or delete the message. You can also see that it was redirected to /dev/null. Let's see if we are correct by going to:
Code:
# cd /home/cptest/etc/cptest.net/newuser
# ls -al filter*
-rw-r--r-- 1 cptest cptest 339 Jan 2 10:13 filter
-rw-r--r-- 1 cptest cptest 232 Jan 2 10:13 filter.cache
-rw-r--r-- 1 cptest cptest 253 Jan 2 10:13 filter.yaml
# cat filter
# Exim filter
# Auto Generated by cPanel.Do not manually edit this file as your changes will be overwritten. If you must edit this filter, edit the corresponding .yaml file as well.
if not first_delivery and error_message then finish endif
#Fubar Rule
if
$header_from: is "[email protected]" then
save "/dev/null" 660
endif
So, any message sent from [email protected] is automatically deleted by saving it into /dev/null
Those by the way are user level filters. For Account Level Filters, you would look in the /etc/vfilters directory.
Blacklisted IP Address
How about if your servers IP address is blacklisted? First of all, you can check to see if your IP address is blacklisted or that of the customers ISP. It's more common that your customer's IP address is blocked.
Code:
SMTP error from remote mail server after initial connection: host mx.ayahooserver.net [xx.xxx.xxx.xx]: 421 4.7.1 [TS03] All messages from xx.xxx.xxx.xxx will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html: retry timeout exceeded
This particular message means that Yahoo's server has blacklisted you (or your customers IP address). You should check with them (using the link they provided) as to why. Notice that Yahoo is using a soft-error code (421)? They *should* be allowing retries and not state that Retrying will NOT success. It's technically against RFC.
Here's a similar one from Google:
Code:
SMTP error from remote mail server after end of data: host ASPMX.L.GOOGLE.COM [74.125.129.27]:
550-5.7.1 [xxx.xxx.xx.xxx 1] Our system has detected an unusual rate of
550-5.7.1 unsolicited mail originating from your IP address. To protect our
550-5.7.1 users from spam, mail sent from your IP address has been blocked.
550-5.7.1 Please visit http://www.google.com/mail/help/bulk_mail.html to review
550 5.7.1 our Bulk Email Senders Guidelines. xa2si10793227pab.345 - gsmtp
Remember that you can use MX Toolbox to check an IP address to see if it is blocked on any other lists. No matter how you get on a blacklist, your first priority is to contact the blacklist and find out how to get removed. some lists, you don’t need to do anything; for others, you may need to make changes and prove that you have made those changes. The procedures are as varied as the lists themselves.
Invalid PTR/rDNS (reverse) record
rDNS (Reverse or PTR). Another common problem with email being returned is if the IP address that you are using to send email from does not have a PTR (or rDNS/reverse DNS) record set up. Let's use cPanel's MX IP address to test this:
Code:
# dig -x 208.74.121.68 +short
mx1.cpanel.net.
That happens to be correct, so we know that mx1.cpanel.net points to 208.74.121.68 and the reverse of that IP address points back to mx1.cpanel.net. If you get back nothing at all, or something other than what you expected, then that can be a problem as some mail servers will reject email if the PTR does not match or is missing. To set up an rDNS properly, your ISP/Hosting provider, NOC or Datacenter must have delegated the IP address to you. The following documentation will show you the steps you need to set up your own rDNS.
If they have NOT delegated the IP address to you, then they will need to set up the proper rDNS entries for you.
Invalid entry in /etc/resolv.conf
An invalid /etc/resolv.conf file can also hinder delivery messages. In that case the log file may look something like this:
Code:
2014-01-02 12:45:59 1VynH0-0006D0-Oc <= [email protected] U=root P=local-smtp S=417 T="Test2" for [email protected]
2014-01-02 12:45:59 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1VynH0-0006D0-Oc
2014-01-02 12:46:00 1VynH0-0006D0-Oc == [email protected] R=dkim_lookuphost defer (-1): host lookup did not complete
So if your /etc/resolv.conf file contains an IP address that can not resolve gmail.com then this is a problem. Once fixed the same message can be seen as having been successfully sent.
Code:
2014-01-02 12:48:52 1VynJq-0006DL-OO <= [email protected] U=root P=local-smtp S=425 for [email protected]
2014-01-02 12:48:52 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1VynJq-0006DL-OO
2014-01-02 12:48:57 1VynJq-0006DL-OO => [email protected] R=dkim_lookuphost T=dkim_remote_smtp H=gmail-smtp-in.l.google.com [173.194.77.27] X=UNKNOWN:ECDHE-RSA-AES128-GCM-SHA256:128 C="250 2.0.0 OK 1388688538 jb8si45146276obb.27 - gsmtp"
2014-01-02 12:48:57 1VynJq-0006DL-OO Completed
Sender verify failures
Here's one where the server shows Sender verify failed...
Code:
SMTP error from remote mail server after RCPT TO:[email protected]: host someserver.com [xxx.xxx.xxx.xxx]: 550-Verification failed for [email protected] 550-The mail server could not deliver mail to [email protected]. The account or domain may not exist, they may be blacklisted, or missing the proper dns entries. 550 Sender verify failed
In this particular case, the problem was that there was no A record for the host.someserver.net account. The error message was pretty much right on the money with the account or domain may not exist.
Once the customer was advised to add an A record, the problems went away.
FQDN Problem
Here's another interesting error...
Code:
2013-12-24 13:08:21 H=mail.someserver.com [xx.xxx.xxx.xx]:56035 F=<[email protected]> temporarily rejected RCPT <[email protected]>: remote host address is the local host
This was caused because the host name was not a Fully Qualified Domain Name (FQDN). To fix this, make sure your hostname for the server is always a FQDN.
Summary
As you can see there are dozens of reasons why an email message may not be delivered. Log files don't lie, and they will always be able to tell you why the message could not be delivered. It just takes a little research on your part. I hope that the above information is helpful to you and if you have anything to add to this please let me know.
As an additional note, if you haven't read it yet, I recommend reading the SPAM Primer by Randy Cassingham. This truly explains in laymans terms what spam is, who sends it, and how they got your email address in the first place.
Please feel free to post questions or comments below. I'm sure there are dozens of other reasons for mail delivery failure that I have not covered in this post. The point however is that you should always check the log files and remember that email does NOT just disappear.
