What is greylisting?Greylisting is a common anti-spam technique. It is an internet standard defined in RFC 6647 (http://tools.ietf.org/html/rfc6647). It complies and fits in nicely with the standard SMTP RFC 5321. Particularly the Retry Strategies section of the SMTP RFC (http://tools.ietf.org/html/rfc5321#section-4.5.4). How does it work?Greylisting is done by the receiving server. It simply issues a temporary deferral to the sending server. This means that the receiving server tells the sending server "I'm sorry. But I can't accept that message from you right now. But I may be able to later. So please retry delivery later. Don't bounce the message because I am not rejecting it. I'm just deferring it". This is the same response (deferral) that a receiving server sends when:A user's mailbox is full & cannot receive the message at this time.The receiving server is overloaded & can't handle processing any more messages at the moment.The receiving server is having some other problem, etc. and is not able to process the message presently but the problem is of a nature that it is likely that the message should be able to be received & processed later. Therefore rather than bouncing (permanently rejecting) the message, it simply defers it. Greylisting only applies to strangers:Greylisting is typically only done when an email is coming from an IP address & sender address (or domain) that we have never seen before (or at least not in the recent past). Odds are that 95-99% of the emails that you receive are coming from people that you have previously received email from. Greylisting engines maintain a database of IP addresses associated with domain names/senders & recipients. The greylisting engine refers to this database each time an inbound email is received. If the combination of IP/IP block and sender/sender domain has never before been seen, then it is not considered a "known sender". In this case, the message is deferred. Once that the sending server retries the message, the greylisting engine allows it through. So greylisting is usually only a 1-time deferral asking for the sender to retry again shortly. Retry:The sending server should be able to handle the deferral without any problem. Internet SMTP standard (referenced above) states that emails servers must retry delivery of deferred messages, generally for up to 4-5 days. Additionally, the first retry should happen less than an hour from the deferral. So in most cases, a person should end up receiving the email less than 1 hour after it was sent. As long as the sender retries the message from the same source IP address, the greylisting server will let it through. Future messages are not delayed/deferred:Greylisting engines do not delay future emails, so long as they are coming from a previously seen combination of IP address/block and sender address/domain. Only the initial such email is delayed. Then you're home free. Why does it work?A few reasons:Some spammers are too lazy to retry sending the spam. Or their email engine just doesn't because they really don't care about following SMTP rules in the first place. They treat a deferral as a failure & just move on.It gives spammers more time to get on a blacklist. If they are sending a mass spam from a newly infected PC, it is likely that the PC's IP won't immediately be on any email blacklists. But give it 5 minutes, and it's likely that it will be. Hence by delaying the message reception by a few minutes, it is much more likely that the receiving server will be able to correctly identify the message as spam because the sender IP addres is now on blacklists. Does it work with IP addresses or blocks? And is it using sender addresses, or domains? Which is it?Both. Ultimately it depends on the greylisting implementation. However, ours takes both the IP block into account as well as the IP address. It also takes both the sender's email address & the sender's email domain into account. If multiple emails from different addresses (but in the same domain) have been received within a certain period of time (60 days as of this writing), then we whitelist the sender's entire email domain for the associated IP address.Also, we whitelist an entire class C IP block rather than just the single IP, so long as the sender IP has a reverse DNS record & doesn't appear to be a home internet DNS record. What problems does greylisting cause?Temporary delays which can be annoying.Bounced or missing emails due to:Some email servers don't follow the rules (referenced at the top of this blog). These emails may be completely bounced immediately, or never bounced or delivered (the treacherous black hole). Note that the receiving/greylisting server does NOT bounce the message. It defers it. However, if the sending server doesn't adhere to internet standards, or it is misconfigured, then one of the above could occur. Note: If your email provider doesn't sent you bounce notifications for any email that it determines will not delivered, you should probably consider changing to a new email provider. The black hole should never happen and is a sign of an email service you probably don't want to use.Some larger email providers may have a pool of outbound servers. And some may actually retry transmission from a different IP address every time. In such instances, the email may never make it through the greylisting engine. The solution here is to notify your email provider and they can probably make an adjustment for the greylisting engine to completely whitelist the IP block (not just a single IP) the message is coming from. We can do this with our system.A novice web designer decides to use a PHP or similar email function that just sends email (like from a feedback form) directly to the destination servers, rather than to its own internal email server. The result is that such a PHP email function is NOT an email server and has no concept or mechanism to queue a message for retry. It fails to be delivered and will never be tried again. Instead, the developer should have configured his web script to send/relay all such emails through his own email server so that deferrals can be handled correctly. What to tell senders who don't retry transmission: They should fix their email server configuration (or replace their server) so that it is in compliance with internet email standards.Not responding correctly to Greylisting is in violiation of:RFC 6647 (Greylisting):http://tools.ietf.org/html/rfc6647andRFC 5321 (SMTP [sec. 4.5.4 - Retry Strategies]):http://tools.ietf.org/html/rfc5321#section-4.5.4The specific violations via the SMTP RFC are:"mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender.""The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP client can determine the reason for non-delivery. Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days." Is greylisting worth the hassle?Yes. When we consider the security risks associated with viruses/malware, as well as phishing scams, etc., you really have to take every reasonable measure to protect yourself. Greylisting can be a bit inconvenient at times, but it works quite well and is a part of a good anti-spam system. We recommend that everyone use an anti-spam system supporting greylisting. Turning it back off just allows too much spam through.
