Kavi Mailing List Manager Help

Chapter 31. Bounces and Automated Bounce Handling

What Is a Bounce?

If an email cannot be delivered, it is returned to the sender, or "bounced." Email delivery can fail for any number of reasons, and the failure may be classified as temporary or permanent.

When an MTA attempts to transfer an email and gets an error back from the receiving MTA, it checks to see if the error message indicates the failure is temporary. If so, it attempts to send the message again. If the sending MTA gets a fatal error message or is unable to complete the transfer after repeated attempts, it wraps the original message in a warning and bounces it back to the sender. For more information on MTAs and the email delivery process, see How Email Works.

Back to top

Why Does Email Bounce?

The most common reason email bounces is that the address of the intended recipient isn't valid. Email addresses change frequently as users migrate to new ISPs, create new accounts, move to new companies, etc., and users don't always update their mailing list subscriptions when their email addresses change. This is a significant maintenance issue for large mailing lists that may have thousands of subscribers. At any given moment, a mailing list probably has some bad email addresses on the subscriber list and this number tends to increase over time. This phenomena is known as "list decay."

Even if an address is valid, a mailbox may be temporarily unavailable because the owner has exceeded its allotted disk space, or a mail server may be temporarily unavailable because it is processing a large volume of mail.

If the address doesn't exist, the failure is classified as permanent.

Back to top

The Problem with List Decay

When a message is posted to a mailing list, an email is generated for every subscriber, then passed to an MTA for transfer to an MTA that receives email for the domain to which the message is addressed. Large mailing lists may generate more than ten thousand email messages to send to their subscribers every time a message is posted.

The mail queue has a limited number of slots available (e.g., 100) to process outgoing messages. Each of these slots is filled with an email and the remains occupied until the message is transferred successfully or is deferred (i.e., fails).

When an MTA encounters problems transferring an email, the email continues to occupy the slot while the MTA makes repeated attempts to transfer it. Eventually the MTA will give up and defer delivery, adding an error message to the mail logs and freeing up the slot for the next message in the mail queue. If there are many bad addresses on a mailing list, the average rate at which email clears the queue can drop significantly, as shown in the following diagram.

Figure 31.1. Mail queue

Slots in a mail queue are filled with
	    outgoing email. Slots that encounter problems transferring
	    a message stay occupied for a prolonged period of time,
	    reducing the number of available slots and delaying the
	    transfer of other messages in the queue.
Back to top

Automated Bounce Handling

Automated bounce handling is the mailing list's answer to list decay. When an email bounces, it is returned to the envelope header's 'Reply-To' address, which has been set to the address of an automated bounce handler. The ezmlm bounce handler sends a series of bounce messages to the addressed that bounced to test whether the bounce was temporary or permanent. The bounce handler tracks what happens to the messages it sends and responds accordingly.

The bounce handler uses the subscriber address and message number (encoded within the sender address) to track which messages have bounced for which address. Since messages can bounce for a variety of reasons, the ezmlm bounce handler does not remove a subscriber from the list immediately. Instead, it waits approximately 10 days after the first bounce (the actual default value is 1 million seconds), then sends a warning message to the subscriber. The warning message lists the numbers of the messages missed. The user can retrieve these from the raw archives, assuming this feature has been enabled. If the warning message bounces, too, ezmlm waits another 10 days, then sends a probe message. If this probe message bounces, the email address is removed from the subscriber list. This process guarantees that a subscriber will not be removed due to a temporary error, while assuring that all nonexistent or permanently failing addresses are removed.

How Bounce Probes Work

If the bounce warning or probe doesn't bounce:

Then it must have been delivered successfully, so the bounce handler determines the email address was only temporarily unavailable, and is actually a good address and takes no further action.

If the warning or probe bounces:

The bounce handler will continue to send probes until one is successfully delivered, in which case it determines the address is good, or it has sent the complete series of bounce messages including the final probe and they have all bounced.

If all probes bounce:

If every one of the series of probes bounces over the course of 24 days, the bounce handler determines the address is bad and automatically unsubscribes the address from the list.

Note

Since the bounce handler only unsubscribes addresses where all the bounce messages prove undeliverable, subscribers who receive a bounce message can be reassured that the fact that they received the message indicates a temporary condition has been resolved and they won't be automatically unsubscribed. If someone gets bounce messages repeatedly, it may indicate a recurring problem with their Internet mail service.

Auto-Unsubscribe

If the final probe bounces, the email address is automatically removed from the ezmlm Subscriber Lists by the ezmlm software without any notice being sent to list administrators.

Subscribers who are unsubscribed from Members Mailing Lists are automatically resubscribed the next time a message is sent to the mailing list, since the Regular Subscribe List is dynamically populated. On regular mailing lists, it is the subscriber's responsibility to resubscribe under a valid email address.

When an address is automatically unsubscribed by ezmlm, it is removed only from the ezmlm Subscriber Lists for that one particular mailing list. This change isn't immediately propagated to other parts of the system. It is propagated by a cron task that runs periodically, checking the Kavi Mailing List Manager database against all the ezmlm Subscriber Lists and adding or removing subscriptions from its database to match the ezmlm Subscriber Lists. This means there will be some period of time (less than 24 hours) after which the address has been unsubscribed from the ezmlm lists but still appears to be subscribed when viewed through Kavi Mailing List Manager tools.

Back to top

Bounce-Related Error Messages

There are different kinds of bounce-related error messages. The discussion so far has focused on bounce warnings and probes sent to test whether a delivery failure is permanent or transitory, but

When email delivery fails, the sending MTA returns error messages to the sender by email, and the server also records an error message in the mail logs. Both of these types of messages indicate whether the failure was temporary or fatal, and describe the type of failure.

Email bounce messages are wonderfully self-explanatory for the most part, but if you aren't completely sure how to identify what type of failure occurred, you can look up the error message in the troubleshooting flowchart What does this message mean?

Ezmlm Bounce Warnings

This is a sample of an automated bounce warning from ezmlm warning a list subscriber of a recent failure to deliver a list message to their company email address. The information in the bounce message indicates that the subscriber's company, example.com, rejected a message sent from a mailing list on June 19th. The fact that the subscriber received the bounce warning indicates that mail is now working correctly. The automated bounce handler recognizes that the warning didn't bounce, so it won't send another bounce message, nor will it unsubscribe the user. The bounce warning contains information and instructions the subscriber can follow to retrieve the missing message.

Example 31.1. Sample ezmlm bounce warning


From: list-listname-help@example.org
[mailto:list-listname-help@example.org] 
Sent: Sunday, June 19, 2005 6:04 PM
To: Username
Subject: ezmlm warning

Hi! This is the ezmlm program. I'm managing the
list-listname@example.org mailing list.

I'm working for my owner, who can be reached
at list-listname-owner@example.org.

Messages to you from the list-listname mailing list seem to
have been bouncing. I've attached a copy of the first bounce
message I received.

If this message bounces too, I will send you a probe. If the probe
bounces,
I will remove your address from the list-listname mailing list,
without further notice.

I've kept a list of which messages from the list-listname mailing
list have 
bounced from your address.

Copies of these messages may be in the archive.

To retrieve a set of messages 123-145 (a maximum of 100 per request),
send an empty message to:
   <list-listname-get.123_145@example.org>

To receive a subject and author list for the last 100 or so messages,
send an empty message to:
   <list-listname-index@example.org>

Here are the message numbers:

   32
          

Note

Depending on list configuration and the subscriber level assigned to their email address, people who receive bounce messages may or may not be able to retrieve missed messages by following the instructions in the bounce warning. For example, if the user attempts to follow instructions in the bounce message on retrieving missed messages from the archives, and uses a public (unsubscribed) address and the list is closed to public administration requests (i.e., the -p switch is set in the ezmlm-make argument string), the list will reject the attempt to retrieve the archives.

Mail Queue Warnings

Here is a sample warning from a mailer daemon that is managing an email in the mail queue that the email is delayed because the sending MTA has been unable to contact the name server (i.e., DNS server) to get the list of mail servers for the address domain (If you don't understand DNS lookup, see How Really Email Works). The server is probably only temporarily unavailable due to an unusually high number of requests or because it has been taken down for maintenance. Mail server failure is a relatively rare occurrance.

The message hasn't yet bounced, and there isn't any action required on the part of the sender, but the mail server is warning the sender that the message is delayed.

Example 31.2. Message delayed in queue because of unavailable DNS server:


    From: MAILER-DAEMON@mailserver.example.com
    Subject: Warning: could not send message for past 4 hours
    Date: July 27, 2005 4:06:14 PM PDT
    To: username@example.com

    **********************************************
    ** THIS IS A WARNING MESSAGE ONLY **
    ** YOU DO NOT NEED TO RESEND YOUR MESSAGE **
    **********************************************

The original message was received at Wed, 27 Jul 2005 12:03:50 -0700
from mailserver.othercompany.com [127.0.0.1]

   ----- Transcript of session follows -----
451 example.org: Name server timeout
Warning: message still undelivered after 4 hours
Will keep trying until message is 5 days old
Reporting-MTA: dns; mailserver.example.com
Arrival-Date: Wed, 27 Jul 2005 12:03:50 -0700
        

Error Messages in the Mail Logs

If you are troubleshooting a bounced message, you can use the Mail Delivery Logs tool to retrieve information about the message, including the error message provided by the local MTA.

Here is an error message extracted from the mail logs indicating the reason an email bounced (i.e., was rejected) by a mailing list. In this case, the sender tried to post an email that exceeded the list's file size limit, so the mailing list's MTA (i.e., 'remote host') rejected the message and provided the sending MTA with this error message.

Example 31.3. Message rejected because of file size limits:

Remote host said: 552 5.2.3 Message exceeds maximum fixed size (5242880)

Back to top

Troubleshooting Tools

Kavi Mailing List Manager provides a Bounce Report tool. This report can be viewed online in real time or the application can be configured to automatically generate and email a Bounce Report on a periodic basis. The Bounce Report contains information on subscribers who have been automatically unsubscribed from one or more mailing lists by a bounce handler.

Bounce Report

Click here to visit the Bounce Report tool.

Administrators can use this tool to search for addresses that have been unsubscribed by the automated bounce handler. The Bounce Report tool results page provides information about which mailing lists have been unsubscribed and links to other tools that can be used to manage this subscriber. By default, the subscriber's phone number is included in the report.

Super Admin Tool: Configure Bounce Report Summary

Click here to visit the Super Admin Tool: Configure Bounce Report Summary tool.

Super Admins can use the Configure Bounce Report Summary tool to schedule a weekly or monthly (or even daily!) roll up of Bounce Reports sent via email.

Back to top

Troubleshooting Hopcount

Why Hopcount Matters

The most common causes of bounces have already been described, but some spam filters bounce mailing list messages based on whether the message's "hopcount" exceeds some presumably reasonable number.

Most mail hubs still use Sendmail—a venerable email program with somewhat primitive loop detection. When Sendmail receives an email message, it examines the envelope header to try to determine whether looping has occurred. It can't identify looping directly, so it attempts to identify it indirectly by looking for one of the symptoms of looping: a high number of hop counts. Sendmail counts the number of MTA-to-MTA hops the email has made during the delivery process by counting how many 'Received' fields have been added to the envelope header during the email delivery process. If the hopcount exceeds some preconfigured number, Sendmail bounces the email message.

List Messages Have Higher Hop Counts

This can be a problem when delivering mailing list messages. By default, ezmlm propagates 'Received' fields to facilitate message tracking, so messages distributed by mailing lists tend to have a higher number of 'Received' fields than most email messages. These extra 'Received' fields can be enough to tip the balance and exceed the hopcount set on poorly configured Sendmail MTAs.

Diagnosis and Treatment

Since subscription confirmation requests, warnings and probe messages have fewer 'Received' fields, a subscriber may find they are able to receive these sorts of messages, but aren't receiving messages posted to the mailing list.

The ideal way to resolve this problem is to adjust the configuration of the receiving Sendmail MTA, but this is often outside the control of the subscriber or the list owner. To compensate, ezmlm can be configured to remove all of the 'Received' fields except for the first.

Back to top