Kavi Mailing List Manager Help

Chapter 12. How Mailing Lists Work

Mailing List Characteristics

A mailing list is similar to an alias because it works by distributing email messages it receives to a list of subscribers, as discussed in Introduction to Mailing Lists and Aliases. But mailing lists differ from email aliases because of their ability to apply rather complex rules that determine how they process the messages they receive, whereas aliases simply forward every message indiscriminately. If you need to know how mailing lists work because you are responsible for managing, troubleshooting or creating mailing lists, this document is for you.

Mailing lists can be configured in a variety of ways according to the organization's objectives for each list. There are two main divisions of mailing lists: those that support ongoing group discussions among subscribers and those that support the one-way distribution of information from the organization to subscribers. Lists can be configured to allow the public to subscribe directly or to restrict the ability to add subcriptions to organization members or even administrators only. There are multiple factors that can be considered when matching configuration to list objectives. These are discussed in the List Types Concepts document.

Configurable Rules

Most mailing list rules are configured in the List Type through a combination of options available in the ezmlm-make argument string and the ezmlmrc rules file. These are extended by list-level options that control Web availability, Web archives and Subscriber Lists. Since all Kavi Mailing List Manager lists use the same default ezmlmrc file, the help focuses on the ezmlm-make arguments. The only reason you need to be aware of the ezmlmrc file is that your organization may have a custom List Type that uses a custom ezmlmrc file, although this is not common. If you are managing or troubleshooting a mailing list based on a List Type with a custom ezmlmrc file, the behavior described in the help won't necessarily apply. Consult the List Type description for an explanation of the behavioral differences.

Multiple Mailboxes

Mailing lists use multiple mailboxes so that messages can be filtered into different mailboxes and stored or processed rather than simply forwarded. For example, when a message is posted to the mailing list it is forwarded to all subscribers, but if the digest feature is enabled, the message is also sent to a digest mailbox where it is eventually processed into a digest and sent to addresses on the Digest Subscriber List.

Multiple Subscriber Lists

Unlike an alias, a mailing list can have several different kinds of Subscriber Lists, and apply different rules when it receives an email message depending on which of the Subscriber Lists the envelope sender's address appears upon (e.g., Regular Subscriber, Digest Subscriber, Moderator, Allow or Deny). So in the previous example, when a message is posted it is immediately sent to the email addresses on the Regular Subscriber List. When the digest is ready to send out, it is sent to the addresses on the Digest Subscriber List. See the Concepts document Subscriber Lists for more information.

Incoming Message Processing

Since the primary task of a mailing list is to receive email messages and forward them to subscribers, many of the configurable rules govern the message posting process. The main mailing list mailbox is the one that receives messages submitted for posting. When the mailing list receives a message at this address it takes the appropriate action based on its configured posting rules, possibly factoring in which Subscriber List the sender's address is on. Posting rules and other configuration settings determine how the message is then routed. For instance, if this is a moderated mailing list, the message may be sent to the moderation queue. For more information on posting rules, see the Concepts document Access Control and Posting Access in the Appendix.

Mailing lists can do more than filter and post email messages. Users can send an email message to a specific email command address to subscribe or unsubscribe, change their subscription type or retrieve archives. As with posting access, access to email commands is governed by configurable rules that may be applied according to the user's Subscriber Level. For more information, see the Concepts document Email Commands.

Forwarding and Generation of Outgoing Messages

Besides forwarding posted messages to subscribers, mailing lists can send messages in response to email commands, send moderation messages to moderators and as part of the automated bounce handling process. These messages are described in more detail in Bounces and Automated Bounce Handling and Mailing List Moderation. These standard ezmlm messages use default text that can be customized through the Super Admin tool Edit Mailing List Text.

Back to top

Email Access Versus Web Access

There are two ways to access mailing lists. Posting and most other user interactions are handled through email, but users may also be able to manage subscriptions or access Web-based archives through the website. Administrators see a much more Web-centric view of mailing lists than subscribers do. This can be an issue when trying to gain a comprehensive understanding of mailing list behavior regarding the subscription process and rules governing archive access. There is a tendency for administrators to focus on Web availability and overlook the fact that a different set of rules applies to access via email commands. For more information, see Access Routes: Email Versus Webtools (in the Concepts document Access Control).

Back to top

How the Posting Process Works

Kavi uses qmail as its MTA, so the qmail software manages messages sent to and from Kavi Mailing List Manager mailing lists. For more information on qmail, see The Big Qmail Picture.

Posts Are Screened at the Firewall

Every company or organization that handles email must be protected by a firewall, so every email sent to a mailing list passes through at least one firewall where it is subject to screening by virus and spam filters. If the email isn't well-constructed (e.g. the subject line is blank), it may be classified as spam by the spam filter. Spam messages are usually deleted automatically without any notification being generated.

If the organization has virus scanning enabled, messages sent to a mailing list may be scanned. If the message contains a virused email attachment, the email is deleted, but an email notification is sent to the sender. If the email message passes the tests for spam and malware, it is sent to the mailing list, then routed according to list rules.

How Posting Rules Are Applied

When a mailing list receives a message to be posted, it follows its configured rules and processes the message according to those rules. Most rules require the mailing list to identify the envelope sender and check whether the envelope sender's address is on a specific Subscriber List and then applying the posting rules for that Subscriber List.

When certain sender messages are rejected:

The mailing list checks its rules to see what kinds of senders it should reject. If it is configured to reject messages from senders who aren't on a specific Subscriber List (i.e, moderators only) or set of Subscriber Lists (moderators and other subscribers), it checks whether the sender's address is on any of the required Subscriber Lists. If the sender's address isn't found, the post is immediately rejected.

When the Deny Subscriber List exists:

If this mailing list has a Deny Subscriber List, the mailing list checks whether the sender's address is on this list. If found, the post is immediately rejected. The following steps apply if the sender's message wasn't rejected or denied.

When the rules are the same for all senders:

If the posting rules say to post all messages directly, then the message can be sent directly to the list without checking into the sender's identity. If the posting rules say that anyone can submit posts but all posts are moderated, then the message can be sent to the moderation queue without checking the sender's identity.

When only moderator posts are accepted:

The mailing list determines the identity of the envelope sender, then looks for this address on the Moderator Subscriber List. If it finds the address, it applies the rules to determine how to route the message and either posts the message or sends it to the moderation queue. If the sender's address isn't on this list, the post is rejected.

When different rules apply for subscribers versus public senders:

The mailing list determines the identity of the envelope sender. If this is a moderated list, it searches for the sender's email address on the Moderator Subscriber List and posts the message (assuming moderators are allowed to post directly). If the address isn't found there or if this isn't a moderated mailing list, the mailing list looks for it on the Regular Subscriber List, Digest Subscriber List and Allow Subscriber List (assuming these all exist). If the address is found on any of these Subscriber Lists, the message is posted directly or sent to the moderation queue, depending on what rules apply to posts from subscribers. If the address isn't on these Subscriber Lists, the sender is classified as public and the mailing list applies whatever rules apply to public posts, either sending the post to the moderation queue or rejecting it outright.

How Posts Are Routed

Posts are routed according to the configured posting rules, and the rules usually apply differently to senders whose addresses are on different Subscriber Lists.

What the list can do with a message it receives:

Post it directly to the list

If subscribers are allowed to post directly, but public posts are moderated or rejected, only posts from subscribers, posters and moderators will be sent to be posted.

Send it to the moderation mailbox

If the list is configured to moderate posts from this level of sender, the message is sent to the moderation mailbox. Messages received by the moderation mailbox are sent to moderators for review. If the sender is a moderator, the moderation request is sent to this moderator only, so the moderator can approve their own message. A stub is added to track the message and record its status in the moderation queue.

Send it to the rejection mailbox

If the list is configured to reject post from certain levels of senders, it routes messages from these senders to the rejection mailbox. When a message is sent to the rejection mailbox it is packaged into a rejection notice and returned to the sender.

From the moderation queue, messages can be:

sent to the accept alias

Messages sent to this mailbox will be posted to the list, and forwarded to the moderation queue where the moderation stub that stores information about the action taken on each message in the queue will be updated.

sent to the reject alias

Messages sent to this mailbox will be returned to the sender with a rejection notice. The moderation stub that stores information about the action taken on each message in the queue will be updated.

automatically removed

Messages in the moderation queue that aren't acted upon will eventually timeout and be returned to the sender with a notice. The moderation stub that stores information about the action taken on each message in the queue will be updated.

When a message is posted to the list, it is sent to the:

Regular Subscriber List

The Regular Subscriber list alias mailbox distributes the message to every user on the Regular Subscriber list.

Digest Subscriber List

If the list has a digest, the message is forwarded to a digest alias mailbox, where it is stored until the next digest is generated. Digests are configured so that they are generated after a set period of time or when the volume of accumulated messages reaches some preconfigured number of bytes.

raw ezmlm-idx archives

If archiving is enabled in the List Type on which this mailing list is based, the ezmlm-idx software creates and manages an archive mailbox. The message is forwarded to the mailbox that stores and indexes these messages. If this mailing list has Web-based archives, the ezmlm-idx software that manages these raw archives contacts the MHonArc software that creates the Web-based archives so that it can add the new message to the Web archives. This can happen instantly or periodically, depending on list configuration.

Back to top

Constructing Outgoing Messages

Mailing list email has certain features that distinguish it from email correspondence sent directly from person to person. Before a message is distributed to the list, the mailing list resets the information in certain fields.

A mailing list message has these distinguishing features:

'To:'

First, the message is addressed to the list, so it has the listname in the 'To:' field instead of the individual subscriber's address. This keeps a subscriber's email addresses from being revealed to every other subscriber on the mailing list.

'From:'

To protect the poster's email address, this field contains the mailing list name.

'Reply-To'

This field contains the address of a bounce handling mailbox. If an email sent to a subscriber bounces for any reason, the bounced email is returned to the address in the 'Reply-To' field rather than being sent to the list and distributed to all the list's subscribers. Instead, it is sent to be processed by an automated bounce handler. For more information, see Bounces and Automated Bounce Handling.

Note

The fact that the 'To:' field doesn't contain the address of the intended receiver, or that the 'From:' and 'Reply-to' fields don't match sometimes results in list messages being misclassified as spam and deleted by overzealous spam filters.

Back to top

Subscription Processes

Depending on configuration, list users may be able to subscribe to the Regular or Digest Subscriber Lists through ezmlm email commands, through Kavi Mailing List Manager User Tools or automatically.

Automated Subscriptions

Subscribers are added automatically to Members Mailing Lists, which are populated by a configurable query run against the Kavi Members database. Subscribers cannot manage their own subscriptions on this kind of mailing list, which includes the default 'members' mailing list. Members Mailing List queries are customizable, so subscribers may be selected based on assigned types, purpose or other criteria.

Assuming the 'Provide Members Email Option' is enabled in Configure Privacy Options, Members Mailing Lists may be configured to respect user opt-out or not. Mandatory mailing lists do not respect user opt-out, so subscribers cannot be removed from these lists. If the mailing list is configured to respect opt-out, users or administrators can remove the user from all optional Members Mailing List by setting the opt-out option in the user's record. If a subscriber is removed from a Members Mailing List by an automated bounce handler, the subscriber is automatically re-subscribed.

Subscribing Via Email

If a mailing list accepts subscription requests via email, users can subscribe themselves directly to the Regular Subscriber List or Digest Subscriber List, or unsubscribe from either of these lists, using ezmlm email commands.

The list user must usually confirm the subscribe or unsubscribe request by replying to a confirmation request sent to the address for which the request was made. If the mailing list accepts public email commands but has a moderated subscription process, the moderator also needs to confirm the request before the subscription will be added.

The user may subscribe under any email address, so even if this subscription belongs to an account holder, the subscribed address may or may not be in the Kavi Members database. When an account holder subscribes via email and the email address matches one in the Kavi Members database, the software automatically associates this subscription with the appropriate account holder. If the account holder later updates this email address in the Kavi Members database or their account status changes, all associated subscriptions are automatically updated as well, even those added via email.

When an account holder subscribes under an email address that isn't in the Kavi Members database, the subscription can't be associated with the account holder. These disassociated or orphaned subscriptions can't be maintained with the same degree of accuracy as those that are associated with an account holder. List users subscribed under email addresses that aren't in the Kavi Members database are classified as Public Subscribers. Public Subscriber subscription information is stored in the ezmlm Subscriber Lists and Kavi Mailing List Manager Database, rather than the Kavi Members database.

Subscribing through Webtools

When an account holder is subscribed through Kavi Mailing List Manager tools, the primary email address is pulled from the Kavi® Members database. If the list is configured to use an Allow Subscriber List and there are alternate email addresses stored in Kavi Members, these alternate email addresses are automatically added to the Allow Subscriber List to confer subscriber-level posting privileges. If an account holder email address changes, the address is updated on all mailing lists to which it is subscribed.

User access to webtools used to subscribe to a mailing list is configured in the Web Availability option. The public may be allowed to subscribe (although this requires customization, since User Tools are ordinarily protected and require login), or any account holder may be able to subscribe directly. More private lists require new subscribers to be added by administrators. The most private lists of all, those that use the 'Administrators Only' setting, actually prevent subscribers from unsubscribing themselves from the list.

Administrators can use tools available through the Admin Tools menu to subscribe or unsubscribe addresses from any Subscriber List, including the Moderator, Deny or Allow Subscriber Lists. If an administrator unsubscribes an address from a mailing list that is maintained through automated processes, such as the Members list, the automated process will respect the administrator action and not re-add the subscription.

Back to top

Subscriber Data Issues

There are a couple of data management issues that you should be aware of if you are managing mailing lists. Though this pertains to mailing list management, it is included here rather than the document Managing Mailing Lists because it is a natural outcome of how mailing lists work and the explanation is based on information discussed in detail in the preceding sections of this document.

Data Synchronization Issues

As already discussed, Subscriber Lists are actually managed by the third-party ezmlm software. Email addresses can be added or removed from these Subscriber Lists through ezmlm email commands, through automated processes, and through Kavi Mailing List Manager tools. Subscriber data is actually stored in three places: the ezmlm Subscriber Lists, the Kavi Members database and the Kavi Mailing List Manager database. When a subscription change is initiated through webtools, the ezmlm subscriber lists are immediately updated. But when the change is initiated through ezmlm email commands, the information isn't immediately propagated to the databases. Instead, the information in the ezmlm Subscriber Lists is propagated to the application databases on a nightly basis, or by an administrator performing a manual database synchronization operations through the Synchronize Lists tool. For more information, see the page help for the Synchronize Lists tool.

Issues with Public Subscribers

The existence of Public Subscribers on a mailing list gives rise to several issues. A Public Subscriber is any subscribed address that isn't in the Kavi Members database. As stated previously, a Public Subscriber may be an account holder using an email address that isn't listed in their Kavi Members account, or may truly be a member of the public with no affiliation with the organization other than as a consumer of information being distributed through some organization mailing list.

The issues mentioned here are primarily associated with public mailing lists that accept email subscribe commands, but these lists are the best way to facilitate a large public subscribership, which is ideal for newsletters. This is also a convenience for members because they can easily subscribe under a personal or temporary address.

Uncaptured Participation Data

If an account holder is on a mailing list as a Public Subscriber, the account holder's participation in this list isn't recorded.

Obfuscated Troubleshooting

It is more difficult to help an account holder who is experiencing problems with mailing list subscriptions if the account holder is on one or more lists as a Public Subscriber. The account holder may not remember which email address was used to subscribe to a mailing list and since these subscriptions aren't associated with the account holder's record, there is no information about this subscription in the account holders record. The administrator may have to check multiple email addresses individually to troubleshoot the issue. For example, a user who wants all their subscriptions canceled may use Web tools to cancel all those associated with their account, but will continue to receive email from the dissociated Public Subscriber subscriptions until these are tracked down and removed.

Difficulty Updating Email Addresses

When a list user needs to update their email address, they need to update every Public Subscriber subscription individually or contact and administrator (administrators can update this email address globally, so if the Public Subscriber is aware of this, there shouldn't be an issue). If an account holder is on a mailing list as a Public Subscriber, updates made to email addresses in the Kavi Members account won't affect the Public Subscriber subscriptions (unless the Public Subscriber address is added to the Kavi Members account in which case the software will be able to associate these orphaned subscriptions with their rightful owner). If a Public Subscriber email address becomes invalid, the user won't be able to unsubscribe that address via email commands because they won't be able to submit the unsubscribe request or confirmation from that account. Eventually mailing list messages to this orphaned subscriptions will begin to bounce and the address will be unsubscribed by automated bounce handlers.

List Rot

Public Subscriber email addresses are more likely to be obsolete than email addresses associated with Kavi Members accounts. This can lead to the dreaded condition known as "List rot"—a corollary of the second law of thermal dynamics regarding an ordered system's tendency towards entropy. Every now and then a subscriber moves to a new ISP and the Public Subscriber address they've been using becomes invalid. You can expect some subscribers to update their addresses all of the time, and you can expect others to update their addresses some of the time, but you can't expect all subscribers to update their addresses all of the time. The tendency of the number of bad email addresses to grow over time is called list rot. Lists combat list rot with automated bounce handling. However, mailing list messages aren't sent to addresses on the Allow Subscriber List, so this type of Subscriber List is especially vulnerable to List Rot and may have to be cleaned out manually from time to time.

Back to top