the foobar zone - rants - why dialup smtp is a bad idea

Why dialup SMTP is a Bad Idea.

Lots of people ask why I regularly let off steam about dialup SMTP as a method for downloading email from their ISP. Here's a quick outline of my main gripes about it.

What SMTP is.

SMTP is a "push" oriented mail transport mechanism. This means that when you send an email from your workstation to an SMTP server, that server will attempt to deliver it or push it to the recipient's mail host. If it can't do this, it may send the email to a backup system, which can then forward or push the email to the correct destination at its convenience.

What dialup email needs.

Dialup email, on the other hand, requires a "pull" based protocol, so that it can connect to the ISP's mail system, pull all the mail down from it, and then disconnect. In addition to this, it also needs a mechanism to ensure that once the transaction has been completed successfully, you are guaranteed that

all of the mail at the remote end is delivered to you correctly, and
all of your mail has been uploaded correctly to the remote end.

Virtually no commonly-used production SMTP mailing system currently support this functionality (with the possible exception of CommuniGate Pro which provides some, but not all of this functionality). So in order to make dialup SMTP work at all, kludges are put in place by ISP's to make things work as well as they can.

Square pegs into round holes.

When a dialup customer logs into an ISP with their modem, the mail system must be notified that it should make some attempt to deliver email to that customer. There are normally two ways of doing this: either the ISP can correlate the customer's login to their domain name and then fire off a delivery attempt, or else they can leave it to the customer to tell the mail system that they want email delivered to them now, please, if they wouldn't mind.

Both of these methods are problematic. In the first case, many badly designed pieces of software do not start listening for incoming SMTP connections immediately, and when the mail system attempts to deliver mail to them, it fails and will often mark the host as being unavailable. This can lead to delayed mail which irks customers no end.

In the second case, the normal procedure is to use the extended SMTP ETRN command (documented in RFC-1985). Alternatively, the ISP will listen to queue-run requests on another TCP port (often "finger"). Regardless of which method is used, an unauthenticated signal is sent to the ISP's mail system to start a queue run.

Either way, control methods need to be built in to ensure that one the one hand, denial of service attacks or stupid customers cannot bring down the system or deny service to other people by making multiple requests, and on the other hand, so that legitimate customers can get their mail downloaded in a timely manner. Currently there is no off-the-shelf mail software which will do this.

In both cases, the "thundering hoardes" problem occurs. At certain well-defined times of the day, it is normal to get large jumps in the number of people logging in and wanting to download their email. This is because people use automatic schedulers to kick off mail download sessions, and the login/download times do not vary from day to day for each customer. A few minutes past the hour is a common enough time for half the world to log in. It has been speculated that people log in at a few minutes past the hour, because surely everyone is going to log in at exactly the hour, no? Whatever the psychology, the effect is very real and causes very real problems.

When does the modem get hung up?

Beats me. Most systems figure this out by deciding when the modem link has been quiet for more than a pre-determined length of time. However, in an era of per-second phone billing, this is wasteful of resources. It is also an approximation of whether all of the email has been downloaded, because there is no mechanism in SMTP to tell the remote host how many emails remain to be downloaded. Whatever the method used, the outcome is unsatisfactory.

Overloading

Many SMTP mail systems still cannot limit the number of concurrent connections to a remote host. In the case of dialup systems, if multiple delivery attempts are made to the dialup host - and this often happens - then the dialup link can be completely swamped, rendering it useless for other traffic (e.g. web, ftp, etc).

Kludges

With the exception mentioned above, there are no commonly used packages which implement everything necessary to implement dialup SMTP in anything except an insane manner, hacked together with often fragile kludges. Providing commercial services over kludged systems is rarely scalable and almost always bad for customers.

SMTP mail queueing.

When you send an email destined to a dialup SMTP host, it will normally get put into a mail queue on the ISP's mail system to be delivered to the recipient site as soon as they're on-line. If you're in luck and they're already on-line, the mail might even be delivered to them immediately. But most likely, the email is going to be put into a queue for later delivery. On most mailing systems, there are normally a very limited number of queue, which means in all likelihood, your email is going to be bundled in the same queue as all or most of the other emails on the system

Once the email is in the queue, the mailing system will then attempt to deliver it. Most mail systems are pretty unintelligent about how they manage their queues (that's the subject of a future rant), and normally have a small number of simplistic algorithms for figuring out how to deliver things in an efficient manner. Common to most mailing systems, though, is the fact that they do not sort mail into any useful order before they put it into a queue. This is a serious shortcoming, as we'll see later.

In addition to this, most mail systems also note whether delivery attempts were successful or not. The purpose of this is to create a list of all the remote hosts that the mail system talks to, so that it can make a sensible decision about whether to attempt to deliver another mail to the same host within a short period of time. Obviously, if there's 100 emails in a queue for a particular destination, there's no point in attempting to send each of them in turn if the remote host is refusing mail connections in the first place.

So you've got a queue full of emails which need to be delivered. What happens then is that the SMTP delivery program will sweep down through the mail queue, decide which emails it's going to try to deliver and will then attempt to deliver the emails. For non-dialup recipients, this works reasonbly well in most cases.

Why is intelligent queueing is necessary for dialup SMTP.

Ok, so you've got a queue full of email destined to the world and his wife, including (somewhere in there) mails for your domain. Most mail systems don't maintain a useful index of all these emails, so the queue sweeper program must go down through each mail separately and decide whether it's destined to your domain or not. This sweep normally consists of the following actions:

get a complete list of all of the files in a directory or directory tree,
open, read and close the delivery data file for each email, and
attempt to deliver the email.

If your queue is largish - say 10,000 emails - and you have a large number of dialup users - say one per second, this means that once every second, your queue sweeper program has to fire off a process to do the things listed above. This means huge disk i/o requirements, and more importantly, disk requirements which exist purely because of bad systems design.

This is a problem which is not just restricted to dialup SMTP, but rather which is exascerbated by dialup SMTP, even if it can sometimes be a problem with other mail relay systems. There are some mail systems which are well-designed and which have clever queue management systems, but they are among the mainstream mailing systems yet.

Summary.

In short, SMTP is trying to do a job which it really wasn't meant to do. In any situation, this is going to cause problems and in the case of the Internet, it is inevitable that - however unsatisfactorily - existing protocols are going to be patched up rather than new ones created.

Alternatives exist which include using UUCP, a protocol which is still arguably better than overloading the SMTP protocol to do things that it really wasn't meant to do in the first place.

Light at the end of the tunnel.

If dialup SMTP were a horse, it would have been taken outside a long time ago to be shot. People are forced to use it because it's the only service that their ISP uses for providing domain based email. This is a Bad Thing of itself.

Fortunately, RFC-2645) has taken some good steps to provide an overlay protocol for SMTP to alleviate most of the problems described in this document. Let's hope that people implement this as widely as possible, as soon as possible.