By Dr. Philip Baczewski, Associate Director of Academic Computing

No Will to Fight SPAM

For eight days in June and July 2004, UNT tried another experiment in SPAM prevention. Well, it was more like an experiment in the denial of SPAM, but it turns out that denial has another meaning.

From June 29 until July 7, UNT employed one of the more stringent measures that can be used to deny acceptance of SPAM E-mail. We enforced the SMTP standard (at least part of it.) RFC821  defines the Simple Mail Transfer Protocol. It provides the methodology used by all systems on the Internet to transfer E-mail. The first message exchanged when opening an E-mail transfer session is to transmit the "HELO" command. As RFC821 defines it, "HELO <domain> . . . may be interpreted as saying 'Hello, I am <domain>'". The purpose of the HELO command is "to ensure that the hosts are communicating with the hosts they think they are."

Of course, SPAMmers don't want you to know what host from which they are communicating, so often SPAM message transmission starts with a HELO command and a non-existent domain. During the eight day period, the UNT mail exchange systems checked the value of the HELO domain and rejected acceptance of the E-mail message if the domain provided did not exist. Whether or not the domain provided was the originating domain or not was not checked -- just whether or not it existed.

Now, you would think that "legitimate" mail servers would have no problem providing you a legitimate domain which represents their Internet presence. You would be wrong. It turns out that lots of E-mail transmitted for legitimate business purposes comes from mail servers which are not configured to honor RFC821. Most of the problems probably come from poorly configured software or software which does not support good configuration. Either someone has just filled in a arbitrary value in their E-mail transfer configuration or the software is taking some arbitrary value such as the machine name or machine IP (which may be on a private network or such).

Did the measure work to cut down on SPAM? Definitely. Did we get a lot of complaints about mail that was undeliverable. Definitely! More about both of these topics later, but first an observation about SPAM volumes.

SPAM takes a quantum leap

For several years, I have been using SpamAssassin to filter as much SPAM as possible out of my incoming mail stream. From the logs of my mail system, I can tell how much of the E-mail I receive has been identified as SPAM. In the last year and a half, the amount of identifiable SPAM I receive has gone from 12% of all my E-mail to around 50%. In January of 2003, 225 messages were identified as SPAM. In May of 2004, 1826 messages were identified as SPAM.

In November of 2003, UNT instituted some other anti-SPAM measures on our mail hosts. Identified SPAM had grown to a level of almost 30% of my E-mail by October of 2003, but the anti-SPAM measures seemed to have an effect, with identified SPAM levels dropping to 15-16% for December and January. This respite was short-lived, however, and in March 2004 my amount of identified SPAM took a quantum leap up to 41%, on it's way to the peak of 50% in May. (For you visual types out there, I've included a graph of this activity).

Did blocking invalid "Helo" domains work? It seemed to. This would explain the drop from 50% SPAM in May to 47% spam in June. But also, for the 8 day period preceding the "Helo" experiment, my percentage of SPAM was 47%. For the 8 day period during the blocking of invalid "Helo" domains, my identified SPAM was 37%. That's a 21% decrease in the proportion of identified SPAM (I know, that's a 10% decrease in identified SPAM, but it is a 21% decrease in the amount of SPAM versus the total amount of E-mail messages.) So, while denying invalid "Helo" domains didn't eliminate all SPAM, it was a start.

Unfortunately, we also found out that there are many mail servers on the Internet which are poorly configured. Whether it was a Dean expecting an E-mail from a government agency, or a University business office expecting an E-mail from a vendor, there turned out to be numerous cases where someone just could not do without a particular piece of E-mail, whether or not that E-mail could just as easily look like a piece of SPAM. In spite of our E-mail administrator's best attempts to keep up with the situation by creating an exceptions list where necessary and communicating about configuration issues when possible, the situation became overwhelming and our bold attempt to control SPAM turned into a short-lived experiment in Internet sociology.

Why is Internet E-mail in such a state?

Some (I won't name names) would say that it is because of the "Microsoft Effect" -- that is, you don't need to know any technical stuff to use this technology -- just point an click and our software will do the rest and you don't have to worry your pretty little head about it. Regardless of whether that's true, you have to wonder about such lack of adherence to standards on an internetwork based upon standards. Standards are in place so that we can all get along in a diverse environment and accomplish our activity with minimal conflict. If you don't follow the standards, whether it be RFC821 or the Geneva Convention, you are just making it easier for the "bad guys" out there to take advantage of your laxity.

I'd love to see the elimination of SPAM so I wouldn't have to filter out half my E-mail, but I don't see any general will to do so, either Internet-wide or in individual organizations. The will to do so means that an organization would embrace the idea of being a "SPAM-free zone" and expect to have to educate their E-mail correspondents about the standards used to judge acceptable E-mail. Rather than making exceptions for those who refuse to follow the "rules" it would mean standing firm in our belief in those rules as they have been defined by the Internet community. It means being able to enforce the idea that we just want to know our "hosts are communicating with the hosts they think they are."