posted on August 24, 2004
Overview: Spam Filtering on Incoming E-mail
In an ongoing effort to reduce the amount of unwanted, unsolicited e-mail, Academic Services and Emerging Technologies (ASET), a unit of Information Technology Services (ITS), will activate spam scoring on incoming e-mail going through psu.edu on Thursday, August 26. This latest effort marks an addition to the current spam filtering methods, started during fall 2003, to combat spam.
How It Works
On the server-side a "mail filter" will run on Penn State's incoming e-mail servers to scan for and identify spam. The mail will be analyzed, scored and flagged as spam per the rules used for Penn State's e-mail servers. This mail filter will insert mail headers, as noted below, in order to explain the results of the spam filtering. Users may add/create filters based on these rules to client-side e-mail programs in order to filter/handle messages flagged as spam.
NOTE: E-mail messages will not be deleted before reaching users.
The headers/rules are:
- X-PSU-Spam-Flag
- X-PSU-Spam-Level
- X-PSU-Spam-Hits
For example, this additional header information in a spam e-mail message would appear as:
- X-PSU-Spam-Flag: YES
- X-PSU-Spam-Hits: 5.82
- X-PSU-Spam-Level: ******
The X-PSU-Spam-Flag is the simplest rule and thus is likely the one users will employ. This rule uses an entry of either YES or NO to flag messages. If an e-mail message is flagged as YES, then it has passed enough tests to score as spam. It is recommended that users filter any e-mail messages marked as YES to a mailbox that allows them to review the e-mail. While work is being done to minimize the occurrence of false positives, it is difficult to guarantee 100% accuracy given the diverse community at Penn State.
The rules X-PSU-Spam-Level and X-PSU-Spam-Hits provide the actual numerical values of spam scoring, and are represented by a number and a corresponding value displayed as asterisks. Some e-mail clients cannot use a numerical value to determine whether or not an e-mail message is spam, but they can do so if asterisks are used. X-PSU-Spam-Level includes a number of asterisks depending on the spam level. The spam level is determined by the number of tests it matches. For example, if the spam level is 6, 6 asterisks will be used. Note that the X-PSU-Spam-Flag will flag an e-mail message as YES if it has a score equal to or above 3.
The X-PSU-Spam-Hits rule is a numerical value, either positive or negative, that is the sum total of every test. The higher the number, the more likely it is that the e-mail message is spam. Because this type of filtering is based on Bayesian filtering, which examines e-mail messages based on content characteristics and "trains" itself to recognize a users legitimate mail from spam mail per those characteristics, it is possible to have partial results. In this case, a message could have a score of 6.1.
On the Horizon
The effort to reduce the amount of unsolicited spam received by Penn State's e-mail servers is a top priority, one that is ongoing and is being aggressively pursued. All updates will be posted as available on ASET's home page and through various media avenues. For additional information as well as information about spam filtering trials, please refer to the Help/Resources information noted below.
Help/Resources
Questions, problem reports or requests for assistance may be directed to the ITS Help Desk at helpdesk@psu.edu. Documentation on how to set up client-side filters based on the rules noted above, specifically the X-PSU-Spam-Flag rule, are available via the ITS Help Desk Web site by clicking on the "Creating Client-Side Spam Filters" link. Instructions for X-PSU-Spam-Level and X-PSU-Spam-Hits will be available shortly.
