[FQuest Notice] SpamAssassin Upgrade

[sheila]

[FQuest Notice] SpamAssassin Upgrade

Due to the decreasing effectiveness of the current SpamAssassin filter in recognizing spam email, FutureQuest will be upgrading to the latest version of SpamAssassin, 2.55, on Wed, May 28, 2003

When: Wed, May 28, 2003
Window: 12:00AM - 3:00AM EDT
Duration: 5 - 30 minutes - affecting only users of
SpamAssassin Servers: MX01 and MX02

No email should be lost during this upgrade as a result of the delivery mechanism, however there is a potential for email loss due to SpamAssassin configuration (details below). Delivery of some inbound emails, which require scanning by SpamAssassin, may be briefly delayed. All other email services, including the sending of outbound email, and delivery of inbound email which does not require scanning by SpamAssassin, will be unaffected.

Due to the nature of the SpamAssassin software, it becomes less effective over time, as spammers change the characteristics of their email. Therefore, in order to check for the new types of email techniques that the spammers use, it is necessary to upgrade to a new version of SpamAssassin.

The current version of SpamAssassin running on the FutureQuest servers is 2.43.

FutureQuest will be upgrading to the latest version, which is 2.55.

Note: This is a major version upgrade, and not a minor version upgrade.

It is likely that the new version will assign different scores to your incoming email than the current version does. Therefore, if you have your email currently set to delete, you may wish to visit your CNC now and change this to some other action, or temporarily raise the Required Hits Score, to protect yourself against losing email.

FutureQuest is unable to recover any email that is deleted or bounced from your account as a result of email filters you have enabled.

Some of the configuration and preference options in version 2.43 are not compatible with the options in 2.55. FutureQuest will do their best to translate any preferences that you have currently set for version 2.43 into similar settings in version 2.55.

Specifically, the default option in the 2.55 version of SpamAssassin is to create a new email for the report, and attach the original email to this new report email. Other options include changing the Content-type of the attached email to text/plain in order to prevent auto-launching of dangerous attachments by some email clients, or not modifying the email in any way except to add headers.

For best results, we advise that you visit your CNC soon after the upgrade has been completed, and review the configuration options for yourself.

[Randall]

Great news!

Just realized that I haven't checked our spam box since I reactivated it two weeks ago. :o Now I have to sort through 180 lovely specimens, just in case something useful slipped through...

Nope. :blah:

Randall

[dank]

That reminds me of something I've been thinking would be a great addition to SpamAssassin, if it isn't already in the new version (or on the drawing board): a 2-tier filtering system that allows you to set a spam/tag threshold and a spam/delete one. For instance, it's fairly common to get false positives in the 5-10 range, so you want to keep those around to double check they are really spam. Anything over 20 is unlikely to be desireable, so deleting it on the spot would make sorting through the piles of spam that much easier, not to mention less server load delivering hundreds of unnecessary emails.

Dan

[Randall]

There have been discussions here about running filters against the tagged email, either server-side or in an email program. One of these days I'll get around to actually trying it...

Randall

[dank]

That still would be much less efficient than doing it in the same step as the initial SpamAssassin filter... Redundancy is not our friend in this case. :\

Dan

[Jeff]

That reminds me of something I've been thinking would be a great addition to SpamAssassin, if it isn't already in the new version (or on the drawing board): a 2-tier filtering system that allows you to set a spam/tag threshold and a spam/delete one. For instance, it's fairly common to get false positives in the 5-10 range, so you want to keep those around to double check they are really spam. Anything over 20 is unlikely to be desireable, so deleting it on the spot would make sorting through the piles of spam that much easier, not to mention less server load delivering hundreds of unnecessary emails.
In the config file for the newest version there are options for what to do with both {Spam} (tag, delete, bounce, etc.) and for {HighScoreSpam}

--edit... oops, I installed spamassassin in conjunction with mailscanner, and I'm not sure if the two-tiered option is a default spamassasin option that mailscanner is using and that would be really easy to also use or if it's something that mailscanner brings to the table, but I bet it wouldn't be too hard to do and I agree that it is a nice option.

[sheila]

OK, Jeff, I see you added a qualifier to your statement, as I was going to say...

I cannot find any reference to this two-tier option on this page:
http://useast.spamassassin.org/doc/Mail_SpamAssassin_Conf.html

[Jeff]

I'm sorry sheila, I misremembered when I posted as I first installed spamassassin standalone a month or two ago and then later installed it in conjunction with mailscanner, and that two-teir option is part of mailscanner's config file.

[sheila]

Sounds very nice, and I agree that it would be a welcome addition to the other email filtering options. (I only posted cuz you made me look. :P )

[frankc]

Is it the program that's been upgraded, the rules, or both?

I'd sure like to see some way for the rules to be updated automatically (with an email to site owners notifying us of the actual filter changes).

Is there any way SA can be configured so that the "engine" (for lack of a better description) can opt to load the configs and tweaks from within a domain? I don't want to waste the room and time to install SA myself but allowing site owners to optionally run their own configs would be handy.

[kitchin]

Here's a very small script that matches on the
*{spam _HITS_ }* tag. It's easy to test: just send yourself an email with
*{spam 11.1 }* in the subject.


#!/usr/local/bin/perl
## No warranties, understand before using.
###########################
## Detects mail SpamAssasin has tagged like Subject: *{spam 15.2
## using the SA setting Subect tag: *{spam _HITS_ }* or *{spam _HITS_ anything...
###########################
# Name this file "sa_simple.pl" and upload in ASCII/text to
# /big/dom/xdomain/efilter/sa_simple.pl
# Then set up a Simple Filter in CNC:
# sh -c '822field | /usr/local/bin/perl /big/dom/xdomain/efilter/sa_simple.pl'
# or
# bouncesaying "Spam?" except sh -c '822field | /usr/local/bin/perl /big/dom/xdomain/efilter/sa_simple.pl'
# Note: "822field" means "822field subject".
# Note: Change xdomain to xyourdomain above.
###########################

$maxscore= 11.0; ## change to a good threshold

##############################
## end of config section #####
##############################
$/=''; ## get all subject lines ## -edit-
$subject= <STDIN>;
$score=0;
## SA set to always use single leading space (right?)
while ($subject=~ /^ \*{spam ([\d.]+) /mg) { ## -edit-
$score=$1 if ($1 > $score);
}
exit(($score > $maxscore) ? 100 : 0);

[kitchin]

To redirect high scoring mail instead of bouncing it -- which is a good idea while you're testing -- change the exit line at the end to:

### bounce version:
# exit(($score > $maxscore) ? 100 : 0);

### redirect version:
# Simple Filter:
# condredirect hiscore@domain.tld sh -c '822field | /usr/local/bin/perl /big/dom/xdomain/efilter/sa_simple.pl'
exit(($score > $maxscore) ? 0 : 99);

Also, if you bounce SA mail then you are sending back the score, and the reasons, if you have that enabled.

[frankc]

To redirect high scoring mail instead of bouncing it -- which is a good idea while you're testingFrom everything I've read, there is no good reason to bounce emails. The "From" and "Reply to" is usually spoofed, so one's attempted relatiation against the spammer just ends up spamming some poor uninvolved site owner.

[dank]

kitchin, that looks like a pretty simple solution, thanks for sharing. I'd be interested in hearing Terra's thoughts, though. What happens if a large number of FQ clients start running regexes to filter hundreds of emails a day? I could see that doubling the dynamic server load...

Dan

[Bruce]

Originally posted by frankc:
Is it the program that's been upgraded, the rules, or both? Both are being upgraded. In general, it is difficult to upgrade just the rules, as quite a number of them are embeded in the program.

I'd sure like to see some way for the rules to be updated automatically (with an email to site owners notifying us of the actual filter changes). On minor version changes (that is, version upgrades that primarily change the rules without changing the overall behavior), we will try to upgrade much quicker. This upgrade was complicated by some of the internal behavior changes in SpamAssassin.

Is there any way SA can be configured so that the "engine" (for lack of a better description) can opt to load the configs and tweaks from within a domain? I don't want to waste the room and time to install SA myself but allowing site owners to optionally run their own configs would be handy. The current SpamAssassin engine does load configs from within a domain, however those configs are under the control of the CNC, and will remain so to prevent both resource abuse and misconfiguration problems.

[kitchin]

I made two corrections, marked -edit-, in my script above, for the possible (or not ?) case of repeated "subject:" lines.

Might be lighter-weight to use grep and a shell script, or whatever can do the math comparison. But I haven't tried to pipe in and out of grep.

Anyway... it's a very easy regex, I think, because ^ means typically it only has to look at the string twice. Also, the input is tiny, usually just one line, thanks to 822field. Compare that to scripts that parse the whole email, even if it's 2MB. And it doesn't open any files. That's my guess about the issues involved, anyway.

[kitchin]

Originally posted by frankc:
From everything I've read, there is no good reason to bounce emails. The "From" and "Reply to" is usually spoofed, so one's attempted relatiation against the spammer just ends up spamming some poor uninvolved site owner.
To handle false positives. In case you bounce a good email!
But still I don't like the idea of bouncing to spammers, with all that info for them to parse. I'm sure they do not all discard bounces, because I have one address that gets some spam, and I think they picked it up in the about-five minutes that I was bouncing from it. Other than that, I've never used it in any outgoing mail. It's where I redirect junk mail... instead of bouncing it. %)

[Charles Capps]

Question...

I (finally) installed SA and Vipul's Razor on my own Linux box the other day, after getting fed up with the sheer number of spams being delievered. I noticed that SA now, by default, checks a few RBLs. Will the FQ installation also check these RBLs, or has that been turned off to keep mail flowing quickly?

[frankc]

Originally posted by kitchin:
To handle false positives. In case you bounce a good email!Fair point--never thought of that. Still, I don't think I'd have wanted to bounce the 6,920 spams I've received since FQ added SA here vs the 30 or so false positives I've found (valid but tagged as spam). The chances of causing a lot of unnecessary grief for the clueless spoofed "From" or "Reply to" is far too great.

Nor do I think you can just blackhole inbounds tagged as spam without giving the To/From/Subject a cursory look-see. Even the Bayesian filters--good as they are--do mis-tag good emails as spam.

[frankc]

Originally posted by Charles Capps:
Will the FQ installation also check these RBLs, or has that been turned off to keep mail flowing quickly? H, Charles! Well, if you want that feature, just install EFM (http://diamond-back.com/software/efm/) and use that feature. I'd suggest not using SpamCop's BL due to the large number of mis-ID'd sites. EFM has trapped 2726 spams (and no false positives) due to matching the two RBLs I've enabled.

[Charles Capps]

Well, the idea being that I wouldn't have to install anything else, because RBLs would automagically be checked... %)

It's just one of those "It Would Be Nice If..." features - I have yet to see a spam slip through that was in an RBL that didn't have a score under 30 or so, so it probably doesn't matter that much if RBLs are checked or not... :)

[Bruce]

The upgrade will begin in a few minutes. Users of SpamAssassin may notice that no new mail arrives during this upgrade. This is normal, and is only temporary. Mail received during the upgrade will be delivered after the upgrade has completed.

[Bruce]

The upgrade has been completed. Your SpamAssassin preferences files have been converted to adjust for the reporting options available in the new version. The new scores used by the system have been updated at http://www.FutureQuest.net/docs/SA/

We recommend checking in your CNC to make sure the new configuration properly represents your preferences. For details on the new options, please read the Helpful Hints at the bottom of the SA edit page, which describe the new options. You can find these by scrolling down to the bottom of the SpamAssassin Editing page in your CNC.

Enjoy!

[dank]

I take it one of the changes is that tagged spam now includes the original message as an attachment instead of in-line? That seems like a good approach.

Dan

[Arthur]

Dan, that is correct. The original message is encapsulated as a MIME part and the spam report is in the main part of the message. Although it is possible to have SpamAssassin only tag the headers and leave the message intact. There is an option in the CNC to do this.

Advantages of encapsulating the message are that you don't see the spam message in most mail programs and it makes retrieving the original message easier if you need to have that.

Arthur

[Bob]

Due to a change in the default settings in the new version of SpamAssassin, from the time of Upgrade until 7:55 AM EDT today, any custom subject tags that may have been set were not being prepended to the message subjects.

This has been corrected and any Custom "Subject Tag" you may have had enabled in your SA settings will now be prepended to messages sent after the correction was completed, as they were prior to the upgrade.

We apologize for this snafu.
Bob

[JoeLeBlanc]

Hello,

I got a question about the scoring for the SA, the old SA was average scoring as 5.1, what would be the new average scoring?

Thanks

[frankc]

Bruce, does the new version of SA @ FQ use the Bayes filter?

[Bruce]

Originally posted by JoeLeBlanc:
the old SA was average scoring as 5.1, what would be the new average scoring? The new default score threshold remains the same as the previous default, as the SpamAssassin team has designed their scores around the assumption that spam should be scored higher than 5.0.

Originally posted by frankc:does the new version of SA @ FQ use the Bayes filter? No, it does not. Our system is not equipped at the moment to have seperate learning databases at the domain or account level, and we are still discussing if doing automated learning/filtering at a global level (across everybody that uses SpamAssassin) is even a useful option (it is potentially subject to poisoning, for example).

[Randall]

Well, I've already seen evidence of the upgrade here. Some newsletter stuff that was below the threshhold is now getting tagged, so something's been fiddled with.

Unfortunately, Eudora and Mozilla Mail both display HTML attachments inline -- I don't see a setting to turn it off, at any rate -- so that won't protect me against web bugs or whatever.

Randall

[Bruce]

Originally posted by Randall:
Unfortunately, Eudora and Mozilla Mail both display HTML attachments inline -- I don't see a setting to turn it off, at any rate -- so that won't protect me against web bugs or whatever. The original message is actually attached using MIME content type "message/rfc822", which is a distinct type for HTML ("text/html"). If Eudora and Mozilla Mail are both viewing these message attachments, you can always switch SpamAssassin to encoding the attachments as plain text type, which should not be viewed as HTML.

[Randall]

Eudora shows them even so, if slightly mangled. It's the mangling I really want to avoid -- especially if the upgrade is going to be giving me false positives for a while.

Oh well.

Randall

[Syneryder]

you can always switch SpamAssassin to encoding the attachments as plain text type, which should not be viewed as HTML.
Only problem with that is Eudora then saves the attachment into its attachments folder, so I'd soon have to start clearing the folder out (and going through each file to check if it's spam or not). Oh well, can't have everything, and the upgraded Spam Assassin is *much* better at determining what is spam, I've even been able to reduce my threshold (it's currently at 4.0, down from 6.1). Good stuff!

[Randall]

Only problem with that is Eudora then saves the attachment into its attachments folder, so I'd soon have to start clearing the folder out (and going through each file to check if it's spam or not). Oh, so it doesn't try to render the HTML anymore? Maybe I'll try that then.

Randall

[songdog]

Originally posted by Bruce:
Our system is not equipped at the moment to have seperate learning databases (for Bayes filtering) at the domain or account level... But how awesome it would be if it were so equipped... :bounce: