PDA

View Full Version : Filtering performance: .qmail and scripts


sheila
01-16-2001, 04:54 AM
So, I'm writing a script to filter my e-mail. I'm using Python, but I'd assume performance would be more or less equivalent to a script in Perl.

I tested it earlier this afternoon. Seemed fine.

My plan: replace my .qmail-default file, with a new version, that calls my script. Except for a few, well-chosen aliases, that have .qmail files, that skip over the script.

I started using it for a while tonight. Further testing. It all seemed fine. What it does: checks for certain headers in the e-mail to determine whether it is a "good" email or a "bad" email. If good, it sends it to a private e-mail address that I have never published or shared with anyone. Otherwise, the mail is sent to a spam-bucket address.

Like I say, seemed fine for a while.

Then, a little over an hour ago, my e-mails seemed to be disappearing into the void. I panicked, and took the script down. Before taking it down, I sent myself several test mails. Both ones that would qualify as "good" mails and as "bad" ones. None seemed to be delivered. So, I took my filter script down. Tested if mails could now go through. They did. Just fine.

Strangely enough, these e-mails eventually appeared. Some of them, over an hour from the time they hit the FQ servers until they hit my mailbox. Some of them about a half hour. Some about seven minutes.

Could my little 26 line Python script be causing this? Or did something really wacko happen on the FQ mail servers tonight? Or should I consider that 45 minutes to an hour is about normal for an email to go through? (I never saw it take that long before.)

Here is my script, in case someone is interested:

#! /big/dom/xdomain/Python-2.0/python

import rfc822, sys, string, smtplib

##check if to: and cc: addresses are in my approved file
def goodmessage(addresseeslist, rcptlist):
[nbsp][nbsp][nbsp][nbsp] for addr in addresseeslist:
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]for rcpt in rcptlist:
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] if (upper(addr[1]) == upper(rcpt)):
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]return 1
[nbsp][nbsp][nbsp][nbsp] return 0

## Retrieving data ##

##uncomment the below two lines if message data
##[nbsp][nbsp]is read from stdin
origheaders=rfc822.Message(sys.stdin, 0)
body = sys.stdin.read()

##uncomment the below three lines if message data
## is read from a file
#infile=open("message6.txt", "r")
#origheaders=rfc822.Message(infile)
#body = infile.read()

## Setting forwarding e-mails, sender e-mail
## and file with list of good recipients
fwdgood = "privateaddr@domain.com"
fwdbad = "spambucket@spamcop.net"
sender = "myemail@mydomain.com"
goodrcptfile = open("GoodToList.txt", "r")[nbsp][nbsp] ##textfile with good address
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]## for To: and cc: fields
goodrcptlist = string.split(goodrcptfile.read())
goodrcptfile.close()

addressees =[]
addressees = addressees + origheaders.getaddrlist("To")
addressees = addressees + origheaders.getaddrlist("Cc")

mssg = string.join(origheaders.headers,"")
mssg = mssg + "\n" + string.join(body, "")

server=smtplib.SMTP("mysmtp.com")

if goodmessage(addressees, goodrcptlist):
[nbsp][nbsp][nbsp][nbsp] server.sendmail(sender, fwdgood, mssg)
else:
[nbsp][nbsp][nbsp][nbsp] server.sendmail(sender, fwdbad, mssg)
server.quit()


[This message has been edited by sheila (edited 01-16-01@03:57 am)]

Terra
01-16-2001, 05:28 AM
SMTP is not designed to be fast, but rather reliable...

There could be a number of reasons for a mail slowdown...

Most likely in your case, the filter script caused an error for the local queue delivery which in turn shoved it back into the deferred queue...[nbsp][nbsp]The delay for redelivery follows a quadratic equation, and subsequent attempts to deliver the deferred email could very well be a few hours...

The ultimate goal is to get the email into your POP box, and it will make every best effort to do so...[nbsp][nbsp]The delays are a 'feature', as it will wait until a more opportune time to make the delivery...[nbsp][nbsp]It also helps to prevent mail floods, especially to remote MTA(s)...

When you are working at the filtering level of mail delivery, then you must possess a tremendous amount of patience in the event that your script errors force email back into the 'deferred' queue...[nbsp][nbsp]The delays could easily last a few hours or more, as that is just the way it works...

--
Terra
--The goal is to not superglue your mailbox--
FutureQuest

sheila
01-16-2001, 05:44 AM
Thanks very much for the explanation. I will be analyzing this whole situation much more thoroughly.

sheila
01-17-2001, 03:20 AM
Found my error. It was a bug in my script. Stupid thing is, I had actually corrected this bug at one point. I must've copied an old version onto my web space by accident.

For closure, here is the correction.

My function definition given above as

##check if to: and cc: addresses are in my approved file
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] def goodmessage(addresseeslist, rcptlist):
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]for addr in addresseeslist:
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] for rcpt in rcptlist:
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]if (upper(addr[1]) == upper(rcpt)):
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] return 1
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]return 0


should read
##check if to: and cc: addresses are in my approved file
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] def goodmessage(addresseeslist, rcptlist):
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]for addr in addresseeslist:
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] for rcpt in rcptlist:
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]if (string.upper(addr[1]) == string.upper(rcpt)):
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] return 1
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]return 0

The key is in this line:
if (<font color=#FF0000>string</font>.upper(addr[1]) == <font color=#FF0000>string</font>.upper(rcpt)):

Now everything seems to be working fine. I've also added some code to catch exceptions and print to an error log. :)


[This message has been edited by sheila (edited 01-17-01@02:22 am)]