blackhole email address & comments in .qmail file

[Oink]

It seems like every six months, I would struggle for a few weeks reading up and catching up on spam filter/techniques. Last time I gave up trying to use procmail with our mail system. I have just read several of Sheila's posts on a alternative technique and I am very keen to try out the blacklist goodlist filtering.

The .qmail syntax is very new to me, and I have perhaps a few very simple questions:
Can I put comments in a .qmail (dot-qmail) file? If so what is the correct escape character/syntax?

Also would like to know if there is a legitimate blackhole email address I can use? Can I safely use nobody@spamcop or would I need to set up an email account in my domain to redirect email to dev/null?

Thanks in advance.

Oink

[sheila]

Originally posted by Oink:

The .qmail syntax is very new to me, and I have perhaps a few very simple questions:
Can I put comments in a .qmail (dot-qmail) file? If so what is the correct escape character/syntax?

Yes, comments are allowed. See this file:
http://qmail.org/man/man5/dot-qmail.html

Also would like to know if there is a legitimate blackhole email address I can use? Can I safely use nobody@spamcop or would I need to set up an email account in my domain to redirect email to dev/null?

I would recommend that you set up your own black hole addy. This thread explains how to do it:
http://www.aota.net/forums/showthrea...dev%2A+%2Amail

Good luck!

--would you believe I've turned off all my mail filtering scripts on FutureQuest and am now using my mail client to handle that for me???

[Oink]

echo '#' >/big/dom/xacme/.qmail-sales

Thank you Sheila,

very handy to know, though I am not too certain about the echo command of what it actually does. I initial had my spam .qmail like this:

|forward &>/dev/null

Test mail didn't make it to the mailbox as intented. But lets hope for those 2 hours, it didn't cause any harm!

Btw, just a few minutes ago, I tested your python (whitelist) script. All the test email sent to it failed (exit 1). I wonder if you would reveal the signtax of your "goodtolist.txt" file? This is what my .qmail-test file:

|condredirect passed@sitedish.com /usr/bin/env python /big/dom/xmydomain/mailfilter.py 
&failed@sitedish.com

and I commented out checking the all the call to "goodfromlist.txt" your python script:

import rfc822, string, sys

def formatList(checklist):
    checklist = filter(lambda x: x != '', checklist)
    checklist = map(lambda x: string.strip(x), checklist)
    checklist = map(lambda x: string.upper(x), checklist)
    return checklist

def goodmessage(mssgHeaders, ToFileList, FromFileList):
    ToList = mssgHeaders.getaddrlist("To") + mssgHeaders.getaddrlist("Cc")
    ToList = map(lambda x: x[1], ToList)
    ToList = formatList(ToList)
    for addr in ToList:
        if addr in ToFileList:
            return 1
    FromList = mssgHeaders.getaddrlist("From")
    FromList = map(lambda x: x[1], FromList)
    FromList = formatList(FromList)
    if FromList:
        for addr in FromList:
            if addr in FromFileList:
                return 1
    return 0


# Read in the message headers
 origheaders=rfc822.Message(sys.stdin, 0)


# get the list of good addresses for the To:
# and cc: fields, and close the file
goodRcptFile = open("goodtolist.txt", "r")
goodRcptList = string.split(goodRcptFile.read())
goodRcptFile.close()

# get the list of good From: addresses
# and close the file
#goodSenderFile = open("goodfromlist.txt", "r")
#goodSenderList = string.split(goodSenderFile.read())
#goodSenderFile.close()

# format the lists -- uppercase them and
# remove empty entries and strip whitespaces
goodRcptList = formatList(goodRcptList)
#goodSenderList = formatList(goodSenderList)

# check if this message has any of the good addresses
if goodmessage(origheaders, goodRcptList, goodSenderList):
    sys.exit(0)  # it does have a good address: redirect
sys.exit(1)   # it doesn't have a good address: don't redirect

My goodtolist.txt:

still@doubt.com
kim@sitedish.com

The permissions are correct, Email all went through the filter (to the failed@sitedish.com address) Thanks in advance, I am testing and checking as I type.

p/s btw, can goodtolist.txt contains meta * character?

[sheila]

Originally posted by Oink:

though I am not too certain about the echo command of what it actually does. I initial had my spam .qmail like this:

|forward &>/dev/null

Hmm. I'm not sure what the "echo" command does, either.

Here is how I have my black hole address set up:

I have a .qmail-devnull file set up, where the contents are this:

#/dev/null

(Has to have a blank line at the end of the file.)

Btw, just a few minutes ago, I tested your python (whitelist) script. All the test email sent to it failed (exit 1). I wonder if you would reveal the signtax of your "goodtolist.txt" file?

The syntax is just this:

email1@domain1.com
email2@domain2.com
email3@domain3.com


Basically, one email address per line in a text file.

and I commented out checking the all the call to "goodfromlist.txt" your python script:
...[snipped code]...
The permissions are correct, Email all went through the filter (to the failed@sitedish.com address) Thanks in advance, I am testing and checking as I type.

I think I see what the problem might be...
If the script fails for *any* reason, you will get the effect of a sys.exit(1). Well, maybe not *any* reason. But, if there are syntax errors in the script, or programming logic errors, you will get the result of a failed test.

The way you commented out the code, like this:

# get the list of good From: addresses
# and close the file
#goodSenderFile = open("goodfromlist.txt", "r")
#goodSenderList = string.split(goodSenderFile.read())
#goodSenderFile.close()

and this:

#goodSenderList = formatList(goodSenderList)

You commented out all the lines in the main program that open the from-file-list and read in and defines the goodSenders list.

HOWEVER, when the main program makes this call:

if goodmessage(origheaders, goodRcptList, goodSenderList):

Notice that one of the parameters that it needs, is the goodSenderList. However, this is undefined in your version, since you commented all of that code out. So, when it enters the subroutine/function goodmessage and tries to execute these lines:

 FromList = mssgHeaders.getaddrlist("From")
    FromList = map(lambda x: x[1], FromList)
    FromList = formatList(FromList)
    if FromList:
        for addr in FromList:
            if addr in FromFileList:
                return 1

It is unsuccessful, and will not return the '1', since that is an undefined reference to the FromList.

Therefore, the function goodmessage will ALWAYS return a 0 for you, with the main program code commented as you have done.

Thus, you will ALWAYS get a sys.exit(1) and all your mail will go to failed@sitedish.com

Does this make sense?

p/s btw, can goodtolist.txt contains meta * character?

Not as currently coded. It is possible to make a change in the code, I supposed. This is an exercise for the reader???

[Oink]

Just a moment ago, I had success with one of your earlier scripts, the one calling 4 python processes... and to keep it simple, I used only a process and a goodtolist.txt file. At least I know all files are in correct order/format/astrological alignment.

Your explaination makes very good sense, you are a teacher. I will go back to the original script.

About your going back to email client for filtering... I have also just decided to printand send out snail mail christmas cards this year after 6 years of interactive email attachments

Thank you again

[Oink]

The sigle execution script listed here didn't work for me, kept exit(1) on me, I even tried an unmodified stock copy and paste version.

I do however have success with the simple, though less server friendly script. I have modified it to reject mail from bad domains (badfromlist), redirect to good address (to & cc from goodtolist), reject everything else. Testing still in progress.

Wonder if there is a more streamline and more account frugal way of achieving this? I have an account quota, and it doesn't seem very wise to use two accounts per person just for filtering.

[sheila]

Originally posted by Oink:
The sigle execution script listed here didn't work for me, kept exit(1) on me, I even tried an unmodified stock copy and paste version.

Hmm. I don't know why it isn't working. I copied and pasted from there, and tried it out on my account, testing both good and bad addresses, and it worked fine. Nevertheless, just "It didn't work" isn't enough info. Need error reports. Here, I've pasted a version below that should log errors. Can't debug without debugging info. Try running this one and then posting the recorded errors from the logfile.txt. Make sure to edit
xdomain

to your real domain. OK?

import rfc822, string, sys

import os, traceback
from time import strftime, gmtime, time, localtime, asctime

datestamp= strftime("%a, %d %b %Y %H:%M:%S", gmtime(time()))+" -0000"


# **** edit the line below to be a path to
# **** a file for logging in your web space
logfile = "/big/dom/xdomain/logfile.txt"

def formatList(checklist):
    checklist = filter(lambda x: x != '', checklist)
    checklist = map(lambda x: string.strip(x), checklist)
    checklist = map(lambda x: string.upper(x), checklist)
    return checklist

def goodmessage(mssgHeaders, ToFileList, FromFileList):
    ToList = mssgHeaders.getaddrlist("To") + mssgHeaders.getaddrlist("Cc")
    ToList = map(lambda x: x[1], ToList)
    ToList = formatList(ToList)
    for addr in ToList:
        if addr in ToFileList:
            return 1
    FromList = mssgHeaders.getaddrlist("From")
    FromList = map(lambda x: x[1], FromList)
    FromList = formatList(FromList)
    if FromList:
        for addr in FromList:
            if addr in FromFileList:
                return 1
    return 0

try:
    # Read in the message headers
    origheaders=rfc822.Message(sys.stdin, 0)


    # get the list of good addresses for the To:
    # and cc: fields, and close the file
    goodRcptFile = open("goodtolist.txt", "r")
    goodRcptList = string.split(goodRcptFile.read())
    goodRcptFile.close()

    # get the list of good From: addresses
    # and close the file
    goodSenderFile = open("goodfromlist.txt", "r")
    goodSenderList = string.split(goodSenderFile.read())
    goodSenderFile.close()

    # format the lists -- uppercase them and
    # remove empty entries and strip whitespaces
    goodRcptList = formatList(goodRcptList)
    goodSenderList = formatList(goodSenderList)

    # check if this message has any of the good addresses
    if goodmessage(origheaders, goodRcptList, goodSenderList):
        sys.exit(0)  # it does have a good address: redirect
    sys.exit(1)   # it doesn't have a good address: don't redirect
except StandardError:
    exc, val, tb = sys.exc_info()
    f = open(logfile, 'a')
    f.write('***** '+ datestamp + '\n')
    traceback.print_exception(exc, val, tb, file = f)
    f.write('\n\n')
    f.close()
    os.chmod(logfile, 0660)
    raise

I do however have success with the simple, though less server friendly script. I have modified it to reject mail from bad domains (badfromlist), redirect to good address (to & cc from goodtolist), reject everything else. Testing still in progress.

Hmm. If you can do the single versions, you should be able to do the combined one. Can't see why it isn't working. Hopefully, the error logs will help to figure out what the problem is.

Wonder if there is a more streamline and more account frugal way of achieving this? I have an account quota, and it doesn't seem very wise to use two accounts per person just for filtering.

I'm not sure what you mean, here? Well, one thing you could do...have all the "bad" mail from all the accounts just go to a single "reject" mailbox that one person sorts through later and sees if any of it is worth keeping. What I had been doing, was sending all my "reject" mail to a spamcop account. However, spamcop is going through some changes right now, and I don't want to send my mail to them any more. Or, you could open throwaway hotmail or yahoo accounts, send all the reject mail there, and just check those mailboxes once a week? I know someone else here was simply sending all their "reject" mail to the dev/null mailbox. That's a bit too drastic for me.

Another possibility is to simply add a tag line or header to the "reject" messages and have your mail client filter on that? Actually, I've found that my mail client has very good filtering capabilities, so like I said, I'm using that right now.

[Joe]

I think, or should I say I hope, blackholing email addresses (from the CNC?) is going to be in Santa's sack this year. If not, at least we've been told its on the agenda. It is one feature I am very much looking forward to.

Joe

[Oink]

I took a coffee break at 4pm, reuploaded the previouly working script, and hadn't been able to receive an email since (all test email, thank goodness)! I have just spent the last 5 hours or 6 hours trying to debug the thing, I was wishing there was a form of logging function built in. It wasn't until i did a 'ls -al' to notice that my FTP client has gratuitiously set the 'executable' flag on all the files I have been uploading..... what a waste of time. However I did find the man pages for dot_qmail...

You must have heard my prayer, I just uploaded the new script and eagerly waiting for an error log and guess what? It works this time Arggggggggg! Now I can cut my .qmail file to request 2 python executions, unless I can change your script to make the goodfrom lookup into a badfrom lookup.

Do you foresee any problem is I change

    if FromList:
        for addr in FromList:
            if addr in FromFileList:
                return 1
to
    if FromList:
        for addr in FromList:
            if addr in FromFileList:
                return 0

and

    goodSenderFile = open("goodfromlist.txt", "r") 
        to
    goodSenderFile = open("badfromlist.txt", "r")

I should perhaps also change the order, so the badfrom list is parsed before the goodto list. But I have used up my adventure spirit for tonight.

Thank you Sheila, you have been most helpful.

[sheila]

Oink: Your description of changing goodsender to badsender sounds like it should work. By the way, first time I ever 'heard' someone go "ARRRGH!" over a script actually working.

As for Santa's possible mail filtering:
Would be nice. I would certainly look forward to such a thing.

For me, personally, the mailfiltering was just one aspect of a problem that has allowed me to learn things that were related to other projects I've worked on, like:
writing my own form-mail cgi script
and writing a robot moderator for a newsgroup.