filtering: Whitelists/Blacklists

[sheila]

OK, in this thread:
http://www.aota.net/forums/showthrea...&threadid=8798

There were some questions about Whitelist/Blacklist filtering.

Here are some simple examples using filtering.

Suppose this is your .qmail-default file in your /big/dom/xdomain directory:

|condredirect devnull@domain.com /usr/bin/env python /big/dom/xdomain/sircamscan.py
|condredirect secret@domain.com /usr/bin/env python /big/dom/xdomain/addresscheck.py to ./goodtolist.txt
|condredirect secret@domain.com /usr/bin/env python /big/dom/xdomain/addresscheck.py cc ./goodtolist.txt
|condredirect secret@domain.com /usr/bin/env python /big/dom/xdomain/addresscheck.py from ./goodfromlist.txt
|condredirect secret@domain.com /usr/bin/env python /big/dom/xdomain/domaincheck.py from ./gooddomains.txt
&nobody@spamcop.net

secret@domain.com is a private e-mail address that I share with NO ONE, not friends, family, NO ONE. I filter my mail and the filter decides wether to forward to that addy or not.

Here is what it does: the first line checks whether a copy of the SirCam virus has been sent. (I only got two copies today? It's going away, I think.) If the mail passes that test, it is redirected to a "black-hole" address. (See this post http://www.aota.net/forums/showthrea...0563#post50563 for more details.)

The next three lines check for addresses in a white list. They all run this script:

Corrected Code posted in next Post
ADMIN EDIT: Removing Proof That Sheila is Human by Sheila's Request

I call this script "addresscheck.py". This script takes two parameters:
A mail header field (To, CC or From) and the name of a whitelist file.

In the file goodtolist.txt I have a list of e-mail addresses, that if anyone sends an e-mail TO or CC to that address, I want to get that e-mail. So, lines 2 and 3 of the .qmail file are checking the To field and the CC field to see if those addresses are in my whitelist of "To" addresses.

The fourth line of my .qmail file is checking another file, called "goodfromlist.txt", which is a list of e-mail addresses, that if I receive anything FROM that e-mail address, then I want to get that e-mail.

The fifth line is checking a file called gooddomains.txt, which is a list of domains that I WANT to receive mail from. (I haven't posted the domaincheck.py script here, but if someone wants it, just send me a message.)

The sixth line forwards any mail that has not passed previous filters to a spamcop address.

You could reverse the principles here, and have blacklists of e-mail addys or domains, instead of whitelists, and use the same addresscheck.py script, only I'd assume you'd redirect them e-mail to a different address (like spamcop or a black hole? Or use the "bouncesaying" command to bounce the messages back?)

I don't know how "server friendly" it is to have up to five scripts execute per email received to the default mailbox for a domain. This is just an example, and one should probably be conservative with this type of thing.

Anyway, I hope these examples are useful to some of the people who asked about the Blacklist/Whitelist type of stuff.

[sheila]

The script posted above has a flaw:

If you put
joeuser@yahoo.com in your file of addresses

But joe sends you e-mail with an address like this:
JoeUser@yahoo.com

It will not see that they are the same person.

I've fixed this. Here is the revised script, which checks the e-mail addresses in a way, that upper and lower case do not matter:

#! /usr/bin/env python

import rfc822, sys, string
from cStringIO import StringIO

""" Usage: This script takes parameters on the command line
in the following form:

/path/to/python scriptname.py FieldName filename

where FieldName is the name of an address field in the headers
of an e-mail message (i.e. To, CC, or From) and
filename is the name of a file with a list of e-mail addresses,
one per line. If the filename is not in the same directory as
this script, then you must give the full path to the filename """

if len(sys.argv) < 3:
    sys.exit(1)  # file parameter and Fieldname
                 # parameters were not passed to script
                 # assume conditions fail and
                 # give a one exit code.
try:
    file = open(sys.argv[2])
except:
    sys.exit(1)  # couldn't find or open file
                 # assume conditions fail and
                 # give a one exit code.
filetext = file.read()
file.close()

checklist = string.split(filetext, "\n")
checklist = filter(lambda x: x != '', checklist)
checklist = map(lambda x: string.strip(x), checklist)
checklist = map(lambda x: string.upper(x), checklist)

headers = rfc822.Message(sys.stdin, 0)
FieldName = sys.argv[1]
AddrPairsList = headers.getaddrlist(FieldName)

for pair in AddrPairsList:
    name, email = pair
    if string.upper(email) in checklist:
        sys.exit(0) # you found one! return success
sys.exit(1) # went through the whole list and found nothing

I had been using this script myself, and wondering why it didn't seem to work 100% of the time, and today I finally figured out what the problem was.

If anyone is currently using it, I suggest you copy this revised version, instead.

[sheila]

ADMIN EDIT: Removing Proof That Sheila is Human by Sheila's Request

Now, Bob, that isn't quite the way I asked you to phrase that, in the support e-mail I sent.

--what, me make a mistake???

[Bob]

Sheila,

I cannot tell a lie!
The Devil Made Me Do It!



Or was it Deb

[MTDesigns]

ADMIN EDIT: Removing Proof That Sheila is Human by Sheila's Request

He he he

[Terra]

|condredirect devnull@domain.com /usr/bin/env python /big/dom/xdomain/sircamscan.py
|condredirect secret@domain.com /usr/bin/env python /big/dom/xdomain/addresscheck.py to ./goodtolist.txt
|condredirect secret@domain.com /usr/bin/env python /big/dom/xdomain/addresscheck.py cc ./goodtolist.txt
|condredirect secret@domain.com /usr/bin/env python /big/dom/xdomain/addresscheck.py from ./goodfromlist.txt
|condredirect secret@domain.com /usr/bin/env python /big/dom/xdomain/domaincheck.py from ./gooddomains.txt

I have a problem with this...

You are firing off up to 5 Invocations of Python for each email that comes in... This is highly inefficient, combined with the fact that python is heavier to execute than even perl is...

Please find a way to combine your lists into 1 execution of python by writing each checker as an imported library...

--
Terra
--It just sorta jumped out at me this time--
FutureQuest

[sheila]

Originally posted by Terra:

I have a problem with this...

Right, that's why I wrote, in the original post:

I don't know how "server friendly" it is to have up to five scripts execute per email received to the default mailbox for a domain. This is just an example, and one should probably be conservative with this type of thing.

I probably shouldn't have said, "I don't know how 'server friendly'..."

I know/knew that wasn't. I just was trying to give examples, for those who are interested in this type of stuff.

You are right. It should be combined into a single Python call.

Better yet, it seems to me, would be to take advantage of this library of functions:
http://cr.yp.to/mess822.html

That's now available on all the servers, right? And I assume, much more efficient to use than running any scripting language. I'm looking into possibly switching to some of those utilities, instead.

[sheila]

OK, fine. I shouldn't have posted such a bad example before. All I can think is, it's the "teacher" in me. I didn't want to give out the answer. I wanted to give a clue, and then let others proceed from there. But this is a web host forum. People don't want a script that's only close. They want THE script.

OK, so, here is a script that checks To, Cc, and From fields (on the assumption that if a good "to" address appears in either the To: or Cc: field, you want it).

You have to maintain two separate text files. One for good "To/Cc" addresses (I call this goodtolist.txt). And one for good "From" address (I call this goodfromlist.txt).

Your .qmail file might look something like this:


Code Sample:

|condredirect goodforward@mydomain.com /usr/bin/env python /big/dom/xdomain/mailfilter.py &badaddress@someplace.com

The first line calls the script, checks for one of the desirable address, and if one of them is found, it redirects the mail to goodforward@mydomain.com

The second line forwards the mail to an address for mail that you don't want (or doesn't pass the script's filter test). It will only get to this line in the event that the script doesn't find any good addresses in the To, Cc, or From fields.

and the script mailfilter.py is this:

import rfc822, string, sys

def formatList(checklist):
    checklist = filter(lambda x: x != '', checklist)
    checklist = map(lambda x: string.strip(x), checklist)
    checklist = map(lambda x: string.upper(x), checklist)
    return checklist

def goodmessage(mssgHeaders, ToFileList, FromFileList):
    ToList = mssgHeaders.getaddrlist("To") + mssgHeaders.getaddrlist("Cc")
    ToList = map(lambda x: x[1], ToList)
    ToList = formatList(ToList)
    for addr in ToList:
        if addr in ToFileList:
            return 1
    FromList = mssgHeaders.getaddrlist("From")
    FromList = map(lambda x: x[1], FromList)
    FromList = formatList(FromList)
    if FromList:
        for addr in FromList:
            if addr in FromFileList:
                return 1
    return 0


# Read in the message headers
 origheaders=rfc822.Message(sys.stdin, 0)


# get the list of good addresses for the To:
# and cc: fields, and close the file
goodRcptFile = open("goodtolist.txt", "r")
goodRcptList = string.split(goodRcptFile.read())
goodRcptFile.close()

# get the list of good From: addresses
# and close the file
goodSenderFile = open("goodfromlist.txt", "r")
goodSenderList = string.split(goodSenderFile.read())
goodSenderFile.close()

# format the lists -- uppercase them and
# remove empty entries and strip whitespaces
goodRcptList = formatList(goodRcptList)
goodSenderList = formatList(goodSenderList)

# check if this message has any of the good addresses
if goodmessage(origheaders, goodRcptList, goodSenderList):
    sys.exit(0)  # it does have a good address: redirect
sys.exit(1)   # it doesn't have a good address: don't redirect

I've tested it a bit, and I think it is OK. Please check and test it yourself, before using in any critical situation. I can't guarantee that it is without flaws, but I am using it myself. So, clearly, I *think* it is OK.

[Terra]

Sheila:

Thank you for putting up the combined code...

Even though you did mention the warning, I feared that many would simply copy and paste your example without fully realizing the impact it would cause... For all intents, you did provide a working mail filter system, and for some - that would have been simply good enough...

Anyhow, your first set was a good prototype (proof-of-concept) for the final combined solution which I hope many will see and use instead...

--
Terra
--The difference between the mall, and your local neighborhood downtown shop--
FutureQuest

[manish]

Originally posted by sheila:
Better yet, it seems to me, would be to take advantage of this library of functions:
http://cr.yp.to/mess822.html

That's now available on all the servers, right?

Terra, are they available on NINE? I have seen the 822xxxx commands. I wasn't sure if the libraries were installed. I might want to use these libraries for efficiency as well....

Thanks,
Manish