Bouncing mail based on From field without scripts

[sheila]

OK, using some of the hints in this thread:
http://www.aota.net/forums/showthread.php?s=&threadid=8332

I mucked about tonight with the 822field at the command line, and I read the man pages for grep and so on, and now I think I can use the .qmail bouncesaying command without any scripts with a whole textfile of bad email addresses or domains.

Here is a sample .qmail file:

.qmail-test

|bouncesaying "Bounce Message" sh -c '822field from | grep -iqf badfromlist.txt'
|vdeliver


Where there is one blank line after the |vdeliver line, and badfromlist.txt contains a list of email addresses or domains which you want to bounce back to the sender in the case of a match (one per line).

I seem to have this working on a test address of mine.

My son has been getting spammed up the wazoo by mail coming from pm0.net. He has been getting dozens per day. If he didn't check his mail for a few days, he would have 300+ messages in his inbox. Two or three would be legit, and the rest would be spam. He didn't even want to check his mail any more. I've set up bouncing for his account, and he isn't getting any more spam from these idiots at pm0.net.

Anyhow, this may be easier for some people than messing with those Python scripts that I had put up earlier.

Plus, I'm sure this must be easier on the server.

[BenV]

<aside>Sheila, you need to start an email management information and techniques website.</aside>

Summary:

I wish we had alias wild-card functionality. I have the opposite problem as everyone else here. I want to filter email based upon "To:" addresses that are allowed to receive mail.

Details:

I have watched all these messages on this topic with great interest as of late. This is the way that I handle things...

Everywhere I go that I have to leave an email, I use a completely unique address. These addresses follow a naming convention - prefix.unique.site.name@mydomain.com. For example, I subscribe to Information Week newsletters with: news.informationweek.com@domain.com or something like that. All news related stuff begins with news.[whatever], Delphi stuff begins with delphi.[whatever] etc.. If I EVER get spam from one of those address I know exactly where the address came from and I start bouncing it.

I filter really easily using Outlook Express based upon those prefixes.

This was all real easy until I decided to change my main personal address and disable the catch-all feature. I had to set up aliases for all the addresses I want to continue to use - not fun. But I went from getting 100-150 messages a day to getting perhaps 10. The only junk mail I get now is to the webmaster address.

Anyway...

1) How would I say (in .qmail-alias) allow this email if "the address is in this list", or if "it begins with news.(\w?)" etc.?

2) I also want to be able to look at the recipient in the message envelope, not the header.

3) In all of the above, how can I rewrite the envelopes' "To:" field, leaving all headers intact?

:\

Ben
-- I can't think of taglines like Deb and Terra...

[BenV]

Would this work?


|bouncesaying "Bounce Message" sh -c '822field delivered-to | grep -ivqf goodaliases.txt'
|vdeliver


I am not somewhere that I can test it. [Note the -v in the grep above.]

Ben

[Justin]

The problem with this is that it will print out each Delivered-to: line it finds in the headers. So if any one of them is not listed, it will bounce... note that there will be more than one on most any email.

I myself have been getting a TON of spam from pm0.net as well. I thought it was just me; I personally blocked that entire domain as well. I can't seem to find what address/domain that was, but I do remember fixing it with a Perl script; however, your solution is a lot nicer. I wasn't aware of the '822field' utility, it's quite handy.

[sheila]

BenV:
I think I've found a way to do what you want.

Use the "iftocc" command. This one checks the email addresses in the To: and Cc: fields. If you make a text file containing all the good "To" addresses, suppose it is called goodto.txt, then this command in your .qmail file will bounce anything that doesn't have one of those addresses in either the To: or Cc: fields:


|bouncesaying "Bounce Message" except iftocc `cat goodto.txt`
|vdeliver

edit: added vdeliver line to .qmail file

Note that, so far as I know, the addresses in the text file have to be all on the same line, separated by a single space each. (I haven't tested putting them on separate lines, but the way the man page for iftocc looks, it appears that they must all be on the same line.)

Notice that those are backquotes on `cat goodto.txt`

The purpose is to output the contents of the goodto.txt file.

If someone can think of another way to get the addresses output in a single line separated by single spaces, without using the backquotes on a cat command, especially if it would make it easier to keep the addies on separate lines in a file, it would be nice if you would share with the rest of us!


<aside>Sheila, you need to start an email management information and techniques website.</aside>

LOL. It's on my to-do list. Let's see...um...it's about item #53! ;)

[sheila]

Ben...to follow up in further detail on your questions:

1) How would I say (in .qmail-alias) allow this email if "the address is in this list", or if "it begins with news.(\w?)" etc.?
I think the preceeding post does what you want. (i.e. it checks if the address in either the To: or Cc: fields is one of the addresses in a file that you maintain, of "good" To: addresses)

2) I also want to be able to look at the recipient in the message envelope, not the header.
The environment variable $RECIPIENT is made available by Qmail for you to use in .qmail files. For other environment variables, refer to this documentation:
http://www.qmail.org/man/man8/qmail-command.html

3) In all of the above, how can I rewrite the envelopes' "To:" field, leaving all headers intact?
I wonder, if setting a new value in the RECIPIENT environment variable would accomplish what you want?

Good Luck. Let us know how it goes.

[sheila]

I wonder, if setting a new value in the RECIPIENT environment variable would accomplish what you want?
Actually, this doesn't seem possible. And it makes sense. By the time your .qmail file gets the mail, the mail has been delivered to the message envelope's recipient. So, it doesn't make sense to re-set the recipient. Delivery is finished.

HOWEVER, if you now forward the message to a new address, this will set a new RECIPIENT on the envelope, because the mail is now being delivered again.

To forward to a new address, put a line like this in your .qmail file:

&newaddy@somedomain.com

[sheila]

OK, here is how to:

Keep a file of "good" addresses in a text file, one per line, that if the email does NOT have one of these "good" addresses in either the To: or Cc: fields of the message header, it will bounce the email back.

create a .qmail file with these contents:

|bouncesaying "Bounce Message" except iftocc `tr "\n" " " < goodto.txt`
|vdeliver

edit: added vdeliver line to .qmail file above.

Where "goodto.txt" is the name of the text file with the good email addresses, one per line. [NOTE: those are back quotes in front of tr and after goodto.txt !! ]

[bturner]

Using these same tools, is there a way to filter email using the 822field tools by the Subject line? In my case, it looks like most of the spam never comes from the same email address, but the subject lines are almost all alike, or contain phrases that are the same.

I like the filtering good addresses with the To or Cc fields, but I often receive email from those I don't know(and not being spam), and would worry about those getting trashed.

Just curious. Thanks!
Bill

[sheila]

Originally posted by bturner:
Using these same tools, is there a way to filter email using the 822field tools by the Subject line?
Yes.

The 822field command returns the contents of a message header field. You can try this out at the command line in a telnet session.

Here is a little tutorial for those who are not familiar with command line utilities:
Try this:

create a textfile, say you call it message.txt, and put it in any directory in your webspace, with these contents (cut and paste):

From: someaddy@mydomain.com
To: myfriend@hisisp.com
Subject: testing ignore
Date: Sat, 08 Sep 2001 12:24:07 -0700
X-Mailer: Forte Agent 1.8/32.548

From a friend.

The end of this test is near.
It is at hand.
END.


Then, at a command line prompt in a telnet session, go to that same directory and try this:

$ 822field from < message.txt

or

$ 822field subject < message.txt

or

822field x-mailer < message.txt


In each case, you will see that it prints out the contents of that message header. (Play around with it, experiment until you see how it works.) The < message.txt that I'm putting at the end of each command, means that the contents of the file message.txt will be used as the input (whereas, in your .qmail file, it would be the email which was being delivered), so this serves as a good way to test instead of testing by sending yourself actual emails.

Now, if you don't already know how the grep utility works, you should read the man page

$ man grep

and try some of these experiments:
$ grep end message.txt
$ grep -i end message.txt
$ grep from message.txt
$ grep -i from message.txt
$ grep -ic from message.txt
$ grep is message.txt

Anyhow, the idea is that grep looks for the first argument that you pass to it in the file that is the second argument. Default behavior is to print each matching line. The -c switch prints the count. The -i switch makes it match without regard to case. The -q switch makes it silent.

If there is a match, the returned exit code is 0. If there is no match, the returned exit code is 1.

When using in .qmail files, you probably want to use at least the -iq switches, since you don't need to print out the matches, you just want to know if there is one.

Now, in order to use "bouncesaying" , you need an exit code. If zero, it will bounce, if non-zero, it will not bounce.

So, let's say that you want to bounce messages where the subject contains the string: "test" (without the quotes). You could do something like this:

|bouncesaying "bounce message" sh -c '822field subject | grep -iq test'
|vdeliver

The pipe symbol | takes the output of the 822field command and uses it as input for the grep command. So it will determine whether there is a match between the contents of the subject in the message header and the string "test" (without quotes). If you get a match, it will give a zero exit code, and execute the bouncesaying command. Otherwise it will give a non-zero exit code, and skip the bouncesaying command, and go to the next command (vdeliver) and deliver it to your mailbox.

[Note: originally, instead of using the string "test" as my match for the 822field subject command, I tried using "$$$", since that would likely occur in the subject of a spam. But the dollar symbol has a special meaning in the command line shell, and I couldn't figure out how to get it to work in this case. Again, knowledgeable others are welcome to share.]

That's the end of tonight's tutorial. ;)

[Tom]

Whoa! This works like a champ. Now I'd need to see about finding a good (current) known spammer list to start out with. Too bad I can't just bounce all of hotmail, mail.com and verizonmail... ;) Questions:

• Is there a limit to the reasonable size of badfromlist.txt?
• I wonder what the best "Bounce Message" would be... "Invalid Mailbox" or "Mailbox closed" or .. ? Just some way to kill it off the spam list is what we want.

This is getting very interesting...
Tom

[sheila]

Originally posted by Tom:
Whoa! This works like a champ.
mE
Now I'd need to see about finding a good (current) known spammer list to start out with.
You might try the RBL or ORBZ. Maybe you could grep against the IP numbers in those lists on the Received lines in the message headers. You could try hanging out in the spamcop newsgroups and asking there about such lists. I know that topic has come up before.
Too bad I can't just bounce all of hotmail, mail.com and verizonmail... ;)
Well, you can. (Maybe you mean that you don't want to, just in case someone sends you legitimate mail?) To match an entire domain, instead of just an address, just list the domain only in your file. Grep will see whether you get a match against that, then. (I'm currently bouncing all mail from pm0.net.)

• Is there a limit to the reasonable size of badfromlist.txt?
You asking me? Or just anyone? This is outside my area of expertise. I'd guess there probably is a limit, but it's fairly large. Certainly you could probably have a couple hundred without a noticeable performance degrade.
• I wonder what the best "Bounce Message" would be... "Invalid Mailbox" or "Mailbox closed" or .. ? Just some way to kill it off the spam list is what we want.
If you'd rather just silently drop the messages, instead of bouncing them (which is probably advisable in the case of spam anyhow), use condredirect instead of bouncesaying:
.qmail file:

|condredirect blackhole@mydomain.com sh -c '822field from | grep `tr "\n" " " < badaddylist.txt`
|vdeliver

should send all messages where the From: field address is listed in the badaddylist.txt file to the address blackhole@mydomain. (See previous discussions for how to set up a blackhole address.)
This is getting very interesting...
Tom <tee, hee>. Glad we finally found something that is working for you, although it sure beats me why those Python scripts wouldn't run for you. Well, this is better and faster anyway.

[Tom]

Yeah, that's the rub on blocking all of mail.com or verizonmail - there may be legitimate senders out there but those two sure seem to be commonly spoofed or used.

I found a list of about 400 addresses & domains -- about 25K to start out with on a side address that gets spammed.

Question: wouldn't it do better in the long run for spam to bounce back and look to the evil ones that it's a bad address rather than appear to them that it went through? That way maybe some of them might remove it as a clean up process?

Side note: I can't believe how many say, "you're getting this because you opted in..." or "get results from our list of hundreds of thousands of opt-in addresses to willing recipients." Yeah, right.

Tom

[bturner]

I tried the following in a .qmail file


|conredirect null@brilliantcorners.org sh -c '822field subject | grep -iqf subjects.txt'
|vdeliver



And that didn't seem to work. I also tried variations on the grep switches (based on some of Shiela's other posts), like -ivqf, -icqf, etc.

The mail still gets through.

In subjects, I have a list of the common phrases used in the spam email subjects, like

mortgage rate
rate quote
credit card
cell phone
SPAM:
ADV:
harrassment
work from home


And I sent an email to the account (bill) that I set up the qmail file (.qmail-bill) for, with a subject line of "ADV: Mortgage Rates falling". I have the null@ email address set up as a standard pop email address, until I get this working, then I'll make it a blackhole address. (using the '#' in the qmail trick from another thread.)

Any ideas?

[sheila]

A quick thought, for bturner, since I'm on my lunch break and have to leave shortly:

I thought it might have something to do with the fact that you have lines in the subjects.txt file with multiple words per line. I'm trying to test this at the command line, and having no success, and I even went in and changed the subjects.txt file so that there was only one word per line. Still no success.

I think we need someone who is a little bit more of an expert with bash and the grep utility than I am to help us out here.

Back to this problem later...?

[sheila]

Doh! I was doing something stupid!

I think what you need to do is this...
in your subjects.txt file, any line that has more than one word in it, enclose in double quotes, like this:

"mortgage rate"
"rate quote"
"credit card"
"cell phone"
SPAM:
ADV:
harrassment
"work from home"

This worked for me at the command line with the following file as a test file (sample6.txt):

From: jking@myisp.net
To: devnull@thinkspot.net
Subject: ADV: testing ignore $$$
Date: Sat, 08 Sep 2001 12:24:07 -0700
X-Mailer: Forte Agent 1.8/32.548

From a friend.

The end of this test is near.
It is at hand.
END.

and the following command line instruction, with its result:
822field subject < sample6.txt | grep -f subjects.txt
ADV: testing ignore $$$


This should fix it for you. Your .qmail command should be (I think...don't have time to test it right now...)

|conredirect null@brilliantcorners.org sh -c '822field subject | grep -iqf subjects.txt'
|vdeliver

I think that will work. (Barring you use any special characters like $ and so forth that bash reserves for special meanings. I haven't figured out how to escape those, yet.)

Good Luck. Let us know if it works.

[bturner]

Sorry, Sheila, that didn't work.

I'm now trying to work out a Perl script based solution, since I keep hitting roadblocks with grep.

UPDATE: I'm a complete moron! I've been creating all these .qmail files in the default user's directory. I moved some of the stuff into the root domain directory /big/dom/xdomain/ and my new perl-based script now works.

Here's the script if anyone is interested:

#!/usr/bin/perl -w

use strict;

my @mail = <>;
my @keywords = <DATA>; # at the bottom of the script
my $subjectLine;

for (0..$#mail)
{
$subjectLine = $mail[$_] if ($mail[$_] =~ /^Subject:/i);
}

my $foundIt = 0;
for (0..$#keywords)
{
my $keyword = $keywords[$_];
$keyword =~ s/\s//g;
$subjectLine =~ s/\s//g;
if ($subjectLine =~ /$keyword/gi)
{
$foundIt = 1;
last;
}
}

if ($foundIt == 1)
{
exit(0);
}
else
{
exit(1);
}
__DATA__
mortgage rate
rate quote
credit card
cell phone
SPAM:
ADV:
harrassment
work from home
as low as
background check
life insurance
download your copy now
more spendable income
government knows how
making serious money
feeling lucky
customized gift
password request
online business
secretly monitor
how can this be
make money
internet speed
interest rates
attention travelers
saw your website
the future is now
uninsured
confirm your subscription
perfect gift


and the .qmail file:

|condredirect null@brilliantcorners.org /usr/bin/env perl /big/dom/xbrilliantcorners/bin/filter.pl
|vdeliver



I'm going to leave null@ as a regular email address to see what kind of stuff it catches.. and if it works well, it becomes a black hole!


DOH!!
Bill

[sheila]

Well, I am just baffled by the whole filtering on the subject thingy. I'm pretty sure it's because of multiple words on the same line in the subjects.txt file. I'm so sure that there has to be a way to do this, that I spent a good hour or so messing around with it, trying to find a way to accomplish it. But in the end, I gave up. Someone more knowledgeable in bash and shell utilities is needed here.

Definitely, a script will work fine. Perl script like yours, or here is my Python version:

import sys, string, rfc822

headers = rfc822.Message(sys.stdin)
subject = headers.getheader("subject", "")
f = open('subjects.txt')
subjectlist = f.readlines()
f.close()
subjectlist = map(string.strip, subjectlist)
matches = filter(lambda x: string.find(subject,x)>-1, subjectlist)
if matches:
sys.exit(0) #success -- found a match
else:
sys.exit(1) #failure -- no match

where I saved the above code as subjectscreen.py in my /big/dom/xdomain folder, and saved the offending subject substrings in the same folder under the file name subjects.txt (same as you had before) with this .qmail file:

|bouncesaying "bad subj" /usr/bin/env python /big/dom/xdomain/subjectscreen.py
|vdeliver


This I have tested, and works. Still bugs me that I can't get that darned way to do it without a scripting language like Python or Perl. But, I guess I have to get on with my life, now.