PDA

View Full Version : A life without spam


manfred
03-01-2010, 04:25 AM
Today, I want to write down a few words about a spamfilter, which did made my life much easier. Normaly I don't write such Testimonial, but now I MUST do it. :)

Until three months ago I had also a big problem with spam. I had activated Spamassassin, EFM and the "received line" IP-address filter to stop the flood of spam.
All of this antispam software and technique is good but not good enough.

Spamassassin: Good if you can manage and tweak the config. If you are on a shared host, you can't change the config. You can only tweak the cut-off-score and do some whitelisting. But, what to do if you whitelisted your own domain and the sender forged any email address at your domain? I think 99% of the spam mails do have forged sender addresses.

EFM: You have to edit the "banned word" lists on a daily base.
Examples: V1agra, V1@gra,.. Pills, P_i_lls,... and so on

"received line" built-in filter: Nice and effective. But, how do you block an IP-range if your customer is within this range? Block every single IP-address? My "blocked-IP list" was three A4 sites long.

And now the best question. Delete, Tag, Redirect or Bounce the email?

Delete: What is if the filter deletes a good mail?
Tag: No email lost, but you still have to review the tagged messages.
Redirect: see "Tag"
Bounce: STRICTLY NO! Innocent people get the bounced emails, and maybe your emailserver will be blocked.

You can kill a big amount of spam, but you will also kill good emails and you will still receive spam. For this reason I started to search the www for a better (effective) solution. Greylisting, software to install locally on your desktop computer,...

And then I found IpNation! I read the introduction and wrote an email to the developer (a FQ customer) to become a volunteer. Only a few hours later, I was a "Padawan Learner" for this incredible effective spamfilter.
I only can say, try it and love it.

The filter also does have a remailing function to remail killed ham (false positives). !! No email is lost !!

Each email is saved to one of four daily log files on the server by the filter. You download all the data files with the provided viewer software and start the so called "Friends Finder". Now the viewer is doing what it has to do, find the good emails, the false positives and ask you if some (just a very few!) emails are spam or ham.

If a good email was killed, you can simply hit a button to remail it. The viewer takes the original email, uploads it to the server and the remailer software puts it directly back into the email queue. So, you can see, no email is lost.

Some more important points:

1. The "Friends Finder" tool takes most of the tedium AND risk out of searching for False Positives
2. We all contribute our low scoring spam which means we can more rapidly create and deploy new rules
3. It comes with a complete "IP-to-Nation" data file, and a set of pre-defined "kill worthy" nations

The developer and the IpN community will help you through the installation and configuration process.

Where can you find this incredible tool?
At http://www.IpNation.org (http://www.ipnation.org/)

-Manfred
--Die Spam, die! :)

P.S.: The filter is also for non english users. I'm sitting here in the heart of Europe, Austria, and it works perfect against french, russia, english and german spam.

Buck
03-01-2010, 11:55 AM
I have been using the IpN filter & Viewer for several years now. I have killed more than 1.5 million SPAM messages since I started, and I can honestly say that no SPAM gets through to my end-users. I have a dozen e-mail addresses, and none of them see SPAM anymore. I even have the catch-all feature enabled, and still nothing comes through. Backscatter is a common problem, and here's what I encountered: The biggest month was July 2009, with 702,632 pieces of Backscatter. That's almost one thousand per hour...
... for every hour, of an entire month. The filter caught all of that, none of it came through to me!

I would recommend the IpNation system to anyone!

hobbes
03-01-2010, 12:46 PM
What is FQ's take on IpNation? Is there much of an impact on the server? Any chance its use would be forbidden? Bruce?

Bruce
03-01-2010, 07:54 PM
I do not recall nor have on record any incidents of IpNation ever being fingered for server resource abuse. It could be forbidden based on that criteria, but I don't forsee that happening given that we haven't had a significant problem with other such filters (most notably EFM, which appears to be more likely to trigger high loads). The comment about being able to "remail killed ham" makes me a little nervous, as that could potentially double the load on the POP toasters if it comes into widespread use, but I'm willing to wait and see on that one.

In short, it is not a problem now, and does not appear to be something that would become a problem in the future.

hobbes
03-02-2010, 08:29 AM
Thanks!

Buck
03-02-2010, 11:43 AM
I remail maybe one or two FP's a week at the most. The filter gets smarter & smarter over time, which results in less & less FP's, so it's not really much of an issue, in my opinion.

CamFraser
05-31-2010, 05:23 PM
Today, I want to write down a few words about a spamfilter, which did made my life much easier. Normaly I don't write such Testimonial, but now I MUST do it. :)

Until three months ago I had also a big problem with spam. I had activated Spamassassin, EFM and the "received line" IP-address filter to stop the flood of spam.
All of this antispam software and technique is good but not good enough.


+1

I've been a very satisfied user for more than four 1/2 years, with more than a million dead spam across four domains. :yeah:

Like manfred said, it's not just about killing spam, it's about reliably finding FPs. Digging thru a redirect mailbox is too error prone, and too soul sucking.

IpNation's FriendsFinder is a bit weird, and kind of homely, but it gets the job done quickly and leaves me with peace of mind. It reminds me of a Firefly class freighter. :P

There is a learning curve, and if your domain has accounts that have been signed up for a bunch of crud, it'll take a lot more effort in the beginning, but it'll be worth it. Of my four domains, three were easy (stabilized within a month), one was a grind, taking more than three months to get under control. It's still more effort than the other three combined.

The tools are better now, but figured I'd discourage those who easily can be.

We could really use a graphics designer or artist to beautify parts of it. Anyone interested?


What's better than massive spam slayage, and rock solid FP detection?!?
SHARING ALL YOUR BIG DANG SPAM DATA!

Seriously. That's almost kewler than looking at graphs of your body count (http://IpNation.org/data/graphs.htm).
On the rare occasion that a spam sneaks thru, or on the more common occasion that a brand new spam campaign hits you first, Chipmunk autopsies the corpse, writes new rule(s), and sends that out to everyone on the Team. We're small, but fast. :P

It's not just your fellow IpNationers who benefit. At least three times, Chipmunk has spotted general or FQ specific issues, that have been reported by an FQ teamie, and helped every FQer.

For example, the team's official Knitting Granny was the very first teamie to notice the SpamAssassin "Y2010" bug, reported it to Chip, and a volunteer kindly posted it here.
Hoorah for Grannies and all volunteers! :P

Buck mentioned his backscatter bonanza. For years, Chipmunk had been carefully cataloging backscatter, planning to turn it into a "smart" DNS info list. We didn't have anywhere near enough data. Then in one month, Buck got more backscatter than the entire team had received to date, and it kept coming. With abundant help from Buck (he didn't mention he had to slog thru much of that manually - thanks dude!), there's now enough data to seed that info list, and some day the entire world will benefit from that.

When I read Buck's post, I had a mental image of him grinning contentedly instead of groaning about that one month spike. :P



In short, it is not a problem now, and does not appear to be something that would become a problem in the future.

Bruce, check out the "Safety options" stuff at the bottom of the IpNation filter features page (http://IpNation.org/filter/features.htm).

All of those "Safety options" have been thoroughly tested, and were in the original design.

Chipmunk is considering expanding the environment variable's throttle options. Right now, it mainly limits DNS lookups and lets you shorten the Linux signal thingie timeout (um, don't ask me about Linux stuff, it's above my payscale).

Terra is credited as the inspiration for the date lockout.
You sysadmins don't have to worry about anyone running an old copy of IpNation. In the early, experimental days, that lockout ranged from 3-6 months. Now, it's a year. The server filter is so stable and solid, it went more than a year (Jun'08 to Nov'09) without any changes (except for one date lockout release). It has its own mini scripting language, so all spam morphs were dealt with by team-wide rule updates.

The latest version of the filter automatically tracks all DNS timeouts, and if there's too many for one blocklist, it disables just that one list for the rest of the day. That's been thoroughly tested, thanks to FQ's current lockout problems with Uribl.