FutureQuest, Inc. FutureQuest, Inc. FutureQuest, Inc.

FutureQuest, Inc.
Go Back   FutureQuest Community > General Site Owner Support (All may read/respond) > Email & Mailing List Management
User Name
Password  Lost PW

Reply
 
Thread Tools Search this Thread Display Modes
Old 02-16-2002, 09:17 PM   Postid: 62147
sheila
Site Owner
 
sheila's Avatar

Forum Notability:
0 pts: Even-handed
[Post Feedback]
 
Join Date: Aug 1999
Location: Metro Los Angeles Area
Posts: 7,398
Using Vipul's Razor at FutureQuest

OK, following up on Arthur's post in which he sketched out how one might do a razor-check on an email message with Python (instead of the standard Razor package, I've made some progress on another one of the parts of Vipul's Razor.

Essentially, the client part of Vipul's Razor comprises three parts:
razor-check: This is the part Arthur worked on, which simply checks whether an individual email message has been reported as spam.
razor-report: This part reports a message that is not already in the database. This part I have not (yet?) worked on.
razor-discover: This part goes out and "discovers" razor servers, and determines which one is closest (by timing pings to the servers).

Here is my Python code which is essentially equivalent to the razor-discover package available from the links above. It would have to be set up with a cron job (which I don't much know how to do...well, OK, I haven't tried yet). Anyhow, FWIW, here is my Python code:
Code:
import socket, string, os

subdomainlist = list(string.ascii_lowercase)

zone = 'razor.vipul.net'
host_IP_list =[]
response_times = {}

for subd in subdomainlist:
    host = '%s.%s' % (subd, zone)
    try:
        IPaddy = socket.gethostbyname(host)
        host_IP_list.append(IPaddy)
    except socket.error, e:
        pipe = os.popen('dnsquery -t CNAME %s' % host)
        data = pipe.read()
        if data.find('list.terminator') > -1:
            break


for IPaddy in host_IP_list:
    pipe = os.popen('ping -n -c 4 %s' % IPaddy)
    result_list = pipe.read().split('/')
    avg_resp_time = float(result_list[-2])
    response_times[avg_resp_time] = IPaddy
    time_list = response_times.keys()
    time_list.sort()
    sorted_host_list = []
    for t in time_list:
        sorted_host_list.append(response_times[t])

homedir = os.getenv('HOME', '')
f = open(os.path.join(homedir,'.razor.lst'), 'w')
for IPaddy in sorted_host_list:
    f.write("%s\n" % IPaddy)
f.close()
The main problem that I find, right now, is that I can only discover two servers. That sure isn't very many!!! I think that Arthur's suggestion, that FutureQuest set up a Razor server right here, on one of their servers, is an excellent idea. Hmm...

One of the two servers is about 40 ms ping away, and the other is more like 116 ms ping away.

Anyhow, this all looks promising. Reading the Perl code and trying to figure out what they are doing is a little tough for me (since I am not JAPH). I cannot see at all how Arthur decided that the correct telnet command to give to the razor server for the razor-check was:
Code:
telnet.write ('key:%s&action:lookup\n.\n' % key)
even though I've looked over the Perl code from which he adapted it several times.

Which kind of stumps me, then, when I try to see how to adapt the razor-report module.

Ah, well. Enough coding for one day, probably, anyways.

<edited code...removed a "raise" statement after the "break" in the exception block>
__________________
sheila
http://www.thinkspot.net/sheilaruns/

Last edited by sheila : 02-16-2002 at 10:39 PM.
sheila is offline   Reply With Quote
Old 02-17-2002, 03:29 AM   Postid: 62167
 Arthur
Developer
 
Arthur's Avatar
 
Join Date: Nov 2000
Location: The Netherlands
Posts: 2,212
Wow, you've been busy
Quote:
The main problem that I find, right now, is that I can only discover two servers.
That is correct, that's also how many razor-discover finds when you run it from a FutureQuest server. One is located at NYI in New York, the other is located at XS4ALL in Amsterdam.
Quote:
I cannot see at all how Arthur decided that the correct telnet command to give to the razor server for the razor-check was:
I kind of cheated on that one I didn't feel like going through all the Perl modules that come with Razor, so I used Ethereal to sniff out what Razor was sending over the wire and what the response from the server was.
I'll have a look at the razor-report script and try to figure out what it does and how it does it.
Arthur is offline   Reply With Quote
Old 02-17-2002, 05:35 AM   Postid: 62172
 Arthur
Developer
 
Arthur's Avatar
 
Join Date: Nov 2000
Location: The Netherlands
Posts: 2,212
I figured out how razor-report works, it's almost the same method as the razor-check, the query just says 'report' instead of 'lookup'.
Here's the conversation between client and server:
Vipul's Razor 1.11, protocol version 2.
key:c47d03b6b46479086848c13deae57878bb254539&action:report
.

Accepted c47d03b6b46479086848c13deae57878bb254539a

The answer can be 'Accepted' or 'Rejected' (also see Razor::Client). razor-check also checks the protocol version and has error handling (tries the next server if the current one fails), but the basic thing it does is the above query after calculating the hash.
Arthur is offline   Reply With Quote
Old 02-17-2002, 10:47 AM   Postid: 62173
sheila
Site Owner
 
sheila's Avatar

Forum Notability:
0 pts: Even-handed
[Post Feedback]
 
Join Date: Aug 1999
Location: Metro Los Angeles Area
Posts: 7,398
Last night I also subscribed to the razor-users mailing list and browsed the archives of the list a bit, as well. They are discussing modifying the encryption method a bit (I think?). They are having problems with too many false positives or something.

This would be a problem, of course, with not using their authored package.

I posted some questions to their list but haven't received any responses yet.

Yeah, I was busy on that code yesterday (should have been correcting papers...oh well, guess I'll do that today!). One thing: Python doesn't have built in DNS/ping functions to the degree that I needed them, so had to resort to calling system commands on the FutureQuest servers (man pages are your friend). Doesn't make the code very portable, though.

Quote:
razor-check also checks the protocol version and has error handling (tries the next server if the current one fails), but the basic thing it does is the above query after calculating the hash.

Could you post more info about the conversation on the razor-check. Last night I was so frustrated. b.razor.vipul.net kept giving me a Negative on EVERYTHING I sent it, no matter what. I was beginning to think there was an error in our code for checking the hash. Then I tried a.razor.vipul.net and got a Positive (like I would have expected) on something that b.razor.vipul.net had already told me was Negative. Maybe that error handling would have helped. I did see the protocol check in the code.

Where can I get Ethereal? Is it free? I don't think I want to actually install all the Perl modules for Razor and Razor itself, though.
__________________
sheila
http://www.thinkspot.net/sheilaruns/
sheila is offline   Reply With Quote
Old 02-17-2002, 12:54 PM   Postid: 62176
ryount
Site Owner
 
ryount's Avatar

Forum Notability:
94 pts: Helpful Contributor
[Post Feedback]
 
Join Date: Jan 2001
Location: Clearwater, FL, USA
Posts: 825
http://www.ethereal.com/
ryount is offline   Reply With Quote
Old 02-17-2002, 02:03 PM   Postid: 62180
 Arthur
Developer
 
Arthur's Avatar
 
Join Date: Nov 2000
Location: The Netherlands
Posts: 2,212
Like ryount said Ethereal is at that address, it's free and it's available for Linux (and other *nixes) and Windows.

I checked with a small number of spam messages and both the Razor servers (b.razor.vipul.net and a.razor.vipul.net) gave the same result. I'll throw some more messages at both, to see if there are any differences.

The error checking isn't very sophisticated stuff, it just checks if a connection to the first server can be made and if it cannot, the other server is tried.
I didn't realize before, but there are just two servers in total.
Arthur is offline   Reply With Quote
Old 02-18-2002, 05:32 PM   Postid: 62215
sheila
Site Owner
 
sheila's Avatar

Forum Notability:
0 pts: Even-handed
[Post Feedback]
 
Join Date: Aug 1999
Location: Metro Los Angeles Area
Posts: 7,398
Well, I think I'm going to leave off with spending any more time on trying to adapt Vipul's razor for use within a Python script. It turns out, that they are going to change the encryption method for generating the keys some time in the not-too-distant future. Instead of using sha they will be using something called nilsimsa. I don't want to spend time on this only to have it become incompatible in a short time.

For references see:
http://www.geocrawler.com/lists/3/So...539/0/7872330/
http://www.geocrawler.com/lists/3/So...39/75/7835683/
http://freshmeat.net/projects/nilsimsa/
__________________
sheila
http://www.thinkspot.net/sheilaruns/
sheila is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 visitors)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -4. The time now is 09:51 PM.


Running on vBulletin®
Copyright © 2000 - 2013, Jelsoft Enterprises Ltd.
Hosted & Administrated by FutureQuest, Inc.
Images & content copyright © 1998-2013 FutureQuest, Inc.
FutureQuest, Inc.