Making a Web Directory

[Kayge]

I run a site about computer games. I'd like to make a directory of related sites. I was hoping I could get some kind of bot to index submitted sites so users could search the directory. Anyobdy know where I can get a bot that can do this?

[Benson]

I'm not sure if this is what you're looking for (it *might* be a little complex for what you need), but this script can give you a miniature Y*hoo! on your site, fully searchable:

http://www.gossamer-threads.com/scripts/links/index.htm (http://
http://www.gossamer-threads.com/scripts/links/index.htm)


------------------
ICQ#8418658-alba gu brath!

[Kayge]

Thanks for the suggestion. But do you know how it searches sites? I tried some searches that should have showed hits and nothing came up. I think it only searches the description the person enters.

That's actually too simple. I need something that will actually spider a site after it is submitted. You know, look for meta tags or what have you. Their program deosn't seem to do this.

[Benson]

You're right, it's not what you're after. Links works like Y*hoo!, by organizing, classifying, and rating submitted sites. It's not a spider or bot tool.

[Justin]

http://www.xav.com/scripts/search/index.html

Try that one out. I've used it myself. It uses meta tags as well as the page title, and it creates an index file whenever you run build.pl. It will spider anything in the directory you specify.

Oh, I just realized... it only works on the site that the script is installed on, not other sites... sorry http://www.aota.net/ubb/smile.gif There are 2 other search scripts there though, and one of them may help you.


------------------
Justin Nelson, SFE Software
www.vdj.net (http://www.vdj.net)

[Del]

Take a look at ftp://ftp.xav.com:21/search.txt (http://ftp://ftp.xav.com:21/search.txt) It's from the same guy Justin mentioned, but this one's a bit 'hidden' from normal view (I don't think he links to it anywhere). The code may or may not work properly, I've never tried it, although I've used quite a few of the other scripts from xav.com and am quite happy with them.

Anyway, that one sounds kinda like what you're asking for (if I remember correctly anyway).

Del

[Jacob Stetser]

I use Xavatoria II, I think, on my site for local searching, but it does remote indexing as well. In fact, I was having problems indexing locally so I'm actually telling it to index my site as if it were a remote site, that is, via http.

Before I modified the outgoing links so that Xav wouldn't follow them, it spidered every page it found. Be careful, a spidered index grows very fast!

Jake

[Kayge]

Jacob, can Xavatoria 2 be told to spider a certain site, and only that site.

Like if I told it to spider http://www.aota.net would it be able to spider that site but not follow links to external sites?

[genc]

I am not sure if I understood the question or not?
For indexing, webinator is a great program. I doubt that any other programs can beat it. it follows all the links and indexes all the pages in that web pages unless you ask for something else. Very customizable. I only have problems with international characters,, but this is because of my lack of knowledge. Best of all it is free up to 10,000 pages.
the guys from tech support is really nice and they answer the questions so detailed, I am learning more than I expected.. They could have ask me to look at the manual, which answers most of my questions.
take a look at:
www.thunderstone.com