Robots

[JoeZychik]

Could some please tell me how the robots.txt file works?

Thank you very much.

[Deb]

The default robots.txt that is pre-installed into your directory contains:

User-agent: *
Disallow: /stats/

What this is doing is telling search robots (like altavista) to NOT spider your stats directory. If you want to allow it to index your stats directory you just remove it.

You can add anything you like to robots text. I have a site that I do not want many areas spidered and I use my robots.txt file to accomplish this (at least with spiders that follow the rules)

A bot will first look for robots.txt and see if there are any directions in there that it should abide by. If there are no robots.txt it simply moves on.. in your case there is one and now altavista knows not to spider your stats directory.

A good explanation of robots.txt can be found at http://webinfo.inch.com/server/apache/robots.html

A cute robots.txt generator can be found at http://mattsmxpxpg.virtualave.net/robots/

Hope this helps
Deb

[JoeZychik]

It helps, as your advice always does.

jz