PDA

View Full Version : blank vs. no robots.txt file


Jeff
10-22-2006, 09:17 PM
If you have no robot exclusions for a site, is it better to have a blank or no robots.txt file, or does it make no difference at all?

Wassercrats
10-22-2006, 09:25 PM
It might be a good idea to exclude a special page created just for bots to see which ones disobey so you'll know the good bots from the bad.

Kevin
10-23-2006, 12:24 AM
I have never setup a robots.txt file myself (I always figured that anything I don't want indexed I don't want to advertise either) and I have never had any problems because of it.

However, if you do something special for 404 errors you may want to create an empty robots.txt file simply to avoid the 404 errors.

The same could be said for favicon.ico which IMO is worth creating (or downloading).

Melissa
10-23-2006, 07:03 PM
According to Google:
http://www.google.com/support/webmasters/bin/answer.py?answer=40360
You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).