Googlebot blitz ?

[kennylucius]

The last few days, my site has been hit hard by googlebot. I typically get 1000-2000 hits per day from bots, but google has been hitting me 10,000 times/day the last few days.

I sent them a polite email about it, with a hint that they should support the Crawl-delay directive in robots.txt (I know it isn't standard).

I guess I have two questions:

Why is this happening? Did Google change something recently? I've never noticed a spike like this before, but my site is less than a year old.

How will Google's respond to my email? I hope they don't turn it off completely.

I've noticed psbot has done the same thing. I have tolerated their bot because it only hit me several hundred times a day, but they shot up to several thousand this week, so I disallowed them and added them to my htaccess "forbid" list. Even so, they have been pulling thousands of FQ's default "forbidden" pages every day.

It's probably the recent changes in my page design--it's much more attractive now ;-)

Ken
Book Award Annals

[Andilinks]

How many pages are on your site?

Did Google change something recently?

They are constantly changing.

How will Google's respond to my email? I hope they don't turn it off completely

I have heard of cases where such an email has had an effect far beyond what was wanted. It is best to keep your tone respectful...

I will be perhaps blocking Gbot from parts of my site soon, selectively. I'll be posting my results and observations about this soon in another thread on this forum (titled "Banned from Google")

Yes your site is quite attractive.

Andi

[Winters]

Maybe they're putting a little harder for the next update?

http://news.com.com/2061-10803_3-5824161.html

[Stecyk]

Hi,

A great site to keep on top of the latest developments in the Search Wars, including Yahoo and Google, is John Battelle's Searchblog. Just look at some of his recent posts concerning to find the Google Yahoo! stuff.

Just passing this information along.

Best regards,
Kevin

[kennylucius]

Originally Posted by Andilinks

How many pages are on your site?

I really have no idea. It's dynamically generated from a database. Probably about 10K directly from my database, and close to 50K from Amazon's database. Perhaps I should add a nofollow directive to links that rely solely on Amazon. That should reduce the crawlable pages to 10K.

Originally Posted by Andilinks

I have heard of cases where such an email has had an effect far beyond what was wanted. It is best to keep your tone respectful...

I know what you mean. I was fairly respectful, though I mentioned the Crawl-delay thing.

[kennylucius]

I have added "rel='nofollow' " to links to pages that are largely Amazon content, and also added to those pages "<META NAME='robots' CONTENT='noindex,nofollow'>" to the header. Hopefully that will slow down googlebot.

Which begs the question, will that affect my pagerank? Those pages contain many links to other pages on my site. I am under the impression that such links count for very little.

[kennylucius]

Just to follow up, this is the reply from Google.

Thank you for bringing this to our attention. We've reduced the load on your servers. We apologize for any inconvenience and appreciate your patience. If we can assist you further, please let us know.

Regards,
The Google Team

[Wassercrats]

Is it possible it was some agent other than the Google's Googlebot that's just named Googlebot? You have to confirm that it's Google's IP address, right?

[Andilinks]

Google's IP's begin with 66.249.

Though on rare occasions I've seen Gbot use an IP beginning with 80.

But others report this too:

http://forums.seochat.com/t45601/s.html

[Andilinks]

This may be of some interest:

http://www.webmasterworld.com/forum30/30268.htm