The last few days, my site has been hit hard by googlebot. I typically get 1000-2000 hits per day from bots, but google has been hitting me 10,000 times/day the last few days.
I sent them a polite email about it, with a hint that they should support the Crawl-delay directive in robots.txt (I know it isn't standard).
I guess I have two questions:
Why is this happening? Did Google change something recently? I've never noticed a spike like this before, but my site is less than a year old.
How will Google's respond to my email? I hope they don't turn it off completely.
I've noticed psbot has done the same thing. I have tolerated their bot because it only hit me several hundred times a day, but they shot up to several thousand this week, so I disallowed them and added them to my htaccess "forbid" list. Even so, they have been pulling thousands of FQ's default "forbidden" pages every day.
It's probably the recent changes in my page design--it's much more attractive now ;-)
Book Award Annals