Re: Yahoo spider crashing mysql on non FQ site
It never occurred to me that you could slow a spider that way. I assumed from G's spastic crawling that they sent for pages on an inhuman schedule regardless of the response they receive (at least in the short term).
Yahoo has always been well-behaved, and is supposed to honor the "crawl-delay" directive. Have they been ignoring that?