what numbers are we talking about as far as making it "safer"...the script, that is.
Two posts to reference first:
Rate Limiting Metrics:
General Goal for Rate Limiting:
No more than X 'search' instances per IP
No more than Y 'search' instances in operation total (all IPs inclusive)
Deny 'search' request if (1 min OR 5 min) server load is above 3.0 (groked via /proc/loadavg)
**Using either 1min or 5min, both have their pros and cons due to differences in rise and decay rates
if I can make it friendly only to humans, I will. Even it if means blocking the great Google in the sky.
Extremely difficult to do, I can assure you... Most of the problems now are caused by 'ill willed' spider operators that try to cloak themselves as 'humans'... This forces server/script operators to look at the heuristics of their behavior instead of relying on specific items or pattern recognition...
"If it walks like a duck and talks like a duck, then it must be a duck"
no longer applies...
The main problem with Googlebot now is that it comes in via spider (wolf) packs... Each spider alone is server & script friendly, but have about 10 Googlebots (nicely) spidering your site, and you have a resource intensive situation happening...
I am not faulting Google over this, as the vast quantity of information they have to spider and index is astronomical... Therefore they do have to scale the task by issuing multiple bots to keep up with the web... I for one do not see an easy solution for Google, as what they are doing for the Internet is a GoodThing™...
if you can give me some hard numbers, I can go to the script support forum and begin to get it sorted this evening.
X = 3
Y = 10
SL <= 3.0
The above values will offer your script a bit of latitude to operate, without thrashing the server...
'Y' is sufficiently high enough to cover bursting, however this is not to mean it can be a 'sustained' load with 10 of your scripts in memory all the time... The 'SL' should help to maintain that the 'Y' is for burst only.
Disabling a single script is extremely dangerous
One other thing....was it not possible to disable just THAT script last night?? Or is that an ignorant question?
, therefore it is now our policy to completely disable any/all CGI activity...
The main problem, is that another one of your scripts may be dependent upon the script we just deactivated... If it cannot execute that deactivated script, then it may not handle that condition gracefully (lost $$$ Orders)... The potential of data corruption (or lost data) is at a much higher risk when individual scripts are deactivated... By freezing the CGI, we remove pretty much all risk (and liable) of a BadThing™ happening...
Think of a shopping cart where the main cart scripts are active, however we have just deactivated the 'credit card processor' script... This is just one simple scenario, of millions, where things can just plain go wrong when a single script has been deactivated...
In short, we play it safe for all parties involved...
--But your honor, FQ got their Peanut Butter in our Chocolate - we'd like 1 million please--
<EDIT: libel != liable>