Spider Rate Control questions

[Randall]

Yesterday I had my first run-in with the SRC. I'm working on an experimental AJAX script that calls two files on the server every 15 seconds (I could probably afford to stretch that out a bit), but last night something went berserk. The logs show dozens of requests over the course of just one minute -- I killed it at that point, or I'm sure it would have gone on indefinitely.

Not sure if it was the client-side script screwing up, or if the AJAX requests were being retried immediately after getting the SRC error message, or what.

Anyway, some questions:

Can activity happening on multiple IRMs trigger the SRC? The sites on that package generally don't get hit very hard.

Would multiple people requesting the same page (once each) in a short span of time trigger it, or does SRC look at individual IPs?

Finally, about the script I'm working on. If I staggered the two page requests -- say 3 seconds apart -- would that be less likely to create server problems? And are two simultaneous page requests really enough to trigger the SRC? I'm wondering if there might be something else involved.

As I work on this I'll probably get smarter and cache the second request (data) if the first one (status) reports no change, meaning fewer double-hits...

Does the SRC return a unique HTTP status code? If I knew to look for it, maybe my script could handle the situation more gracefully (certainly couldn't get any worse).

It goes without saying that this AJAX stuff is all new to me, and I know just enough to be dangerous. :EG:

Randall

[Terra]

SRC only works on Dynamic Content (PHP, SSI, and CGI)...

SRC works via a Token Bucket based on source IP + destination IP...

The Token Bucket is defined as:
1) 30 initial/max tokens (burst)
2) bucket refills at a rate of 1 token per second

The SRC HTTP FutureQuest specific result code is: 503

--
Terra
sysAdmin
FutureQuest

[Randall]

The files in question are PHP scripts, so that much makes sense... The Token Bucket is defined as:
1) 30 initial/max tokens (burst)
2) bucket refills at a rate of 1 token per second
That's a pretty big bucket. The script as written couldn't possibly generate that many requests without going into a full-on freak-out loop, so I'm thinking that it may have been a fluke browser malfunction. I'll have to experiment (carefully, very carefully) and see if it was just a one-time thing.

Thanks for the details -- now I have a better idea of what to be looking for. :winky:

Randall

[kitchin]

The AJAX calls http, so it would be in the access.today, etc., logs, right?

[hobbes]

The AJAX calls http, so it would be in the access.today, etc., logs, right? Yes.

[Snarpy]

Was it me, Randall? Opera wouldn't show the page and said Status: checking. Maybe it was looping. I will PM you my IP.

[Randall]

Was it me, Randall? Opera wouldn't show the page and said Status: checking. Maybe it was looping. I will PM you my IP. Nope, it was definitely me. I could see that from the logs. :blah:

Randall

[Randall]

Starting to look like something happens when my Mac comes out of sleep mode and I try to load the page. Might be peculiar to Firefox, even. Maybe I just need to give it a few minutes before I run it -- it's working OK now. :ytthink:

Randall