FQuest Alert: Downtime

[Terra]

Total Downtime: 20 minutes

Today at 1:08pm EST, the server froze due to a resource contention...

One of our site owners started a very memory intensive CGI script, at the same time I started an Apache server rehash to bring in some new enhancements... We clashed with each other and exhausted all memory... I checked right before I did the rehash, and saw all green lights - in the middle of the server re-stabalizing, he had started his script which quickly began a domino effect...

I will be in contact with this customer, and I have already limited the amount of memory that he can use - so that this will not happen again...

I am also reconsidering how I setup new accounts - and place resource restrictions in the beginning - instead of waiting to see if restrictions need to be invoked... I have held off until now, as I did not want the extra overhead associated with the limits from the get-go... In light of todays server freeze, being this is the second time a runaway script has taken the server down - I will have to trim the fat in areas, to compensate for the extra overhead...

Uptime is paramount to us - and will take measures to ensure this...

We apologize for any inconvenience that this has caused anyone...

--
Andrew Gillespie
Systems Administrator
FutureQuest.net

[Mari]

Andrew,

Can you tell us what types of scripts would cause the problem? I wouldn't want to do anything to mess you up but if I did, it's because I didn't know any better. If you can educate me a bit, I'd appreciate it! Are the scripts a certain size or do specific things? (Sorry for this ignorant sounding note!)

Mari Doerr

[Deb]

Hi Mari,

It's not really "this or that" script.. it's usually faulty/sloppy coding in a particular script that causes it....

Scripts that need to DO a lot are the ones to be extra careful of.. Chat scripts are common for this type of thing.

Now the script that caused the glitch today was such that I think it creates an index of your site on the fly... on the norm this is not too much of a problem.. but if the site is huge then the script has to go through and index each and every file on the site, create titles and links and file sizes for each of them and return an html page showing them all... to the best of my knowledge this script could be coded differently to optimize it for a large site, and/or limited on how often it is allowed to run. For which we're working on that.. the site owner is more then cooperative so no worries there.

This issue shouldn't be a problem in the future as it seems Andrew is going to go ahead and limit all of us so we can not over extend the resources. Who will this cause a problem for?

Noone unless they try to run a script that would potentially take the server down http://www.aota.net/ubb/wink.gif If a script does become 'out of control' then the script itself will be halted rather then the entire site or server...

Hope this helps a bit
Deb

[This message has been edited by ccTech (edited 01-04-99).]

[Terra]

On a server wide basis, I am initiating 2 limit methods...

1) Max Memory: 10Meg
2) Max CPU time: 30 seconds

This should be plenty of room for a script to run, and the 30 seconds is like a human lifetime for a script to execute... http://www.aota.net/ubb/wink.gif

If a script should exceed that limit - then it will be automatically killed... Another reason I did not want to invoke this was because the error message are very vague, and does not always point to the above restrictions killing it... I will have to work on more verbose error reporting on this...

One of the main memory hogging programming method is to slurp a file into an array...

For example:

open FH, "./reallyreally_big_file" or die "blah: $!";
@slurp = <FH>;
close FH;

foreach $line (@slurp) { do something with each line here }

A better way to write the above, and avoid reading everything into the @slurp array (stored all in memory), is to use a 'while' loop...

I always rewrite these memory hogs like this...

open FH, "./reallyreally_big_file" or die "blah: $!";

while (defined($line = <FH> )) { do something with each line here }
close FH;

As you can see, we step over each individual line, instead of reading it all into memory then foreach'ing through the array...

This alone is a very powerful technique to optimize your script, and reduce the memory consumption...

--
Terra
Systems Coder
FutureQuest.net

[meikel]

1) Max Memory: 10Meg
2) Max CPU time: 30 seconds


Very good!!! I like solutions taking care of problems before they occur. If one needs more, he is always free to ask for an exception.

But now I feel a lot more safe when programming perl. An endless loop is coded easily and if this can kill the server ...

I believe that these restrictions leave a lot of room. If one needs more, no problem, but let him ask so that you know that there is something more resource intesive going on.

You have my full (110%) support on this topic.

Greetings from the epidemic
[nbsp][nbsp]Meikel Weber
http://www.hp.europe.de/weber