FutureQuest Alert: Downtime

[meikel]

PS: I have been warned that if I keep having to post downtimes - then I will have to go sleep on the couch at the server room...

As long as you're allowed to sleep ... ?!?

Meikel

[Deb]

HAHA!!

I'll have to re-think my threats... When you look at it that way he just might remove the power cord :P

Deb

[Terra]

Today at 9:46am EST, I inadvertantly brought the server to a halt...

As part of our ongoing optimizations, I had taken our server logging sub-system (18% of overall server load) offline for implementing new procedures... This was done to help ease the current load so that the Apache server had more to work with...

Bad timing snagged me as someone started to run a mailing list - generating an enormous amount of system logs, and with that sub-system down, the logs had nowhere to go but consume system memory... I scrambled to get the syslog daemon back online to handle it's influx - but missed the boat by 40 seconds which eventually caused memory starvation...

This forced a server reboot to get everthing back online...

Down: 9:46am EST
UP: 10:30am EST

My apologies for this - I should have seen it coming...

--
Andrew Gillespie
Systems Administrator
FutureQuest.net

PS: I have been warned that if I keep having to post downtimes - then I will have to go sleep on the couch at the server room... http://www.aota.net/ubb/frown.gif