FQuest Alert: Web Logs and Stats

[Terra]

A new Apache server build that was brought online Feb 10th had some subtle problems with logs... During some logfile maintenance that I was performing tonight, I found that there were time gaps in the logs from the 11th to 14th... I believe that I have isolated and fixed this problem within Apache... The old Apache builds have been brought back online until I can further test the new builds... This is the first time that I have ever seen Apache logging subsystem not work correctly...

Your stats will reflect below average results between the 11th and 14th... There is no way to retrieve the gaps because they were never written correctly to the harddrive...

February has been a month of 'behind-the-scenes' upgrades to much of our operation in preparation for the upcoming NOC move, and retrofitting the servers to handle the new environment... Apache was on the list to upgrade, but did not see the problems with the logs until it was too late...

Stats will be offline until tomorrow night, until I can sort this all out... Reverse Resolves (RARP) will also be shutdown for a few nights until I can get the new optimizations in place... I hope to have everything caught back up by tonights run...

Our apologies for any inconvenience that this has caused...

--
Andrew Gillespie
Systems Administrator
FutureQuest.net

[Justin]

I was wondering about the stats - was far below average last couple of days, but I didn't want to ask again... What build of Apache was that, anyway? 1.3.4? Just curious.


------------------
Justin Nelson, SFE Inc.
http://www.vdj.net

[Terra]

I tried to bring the Apache 1.3.4 build up about 1.5 weeks ago, but it had some serious memory leaks in it... I will try once again with a different static/dynamic module combination after the new Linux 2.2.1 server comes online (stress testing it now as I type)... I will create a different thread on the progress of Linux 2.2.1 server as it nears deployment...

The Apache in question, was one that I had applied some high-performance patches too in an effort to remove some unnecessary overhead... Somehow the modifications of the strftime() functionality conflicted with the logging module... There was good odds that the patches would have worked for FQuest, but I believe that our Apache is modified to a point that it did not apply cleanly... Initial testing showed all was well, but when put into *heavy* production use, sporadic results in the logs occurred...

--
Terra
--Went three steps forward, forced two steps back--
FutureQuest