PDA

View Full Version : FQuest Upgrades: Phase2 - Part 2


Terra
03-03-1999, 04:58 PM
CONTINUED from www.aota.net/ubb/Forum4/HTML/000086.html (http://www.aota.net/ubb/Forum4/HTML/000086.html)

We have received the green light from UUnet upstream to begin the Phase 2 upgrades...

Below is a brief itinerary of what Phase 2 entails...

2/16:done: Finish up stress testing on new kernel and server
2/17:done: Recompile all low-level network tools and shared glibc2 libraries
2/18:done: Finish upgrading and converting all DNS Bind entries 4.9.4 to 8.1.2 // finish setup and testing of eth0:2-254 aliasing for new C-Block
2/19:done: Plug up new server 'six.futurequest.net' to UUnet and reconfigure for full C-Block

2/20: ARRrrrggghhhhh, Internic has thrown a curve ball by delaying the registration of 'ns2.futurequest.net' and 'ns3.futurequest.net'... Guess it's our turn to throw our glove in the ring with Internic... I cannot complete the move successfully until these new hosts are registered... This was started 2 weeks ago, and today I find they have put it in their manual queue... gggggrrrrrrrrrrr!
2/22:Internic just approved ns2.futurequest.net, still waiting for ns3.futurequest.net
2/23:ns3.futurequest.net just got approved
2/24:waiting for DNS/RARP reverse delegations to be mapped to new nameservers

2/26:done:Move 1 (low bandwidth sites over) and test
2/26:done:Move 2 (large bandwidth sites) and test
2/27:cleanup nasty mess left behind from SprintLink
2/28:done:Move 15 (low bandwidth sites over) and test
2/28:done:Move 5 (large bandwidth sites) and test
3/01: Watch operations and perform more sanity checking
3/03:almost:In coordination with ITC DeltaCom, we now have makeshift RARP reverse resolves, we are still waiting on ARIN.net to authorize the final changes... For now ITC has agreed to do our 'in-arpa' till ARIN.net comes through...
3/03:done:Update all domains Internic Templates to reflect the new nameserver information... (Thanks Deb!)
3/04:done:Rewrite the new account setup routines / run a batch of 12 and observe
3/??: Start moving the rest over
3/??: Pull 'old' TAZ offline, retrofit with Linux 2.2.1, and rechristen it as (new) 'taz.futurequest.net' for the T-3...

**New Section: Planned (Major) Software Upgrades**
2/17:done:RedHat 5.2 (base only)
2/18:done:glibc2 (fast compile)
2/20:done:Bash 2.02.1
2/20:done:Python 1.5.1
2/21:done:Perl 5.005_02 (Seeing if I can maintain, in production usage, 5.004_04 or not)
2/21:done:recompile all installed Perl 5.004_04 modules to new 5.005_02 binary compatability synchronizing both code trees... (If I missed any, please let me know)
2/22:done:Apache 1.3.4
2/22:done:PHP 3.0.6 (Apache 1.3.4 primary engine pool)
2/25:done:Apache mod_SSL 2.2.3-1.3.4 (My brain hurts!)
3/03:done:PHP 3.0.7 (Apache 1.3.4 primary engine pool)
3/03:done:MySQL 3.22.19a-gamma
3/04:done:recompile Perl DBI::DBD (MySQL) Modules
3/12:done:Gandalf 2.0 (Thawte CERT registrations)
3/??::PHP 3.0.6 (Apache 1.3.4 backend engine pool)
3/??::Apache mod_perl (backend engine pool)
##?/??::QMail 1.03 (possible, if have time)
##why upgrade what isn't broken...
**END Major Software Upgrades**

Most of your visitors will not notice that the move has taken place, except for faster response times... http://www.aota.net/ubb/wink.gif

On 3/?? (If all testing goes A-OK) then for a period of 4 hours, all passwords will be disabled on the old server (for the domains being moved)...

EMail should all flow to the new mail server as well...

If you find that you get an invalid login, then either wait for the DNS propagation to take place (3 hours or less), or use our new IP to login with: 209.192.41.100

Our new C-Block is: 209.192.41.x

There will be a total of 6 Phases that our upgrades will go through, over the next 4 to 6 months, as we are on track to our final destination of the OC-12 NOC...

I will post more information concerning Phase 2 as it becomes available to me...

--
Andrew Gillespie
Systems Administrator
FutureQuest.net

[This message has been edited by ccTech (edited 03-03-99).]

[This message has been edited by ccTech (edited 03-05-99).]

[This message has been edited by ccTech (edited 03-13-99).]

Terra
03-03-1999, 06:35 PM
ALERT: Apache_Logging
Priority: High
Severity: affects the STATS processing subsystem
WorkAround: sort the daily log at the end of the day (not acceptable)
Fix: Kill BUFFERED_LOGGING, and recompile the Apache daemons...
Status: FIXED (03/03|9:33p EST)

I have just discovered a major glitch in the SIX Apache logging sub-system...

In an effort to reduce overhead, I initiated 'BUFFERED_LOGGING' with the Apache daemon... While it is working as desired (reducing overhead) there is a nasty side effect, it is not writing the log entrys in pure chronological order, it seems to be grouping the 'KEEPALIVE' connections and writing them out in chunks upsetting the Date/Time order...

As the logs are still being written to file, the order of writes is critical (Atomic Writes)... Later when Logs get piped to MySQL database, order will be moot point...

All domains on Server SIX will experience some downtime today as I make the repairs to the log files... Expect a 30 minute outage...

Oh well, might as well upgrade the PHP3 and MySQL in the process... http://www.aota.net/ubb/wink.gif
PHP3: 3.0.7
MySQL: 3.22.19a

--
Terra
--Okay, who hid my can of RAID???--
FutureQuest

[This message has been edited by ccTech (edited 03-03-99).]

hearts
03-03-1999, 06:53 PM
DEB DID IT.. she told me in email.. http://www.aota.net/ubb/wink.gif

ooops.. meant to ask.. an estimated time of downtime is?

DUH.. i shoulda finished reading.. hehehe. 30 minutes.. and did ya do it already? Or are ya gonna wait til later.. LATE?

[This message has been edited by hearts (edited 03-03-99).]

Terra
03-03-1999, 07:12 PM
Ummm, when you can't reach your site is when the repairs are taking place... http://www.aota.net/ubb/wink.gif

Everyone who volunteered for the Phase2 Beta project, please bear in mind that things like this do happen, and your domain can go offline at any time for any reason... I do my best to notify, but at times it's not possible... This is what Beta is all about... http://www.aota.net/ubb/biggrin.gif

--Terra

Terra
03-03-1999, 07:17 PM
One more note, as the previous thread grew to over 60 messages, I would like to ask everyone to maintain a *high* Signal/Noise ratio - as I use these forums to track the progress and any problems that pop-up...

If anyone has a problem report, please post as brief and focused as possible... I want to avoid this thread turning into another conversation...

Thank you for understanding...

Terra
03-03-1999, 10:36 PM
The above alert has been fixed... PHP and MySQL have been upgraded as well... The logging has returned to desired behavior... I still need to resort the log files so Stats run may not happen tonight...

jenili
03-04-1999, 01:40 AM
Two very small things that aren't working the way I'd think they would....

Access logs are rotating nightly, but error logs aren't rotating at all. Is this intentional?

The icons for fancy directory indexing aren't there -- my error log indicates they're expected in /usr/apache134/icons/

Side note: I'm also getting an interesting error message, BUT I've been getting it since 2/6 and it doesn't interfere with execution. Could just be related to my Unix permissions (700) on my CGI stuff.
"Permission denied: mod_mime_magic: can't read `/path/to/script.cgi`

FQ-Six rocks!
jeni

Terra
03-04-1999, 02:48 AM
1) This is the updated Stage2 logging sub-system, only the access logs are rotated, not the error logs... I haven't decided yet how I'm going to handle the error logs, probably rotate the entire file at the end of the month and store in 'last_month'...

2) I will fix the icon problem... Somehow I lost them... http://www.aota.net/ubb/frown.gif

3) I assume that 'mod_mime_magic' is probing while still uid/gid 'apache/apache', before the SUEXEC kicks in and switches to your effective uid/gid... It is safe to ignore...

jenili
03-04-1999, 03:51 AM
1) thanks.
2) thanks. You can probably copy the icons from aota.net, where fancy indexing works fine.
3) makes sense to me.

Got a new thing, FWIW. Counters don't seem to update, at least for jenili.net or bestlink.org. I always assumed our dat files were kept in xdomain/count-data, but that directory is empty for both accounts.
jeni

[This message has been edited by jenili (edited 03-04-99).]

Deb
03-04-1999, 04:27 AM
Both of the counters are working.. there is a reason why you 'couldn't tell'...

Check out http://www.aota.net/ubb/Forum11/HTML/000012.html that'll explain it http://www.aota.net/ubb/smile.gif

Deb

hearts
03-05-1999, 11:11 AM
Hey Terra, Is it okay, to post that I went to CNC last night, and all seemed to work ok? I had no problem with stats or anything. http://www.aota.net/ubb/smile.gif

Terra
03-14-1999, 01:02 AM
Current Status: Phase2 is currently on Freeze/Thaw condition...

Freeze == very low priority
Thaw == will upgrade domains as time permits

Push came to shove on the apacheSSL (Secure Server) that a 100% refocus of dedication to finishing this...

We apologize for any inconvenience on this, but the original reason for the upgrades was to enable the SSL capability and to gain the IP's to make this necessary...

Currently both servers are running quite well, and everything seems to have stabilized... We have accomplished our primary objective...

--
Terra
--SSL == Severe Stress & Labor--
FutureQuest