FQuest Upgrades: Phase 2

[Terra]

We have received the green light from UUnet upstream to begin the Phase 2 upgrades...

Below is a brief itinerary of what Phase 2 entails...

2/16:done: Finish up stress testing on new kernel and server
2/17:done: Recompile all low-level network tools and shared glibc2 libraries
2/18:done: Finish upgrading and converting all DNS Bind entries 4.9.4 to 8.1.2 // finish setup and testing of eth0:2-254 aliasing for new C-Block
2/19:done: Plug up new server 'six.futurequest.net' to UUnet and reconfigure for full C-Block

2/20: ARRrrrggghhhhh, Internic has thrown a curve ball by delaying the registration of 'ns2.futurequest.net' and 'ns3.futurequest.net'... Guess it's our turn to throw our glove in the ring with Internic... I cannot complete the move successfully until these new hosts are registered... This was started 2 weeks ago, and today I find they have put it in their manual queue... gggggrrrrrrrrrrr!
2/22:Internic just approved ns2.futurequest.net, still waiting for ns3.futurequest.net
2/23:ns3.futurequest.net just got approved
2/24:waiting for DNS/RARP reverse delegations to be mapped to new nameservers

2/26:done:Move 1 (low bandwidth sites over) and test
2/26:done:Move 2 (large bandwidth sites) and test
2/27:cleanup nasty mess left behind from SprintLink
2/28:done:Move 15 (low bandwidth sites over) and test
2/28:done:Move 5 (large bandwidth sites) and test
3/01: Watch operations and perform more sanity checking
3/03:almost:In coordination with ITC DeltaCom, we now have makeshift RARP reverse resolves, we are still waiting on ARIN.net to authorize the final changes... For now ITC has agreed to do our 'in-arpa' till ARIN.net comes through...
3/??: Start moving the rest over
3/??: Pull 'old' TAZ offline, retrofit with Linux 2.2.1, and rechristen it as (new) 'taz.futurequest.net' for the T-3...

**New Section: Planned (Major) Software Upgrades**
2/17:done:RedHat 5.2 (base only)
2/18:done:glibc2 (fast compile)
2/20:done:Bash 2.02.1
2/20:done:Python 1.5.1
2/21:done:Perl 5.005_02 (Seeing if I can maintain, in production usage, 5.004_04 or not)
2/21:done:recompile all installed Perl 5.004_04 modules to new 5.005_02 binary compatability synchronizing both code trees... (If I missed any, please let me know)
2/22:done:Apache 1.3.4
2/22:done:PHP 3.0.6 (Apache 1.3.4 primary engine pool)
2/25:done:Apache mod_SSL 2.2.3-1.3.4 (My brain hurts!)
3/??::Gandalf 2.0 (Thawte CERT registrations)
3/??::PHP 3.0.6 (Apache 1.3.4 backend engine pool)
3/??::Apache mod_perl (backend engine pool)
##?/??::QMail 1.03 (possible, if have time)
##why upgrade what isn't broken...
**END Major Software Upgrades**

Most of your visitors will not notice that the move has taken place, except for faster response times... http://www.aota.net/ubb/wink.gif

On 3/?? (If all testing goes A-OK) then for a period of 4 hours, all passwords will be disabled on the old server (for the domains being moved)...

EMail should all flow to the new mail server as well...

If you find that you get an invalid login, then either wait for the DNS propagation to take place (3 hours or less), or use our new IP to login with: 209.192.41.100

Our new C-Block is: 209.192.41.x

There will be a total of 6 Phases that our upgrades will go through, over the next 4 to 6 months, as we are on track to our final destination of the OC-12 NOC...

I will post more information concerning Phase 2 as it becomes available to me...

--
Andrew Gillespie
Systems Administrator
FutureQuest.net

[This message has been edited by ccTech (edited 02-19-99).]

[This message has been edited by ccTech (edited 02-19-99).]

[This message has been edited by ccTech (edited 02-20-99).]

[This message has been edited by ccTech (edited 02-21-99).]

[This message has been edited by ccTech (edited 02-21-99).]

[This message has been edited by ccTech (edited 02-21-99).]

[This message has been edited by ccTech (edited 02-22-99).]

[This message has been edited by ccTech (edited 02-23-99).]

[This message has been edited by ccTech (edited 02-25-99).]

[This message has been edited by ccTech (edited 02-26-99).]

[This message has been edited by ccTech (edited 03-01-99).]

[This message has been edited by ccTech (edited 03-02-99).]

[Justin]

Is this a voluntary thing?? If you need volunteers for anything, as long as the chance of long downtimes are relatively slim, I'm in http://www.aota.net/ubb/smile.gif I know that there's always a chance of downtime, so I won't be complaining too much if it happens. Speedy sounds good to me http://www.aota.net/ubb/biggrin.gif


------------------
Justin Nelson, SFE Inc.
http://www.vdj.net

[Terra]

Most of your visitors will not notice that the move has taken place, except for faster response times...

If you want to volunteer your domain to be one of the first golden few to scoot over - that would be fine with me...

--
Terra
--So much to do, So little time--
FutureQuest

[hearts]

can i play too? *giggle*

hey.. count me in..

[Justin]

Just to confirm, I'm in. I just got a complaint from a guy with a cable modem getting only 3k/sec downloading. Only the third complaint in 2 months, though, so I'm not worried - he's in Canada, too, if that makes any difference. The other two were in Germany and California.

Justin
-- Wishing I could get 3k/sec on anything --

[Del]

Mess with downinit.com all you want :-) And djweb.net/.com and splitangels.org if you need any more volunteers.

Del

[Armand]

What can I say I'm like a big kid that doesn't want to left out. Count arcanumcafe.com in if you want (though I'm probably on the low bandwidth category).

------------------
Armand (aka Joe)
joe@arcanumcafe.com
www.arcanumcafe.com (http://www.arcanumcafe.com)

[Paul]

Sign me up!

[Jacob Stetser]

Heh. I'm probably too late :P

But if you need another, icongarden.com is okiedokie.

/me recalls the early days of AOL on the net... 'me too'

[Terra]

Thanks everyone for pitching in the best way you could... I will handle your domains with kid gloves...

Well, I'm off to Gainsville to start preparing initial hookups...

Enjoy the day...

--
Terra
--Looking forward, learning from the past--
FutureQuest

[alexandra]

I'm a whole day late, but I'm more than happy to be moved sooner rather than later. During the day, the site (especially the UBB) is s-l-o-w. I've gotten quite a few (mostly gentle) complaints, but the bottom line is I've lost several of my more interesting posters, so I'm anxious to know when I can tell them it's safe to come back. Unfortunately, I put up an announcement the last time there was a problem saying it had been solved (when you changed servers). So many people were reporting the speed had improved so -- I didn't realize there were different stages.

Query -- you will tell us, as accurately as possible, exactly when we're going to be moved, won't you?

Thanks.

alexandra

[Terra]

Due to some unforseen problems with the new Linux Kernel, and all the programs that have to be recompiled to take advantage of it's new features I have had to revamp the above itinerary...

I will be making another attempt tomorrow to get full connectivity and hopefully the network issues and glibc problems have been solved... I feel the new Kernel is worth all the effort as it opens up many new capabilities in system design and load handling...

As far as fast/slow issues... We have been getting mixed reports after Phase 1... It's been pretty much 50/50 with many saying the response picked up greatly, to others saying their sites slowed down... I think much of it is that the Internet as a whole is getting slower, as more and more people are coming online and greatly depends on what routers you have to get through to make it to your destination... We do have 1 very slow hop during primetime on SprintLink, which is the reason we are dropping it...

I have also always been upfront and honest about any issues that we are having and the steps being taken to resolve them... I will change the '??' in the dates above as they get completed... The move should be mostly transparent for almost everyone... I am **not** rushing this move, as transparency is paramount and it takes time and careful planning to pull it off...

Notifying everyone that they are being moved is next to impossible, hence the transparency... If you should try to login and get access denied, read the first post and you will find the directions to login to 'six.futurequest.net' / 209.192.41.100...

--
Terra
--Walking under a microscope--
FutureQuest.net
Time will tell how our new connection will fair, as we are also looking to get connected with a T-3 to spread out the load more evenly which should hold us till the OC-12 NOC...

[Terra]

bump: 2/20

[Terra]

bump:2/21:new_section

[Terra]

WOW! I have just spent the last 11 hours, tweaking and fine-tuning the new FQuest Perl 5.005_02 build and it is FAST!... I have also gotten it compiled with the Threads library, for those that know how to use it (Use threading support at your own risk, no support will be offered)... http://www.aota.net/ubb/wink.gif

I'm showing improvements overall with over 600% in execution speed... Best thing is, all 183 test scripts, running 6498 sub-tests all came out successful... I'm feeling lucky today... http://www.aota.net/ubb/smile.gif

I used pgcc with heavy Pentium-II optimizations, and removing the extra (legacy) fluff... This will be the base for CGI scripts, and also design template for the Apache mod_perl engines... Once and for all I have finally gotten rid of my NDBM database problems... This is what messed up the new Apache 1.3.4 builds and forced delays in their deployment...

Ok, I'm rambling now -- this was just a high point for my day getting this project completed as it was a pivotal point...

--
Terra
sysAdmin
FutureQuest

PS: On the new six.FutureQuest.net server
/usr/bin/perl54 == 5.004_04 (will use correct module libraries)
/usr/bin/perl == 5.005_02
/usr/local/bin/perl == 5.005_02

[This message has been edited by ccTech (edited 02-21-99).]

[Dean B]

Terra,
You suggest we amend our perl paths in our cgi's to tke advantage of the speed increase ?

Dean.

[Terra]

I am still working on the Perl system, and details are still yet to be determined...

I will try to maintain the existing 5.004 perl, and run a parallel 5.005 release tree... (Only on the new *six* server)

I use #!/usr/local/bin/perl, in almost all my scripts, so I will automatically switch to the new tree - but if I get all my libraries recompiled in time, I may go ahead and pull the 5.004 to remove any possible conflicts... 5.004 libraries (XS modules) are not binary compatible with 5.005...

---How can I tell which server I'm on?
When you login via Telnet, you will see the new prompt:
[username@FQ-Six:/abc/def/ ]$

I am held up on HOST registration's by Internic at the moment, and finalizing on my Upstream to redelegate inverse queries to my name server... I took advantage of this delay and decided to do some major core upgrades, as added in the new section above... 5.005_02 was planned, but not this soon as the FQuest core has been pretty much rock-solid - and I don't like mucking/fixing things that aren't broken... It causes *ton's* of 'support@' emails, which answering them takes precious time away from system coding/development... http://www.aota.net/ubb/wink.gif

I am still ironing out some details on the new server, and writing all my tools to control multi IP'd daemons... Our administrative complexity of operation has just increased 5-fold, but overall I think that there will be vast improvements... This is our next level in growth, and I must say that the initial design of FQuest is going to carry over quite nicely as I have always designed with forward mobility in mind...

--
Terra
--Locked away in the coding closet--
FutureQuest.net

[Dean B]

My telnet prompt is still the same -
[dbarry@FQ-Taz:~ ]$

Dean.

[Terra]

The transfers got delayed, as described above...

six.FutureQuest.net is a closed system right now, as I am still developing and putting the finishing touches on it...

After Internic comes through, and reverse delegations are in place, the moves will begin to commence...

The @FQ-[Six|Taz] was just meant to show how you can tell which server you are on...

You can also do a $hostname, at the command line...

--
Terra
--Where did you want to be today?--
FutureQuest

[hearts]

is that server six.futurequest.net our server? I am confused? how do we know?

and welcome back from where ever you have been!

[Terra]

bump:2/22:updates

[Terra]

bump:2/23:updates (nameservers)

[Terra]

Woohoo, finally got this **monster** compiled... It is probably the most difficult Apache Daemon I've every had to assemble... It is fully featured and stocked full of goodies... http://www.aota.net/ubb/wink.gif

This puppy was rough! Over 18 straight hours went into it's assembly to get it just the way I wanted it...

Next item up for bids, is a companion program called Gandalf 2.0 that helps streamline Thawte CERT registration and processing...

-----
Apache Server Status for secure.FutureQuest.net
Server Version: Apache/1.3.4 (Unix) mod_macro/1.1.1 PHP/3.0.6 mod_ssl/2.2.3 SSLeay/0.9.0b AuthMySQL/2.20-FQuest/1.11
Server Built: Feb 25 1999 05:18:26
-----

You can bounce off it at:
https://secure.FutureQuest.net/
or
https://209.192.41.6/ (*Ignore the Security Alert, it's due to using the IP addy*)

--
Terra
--Special Thanks to my wife, for not throwing things at me everytime I said, "Not now Deb, I'm busy"--
FutureQuest Virtual Hosting

[This message has been edited by ccTech (edited 02-25-99).]

[Benson]

Fine business Andrew!

[ddunlap]

Outstanding, Andrew (patting on back)!

[Terra]

Due to some stability issues yesterday, I went ahead and bit the bullet and moved 2 heavily CGI'd domains, and 1 small domain...

SIX is far enough along to begin testing with live domains...

I'm holding out at 3 right now, making sure that all the new CGI/PHP3/etc configurations are working properly... This particular domain will pretty much shake the bugs out of the CGI sub-system...

There is still much 'sysAdmin' related tools'n'stuff that I need to write for handling the new 3rd Generation FQuest engines... Overall the new engines are looking awesome and will become the core structure for easy migration to clustered servers...

--
Terra
--I have got a plan--
FutureQuest.net

[This message has been edited by ccTech (edited 02-26-99).]

[Terra]

I would like a volunteer, to make a mid-afternoon move... Your domain needs to be pushing (bandwidth) around 250Mb/day...

The only caveot is there will be approximately 2 hours of your domain being blacked out while DNS propagation takes place... It would be scheduled to move ASAP...

Thanks!

--
Terra
--Get those doggies rollin' Rawhide!--
FutureQuest

[hearts]

well.. i don't know a things about bandwidth, and don't even think i use much of it, but i will vounteer, just let me know if this means I need to hold off on working on my site? Will I still be able to upload and all during this propaagtion? I had another question, but i forgot? Anwyays.. move me. http://www.aota.net/ubb/wink.gif

[Terra]

Hearts, I'll be moving your domain late at night in the batch of 5 small domains...

Thanks anyways...

--Terra

[Justin]

After today, might as well do another couple hours down http://www.aota.net/ubb/smile.gif I think I meet the bandwidth category, if plans aren't ruined by today's events. Besides, I liked the trace route on futurequest.org http://www.aota.net/ubb/biggrin.gif Very impressive.

Thumbs up here.


------------------
Justin Nelson, SFE Inc.
http://www.vdj.net

[flowersource]

It would be great if mine could be moved BEFORE I go to work Monday.

Thanks,
Mike

[ubu]

You can move my domain any time you want. This message might be a little late, I haveb't been on the board in a while because of being sick with the flu.

Jamie

[hearts]

Andrew, i have just one *little* word

vrooooooooooooom http://www.aota.net/ubb/biggrin.gif

vroooooooom .. wooooooooooooo hooooooooo

(ya can never limit a woman on her words http://www.aota.net/ubb/wink.gif )




------------------
~~~ hearts ~~~

Heartistic Designs www.heartisticdesigns.com (http://www.heartisticdesigns.com)

Hearts www.heartsangels.com/hearts (http://www.heartsangels.com/hearts)

Heartistic Wishes www.heartsweb.com/heartisticwishes (http://www.heartsweb.com/heartisticwishes)
















[This message has been edited by hearts (edited 02-28-99).]

[Terra]

Glad you like the new server Hearts, I've been tweaking it all morning... Your daemon pool 'x21', needed a slight overhaul in the PHP3 department but I think we've got it all sorted out now...

--
Terra
--Speed Buggy rules--
FutureQuest

[Justin]

I'm not complaining, that's for sure! Awesome! And the PHP is working great now, too.

Ya won't find me complaining http://www.aota.net/ubb/biggrin.gif

Justin
-- Life in the fast lane --

[hearts]

hey Andrew... i sent ya email on what i have found. I don't run PHP so I don't understand what ya mean. CGI ain't running and I cannot log in. *that is what the email is about*

Will ya post when ya got it figured out?

BTW.. thank you. I am impressed. http://www.aota.net/ubb/smile.gif

Ya'll are gonna love this.. woooo hooooo


[This message has been edited by hearts (edited 02-28-99).]

[hearts]

hey Andrew, i was able to log in via ftp.. just changed something.. http://www.aota.net/ubb/smile.gif I put a noitce up on my site, about being upgraded and that this should be temporary. I will assume you are burried in this situation, and feel confident you are breathing. I don't know what time the XFILES come on.. cuz I don't watch TV.. but I hope ya get it done before then.. grin. I would hate for ya to miss an episode. *EG* --- HUGS

[This message has been edited by hearts (edited 02-28-99).]

[hearts]

also.. andrew, I am unable to use Mail Mgr in CNC. I was trying to set up a new POP and i can't. (not complaining) just thought whatever i discover would either help or annoy ya! http://www.aota.net/ubb/wink.gif

[Justin]

Just to add, it's not a Perl problem - I have a C cgi that's dead too - premature end of script...

Oh, and it did work before - and it wasn't uploaded, but compiled on the server, so it's not an ascii/binary thing.


------------------
Justin Nelson, SFE Software
www.vdj.net (http://www.vdj.net)
The individual formerly known as SFE Inc.

[This message has been edited by Justin (edited 02-28-99).]

[Terra]

The CGI issues have been fixed... It was a problem with the core operations...

--Terra

[hearts]

ANDREW.. I love ya man. http://www.aota.net/ubb/biggrin.gif

we got our cgi back! Woooooo hooooo. Sorry. just a bit happy about this move. Can't ya tell? http://www.aota.net/ubb/smile.gif

[Jacob Stetser]

I wanna go to Six tooooooo.. http://www.aota.net/ubb/smile.gif

Whew.. busy weekend in the real world. Finally got back to the net.. http://www.aota.net/ubb/smile.gif

Jake

[Terra]

Jacob, ubu, Mike, your upgrades are starting at 10:50pm... Expect a temporary blackout for 2 hours, for DNS to propagate fully... Each domain takes me about 30 - 45 minutes to complete... Still doing it the old fashioned way and building the 'How-To' for the automatic bulk upgrades...

--Terra

[flowersource]

Which Mike? http://www.aota.net/ubb/biggrin.gif

mikee

edit:
Just tried to go to my site "This server is currently being upgraded. Please try again within the hour. Thank you"

I think I know which Mike http://www.aota.net/ubb/biggrin.gif


[This message has been edited by flowersource (edited 03-01-99).]

[Terra]

bump:3/01:updates

[Terra]

I managed to upgrade 5 domains tonight... 2 of them *extremely* heavy...

Mike, yours was one of them... http://www.aota.net/ubb/wink.gif

--
Terra
--and this concludes our broadcast schedule--
FutureQuest

[hearts]

good morning andrew.. I thought I would tell ya that I cannot telnet in. My account has expired. (but I could yesterday)--just reportin' in cap'n.. http://www.aota.net/ubb/biggrin.gif

[Jacob Stetser]

http://www.aota.net/ubb/smile.gif

[Terra]

The domains that were upgraded had their logins 'Expired' on TAZ... You should be able to login to server SIX without a hitch... (or so I hope) http://www.aota.net/ubb/wink.gif

I have also made an addition to the Login Screen today, stating which server your are connecting to...

If you are connecting to Server: SIX, and still having problems logging in, let me know ASAP via email...

--Terra

[Dean B]

Terra,
When are the rest of us likely to be moved ? Apologies if that's already been asked.

Dean.
--- No longer lost in space, back on planet earth ---

[Terra]

My target completion date is 2 weeks... This is a massive upgrade (largest we've done yet) and takes time to do it *properly*... Right now we are shaking the bugs out of the new SIX server (3rd Generation FQuest engines), and I am seeing light at the end of that tunnel...

On the surface TAZ and SIX appear alike, but under the hood, they are radically different...

What is going to bite very hard, is Updating over (insert very large number here) domain templates to reflect the new nameservers with Internic... I will say that Internic has approved every domain modification within 4 hours of my submission, but that's because I use the CRYPT-PW guardian scheme with them so my changes don't get held up...

All in all, it's been relatively smooth - few glitches here and there, and whatever gremlins did popup I have been able to seek'n'destroy the problem rather quickly... (insert 2 snaps and a twist here)

Site owners with MySQL accounts also require special handling and will probably be moved last...

--Terra

[This message has been edited by ccTech (edited 03-01-99).]

[Jacob Stetser]

Hearts, Terra:

I discovered a solution to the problem of not being able to telnet in..

Telnet to 209.192.41.21.

If you're using SSH, the client will recognize this as a separate server and get a new key for it. If you're using a regular telnet client, it will make sure you go to the correct server.

Apparently, there are some pockets of cached DNS entries out there that we're hitting and it's causing some of us to be redirected to taz, even though we don't wanna.

Jake

[jenili]

Terra, dunno if you still need low-bandwidth sites for this. Mine is mostly unannounced and about as low-bandwidth as they come, and downtime will not hurt me in the slightest. If you do it, let me know when and I'll check everything out on it.

I have some CGI that refers to stuff outside of Web space, might be an interesting thing to check. Everything else is plain vanilla (but I don't mind writing some weird CGIs to test specific things if you want).
jeni
-- slept a lot this weekend and ready to play --

[Terra]

Thanks Jenili, sounds like a winner to me... I will put you in the batch that move tonight... Just look for the 'FQ-Six' at the SSH/Telnet prompt... http://www.aota.net/ubb/smile.gif

--Terra

[Tom]

If you're still takin' requests, my two puny domains (avionflight & perrymanagement) would like to make the jump.

Tom

[Jacob Stetser]

Well, if hearts is speaking up, I get the same problem trying to log in from work. If I go to another telnet account of mine and try to login UNIX-UNIX with SSH or regular telnet, I'm fine. But six won't accept my SSH telnet client here http://www.aota.net/ubb/frown.gif

Jake

[hearts]

hey Jake, don't frown, we are the first to get over to six, and so it is our duty, to tell Andrew where we find errors. Like Deb said, we are paving the way for others to move flawlessly. Be happy.. you are off sprintlink! http://www.aota.net/ubb/smile.gif http://www.aota.net/ubb/biggrin.gif http://www.aota.net/ubb/wink.gif

[Justin]

A couple of us were wondering why aota.net wasn't one of the first to move? Seems it would be a good stress test, considering the UBB. I know it's a great test for speed when navigating the forums, too.

Just curious...


------------------
Justin Nelson, SFE Software
www.vdj.net (http://www.vdj.net)

[Deb]

We've put the site owners as a priority and allowed those who really wanted to move to go first...

We also could not risk putting our primary communications point (outside of email) if there were problems... which is why we left the move up as a voluntary thing for those who were willing to go for it.

Additionally -- aota.net is not setup like the rest (it's on it's own apache and does not even have php yet) we'll have to do some 'work' on it when we move it over to the new server as it requires more upgrades then any other domain on the server.

It's a number of reasons really ....

Deb

[jenili]

All checks out, including the CGIs, and performance is great. Sssssweet! Thanks! -- and let me know if I can help test anything.

jeni

[Terra]

http://www.aota.net/ubb/biggrin.gif