FQuest Status: Slowness

[Terra]

Many of you may have felt a slowness in the past few days during our prime time hours...

Prime Time: 9:00am - 1:00pm EST
and 8:00pm - 12:00am EST

We are working on fine-tuning the server to adjust to the heavy loads that are upon us... 94% of the problem is in the realm of *heavy* CGI scripts that pound on the server...

In an effort to prevent downtime - several brakes have been implemented in the server to stop CGI scripts from overloading and bringing the server down... FQuest has had it's fair share of downtime this month, and we felt it was in the best interest to start being more restrictive on the overall operations...

A custom FQguardian program has been written to instantly shutdown the Apache primaries when they become overloaded... It will then wait for the server load to subside and bring them back online (consider it a self-healing fuse)... This has basically stopped the 30 - 40 minute downtimes (server reboot), and now only causes a 2 min maximum blackout of all domains (except for AOTA)...

Hmmm, 40 minutes or 2 min -- I hope you see now why we made that choice... http://www.aota.net/ubb/wink.gif

We are about 2 weeks from getting the new server online, at which point I will be shifting all SSL customers, and some of the bandwidth/CGI intensive domains to...

Please bear with us during the 2 weeks, as we are working very hard to overcome our current operating status (slowness)... I am finding the right balance between serving pages efficiently while not overloading the server in the process...

A new redesign of the server core is in the works as well, which will greatly reduce load... A glimpse of that will be dedicated Apache daemons that will be specific in duty (e.g. Seperate PHP3 / mod_perl / Java daemons) backend processes - with the *original* TAZ server being placed at the frontend running a very sophisticated SQUID cache...

FQuest is now at the point to start building a backend server farm... We were not sure at what point that would be - now we know... http://www.aota.net/ubb/wink.gif

I had higher hopes for the Pentium-II 400 processor, but it has fallen short of my expectations... http://www.aota.net/ubb/frown.gif We will be replacing it with a P-II 450 (gaining inches now)... We are basically stalled on CPU choices at the moment until the AMD K-7's are released second quarter of 1999... I have also investigated going dual cpu SMP server, but the current Linux kernel is not up to snuff on this quite yet, for which the new Linux 2.2 kernel is supposed to remedy (2 - 3 months away)...

If you will stick with us through these growth spurts, you will see many wonderful things happening (uptime and reliability - more features and capability) as a result of all this...

We take Web Hosting very seriously, and understand that domain owners don't always understand everthing that happens behind the scenes... We made a choice to open up this facet of our business to our clients, and keep everyone informed, so that they may see that we really do care....

I would like to extend my appreciation to everyone that has stuck with us through the good and bad times that we have had...

--
Andrew Gillespie
Systems Administrator
FutureQuest Virtual Hosting
--Another day - Another novel--

[Justin]

I'm curious, as an AMD user. Is the K-7 a socket-7 chip or have they finally gone with the slot 1 & 2 architecture? I like AMD, but a 350 MHz socket 7 chip won't be good enough for my next system, and this is the first I've heard of the K-7. And surely it will be a lot cheaper than a PII of the same speed.

Justin
-- And stop calling me Shirly--

[Terra]

Tis better to read, than for me to explain...

www.tomshardware.com/releases/98q4/981015/index.html (http://www.tomshardware.com/releases/98q4/981015/index.html)

FQuest Future == [Dual|Quad] AMD K-7 [8|9][0|5]0Mhz servers w/2 Gig Direct RDRAM and backed by Digital Alpha's EV6 BUS protocol (whooooo)

SideNote: Intel, Be Afraid - Be Very Afraid...

I'm so tired of playing games with Intel - I want a bigger sandbox...

--Terra
--Feels the need for sp e e d--
FutureQuest.net

[This message has been edited by ccTech (edited 01-19-99).]

[alexandra]

Thanks for the explanation. We have been experiencing a slowdown. It affects mostly the message board, but our posters have been . . .troubled. They think it's us, of course, and some have sent emails saying they tried to get on the message board but it was too slow, and what's going on, and they don't want to be bothered.

So now I can post an announcement saying it's not us, and our web host is adding capacity, but there might be slowness over the next two weeks until things are sorted. Right?

alexandra

[Rich]

Or should that be:

FQuest Future == [redundant cluster]([Dual|Quad] AMD K-7 [8|9][0|5]0Mhz servers w/2 Gig Direct RDRAM and backed by Digital Alpha's EV6 BUS protocol)+

?

http://www.aota.net/ubb/smile.gif

"computers run on smoke--I turned mine on once and a bunch of smoke came out and it never worked after that"



------------------
Rich

"What time is it in _____?"
www.timezoneconverter.com (http://www.timezoneconverter.com)

[Terra]

Redundant Cluster == waste of very expensive hardware...

What I propose to use is a modified Beowulf cluster, that runs at proportional capacity...

3 servers will be pushed to 2/3's of capacity... Then in the event of failure - the remaining 2 systems will split the load of the server that failed...

As we grow to 4 servers - then the margin of capacity increases to 3/4th... 5 servers to 4/5th, up to 6 servers per cluster pack...

This will effectively provide the redundancy that you are looking for - while maximizing our hardware purchasing dollar...

--Terra
--Barking at the moon--
FutureQuest.net

[Terra]

That is correct Alexandra, we are in a state of transition...

During the last 2 days, I have been trying different techniques of optimization - some worked - some didn't... What you felt were the ones that didn't work out so well...

I believe I have a good setup right now, that will hold us till the new server goes online... Within a few days after that, you should see everything return to normal...

--Terra
--Listening to Deb point out that I missed one--
FutureQuest.net

[Justin]

You don't know how bad I want on (or two) of those!!! Man, up to 8M L2 cache!! I though my little 512k was good on this old K6 233. Sometimes I hate this thing, if I'm running multiple apps, which is all day (Visual Basic/Paint Shop/IE4/Outlook/Virtual DJ/Notepad/Cute ftp)

K7 sounds like it's going to blow the PII out of the water! I think I'll stick with AMD. Mine beats my friend's Pentium 233 MMX badly, and he's got a better video card w/3D acceleration and 4M @ 235 MHz DAC, where mine's 2.25 M @ 135MHz DAC. Mine does better Quake 2 performance! I love AMD. Also much faster mp3 compression on the AMD using the same software.

Do you know what they cost and when they'll be available? I guess I should just go to AMD's site...

Justin

[Terra]

Another domain that we proudly host... http://www.aota.net/ubb/wink.gif

www.3dnow.net (http://www.3dnow.net)

Check 'em out and catch all the buzz!

--Terra
--My mind is a blank check--
Futurequest.net

[Tom]

What sorts of CGI scripts can kill your server?

I don't see my 2 domains as causing any trouble at all, yet we fall victim to others?

2nd Q: does the PIII/Xeon (Katmai) come into consideration, or is it too far off?

[Terra]

It's hard to give a definitive list of the types of scripts... Hosting companies, over time, have come to realize that CGI scripts have matured beyond the simple scripts of yesteryear... Now they are extremely complex, cpu / disk / memory intensive... Some of the more complex ones, spawn multiple children to produce the result...

There is a lot of overhead with running CGI, due to the startup and teardown times for each CGI request... When you start stacking them up, they consume enormous amounts of server resources... All this for just serving one browser's web request...

For example, if I dissallowed *all* CGI script execution, I could **easily** serve static pages from 8000 - 10,000 domains on our current server... When you toss in all the CGI overhead, that number drastically reduces to around 300 domains max per server...

Some of the heaviest scripts are:
search engine scripts (internal and external)
chat scripts (Very Heavy - to the point we don't allow them)
forum/message board scripts (dynamic ones)
banner ad scripts (with detailed logging)

Right now, FutureQuest is handling the execution (average) of 25 CGI scripts per second... Thats a lot of overhead...

CGI Perl Script lifetime (for 1 request):
Browser sends request to Apache Server
locate script on hard drive
check internal rules for access
suEXEC security checks
Apache logs the request (access log)
read script from harddrive
spawn child perl process (medium load)
compile program (heavy load)
**log error if any, to error and script logs
execute program (heavy load)
pipe results
script dies and becomes a zombie (medium load)
Apache returns results to browser

The above is only a very brief overview of a CGI scripts life, each item above has much more Kernel level processing to handle the request (pipes / disk reads & writes / network reads & writes / etc... etc...)

Apache without CGI scripts (1 request):
browser --> Apache
locate file on hard drive
check internal rules for access
read file
log to either access or error log
Apache --> browser

---

2Q) I am not impressed by the Xeon's performance or it's price... From all reports that I've read - the AMD K-7 will smoke it due to Digital Alpha's advanced BUS design... The main benefit of the Xeon processors is that the L2 cache runs at core CPU speed, instead of 1/2 speed... AMD K-7's using the EV6 bus will be able to pull this off as well... http://www.aota.net/ubb/smile.gif My money is on the AMD K-7 when it hits the .18micron fab production level, and Digital Alpha's BUS design is already proven technology...

--
Terra
--Juggling the complexities of CGI--
FutureQuest.net

[Terra]

I don't see my 2 domains as causing any trouble at all, yet we fall victim to others?

Unfortunately yes... The only way you can get around it, is by going with a dedicated server which costs around $200.00+ per month...

--
Terra
--Inquiring minds want to know--
FutureQuest.net

[hearts]

Andrew,

with these upgrades and such over the next couple of weeks, will this mean that there would possibly be downtimes as well? like this morning?

Not complaining, just curious and wanting to understand.

Thanks...

hearts

[Benson]

Terra,

http://www.bigbrotherinside.com/

Another reason to go with AMD!

[Kenny]

>There is a lot of overhead with running CGI, due to the startup and teardown times for
each CGI request...

Isn't that's why they have this thing called Mod_Perl http://www.aota.net/ubb/smile.gif

You should seriously really look into mod_perl as an alternative to (perl) CGI, in the meantime HURRAY for mod_php3 http://www.aota.net/ubb/smile.gif

[Terra]

Believe me - mod_perl is very high on my priority list to get back in the server...

The problem with it was it's memory usage where it's memory footprint would add upwards of 8mb per each child... Multiply that by 300 children that can be operational at any given moment and you'll see why I had to drop it's usage...

Getting our C-Block of IP's will help this situation tremendously as I can finally put the power of mod_proxy and mod_rewrite to full usage by reverse proxying all perl requests to a dedicated Apache/mod_perl engine... The same will be done for PHP and Java Servlet engines...

This is truly the beginning of FutureQuest's growth path and all of my engine designs can become reality... Being forced to only utilize only 9 IP's had forced me to do some very ugly contortions to make all this work... Unfortunately, in the last month - those contortions finally began to break consistently and our order form had to be shutdown as the system just couldn't handle any more new accounts...

I still like and believe in the HTTP/1.1 protocol, but it does have it's limitations (answered many places elsewhere in this forum), and FutureQuest has run headfirst into them... Now I can begin to spread out the load by allocating 30 domains (initial) per IP...

Also everyone reading this, please do not ask for a dedicated IP, we are monitoring the situation very closely and will be contacting those domains that have shown necessity (confidential) for a dedicated IP... Right now there are 6...

--
Andrew Gillespie
--Unleashing the power of Linux and Apache--
FutureQuest.net

[Tom]

What's the latest?

We're back to long periods of nothing happening during a simple page load.

Thanks,
Tom

[Dean B]

Yep, I'm noticing it too Tom. We've had two days of normal operational speed, ie this UBB was on a par for speed along with the numerous others I visit every day. Today however we're back to pre-weekend speed http://www.aota.net/ubb/frown.gif
Say it ain't so Joe !

Dean.

[Deb]

The net in general on this side of the world is slow.. there are quite a few routers that have been congested... for reference view http://www.internettrafficreport.com

The FQuest server has been and is doing great... we are anxious to change the gateways and hope to have that going soon but must wait for the DeltaCom hop to be fixed....

We ofcourse, still have our peak times and during those times the server is just plain being pounded on so it's going to be slower then it is during off peak.

But all in all everything is running along great!

Andrew and I decided to go ahead and purchase more RAM for the new server rather then splitting it back up -- that should arrive Monday.

SSL - is a priority along with getting certain sites moved to their own IPs... the main delays right now are with the DeltaCom hop, and simply time. With the move going as shaky as it did we lost a great deal of time and are having to play catch-up.

What we're doing is trying to slow down the requests for tech support in the area of third party scripts etc so that work on the the priorities can be going full speed ahead. Understandably there are a lot of emails asking "is SSL done yet" etc.. and we are concentrating on getting these things done.... I'm sure Andrew will post tonight sometime giving a status update on everything and getting you all up-to-date.

The move was only the first step of the many upgrades...

Hang in there
Deb

P.s. on the credit emails -- Just about everyone has received an email by now receiving the months credit... there are about 20 site owners that have not yet -- they should get theirs today or tomorrow. As an aside -- surprisingly -- those who were scheduled for de-activation for non-payment and have been de-activated are expecting to be re-activated due to FQ's issues this month -- Please, if you did not pay for Jan -- You will not be credited for Jan and it just simply does not count. This should make sense :P

[hearts]

*whew* 300 children.. and I thought 7 was a lot.. http://www.aota.net/ubb/wink.gif

[Terra]

http://www.aota.net/ubb/Forum4/HTML/000077.html

--
Terra
--one day it's slow, next day it's fast: Grrrrr--
FutureQuest