FQuest Notice: Possible dropping of Linux SMP support

[Terra]

As everyone knows on the SIX/NINE servers, I've been waging a war with SMP for quite awhile but have now conceeded that the war has been lost...

Though it may be true that we are exceeding our uptime guarantee of 99.5%, I just simply want the (frustrating) random lockups to stop...[nbsp][nbsp]At this point, I have done everything possible to fix the SMP problems to where the final fix is to remove it completely...

Our servers are CGI screamers due to the SMP design, but time and time again people have voiced their opinions (publicly/privately) that 'stability' is more important than 'speed' or 'power'...

<<Everything below is tentative, depending on the evaluation!>>
Over the course of the next 4 weeks, I will be evaluating what it is going to take for making the conversion to single CPU servers, instead of the Dual PIII-600's we use now...

There are 3 solutions to this dilema:

Solution 1)
-----------
Stay with the SMP core and hope that the Linux kernel team pull off a SMP stability miracle for 2.4.x (January/February)
Use ResierFS (Fast reboots)
*Have to drop the RAID-1 due to buffer incompatabilites
Follow the below TAE steps
Follow the below SMR steps

Solution 2)
The proposition dictates the solution of this problem is with 'quantity' while sacrificing 'power'...[nbsp][nbsp]I had avoided the 'quantity' solution as it creates an administrative nightmare keeping multiple servers up to date which the 2000 focus on automation was to solve...

Proposed Order:
SEVEN (still unloaded, using DAS::MySQL, light CGI)
SIX[nbsp][nbsp](85% full, migrate to DAS::MySQL, medium CGI)
NINE (95% full, migrate to DAS::MySQL, heavy CGI)

Of all the servers, NINE is going to be the most difficult as it get's intermittantly spiked with extremely heavy CGI...

A new server will be built, and the heaviest domains from SIX/NINE will be moved to the new server for load balancing and reducing the averages...[nbsp][nbsp]The bad side to this is each domain to be moved takes up to an hour for: stop,teardown,pack,deconfigure,transfer,unpack,clash resolution,reconfigure,start...[nbsp][nbsp]The good side is I will be moving entire xIP blocks so there is no DNS propagation...

-=-=-=-=-=-=-=-=-
Things that will affect everyone (TAE):
1) All MySQL users will need to migrate to the new DAS::MySQL server
2) Mail settings will need to change (POP/SMTP)

Steps needed to make this reality (SMR):
1) Launch the SAS::Real_Audio server
2) Launch the new QMail SmartHubs
3) Place all CGI under a microscope and reevaluate the allocated resources
4) Launch the DAS::STATS server

-=-=-=-=-=-
Wrap up

Solution #1 offers the least amount of resistance, but my major concern is that I do not want to drop my RAID setup to roll out the ReiserFS (journalling)
Pros:
Fast reboots using journalling filesystem (reboots <4min, instead of the 18min we have now)
Ultra fast CGI
Minimal amount of work to make this happen
Maintain server capacity easing administration

Cons:
Uptime won't necesarily improve, but the recovery will
Still open to random lockups due to the SMP kernel
Loss of RAID configuration (will be fixed in Linux 2.4.x)
Downtime to convert the RAID to ReiserFS

Solution #2 is a nightmare from my standpoint, but the primary advantage is solid stability...
Pros:
Best uptime results
Stability by removing the SMP glitches
Maintain the ext2/RAID setups
Minimal downtime due to the xIP transfers to new server

Cons:
Slower CGI
Massive amount of work on my part
Increased administration
Increased complexity
The absolute last thing that I want to spend my next month doing

In conclusion:
What is before me is nothing short of a complex task fraught with peril...[nbsp][nbsp]I have heard promises that Linux 2.4.x is going to fix the brokeness of 2.2.x (especially the SMP)...[nbsp][nbsp]The dilema is do I wait to see if this Holy Grail (2.4.x) is to become true, or bow to the pressures of the site owners and turn our servers upside down to fix it...[nbsp][nbsp]I have made tremendous progress on stability, but opinions are voiced that >99.8% uptime is still not good enough...[nbsp][nbsp]The above 2 solutions would bring us closer to the 99.9999% uptimes that are sought/demanded in this industry...[nbsp][nbsp]

This is the brick wall that I am now pounding my head against... :(

--
Terra
--Calgon, please take me away--
FutureQuest

PS: Solution #3, is leave it alone - embrace the improved stability we have, and wait to see if Linux 2.4.x will turn this around...

[elite]

I say go for three take a breather and get some R&amp;R while it does suck to have a server lock up every once in a while the timing between these is getting very big. I dont mind an occasional burp in the means of progress.

The facts to me sound like if you had to convert all this it would for starters give a lot more downtime, and second waste time that could otherwise be put into other tasks..

Anyhow I just think you all are doing a great job, and can live with an occasional hiccup in the name of progess, so keep progessing forward and dont take three steps back to start over :)

[sheila]

What elite said.

[elite]

Ok this is propably a stupid thing to ask, but I heard before that fast ethernet cards at one time had a conflict with this? Does that have anything to do with it or am I just being a idiot..

Either way I think we all should start a collection for a week at the spa for terra :)

[Dan Kaplan]

I haven't been here long enough to know the frequency and seriousness of the burps/lockups, but I agree with elite and sheila.[nbsp][nbsp]It seems that your promised uptime (which you apparently exceed) is already higher than most companies attempt to meet.

Is it just me or are Solutions #1 and #3 effectively the same thing?

Do you have percentage numbers on how much faster CGI's run under the Dual Pentium setup?[nbsp][nbsp]That might be helpful for anyone trying to weigh the options.

I third the R&amp;R sentiment.[nbsp][nbsp]I don't see the need for you all to make changes that will cause a large increase in workload and at the same time slow the system down while offering &quot;marginal&quot; stability increases.

Dan

[Terra]

The last kernel upgrade (2.2.13) finally fixed the ethernet transceiver lockup problems that we got nailed with once...

I am particularly frustrated at this point, because the kernel has traps around just about every OOPS and Panic I could find, yet even on todays NINE lockup, I got -zilch- for results...[nbsp][nbsp]The traps will capture a kernel panic and dump core to a swap drive for Crash Analysis...[nbsp][nbsp]The tools to do this were provided by Silicon Graphics...[nbsp][nbsp]I did not see todays lockup as a bad thing until I found that no kernel core dumps were produced...[nbsp][nbsp]Frustrating to say the least!

With no dumps, this leads me to believe that there is a hardware problem *yet* all of the hardware has been completely replaced with new...[nbsp][nbsp]I have now been through 3 generations of hardware, and they have all experienced the random lockups...

The problem is two-fold:
1) Negative image that damages sales due to stability issues
2) Just plain not knowing exactly why this is happening

My belief, as the traps didn't catch it, is that this problem is so obscure and so deep in the plumbing of the kernel that I may never find it nor know why it is happening...[nbsp][nbsp]I, for one, am not content with unsolved mysteries but there must be a time when to conceed to the fact that it will never be known...

I do know this much, the problem is associated with SMP operations but as shown in my original post the work to move away from that architecture is very involved and complex to accomplish...

This is one time that basing a core around leading/cutting edge technology has backfired and thusly continues to haunt me...

Now I understand why you find many hosts out there running legacy/old stuff...[nbsp][nbsp]Quite simply, it's what _works_ for them...[nbsp][nbsp]I now simply have to find what _works_ consistently for us...

--
Terra
--You just can't win them all--
FutureQuest

[Terra]

#1 and #3 differ in the aspect of using ReiserFS and losing RAID support...[nbsp][nbsp]#1 offers fast server reboots as the lenghtly fsck (like ScanDisk) is bypassed...[nbsp][nbsp]fsck'ing 3 18Gig drives is a finger tapping process...

Reboot time:
#1) 4 min or less
#3) 18 - 30 min

CGI excution speeds: (primetime averages)
TAZ) 70/sec w/bursts to 140/sec
SIX) 150/sec w/bursts to 230/sec
NINE) 190/sec w/bursts to 300/sec

As you can see, SIX and NINE were built to be CGI screamers, which I could only accomplish with Linux 2.2.x and SMP...[nbsp][nbsp]Once I turn SMP off, those performance values will drop close to TAZ's level (+20)...[nbsp][nbsp]This is not even counting the PHP dynamic page creations, only the Perl CGI forking executions...

But once again, the question remains: stability or performance

I can't have them both... :(

--
Terra
--I'm trying so hard not to shake my Etch-a-Sketch--
FutureQuest
[This message has been edited by ccTech (edited 12-22-99@9:53 pm)]

[elite]

Hmmm was the ram replaced with the new servers, or what about the temp inside the case? I have also heard that both of these can do this.

What are the full specs on the equipment used (mobo, ram, etc) maybe we can all see if we can dig anything up)

Oh well I am in for the haul and stand behind the &quot;FutureQuest&quot;!

Dan, The problems are not that often and have occured around once a month I think.

<EDIT>
I wish my windows box could stay up a fraction of the time as these servers lol, I think my uptime record on my brand new system is around 6 hours. :(
</EDIT>
[This message has been edited by elite (edited 12-22-99@9:55 pm)]

[Terra]

The hardware is custom built by Intel, and every component is certified and has been repeatedly tested...[nbsp][nbsp]I can't release the full specs on the box, but I will say it's their best equipment (one step below Xeon)...

I have tested the CPU/Memory theory by waiting until I have scheduled maintenance (extended) and I pulled the CPU/Memory out of TAZ (+1 brand new CPU) and swapped them around...[nbsp][nbsp]Well, TAZ still runs and NINE still has an attitude - now worse than SIX...

BTW: All servers run the exact same hardware, except for TAZ with 1 CPU...

The CPU temps stay between 30 - 38 degrees C...

--
Terra
--Can you tell yet that I've about done it all?--
FutureQuest

[elite]

--Can you tell yet that I've about done it all?--

Hence


I think we all should start a collection for a week at the spa for terra

Are their ways to speed up fsck, I saw one but since I know little about this it propably was for another application but it says fsck/rm are speeded up about 2 times.

http://www.uwsg.indiana.edu/hypermail/linux/kernel/9907.0/0139.html

[JoeRT]

I subscribe to the WebPartner reports that measure server up[nbsp][nbsp]and response time.[nbsp][nbsp]In looking back over those reports, since October 3 there have been six days when my FQ site up time dropped below the &quot;WebPartner 100&quot; (an average of the top 100 e-commerce sites as identified by PC Magazine).[nbsp][nbsp]Four of those &quot;hiccups&quot; happened before the upgrade at the end of October, so that's twice since November 1.[nbsp][nbsp]That WP100 average is usually somewhere between 97 and 99 percent... the servers that run the top 100 e-commerce sites are up 97 to 99 percent of the time.[nbsp][nbsp]And on the days when there were &quot;glitches&quot; I was still within a few points of the average.

It sounds like the overall up time actually would improve with option #1.[nbsp][nbsp]If in each of the six days noted above the server went down once, that's a total of 108 minutes (18 minutes per reboot).[nbsp][nbsp]With less than four minute reboots, that would have been a total of less than 24 minutes.

It looks like the answer lies in either option 1 and 3.[nbsp][nbsp]I'd hate to think of all the work that would go into option 2 only to find out that the 2.4.x lockup problem is fixed by February.

I can say this... when the FQ servers are up, my sites work and work great![nbsp][nbsp]With one of my previous hosts, sometimes even when the servers were &quot;up&quot; CGI (or other things) didn't work and it took forever to convince tech support (30 hours once)... I've NEVER had that problem with FutureQuest.

By the way... my wife says option 3 through the holidays, then look at option 1.[nbsp][nbsp]:)

------------------
Joe Torsitano
www.weatherforyou.com (http://www.weatherforyou.com)
www.tiswest.com (http://www.tiswest.com)

[Dan Kaplan]

#1 and #3 differ in the aspect of using ReiserFS and losing RAID support...
Yeah, yeah.[nbsp][nbsp]But I wasn't talking about the technical jargon...

Stay with the SMP core and hope that the Linux kernel team pull off a SMP stability miracle for 2.4.x (January/February) Solution #3, is leave it alone - embrace the improved stability we have, and wait to see if Linux 2.4.x will turn this around...
That's all.[nbsp][nbsp]:)

Seriously though, the <4 minute reboots sound like a very good option.

Dan

[Terra]

Elite, I tried merging that fsck trick into the 2.2.11 Linux core, as it was written for the 2.3.x development series...[nbsp][nbsp]In short, it blew up in my face and I had to remove it quickly...[nbsp][nbsp](been there - done that) ;)

JoeRT, the problem is amplification...[nbsp][nbsp]Read through our Server News forum and you will see the trials and tribulations we have endured with 2.2.x...[nbsp][nbsp]For the few that even realize we have a problem, the problem is amplified because it becomes public knowledge via these forums...[nbsp][nbsp]Our 99.7% uptime versus the Internet's Top 100 sites (97 - 99%) is commendable, but the fact still remains that something is broken and it's my job to hunt it down and fix it...[nbsp][nbsp]I guess public scrutiny and pressure is difficult at times to cope with and eventually takes it's toll on the best of us...

In short, it will never be good enough until we reach 100%...

--
Terra
--Where there is a will, there is a way--
FutureQuest

[JoeRT]

Oh I know about the forum posts... in my early days here I even put a few of them (please forgive me ;)[nbsp][nbsp]).[nbsp][nbsp]That was before I knew that, unlike my previous hosts, someone really was paying attention![nbsp][nbsp]These forums do &quot;amplify&quot; the problems.[nbsp][nbsp]But maybe that's why other hosts don't have forums like this... they don't want to have to explain their problems.[nbsp][nbsp]You're willing to, and I appreciate that.

Sounds like you're a lot like me... I'm constantly trying to squeeze that last bit of efficiency out of a transmitter or extra two or three dB less noise from an amplifier.[nbsp][nbsp]I don't care if the factory specs say the noise is down 80 dB and no one can hear it there... I want it down 85![nbsp][nbsp]:)

------------------
Joe Torsitano
www.weatherforyou.com (http://www.weatherforyou.com)
www.tiswest.com (http://www.tiswest.com)

[This message has been edited by JoeRT (edited 12-22-99@11:19 pm)]

[Hunkorama417]

I say:

Stay with the SMP core and hope that the Linux kernel team pull off a SMP stability miracle for 2.4.x (January/February)
------------------
-Don't worry, perfection isn't for everyone.

[ubu]

My vote is for number three....

I have confidence in you guys and the service you are giving us now.[nbsp][nbsp]I see no reason to go through that much work for this.[nbsp][nbsp]Every host would love to have a perfect uptime record, but that isn't possible.[nbsp][nbsp]You guys have done the best I have seen.[nbsp][nbsp]I have been with many many hosts in the begining of my site, in my opinion..[nbsp][nbsp]you guys have no competition.[nbsp][nbsp]You're awesome!

Just continue with the fine work you have done so far..

Jamie

[Justin]

I just want to get my opinion voiced out here... I do not want to see the servers all down to single CPU. I am always amazed when I work with the servers at the speed of CGI execution, while thinking about how many other tasks the server is pulling off at that moment - it's truly impressive :)[nbsp][nbsp]Our servers I believe would be classified as Super Computers - over 1 billion ops per second (600 MHz + 600 MHz running in parallel) - I just like knowing that (I'm not positive if this is true)...

At any rate, reducing the boot up time would be great, but I don't know if the amount of work involved is worth it. Either way, I vote for #3, but if you insist, go with #1, as speeding up the boot time would be a nice thing to have even after the kernel is fixed, whereas yanking a CPU would be wasted effort by that point...

------------------
Justin Nelson
FutureQuest Support

[Charles Capps]

My site isn't mission critical...[nbsp][nbsp]I don't care about an hour of downtime here or there.[nbsp][nbsp]Of course, I might be the minority.

Wait for kernel 2.4 gets my vote, but trying a server with the new FS might be wise...

Query: Will 'frequent' clean reboots, i.e. one or two a week, help the situation in any way?[nbsp][nbsp]Has this been tried?[nbsp][nbsp]I'd imagine that the fscking would be a good deal quicker when everything was cleanly unmounted...
[This message has been edited by Charles Capps (edited 12-23-99@02:15 am)]

[Deb]

I'm just sitting here scratching my head over this.[nbsp][nbsp]I can't decide if I'm angry, shocked, or confused.[nbsp][nbsp]For that reason I've not been able to really respond to what has been discussed in this thread until now.

After reading TeRRa's post I went back and surfed through much of the history of these forums, I've gone over our uptime reports, I've gone over the sales graphs, I've looked at downtime credits, I've read the reasons for deactivation requests (http://www.FutureQuest.net/Cancel/), I've re-read the comments of those that have filled out our &quot;progress report&quot; (http://www.FutureQuest.net/GradeUs.php), I've gone over the reports that I receive from Netmechanic, web partner and other various companies that offer this type of service, and I have re-scanned over the pre-sales emails and comments within payments and new orders.[nbsp][nbsp]At the same time I've pondered what I see on public &quot;searching for new host&quot; forums as well as surfed through many of the offerings of our competition.[nbsp][nbsp]Ergo, as an owner of this company, a member of this community, and a user of these servers, I am doing my homework.[nbsp][nbsp]To tell you the absolute truth, the more I look at these items, the more this thread, and other discussions that have occurred to cause this thread, do not make sense to me.

Can anyone help me to understand this?[nbsp][nbsp]I do not understand how a competitively priced community web host, that provides a faster delivery time and more uptime than the &quot;Top 100 Ecommerce Sites as defined by PC Magazine&quot; causes some site owners to complain that the uptime is poor and has a Sys Admin considering downgrading the servers, placing more restrictions on the site owners, slowing down the cgi execution time, etc in an effort to meet this demand.[nbsp][nbsp]Did I miss something here?[nbsp][nbsp]Last time I checked, the uptime on all servers was above 99.5% and improving.

I'll ignore my own stats and just go off the reports most commonly brought up within these forums.[nbsp][nbsp]Let's take a look at the latest reports I've received from Netmachnic and WebPartner.

Netmechanic Reported:

Week Ending: Tue 21 (for the NINE server since it had the worst week of our servers)
Monitoring Station = Huntsville, AL, USA
Overall Rating: Very Good

</font><font face="Courier" size="3">Performance Summary:
Event:[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] Your Server's Avg. NetMech Avg.[nbsp][nbsp] Percentile
------[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]-----------------[nbsp][nbsp]-------------[nbsp][nbsp] ----------
Host Ping[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]78.31 millisec[nbsp][nbsp][nbsp][nbsp]476.48 millisec[nbsp][nbsp]55
DNS Look Up[nbsp][nbsp][nbsp][nbsp] 0.04 sec[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] 0.11 sec[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] 51
Connect Time[nbsp][nbsp][nbsp][nbsp]0.26 sec[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] 1.96 sec[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] 56
Download Time
[nbsp][nbsp] (10k file)[nbsp][nbsp] 0.33 sec[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] 1.58 sec[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] 64</font><font face="Verdana, Arial" size="2">

FutureQuest is doing a great job!

Web Partner Reported:

</font><font face="Courier" size="3">Reporting period:[nbsp][nbsp][nbsp][nbsp] Sun, 12-Dec-1999 thru Sun, 19-Dec-1999
Average store-page accessibility by day:
Sun[nbsp][nbsp] Mon[nbsp][nbsp] Tue[nbsp][nbsp] Wed[nbsp][nbsp] Thu[nbsp][nbsp] Fri[nbsp][nbsp] Sat[nbsp][nbsp] URL
100%[nbsp][nbsp]100%[nbsp][nbsp]100%[nbsp][nbsp]100%[nbsp][nbsp]100%[nbsp][nbsp]100%[nbsp][nbsp]100%[nbsp][nbsp](TAZ)
100%[nbsp][nbsp]100%[nbsp][nbsp]98%[nbsp][nbsp] 100%[nbsp][nbsp]99%[nbsp][nbsp] 100%[nbsp][nbsp]100%[nbsp][nbsp](SIX)
100%[nbsp][nbsp]100%[nbsp][nbsp]97%[nbsp][nbsp] 100%[nbsp][nbsp]98%[nbsp][nbsp] 100%[nbsp][nbsp]100%[nbsp][nbsp](NINE)

98%[nbsp][nbsp] 98%[nbsp][nbsp] 99%[nbsp][nbsp] 98%[nbsp][nbsp] 97%[nbsp][nbsp] 98%[nbsp][nbsp] 98%[nbsp][nbsp] (WP Top 100)

Average Site Response Time by day (in seconds):
Sun[nbsp][nbsp] Mon[nbsp][nbsp] Tue[nbsp][nbsp] Wed[nbsp][nbsp] Thu[nbsp][nbsp] Fri[nbsp][nbsp] Sat[nbsp][nbsp] URL
0.40[nbsp][nbsp]0.65[nbsp][nbsp]0.97[nbsp][nbsp]0.69[nbsp][nbsp]0.44[nbsp][nbsp]0.61[nbsp][nbsp]0.47[nbsp][nbsp](TAZ)
0.40[nbsp][nbsp]0.82[nbsp][nbsp]0.96[nbsp][nbsp]1.00[nbsp][nbsp]0.63[nbsp][nbsp]0.66[nbsp][nbsp]0.58[nbsp][nbsp](SIX)
0.43[nbsp][nbsp]0.79[nbsp][nbsp]0.63[nbsp][nbsp]0.74[nbsp][nbsp]0.46[nbsp][nbsp]0.59[nbsp][nbsp]0.59[nbsp][nbsp](NINE)

0.82[nbsp][nbsp]0.82[nbsp][nbsp]0.72[nbsp][nbsp]1.04[nbsp][nbsp]1.04[nbsp][nbsp]1.08[nbsp][nbsp]0.68[nbsp][nbsp](WP Top 100)</font><font face="Verdana, Arial" size="2">

The above means, according to WebPartner, FutureQuest has provided a better uptime and a faster delivery than the PC Magazine's defined top 100 e-commerce sites on the Internet![nbsp][nbsp]My point is not that there isn't room for improvement.[nbsp][nbsp]Until every number says 100% at all times, there will be room for improvement.[nbsp][nbsp]FutureQuest has vowed to always try to reach that goal.[nbsp][nbsp]But are we failing? No.

I know how frustrating a second, minute, and hour of downtime can be.[nbsp][nbsp]Trust me, we pay for it big time.[nbsp][nbsp]But I also know that FutureQuest is not failing in this area.[nbsp][nbsp]There are problems.[nbsp][nbsp]I wont try to deny that.[nbsp][nbsp]But I do not feel these problems mean we should downgrade.[nbsp][nbsp]This is the second time I recall our Sys Admin considering a downgrade as an option.

Do we honestly want to slow down these servers? Slow down cgi execution times? Put tighter restrictions on everything? I don't :(

I don't know how many members of this community agree or disagree with me.[nbsp][nbsp]But history dictates that a whole lot more of the site owners on these servers are satisfied than the number of those dissatisfied.[nbsp][nbsp]It also tells me that a whole lot more would rather have full cgi and power as well as better than 99.5% uptime than to have less power and less tools to gain a tiny fraction of more uptime.

So my input on the subject is WAIT! Let's see what Linux comes up with.[nbsp][nbsp]Let's look at our uptime reports as a whole.[nbsp][nbsp]There seems to be a strong tendency to concentrate on downtime while not seeing the uptime, speed, functionality, value for the money, and everything else that contributes to FutureQuest as a whole.[nbsp][nbsp]I can not, and will not, condone a decision based solely on the downtime when the downtime is less than .5% of the issue.

At the same time -- I back up TeRRa's final decision 100% and put my complete trust in his very capable hands when it comes to what is best for a server.[nbsp][nbsp]So with that said sweety -- if you want to downgrade, just let me know when and I'll be here to comply.[nbsp][nbsp]But don't you dare consider it w/o seriously considering what all it involves.[nbsp][nbsp]Fixing the less than .5% of downtime can't possibly be worth all of the other problems it would create.

Deb
[nbsp][nbsp]--
We've made it this far, why look back now?

When will I ever learn to use UBB code correctly? :þ
[This message has been edited by Deb (edited 12-23-99@07:38 am)]

[tedloh]

Nuff said.[nbsp][nbsp]I vote for #3 - although I have seen a couple of boots, I think that the overall performance has improved somewhat from before the upgrades.[nbsp][nbsp]I also agree with Justin in that a downgrade to single-CPU servers would be a step backwards, although I understand the frustrations Andrew is going through.

We've waited this long - and things have already improved - and I know that things will continue to improve over time as the FQ team have shown their unwillingness to accept anything less than the best.

Besides which, the next couple of weeks were meant for family and relaxation.

I think we all should start a collection for a week at the spa for terra
... and Deb.[nbsp][nbsp]You guys forget about the collection for the spa - just find enough for a couple of plane tickets to Thailand.[nbsp][nbsp]I'll use my remaining barter credit to get them into the best spa in the world for a few days(did you know that the TWO best spas in the world are here in Thailand?).


------------------
Ted (Chief Do-It-All)
Tygre Systems Co Ltd
Bangkok, Thailand, Land of Smiles :) :)
http://www.tygresystems.com (work currently on hold)
ted@tygresystems.com

[JoeRT]

Well said, Deb![nbsp][nbsp]And something I have to remind myself of occasionally.[nbsp][nbsp]People only complain when they're uncomfortable.[nbsp][nbsp]If the telephone works everyday for a year or the electricity stays on through that terrible storm, people won't call the utilities and thank them for their service.[nbsp][nbsp]But if it goes off for an hour, by God there will be calls and newspaper articles about it!

I know how hard it is to look at the days and days of 100% when you just came through one single 97% day... that's because few said thanks for the 100% dayS, but many are complaining about one single day... in fact, they're complaining about 20 minutes.

I still think #2 is not a solution that would be well accepted here.[nbsp][nbsp]You're doing great compared to the competition.[nbsp][nbsp]That's why we're here.[nbsp][nbsp]:)

------------------
Joe Torsitano
www.weatherforyou.com (http://www.weatherforyou.com)
www.tiswest.com (http://www.tiswest.com)

[Dan Kaplan]

At the expense of &quot;competitor-bashing,&quot; let me just say how refreshing it is to hear the owner's and administrator's wrestling with serious issues like these in a public forum.

During my few months with Addr, virtually every day saw downtime of at least 30 minutes, often much more -- I think my uptime reports go back far enough...[nbsp][nbsp]This was far from the less than 30 minutes a year that tech support &quot;promised&quot; me initially.[nbsp][nbsp]Of course, with no guarantee, no forum such as this, and no comments on the subject, it was rather unsettling from a customer point of view.

By not commenting on their problems, they may have avoided the amplification issue.[nbsp][nbsp]However, I think you can always counter that yourself by highlighting conversations such as these that show how good your record, service, and intent really are.[nbsp][nbsp]Who in their right mind could read this thread and not come away impressed?

&quot;The whole problem with the world is that fools and fanatics are always so certain of themselves, but wiser people so full of doubts.&quot;
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]-- Bertrand Russel

No matter how good it sounds in theory, you simply cannot make everyone happy.[nbsp][nbsp]A dissenter will always cast a negative light on a situation, no matter how minimal it is in the big picture.[nbsp][nbsp]In short, keep up the great work! :)

Dan (who has nothing better to do while waiting for two phone calls...)

[Mike Weck]

I think that it is great that you all discuss such issues with your customers :)

I agree with some of the other members, in that you should not downgrade, and wait things out.

[QUOTE]I do not understand how a competitively priced community web host, that provides a faster delivery time and more uptime than the &quot;Top 100 Ecommerce Sites as defined by PC Magazine&quot; causes some site owners to complain that the uptime is poor[QUOTE]

99% +/-.5% uptime is 100% percent uptime in my book... cause nothing in this world is absolutely perfect, and some people just don't get that :)

IMHO FQ is building momentum, by word of mouth and by this community online. There is a sound base for expanding exponentially, it will just take time.

Keep on doing a great job!!

[ubu]

If the telephone works everyday for a year or the electricity stays on through that terrible storm, people won't call the utilities and thank them for their service.[nbsp][nbsp]But if it goes off for an hour, by God there will be calls and newspaper articles about it!
I tend to think this is the case..[nbsp][nbsp] I see this almost everyday at work.[nbsp][nbsp]I am a Store Manager of about 550 assoicates. We can honor their request for time off almost every time they ask, but then there's the one time when we really need them to work their schedules and that's the only thing they talk about for the next week.[nbsp][nbsp]How mean and harsh we were for not giving them time off..

Don't let just a few people dictate what you do..[nbsp][nbsp]listen to the community you have built.[nbsp][nbsp]We're behind you all the way!

Jamie

[cyberassassin]

Well, I rarely write to these forums, just skim them for news (to much else to do), but this one made feel that I need to say something.

As far as I am concerned, this hosting service has been great.[nbsp][nbsp]I have recommended it to dozens of people, and all are impressed wiht what FutureQuest provides.[nbsp][nbsp]Computers are still machines, and they break, it is a fact, and it will never change.

My opinion, stay with the current set up.[nbsp][nbsp]Wait for kernel changes for SMP fixes (hopefully).[nbsp][nbsp]Investigate a new FS (why has XFS been ruled out?, that is what I would look at)[nbsp][nbsp]Take these serious changes slowly.[nbsp][nbsp]I am not disatisfied at all at the moment, and while I know that there have been hiccups, I have never been harmed from them.[nbsp][nbsp]Granted, I have not been doing a web based presentation and been denied access to my information (saw this earlier) but who doesn't go into a major presentation with at least 3 backup? (Zip disk, good old fashioned paper?, extra laptop?)[nbsp][nbsp]The world is full of uncertanties, and anyone who can't accept that is bound to be frustated and disapointed for life.[nbsp][nbsp]Bad things happen, and for no apparent reason.[nbsp][nbsp]

Be mindful, the linux kernel will be fixed, sometime, if that is indeed the problem.[nbsp][nbsp]I am actually surprised that these servers aren't quad CPU configurations.[nbsp][nbsp]Could have fooled me.[nbsp][nbsp]Anyone take a look at all of the stuff on these beast?[nbsp][nbsp]An it only takes 18 mins for a fsck and reboot.[nbsp][nbsp]To me that is impressive.[nbsp][nbsp]And in the meantime, take a breath, step back, and look at the problem with fresh eyes.[nbsp][nbsp]Free helpful hint from a designer who needs creative solutions to problems everyday.

Keep up the good work.

Brian

www.mtmarchitects.com (http://www.mtmarchitects.com)
www.krunk.org (http://www.krunk.org)
------------------
- Who is the master of foxounds, and who says the hunt has begun?

-Pink Floyd

[Monty]

I don't understand the technical issues, but if it comes down to speed or uptime, I vote for speed.[nbsp][nbsp]My goofy little fishing message board is getting massive hits every day, and is doing great. It is 100% cgi, (Webbbs 3.20 version), and everyone there has commented about how fast it is.[nbsp][nbsp]Well, guess what, it ain't fast 'cause of anything this dumb fisherman is doing, it is fast because of FutureQuest and the fine folks there.

Have a safe and Happy Holiday, and spend some time with family, friends and pets.[nbsp][nbsp]

Mont

[Charles Capps]

Touching on what others have said...

I also began hanging around here when throwing ideas around for my site...[nbsp][nbsp]And I saw something that I **LOVED** - those downtime reports, the complete and total honesty![nbsp][nbsp]No lying, no evading the issue, no little &quot;We rebooted it.[nbsp][nbsp]Sorry.&quot; reports - everyone was told exactly what was happening every time.[nbsp][nbsp]This thread is an example - you're asking *US* what to do![nbsp][nbsp]Where will you find a host that'll do that?[nbsp][nbsp]Eh?[nbsp][nbsp]:)

And the servers... Oh, the servers![nbsp][nbsp]That's another selling point![nbsp][nbsp]Fast, fast, fast, fast![nbsp][nbsp]Bleeding edge.[nbsp][nbsp]Unfortunately.[nbsp][nbsp]I was one of the first on SIX.[nbsp][nbsp]The first SIX.[nbsp][nbsp]The naughty one that went down a few times a week.[nbsp][nbsp]

I really don't mind the downtime <font color=#FF0000>BECAUSE I KNOW YOU'RE DOING EVERYTHING YOU CAN TO FIX IT</font>, and you say so.

Right now, option 1 and option 2 are simply a pain in the rear.[nbsp][nbsp]Will they fix the problem?[nbsp][nbsp]Yes, but at what expense?

Crippling the servers will cripple the sites, and even if you do move off the load, it won't be the same.[nbsp][nbsp]Things WILL slow down.[nbsp][nbsp]You've just lost a selling point.[nbsp][nbsp]

Solution two is better, but how much?[nbsp][nbsp]Will easing load really help the problem?[nbsp][nbsp]Can you verify this beforehand?[nbsp][nbsp]I'd really hate to see everyone to go through such trouble when the outcome is still unknown.

That's why I'm for waiting and seeing.[nbsp][nbsp]The kernel hackers may well pull off a miracle and fix SMP.[nbsp][nbsp]

But what happens if they don't?[nbsp][nbsp]Well...[nbsp][nbsp]

[Terra]

<<
<<Moments of Reflection for the Server Status of 1999>>
<<

After taking time to reflect on everything that has been said, with a good dose of sleep :), I realize that my concerns and proposed actions were driven by the impulse of frustration...[nbsp][nbsp](Picture Calvin standing there with that black squiggly cloud over his head, fists clinched to his sides, when Hobbes really pushes him over the edge)

For those that know, this saga began a long time ago with SIX for which then the problems were hopeless and something I was forced to accept and constantly work around...

With 2000 coming up, I wanted to start with a clean slate and solve the 2.2.x lockups once and for all...[nbsp][nbsp]We have been getting quite a few cancellations and complaints, primarily due to downtime related issues, and the point came to a head that enough was enough...

Server uptime has increased dramatically since the upgrades were shifted into place, but even so, the sporadic nature of their lockups provided no rhyme or reason...[nbsp][nbsp]I've gone from top to bottom to top looking at everything under a microscope...[nbsp][nbsp]The final realization is that most everything points to SMP operations, as many others have experienced the same stability issues with 2.2.x...

We are constantly judged against other hosts, and even though we shine like a bright light at times, we still have the 'stability' monkey on our back that is mine alone to solve...[nbsp][nbsp]Sort of like, 'FutureQuest is great, but their stability is not the greatest by looking at their forums'...

When these servers lockup, I tend to take it personally thinking that either I'm not doing my job right, or there is something within the millions of lines of code that I have simply overlooked...[nbsp][nbsp]Passing the blame off to the SMP brokenness is not an excuse...[nbsp][nbsp]When something on the server breaks, I know that the buck stops with me...

Looking above, I will now touch upon some of the questions that were posed...

) XFS vs. ReiserFS
XFS is my embraced choice, but no one knows when it will be officially released and rock solid...
ReiserFS is available now and quite solid...
The only downfall of XFS is it's performance with small files due to it's B-Tree nature, whereas ReiserFS is highly optimized in this respect...
**Our servers are full of 'small' 1k - 4k files and maximizing the performance for this has been difficult
ext3 is still a ways off from full deployment and does not provide the advanced Hash Tree features of the 2 above FileSystems...
(In all journalling FileSystems above, RAID is incompatible due to buffer issues...[nbsp][nbsp]I am not even sure if this is going to be fixed in time for 2.4.x)

) Test before Deployment
While building SIX, we did not have the time to perform extensive regression tests as TAZ was suffering tremendously and we needed a new server ASAP...[nbsp][nbsp]The server was also late as the financial resources were not there to purchase everything at once and it took us 1 month too long to scrape the funds necessary to purchase a new server...[nbsp][nbsp]This was also the time we were plagued with SprintLink network connectivity issues that I don't even care to discuss...[nbsp][nbsp]In short, when SIX was rolled out, we were surrounded by a Towering Inferno...
SIX was deployed and brought online, she had shown tremendous power and flexibility, more than what we had even guessimated...[nbsp][nbsp]1.5 months after she went live, the lockups began to surface...[nbsp][nbsp]Initially I felt that it was a transient problem that would clear itself in time as the Linux 2.2.x kernel had matured...[nbsp][nbsp]Now I look back and realize I was dead wrong...[nbsp][nbsp]From 2.2.1 --> 2.2.12, the problem persisted and it became apparent there was no end in sight...[nbsp][nbsp]I was lulled into the false sense of reality that the 'next release' would fix it, as I was told by the development team...[nbsp][nbsp]That fix never came...[nbsp][nbsp]Around 2.2.10, I stopped depending on them and took the mindset that 'If you want things done right, do it yourself'...
**Bit of History**
This began my private branch of the kernel called the Phoenix (-P) tree were I spent debugging the code trying to track down this elusive glitch...[nbsp][nbsp]2.2.11-P was the first halfway decent stable kernel that gave us increased uptime from the pathetic 3 to 4 days, improving to 7 - 9 days uptime stretches...[nbsp][nbsp]2.2.12-P went through countless updates, patches and fixes that led us to Phoenix-12...[nbsp][nbsp]This particular one paved the way for 14 days of uptime...[nbsp][nbsp]Bear in mind that even though the Phoenix project was a success, it still failed as out of the blue, and the servers not even under load, still froze up randomly after 3 days or less...[nbsp][nbsp]Sporadic in nature, but the trend was getting better...[nbsp][nbsp]Phoenix-20 paved the way for 18 - 20 days of uptime and was much more stable, with a reduction in random locks...[nbsp][nbsp]2.2.13-P35 uptimes have increased to 30 days, but what has been particularly frustrating is that the random locks have begun to surface again...[nbsp][nbsp]Last night I brought online 2.2.13-P38, and only time will tell with this one...
NINE was rolled out right before the Phoenix project, and I had no choice but to use the 3rd Gen core to build it on...[nbsp][nbsp]We had more time available to build and test NINE but what I discovered was that no matter how hard I tried to create a 'simulated' production environment - nothing could come close to the real thing...
NINE Burn-in:
1) RC5 cracking team (tests the CPU and Integer functions)
2) POV ray (tests the CPU/Memory/FPU)
3) Bonnie (exercises the Hard Drive)
4) Endless loops of Kernel compiles using 'make -j8' (absolute Torture in it's extreme)

NINE passed all the regression tests with flying colors, and was deployed shortly thereafter...[nbsp][nbsp]Another discovery is that after about a month of deployment, the random glitches would start popping back up again... Arghhh![nbsp][nbsp]This turned my sights to malicious activity and DOS attacks, which our servers are bombarded with daily...[nbsp][nbsp]I have patched and plugged every hole I could find but still have the realization that I won't ever find them all (no one ever does)...[nbsp][nbsp]Sophisticated monitoring tools were deployed only to come back empty handed...[nbsp][nbsp]So there I sat back at square one...

SEVEN was also subject to the regression tests, but was built on the new Intel servers...[nbsp][nbsp]I won't even begin to express my amazement at that server...[nbsp][nbsp]She was the first 4th Gen server to go online and I can only smile when I watch her blast CPU/Disk cycles like no other...

SIX and NINE were finally retrofitted for the 4th Gen core with the new servers, thinking heh - it just might be the hardware after all...[nbsp][nbsp]To my dismay, the hardware was not the problem as *everything* was replaced with brand new and extensively tested...[nbsp][nbsp]I am now beginning to curse the very existence of the word 'Deja Vu'...

Now I sit alone, asking myself 'Why?'...

Couple key points in my discussion from above...
The first of which is the realization that no matter how hard you try, you just cannot create a 'simulated' real-world production environment on the bench...[nbsp][nbsp]No amount of (sophisticated/extensive) regression tests are going to ever come close to the duties these servers must endure in the real world...
The second and most blatant is 'TIME'...[nbsp][nbsp]I roll out my work to production and it takes days/weeks to get the results back...[nbsp][nbsp]Very, very frustrating indeed...[nbsp][nbsp]TIME has taken it's toll and with a company trying to provide to it's client base, there has to be a resolution somewhere...[nbsp][nbsp]Explaining the 'TIME' issue has become more difficult as what is perceived is simply 'It is still broken'...

Believe me, I could write a novel on my life with Linux 2.2.x, with how it has affected my private, personal and public life, my families life, and the life of this company and it's clients...

My attitude at this point is 'Take Ol Yeller out back and end it's misery'...

--
Terra
--What doesn't kill me can only make me stronger--
FutureQuest

PS: Thank you for not only enduring this 1999 mind dump, but also for believing in us and supporting our efforts every step of the way...
[This message has been edited by ccTech (edited 02-04-00@04:37 am)]

[teach1st]

About the amplification thing: When I was cruising this forum in hopes of finding a way to escape from my then current hosting company, the downtime reports I saw, the questions about the servers, and especially the responses from FQ, are all what sold me. What a difference from my other hosts! I understand servers going down and other difficulties. I don't understand not being notified about difficulties and about having my inquiries ignored. There's a big difference between you and the other hosts with which I'm familiar. We're informed here, we're partners. This thread is proof enough. I would think many of those potential customers looking at FQ through the forums would feel as I did - way cool - and not be put off. You appear to have a very loyal customer base. I know I ain't leaving. All I have to do is look through the archives of e-mail to my former hosts (This is my 4th e-mail...; Why aren't you answering my questions?; If I don't hear from you within two days I'll have to contact the BBB, etc) to know how good I have it here. None came close to FQ's uptime.
As to proposed changes, I'm not sure if I understand them thoroughly, but I tend to trust the more knowledgeable members of the community and agree that #3 seems to be the way to go.
Happy Holidays!

[frankc]

Stay with the SMP core and hope that the Linux kernel team pull off a SMP stability miracle for 2.4.x (January/February)
Yessireebob--do it![nbsp][nbsp]Test the new core on a separate machine and see if it breaks before installing it here; wait to see what others are finding with the new release before bashing your head against the never-yielding brick wall.[nbsp][nbsp]THEN--based on that evaluation of the new core, decide on a course of action.[nbsp][nbsp]To take action now without knowing how the new core works would be acting on impulse rather than instinct, rarely a wise move.

Sit back...watch the new core and get a history on it.[nbsp][nbsp]If it's stable elsewhere, then upgrade with a fallback plan of attack at the ready.

I, for one, ain't movin' either way.[nbsp][nbsp]I'm a FQuestie fer life with both my sites!

Merry Christmas to all, and to all at FQ, a rest.
------------------
Frank
Hosanna! Lutheran Church www.hosannachurch.com (http://www.hosannachurch.com)
[nbsp]--and--
Pacesetter &quot;Moving Message&quot; Signs www.pace-setter.com (http://www.pace-setter.com)

[Rich]

Regarding FQ stability and reliability:
I have seen FutureQuest make great strides in improving all aspects of server performance and reliability. In addition, I believe their longer-term architectural roadmap is the right one, opting for insfrastructures designed to remove single points of failures.

The data clearly support FQ as a leader in reliability. I believe any rational customer would prefer measurable results to only perceived results. I know of no other hosting company that publicly reports these measured results. The data also clearly show a sucessful track record of consistently improving results over the past 12 months.

Regarding Intel SMP architecture:
I believe this has been proven both robust and reliable with many OS's and apps (just not Linux, yet).

Regarding Linux and SMP:
I know that &quot;they&quot; (whatever &quot;they&quot; means when it comes to an open computing platform) know this is a serious issue. The NT vs Linux shootout sorely demonstrated that this is a critical issue. I also know that many, many, people are agressively working the issue. However, the bottom line is that the problem is still not resolved and this raises serious concerns regarding whether this OS has what it takes to compete. (That last sentence was a very difficult and painful one to write.)

Regarding how FQ should respond to the Linux/SMP issue:
While waiting for a reliable Linux kernel is acceptable if the trade-offs are equally acceptable, downgrading the hardware is not. Regardless of how good this might look on paper, I firmly believe doing this would quickly and steeply degrade reliability, not improve it. If waiting for a reliable Linux kernel is not acceptable, then I believe the correct approach would be to consider a known reliable SMP OS. While this approach may be even a &quot;more massive&quot; undertaking than the hardware downgrade, the end result is the desired one.

For now, I believe waiting is appropriate. But I believe FQ's question that has been raised here is a sound one: &quot;How do we maintain our track record of incremental improvements in reliability if the Linux/SMP issue is not resolved in a 'reasonable' amount of time?&quot;

Rich

[Melprophet]

Heh, I didn't understand at least half the technical stuff on this thread, but I don't care about that, it just makes my brain hurt anyway...

Deb:
Do we honestly want to slow down these servers? Slow down cgi execution times? Put tighter restrictions on everything? I don't
But I understood Debs post, and I echo her sense of puzzlement on this topic. My site visitors let me know when things are slow or down, and I haven't heard a peep out of them in months.

I've had 3 previous hosts which I won't name, and not only is FQ is the first one that has consistantly met their uptime gaurantee, they're also the first host I've been with to admit when they didn't...FQ is also the fastest severs I've ever had a site on.

I'd hate to see server speed and cgi execution slow down but whatever the final decision is on this, I'll be along for the ride.

Mel

[DB]

I can really emphasize with the frustration you're feeling, Andrew. Programmers (at least the good ones) tend to be perfectionists, and just knowing there is a bug hiding in the servers that prevents them from running at 100% can be maddening. And having customers leaving because of it is enough to push an admin over the edge, especially one who really cares about the company and the customers. I've been following the Server forums closely since February and have seen how much effort you have put into these boxes. I've quietly cheered when it looked like the problems may have been resolved, and felt my heart sink when the bugs popped-up again. What matters to me the most is knowing that the setbacks have never stopped you from trying to achieve your ultimate goal. And I believe that one way or another you will get to the promised land... you're immensely talented, hard working and dedicated to making FQ the best possible host it can be. With that in mind, it seems like Option Two is out of the question, at least as long as there are still other possibilities (2.4.x... even if your faith in the developers has been shaken, they too are under a lot of pressure to get it right). Since the downtime hasn't been excessive lately I feel you (we) can afford to wait a little longer before scrapping all the work you've put in. You know what they say, it is always darkest just before dawn, think of how good it's going to feel when the SMP problems are finally solved and you have some of the most powerful AND RELIABLE servers on the net. :)

As for the customers who don't want to wait it out, I have to wonder how many of them have read the posts in this forum and are aware of the amount of effort that has gone into fixing them. As many of the posts in this thread have pointed out, the fact that you have been both working hard and totally honest with us makes a huge difference. In contrast, I've been catching up with the Bellsouth.net ADSL newsgroups and forums, and what I've seen is a tremendous amount of frustration over the canned &quot;alert&quot; posts that offer no real clue as to the nature of outages or what BS.N is doing about them. Most people who have been dealing with cutting edge technology know there are going to be glitches and can accept that, but they don't like being kept in the dark. Andrew, your continual updates and explanations shine bright in an industry were darkness seems to be the rule. And for that I thank you... and Deb and Justin as well.

I'm quite hopeful that all the effort and frustration of 1999 will be greatly rewarded in 2000. Merry Christmas to all of you.


------------------
--Tom aka DiamondBack
[nbsp][nbsp]http://diamond-back.com

[Deb]

WoW

[nbsp][nbsp][nbsp][nbsp] Thank You

Deb
[nbsp]-- Speechless --

[YFS200]

Here is a question. I know all to well that bench testing can't hold a candle to the full blown furry the real world can produce. So.. why not a beta server? Something to try your ideas on, but have it hosting non-mission critical sites. Or hobby sites in my case. For a few more MB of disk space:), I would gladly have my site moved to another server, even if that includes a bit more down time. I know if you ride on the bleeding edge, you are going to get cut. I don't care, and my users don't mind. And you don't have to credit my account for any down time.

I am sure there is people here that would go for this. And with an addition of a few chat scripts and some more CGI programs, I bet we can give it a good stress test.

Whatever you want to do, I am behind FQ 100%.

As for another OS. Got my NT 4 server running for 30 days continues once. A record. But now it has a hard time going 10 days without locking up. And the darn thing does not do anything but use Nortion System Doctor to maintain it's RAID and do backups to CDRs twice a day. It's idle most of the time, and nobody ever uses the thing. I just can't keep it running!!!! AUGH!!! How can anyone use this thing as a web server, I do not know.

Eddie
-Nothing says I hate you like the blue screen of death Win95 computers give you when you hit &quot;SAVE&quot;.