FQuest Notice: Resource Issues

[jimbo]

Wow. That was much more of an explanation than I had hoped for/expected.[nbsp][nbsp]You answered all my questions.[nbsp][nbsp]It is understandable that the switch to dynamic content can be tough on server resources.[nbsp][nbsp]But I think the type of people that flock to FQ are the types that like to use dynamic content - not just myself.

It just that SEVEN had been running smooth for what seems like months.[nbsp][nbsp]And then all of the sudden there are two hiccups less than 24 hours apart from each other.[nbsp][nbsp]I was just wondering why it was that it happened now, all of the sudden.[nbsp][nbsp]Whether it was a customer's fault, or just a general server problem - that sort of thing.

If there is anything that I have, it is faith that you guys are doing everything in your power to keep things running as smooth as possible.

Thanks again for your answer, it cleared lots of stuff up :).

-jim

[Monty]

As a site owner, I feel a responsibility to help keep the servers running with as clean of a load as possible.[nbsp][nbsp]

Is the Webbbs 4.33 bulletin board script I am running creating a problem for the server(s) (seven & nine in particular) in terms of load?[nbsp][nbsp]

We are all in this together, and if a particular script is a problem, I would like to know.

If it would help, I would be willing to change my site(s) to different FQ servers as well.[nbsp][nbsp]

Mont[nbsp][nbsp]

stuck in Atlanta, home of the busy signal, for 12 more hours..

[Terra]

***
I wrote this in response to the following, but felt that it had enough impact that it would become buried in thread...
http://www.aota.net/ubb/Forum4/HTML/000426-1.html
***

Just wondering why it all of the sudden decided to start acting up - and only on SEVEN? This is still being determined...

All servers have the combined personality of the site owners that reside on them...

Each are unique in various ways, as the load patterns are different...[nbsp][nbsp]Some are heavy CGI, others are heavy PHP - while some are both...

NINE and SEVEN fall into the category of (heavy) CGI+PHP+CRON activity...

With everything that is going on, sometimes it is analogous to a misbehaving stop light...[nbsp][nbsp]99.9% of the time, the stop light works as expected...[nbsp][nbsp]There is that very small chance that the stop light might be both red (or green or yellow) and two processes collide with one another...[nbsp][nbsp]SMP kernels are prone to this subtle problem, which is why I stopped using SMP kernels...[nbsp][nbsp]However, even when this does happen - the SMP kernel has special deadlock code that can remove this contention...[nbsp][nbsp]Sometimes it can't, and locks up...

With SEVEN, and this memory issue - there is a contention problem with 'kswapd' and it's attempt at memory reclamation...[nbsp][nbsp]There is a rogue (client) program on the server that is causing this memory condition to happen, and I'm moving in the tools to track it down...

The joys of offering liberal servers for everyones development needs...[nbsp][nbsp]Sometimes too liberal that it can come back to bite...[nbsp][nbsp]I will be taking a closer look at our CPU/Memory policies and most likely start tightening the restrictions...[nbsp][nbsp]The policies of a year ago, are losing ground to the sudden shift of dynamic *everything*, with more and more being asked of servers...[nbsp][nbsp]Even simple web pages are now being created with CGI scripts, when simple HTML with an SSI would do...

So in summation, times are changing and the heavy shift to dynamic content is starting to cause unique conditions in the server due to the underlying sudden spikes that some programs cause...[nbsp][nbsp]New resource policies will be investigated to keep up with the changes in the interest of maintaining quality uptime and stability without sacrificing performance...

Jimbo, you of all people - should be able to fully relate to the migration to Dynamic Content, and the dependence it causes with elevated resource requirements...

Right now, I am plugging the dam with my fingers and toes...[nbsp][nbsp]It's now time to divert the river, and build a new redesigned dam from scratch learning from the previous...

Once again, full disclosure at work...[nbsp][nbsp]These are the problems we are faced with - and the issues that I'm tackling...[nbsp][nbsp]It's a real problem, and I'm sure not many realize that static HTML is a dying breed...[nbsp][nbsp]Most hosts trying to bury (cover up) the issue, and I think that's unfair...[nbsp][nbsp]It would be crazy to assume we are the only host facing this growing resource problem...

So in short:[nbsp][nbsp]We are painfully aware of the problem, and with due diligence, reworking our server strategy to keep up with the growing demands asked thereof...

--
Andrew Gillespie
Systems Administrator
FutureQuest, Inc.

[Terra]

Monty, it's not so much 'this-or-that' specific thing that is the issue...

Overall, what I have to contend/design for is the trend...[nbsp][nbsp]Our servers are specifically built for heavy dynamic content - but even the best designed servers can be overrun with completely random occurance of timing...[nbsp][nbsp]It's all 'instance' methodology, whereas what is _asked_ of a server at a specific moment in time...

For example, consider your own desktop...[nbsp][nbsp]For the most part of your day - your bouncing between various applications getting your work done...[nbsp][nbsp]The there may be an 'instance' where there is a magical Microsoft Office button that starts every single application (Word/Excel/etc) in one go...[nbsp][nbsp]These are the loads becoming prevalent, and increasing with frequency...[nbsp][nbsp]Another way of looking at it, is a message forum is mostly inactive (or has normal/regular traffic), but a specific event - such as the SuperBowl has just been played and everyone is flocking to the message forums of the winning team...[nbsp][nbsp]With the dynamics and instant-on nature of the internet, you can never really predict this coming as there are many events just as important as the SuperBowl, that we never really know about: e.g. Golf, Tennis, World Events, Software Releases, etc...

For places like Yahoo, they see it coming as they have established traffic patterns (with history) for their systems...[nbsp][nbsp]They (Yahoo) are mostly singular in design nature, with a specific focus...[nbsp][nbsp]However, for hosts like ourselves, we handle a plethora of different (and vast) content...

This is the challenge in front of me, one that I'm investing a lot of thought and energy in solving...

The wind is changing, and what the servers were initially designed for at the core is once again shifting...

It really is the evolution of the Internet, and in this case - everything is becoming Dynamic in nature...

In order to achieve the high levels of expectation for FutureQuest, I have to solve these problems to ensure longevity and allowing us to provide the best service as technology permits...

--
Terra
--A brief glimpse into the behind-the-scenes world of--
FutureQuest

[Monty]

If my ignorance shows, so be it.[nbsp][nbsp]But here are my thoughts.

My forums can be "open" or "moderated".[nbsp][nbsp]I have no idea as to how this could affect the loads on the servers.[nbsp][nbsp]In a moderated version, I control the what appears on the index page, which messages are "allowed", ect.[nbsp][nbsp]Open, my users have the say, the posts go up asap.

In addition, these same forums have many "extra features", which rely heavily on the server for support.[nbsp][nbsp]One of which is "views" (essentially a hit counter for each message).

On top of all that, is the length of my index page for these forums.[nbsp][nbsp]I can limit it, or choose not to.[nbsp][nbsp]But load times and server loads are affected by my decision there.

What I am trying to get at, is that site owners make decisions, that affect the server.[nbsp][nbsp]If there is anything I can do, as a site owner, to help things out, I am willing to do that.[nbsp][nbsp]As I stated before, we are all in this together.

A while back, we all took part in a thread about speed vs. uptime.[nbsp][nbsp]Every single time a page is called, or loads, speed is a factor.[nbsp][nbsp]All our users see the results.[nbsp][nbsp]Uptime is way different.[nbsp][nbsp]Being down, 20 minutes, in a month, might affect .01% of my users, at best.[nbsp][nbsp]Being slow 2 days in a month, affects 100% of them.

My forums would have been history long ago, without FQ.[nbsp][nbsp]

FQ provides superior support, service, and all that goes with it, at a price I can afford to pay.[nbsp][nbsp]

I am willing to pay more, to keep all that I have gained. I am also willing to change my habits.[nbsp][nbsp]Whatever it takes.

May the force be with you TeRRa. ;)

Mont

glad to be back in League City, where all the busy signals are on AOL. lol.

[zmax]

I would like to add a comment which may or may not be relevant and assuming I understand all thats going on. I read a bit about the evolution of the internet and next gen applications and the whole fiber/broadband infrastructure etc etc. I read an article which said that it wasnt greater bandwidth that was the entire fix to the current shortcomings of the internet but prioritization of data going through. For example email and video have the same priority now. But if video is ever going to work over the internet like it does on tv, no interuption in signal, then it has to get a higher priority over things like email that can wait a few more milliseconds... I dont know if this is somthing that is already or can be applied to servers here.

David