Engima - Server Load

[phppete]

Is it normal for Engima to have a server load of over 4? I know it has dual CPU but it seems a little high.

2:29pm up 255 days, 13:22, 4 users, load average: 4.04, 3.50, 2.64
5 processes: 3 sleeping, 2 running, 0 zombie, 0 stopped
CPU0 states: 16.3% user, 28.4% system, 1.1% nice, 54.2% idle
CPU1 states: 26.4% user, 8.0% system, 2.1% nice, 66.0% idle

I just checked Samson to see what that was and that is also at 4.25, it is just that once upon a time load average was below 1 but I guess you have to cram on as many sites as possible since dropping your prices. Or is it that everyone is running stuff like Joomla and other bloatware?

[Terra]

Dynamic sites and spiders galore...

I wouldn't stress to much with the 1 min average (4.04), those are just processing spikes that occur throughout the day and is normal... The server guardians keep a close eye on the 1 min average though to clamp down on any runaway loads...

What I primarily watch is the 5 min (3.50) and 15 min average (2.64) as well as the CPU idle states...

As an aside, ENIGMA is at an averaged loading capacity and was removed from the server setup rotation list on 7/27/2007...

If I find that ENIGMA is being taxed too much, then I will shift off some accounts and distribute them to other servers that are being underutilized... As it stands now, ENIGMA is busy doing its job and working hard for you every day...

[phppete]

The 15 min server load a few mins ago was 4.00 according to the CNC but I do realise there is more to all this than server load.

Anyway, one method for cutting the server load in half is by doubling your prices and reducing the number of accounts by half, it gives you the same profit. PAIR seem to be doing well and they haven't gone around slashing their prices like you have and they don't even offer PHP Secure_Mode. Their servers seem to be up for less time than yours as well and they take a day to respond to email questions, where as you take about 1 millisecond :yeah:

[hobbes]

I'm not sure that double the price and a quarter of the customers would work out for FQ:ytbrolly:

-- Hummm, maybe if they secretly exchange the dollar sign in front of package pricing to pounds --

[phppete]

I'm not sure that double the price and a quarter of the customers would work out for FQ:ytbrolly:

-- Hummm, maybe if they secretly exchange the dollar sign in front of package pricing to pounds --

Perhaps FQ seem so cheap to me because of the weak $, at this rate I'll be able dig around for loose change and buy the company. Of course my US clients are having to absorb the increase :umm:, not so long ago it was $1.54 - £1, it hit $2.11 - £1 last week.

[Terra]

The loading issue you saw on ENIGMA was from spambots posting comment spam on a users dynamic site... We have already taken action by blocking the source IPs and the site owner has been notified...

Most of our day is spent playing whack-a-mole against zombies and botnets to keep the server loading within acceptable parameters... The biggest headache we have now is all the comment spammers sledgehammering trackbacks and comments... Their tactics have changed where they will pummel sites from 100s or 1000s of different source IPs, which in turn makes it next to impossible to determine (progmatically) if the POST is from a spambot or from a valid visitor... Now, 9 out of 10 times, it requires an admin to inspect the traffic and figure out how to handle the situation...

I have been watching ENIGMA closely for the last few days and I am seeing a loading trend that needs to be addressed... I have isolated a couple accounts that need to be shifted to another server that are almost constantly receiving traffic due to their popularity...

Here is a 26 hour loading view of ENIGMA:
http://www.futurequest.net/overloads/ENIGMA_uptime-26-20071121.png

Black == 1 min
Purple == 15 min
Blue == scaled number of processes

The loading is not really bad at all, however my goal is to keep it within the green lines - also - disregard the blue portion which is stacked on top of the 15 min average...

[phppete]

Hi Terra,

Thanks for the info. Just a question about the spam bots, I thought they wouldn't report a valid browser as a user agent? So surely if you block anything from a form other than a valid browser then they can't post right? I only start a session if a valid browser is detected and then only allow a form post if a session token matches the preset session token in the post (put in when the form loads). This method seems to have cut all spam on all sites I am involved in or am I missing something?

[Terra]

Well, most of the latest spambots are returning valid UserAgents as well as forging the Referrer... It is pretty much trivial to do and doesn't require much effort on the spammers part...

Also, your idea of testing for a 'valid browser' is application specific, and there is no easy way to implement this as a global protection for all software running on our servers... In the end, it is up to the app developers and/or site admins to ensure they have protective measures in place... This is something that we simply don't have the resources to do for them once you realize that our servers run tens of thousands of different scripts...

[phppete]

Even if a valid user agent is returned it still wouldn't maintain a session so therefore the token set when the form loads would not match the one checked when the form is submitted which means the form isn't sent. So far I am having 100% success rate for my clients against spam on web forms. One site was getting about 30 a day, after updating to the above methodology it completely stopped.

I'll keep doing what works until it stops working but hopefully that won't be for a long while.

[Jeff]

and they haven't gone around slashing their prices like you have
I wasn't aware of any price slashing announcement in the last 6 months... what did I miss???

The 15 min server load a few mins ago was 4.00 according to the CNC but I do realise there is more to all this than server load.
It would be interesting to document the actual wall-clock time effect on a typical php script's execution time of the load being 1.0, 2.0 or 4.0 on the current generation of hardware.

[Arthur]

Even if a valid user agent is returned it still wouldn't maintain a session so therefore the token set when the form loads would not match the one checked when the form is submitted which means the form isn't sent. So far I am having 100% success rate for my clients against spam on web forms. One site was getting about 30 a day, after updating to the above methodology it completely stopped. Your method, which sounds like it is similar to using a CAPTCHA, will stop the spam from going through. However, that doesn't necessarily mean it stops the spamming. There is another side to it.

Once a URL is loaded into a botnet, it will stay there for a long time (months). If you change the script that is receiving the spam, or even if you remove it, the botnets will keep pounding on it anyway, even if the attempts are not successful. The resources of the botmaster are plentiful and he doesn't care about a URL more or less. I've seen sites, that have never even had comments or trackbacks enabled, receive lots of spam traffic.

The server will still have to deal with the (attempted) spamming, the script has to be executed, or if it was removed or disabled, a 404 or 403 response has to be sent.

-Arthur