FutureQuest Alert: Network issues

[sheila]

FutureQuest is currently experiencing network issues.
Staff is aware and is working to determine the problem and a solution.

More details will be posted as they become available.

[Bradley]

maaaaaaaaan Sheila! Beat me to the punch! I believe it was where I downloaded mysql 4 on my server at 1.25mb/s :P

Thanks for the clarification though, you can dispose of my post in Server questions/suggestions if needed.

Brad

[phppete]

Thanks, I hope you sort it out VERY soon because its Christmas and I am trying to run a business and this is a very busy time and I am losing money and customers.

[Bradley]

Rest assured Pete that FQ is on top of this, there are quite a few businesses on FQ and unfortunatly everyone felt this, it seems things are clearing up now though.

I thought it was a just me problem as I was on my server at the time and though some runaway process or something tore through it. ;)

[Terra]

The FutureQuest network came under a massive DDoS attack (mostly focused at the ENIGMA server) that was causing roughly 8% packet loss on the internal core links... This would have caused choppy access to all, even though sites were still responding and 92% of the packets were still making it through...

Once we identified the focal point of the attack, blockades were put into place at the edge routers which in turn mitigated the attack and brought our throughput back up to full speed...

When we get hit with attacks of this size, the best we can do (or anyone else for that matter) is batten down the hatches and ride the storm out... There is no magic bullet solution to these attacks, and it usually involves good old fashioned elbow grease in trying to figure out the scope of the attack and the shoring up of the beachhead against the torrential tidal wave...

Our sincerest apologies for any inconvenience this network event had caused anyone...

--
Terra
sysAdmin
FutureQuest

[Terra]

I am trying to run a business and this is a very busy time and I am losing money and customers.
To be blunt, we feel your pain - really - as we are at ground zero...

Our EMS is lit up like a Christmas Tree, with not so good Yuletide Cheer vibes...

--
Terra
sysAdmin
FutureQuest

[LightGuide]

Any operation that can fish-tank a DDoS in that amount of time probably shouldn't be apologizing... lol.

Thanks, Terra.

etLux

[Neo541]

mostly focused at the ENIGMA server

What did my poor little server do to deserve that? Poor thing! :(

[Deb]

Originally posted by Neo541:
What did my poor little server do to deserve that? Poor thing! :( *Ahem* from Dictionary.com

e·nig·ma ( P ) Pronunciation Key (-ngm)
n.
One that is puzzling, ambiguous, or inexplicable.
A perplexing speech or text; a riddle.


Do you actually expect an answer to that question? Sorry hun...you're on the wrong server for that :P

Deb
- Santa is not going to bring those mean Internet People any presents this year!

[phppete]

Can you find out which site was targetted? I thought Attack Mitagators were the answer to DDOS attacks?

Our main site is on Enigma :( ... probably some scumbag trying to ruin the xmas competition for someone on that server, can't be our site we aren't big enough to warrant any attention... jewellery is very seasonal so perhaps the DDOS'ers could wait until after January...lol.

As I type a new customer just signed up and ordered :) , hope we didn't lose many tonight, makes me very angry to think some little rat could cost me orders!... don't ya just hate people :P

[phppete]

Originally posted by Terra:
I am trying to run a business and this is a very busy time and I am losing money and customers.
To be blunt, we feel your pain - really - as we are at ground zero...

Our EMS is lit up like a Christmas Tree, with not so good Yuletide Cheer vibes...

--
Terra
sysAdmin
FutureQuest

When I see network issues or the words DDOS I think back to a huge thread on the Macromedia forums a few months ago. A respected host, chicagowebs were down for 4 days with a DDOS... I did use to host with them but switched from ASP to PHP a good while ago...

[Bradley]

Originally posted by phppete:
don't ya just hate people :P

Now *THAT is the Christmas spirit! :P

I do agree with LightGuide, FQ handled this situation VERY quickly, I would have never noticed execpt for the fact it started happening right after I downloaded mysql 4 (which I upgraded in 10 minutes successfully), had me worried that I broke my server ~#

Again a great big Thank you to the team for getting this situation under control..

[Benj]

The *******s !!
I’m sorry but i’m really upset about these dos attacks.
I can’t access vbulletin.com neither because they BANNED a huge range of IPS (including mine) because of a Dos attack to. For sure they are not hosted on FQ since its been over 24 hours that I can’t access their site and they don’t seem in a hurry to deban. It seems their host is totally unable to properly deal with Dos attacks.

These dos attacks are awfull. Why is that, anywhere, anytime, delinquent must attack people trying to work ?

[Mandi]

What I *would* like Santa to bring to The People (mean or otherwise - mostly just ignorant) is a load and a half of firewalls.

I see that as One Small Thing we can each do for anti-DDOS; preach Firewall Gospel wide and far (ie, away from our Tech Central web haunts) - explaining Why Everyone Needs a Firewall, and What a Trojan Horse Is (as they are often used in DDOS attacks, no?)

Thanks for the super rapid response, FQ'bies!

[Kevin]

Originally posted by phppete:
Can you find out which site was targetted?

We do know which site was attacked however we do not generally give out such information except to the site owner in question. Besides, you can't look at it right now anyways :P

--
Kevin

[Benj]

By the way, did FQ banned a range of Ips to prevent new ddos attacks ?

Cordially,
Benj

[LightGuide]

Originally posted by brnoe:

I do agree with LightGuide

Wow!

Nobody *ever* agrees with me.

Nonetheless...

Part of my business is managing sites for others -- and, of course, a number of them are on other hosting companies (much as I advise them otherwise).

I've seen these things go on for days... and Days... and DAYS... AND DAYS.

Granted, there are situations where even FQ can't clamp a DDoS quite that quick -- but they've got a helluva record for squishing the boogers in nothing short of record-breaking time.

etLux

[Terra]

I thought Attack Mitagators were the answer to DDOS attacks?
Yes and NO...

When a flood comes in, there are two primary components and remedies that presents itself...

1) Source IP (9 out of 10 times spoofed)
2) Destination IP

The only 'truth' item above is #2...

So you can either block #1 or you can block #2...

If you try to 'shape' the flow, then there is nothing that would prevent the DDoS attacker to just start modulating his attack against random IPs that the endpoint is responsible for... Now you start to have rolling blackouts all over the place... It can get ugly real quick...

In short, there is no solution to DDoS attacks - it is just a simple architectural fact of how the TCP/IP protocol was designed... These attacks are the dark side of the protocol, because just like SMTP protocol, they were designed to be resilient and to also provide and freely promote the widest arena of communication...

Zombies pretty much make DDoS attacks a reality, and the true solution to DDoS attacks are:
1) kill all the zombie machines
2) all traffic bound for the internet employ egress filtering to prevent spoofing...

--
Terra
sysAdmin
FutureQuest

[phppete]

brnoe,

My frustration is at the people causing the problem, I do realise FQ are the best host on this planet (I know that because I have been with many of the others!), I sincerely appreciate their hard work and outstanding customer support even when faced with miserable angry whiners like me!.

We are a small business competing with giants in the jewellery industry here in the UK, 30 orders a day is a lot for us so if we lose 3 or 4 due to some DDOS'er it is a large percentage of our business.

I realise many are in the same boat though, although perhaps not everyone depends on it to pay the bills.

Also for what its worth this afternoon I experienced 2 3 minute periods where I couldn't connect to my FQ email, FQ or my FQ sites... although I didnt run tracerts then so it may have been my ISP or a wider network issue.

[Bradley]

You got any of those nifty rrd/mrtg graps of the attack Terra? Just curious as to what a d/dos attack looks like on one.. :)


--
Brad
*curiosity killed the cat*

[Terra]

did FQ banned a range of Ips to prevent new ddos attacks ?
No, because there is no way to tell which IPs were spoofed, and which were not... This would just end up causing a lot of innocent bystanders to be harmed...

--
Terra
sysAdmin
FutureQuest

[Bradley]

Originally posted by phppete:
brnoe,

My frustration is at the people causing the problem, I do realise FQ are the best host on this planet (I know that because I have been with many of the others!), I sincerely appreciate their hard work and outstanding customer support even when faced with miserable angry whiners like me!.

I understand Phppete, and I do apologize for my response, I did mean to come off as trying to be rude.

Again I apologize.
Brad

[Benj]

Originally posted by Terra:
did FQ banned a range of Ips to prevent new ddos attacks ?
No, because there is no way to tell which IPs were spoofed, and which were not... This would just end up causing a lot of innocent bystanders to be harmed...

--
Terra
sysAdmin
FutureQuest

Great. Now go tell this to vbulletin's host.... ;)
I'm currently a "collateral" damage there.

[phppete]

brnoe,

No need to apologise, at the end of the day there wasn't much sense in me posting my first comment since I knew FQ would be doing all they could but you know what message boards are like, its easy to post without thinking.

Anyway, all seems fine now :)

[Deb]

but you know what message boards are like, its easy to post without thinking. ....

hmmm

Now I forgot what I wanted to say....

[LightGuide]

Originally posted by Deb:
....

hmmm

Now I forgot what I wanted to say....

I was going to tell you what it was, but it seems to have slipped my mind.

etLux

[Bradley]

Originally posted by phppete:
brnoe,

No need to apologise, at the end of the day there wasn't much sense in me posting my first comment since I knew FQ would be doing all they could but you know what message boards are like, its easy to post without thinking.

Anyway, all seems fine now :)
Very true, very true! I have been on my wits end working on how to pull this mysql upgrade off and decided to do it right about when the ddos attack happened, now everything is normal, upgraded fine, now just to find an rpm of php 4.3.3/4 to upgrade my old rpm with (I dont wanna have to muck around with red hats rpms to install from source (which I dont mind doing anymore)

:)

[Terra]

Just curious as to what a d/dos attack looks like on one..
I'm sure I can carve you off a piece from the holiday mincemeat pie...

http://www.aota.net/4F/ddos-20031216.gif

I like pie, just not this particular flavor...

--
Terra
sysAdmin
FutureQuest

[Wassercrats]

Zombies pretty much make DDoS attacks a reality, and the true solution to DDoS attacks are:
1) kill all the zombie machines
Unfortunately, that could require cooperation from ISP's, which sometimes you can't get. From http://www.dshield.org/fightback_results.php:Date: Sat, 7 Jun 2003 14:16:03 +0200

> This is an abuse notice meaning that one of your machines might
> be infected with a virus and is trying to infect other machines.
>
> See http://www.dshield.org/ for more information

We don't care, the major issue is that we don't want to receive this kind of mail, because we're a large ISP and we have no control about our multiple clients and their Windows systems.

Your mail was annoying, so we simply filtered it out. We know that many of them are infected even if we don't receive your mail.Spamhaus (http://www.spamhaus.org) has had success with iSecure (http://www.ddos.com/). You could read about the attack that spamhaus experienced here (http://www.spamhaus.org/cyberattacks/index.html).

[Bradley]

Originally posted by Terra:
Just curious as to what a d/dos attack looks like on one..
I'm sure I can carve you off a piece from the holiday mincemeat pie...
..

I like pie, just not this particular flavor...

--
Terra
sysAdmin
FutureQuest

Interesting (thanks for the pic!) I am assuming the orange/red spike would be the attack since it seems blue (incoming traffic) drops, again I am guessing at the colors as you may have them set differently..

Still amazes me how much computers are involved in everones lifes, kinda interesting to think how it'd be if we were not as advanced as we are currently.

[Terra]

Wassercrats:
ok, my all time favorite one on that page is most definitely this one...
Date: Wed, 4 Jun 2003 11:32:02 +0200

Okay, I kick his butt.
I'd sure loan that person my prized LART for a day!!!

--
Terra
sysAdmin
FutureQuest

[Terra]

seems blue (incoming traffic) drops
green == inbound
blue == outbound
orange == inbound to outbound ratio

--
Terra
sysAdmin
FutureQuest

[Bradley]

Originally posted by Terra:
seems blue (incoming traffic) drops
green == inbound
blue == outbound
orange == inbound to outbound ratio

--
Terra
sysAdmin
FutureQuest

*Checks his graphs*

Doh! I should've known that.. Thanks for the clarification! :)
--
Brad
psst.. anyone know if there is a reliable php rpm available? (prefeably latest release and built for RH ES)

[Monty]

ah come on Terra, what about...
This user has been shot.

[MichaelC]

No, no, my favorite is this guy, who despite writing...

Your mail was annoying, so we simply filtered it out. ...isn't really fooling anyone.

MC