[FQuest ALERT] Network Connectivity Issues

[Bob]

At approximately 1:46 PM ET the FutureQuest network started experiencing issues with its upstream providers. Immediate contact was made to begin troubleshooting. As of this time I am not sure of what the issues may have been, however, technicians have been, and still are, working to bring the network back to full functionality. In the meantime the failed Internap link is being rerouted manually due to a problem with BGP4. As soon as additional information is available we will update this thread.

Our sincere apologies and we are all working as hard as possible to restore normal service.

-Bob

[SteveYoung]

Thanks Bob,

We (actualy you) never go down.

It is now 2:27 here and I was able to connect again (I think your time in the posting is off)

Steve

[Vairkjosa]

Got it! Scared me for a second. Couldn't connect from here.

Thanks for the quick response!

[DogAndPony]

Thanks, Bob...

I guess I should have waited another minute before posting my question (thanks for the quick link, Deb!).

It didn't occur to me that maybe you guys couldn't post, either! :smile:

Thanks for doing the manual thang!

[Edit...] Forgot to ask... Any info on how far upstream this was? I'd like to be able to tell any clients who ask whether this was exclusive to FQ's Internap link, or whether others on the Net were just as lucky...

:EG:

Thanks, guys!

[Andilinks]

It's only money... :)

http://www.andifashion.com/image1.gif

edit: updated image.

[stan]

FYI: My site was unreachable 19:45-20:26 CET. Checked from several places in Europe.

[hansm]

Bob,

Could it be that also mail is coming through very slow now?

Hans

[msealey]

We might have known you'd be onto it so quickly - as usual.

Thanks: you're the best!

Am lining up another client for you as we speak :-)

[Kevin]

Bob,

Could it be that also mail is coming through very slow now?

Hans
The outage wasn't long enough to cause a serious backlog of email however remote email servers attempting to send to our email servers would have timed out. They then would have waited a predetermined amount of time before trying again. That time is often 15 minutes however it is a per server setting so it varies wildly across the internet.

[hobbes]

Any word on the cause, fix, and future prevention?

[Terra]

Progress Telecom == Local Loop
Internap == Internet upstream

At 1:26PM ET today, a contractor working for Progress Telecom misunderstood his instructions and disconnected all network cables from a router that serviced our Local Loop to Internap...

That would not have been so bad, until our Quagga BGP daemon did not properly recognize the link going down and advertise this to our other routers... At that time around 30% of our bandwidth was going out towards Internap... Since Internap uplink was down, the packets stopped right there...

To resolve, I gained access to the routers and manually killed off the BGP daemon on the NOVA router (Internap) and manually reconfigured BGP to reroute through our backup TWTC uplink... Unfortunately during this time, our Level3 uplink had become saturated due to the imbalance... I was able to get packets flowing pretty quickly through Level3, however balancing between Level3 and TWTC took a bit longer than I had expected... At 1:46PM balance was achieved between Level3 and TWTC...

As of 2:40PM ET, our Progress Telecom local loop has been reestablished and Internap upstream is now flowing packets once again... I am working now to restore the original balance to the routers and you should not experience any further disruptions...

Work has been underway for the last two weeks in redesigning our core routers to better prevent this type of mishap and we are waiting on the new hardware to arrive... Quagga will be dropped from service in favor of OpenBGPd, which should go a long ways to eliminate this type of problem from occurring again as well as providing better overall redundancy...

Our sincerest apologies for the inconvenience this network outage had caused...

--
Terra
sysAdmin
FutureQuest.net

[hobbes]

Thx Terra.

[Andilinks]

FQ outages are so rare that when I can't get through I start worrying about the whole state of Florida. Thanks for the great service! :)

[kitchin]

Fast work, sounds like.

[Randall]

And I missed all the excitement, as usual. Nuts. :hrmm:

Randall

[Jerome]

FQ outages are so rare that when I can't get through I start worrying about the whole state of Florida. Thanks for the great service! :)

Funny, I had the same thought process when I couldn't reach FQ's homepage to check this support forum. I kept expecting to see a Breaking News flash about Florida on Yahoo's homepage! :vday2:

[Snarpy]

Excitement? You're nuts.

Time out on email....that's funny.:dunno: Try again. Same problem. Hmm. I wonder if there's a notice in the forums...can't reach the forums. :umm: Hmm. Can I view my CNC? no. Quick run through of all my sites and other FQ sites...can't reach them either. :hrmm: Is it me? My ISP? The backbone somewhere....no...I can reach anyplace else, it seems. Husband can't see FQ from work. Remote sites can't reach FQ either. Nothing significant on the Internet Health Report. Have the FL fires done something? Hmmm....well, I'm sure FQ knows and is working on it. I guess there's nothing I can do. Well, I'd like to know what's up if client X calls.:hrmm: Hmm. I wonder if the florida government site works? yep. Orlando? yep. Hmmm. Maybe its back up? Oh, I can get FQ now! And look, there's a message from Bob. And an email in my notice-subscribed gmail account. :yeah:

[Andilinks]

Excitement? You're nuts.I agree, I would rather have been out to lunch when all this happened. Overall traffic and earnings are not noticably down, without the hourly graph or these threads I'd have been blissfully ignorant of the whole event.

[Randall]

Excitement? You're nuts. Hey, it was a slow day at work. So slow that I had time to reconcile the Chamber's checking account ... going back to last August.

Thought I subscribed to this forum, but I didn't get an email. Hmm. :ytthink:

Is there any visual indication that you're subscribed to a forum? All I can see are my subscribed threads.

Randall

[DogAndPony]

At 1:26PM ET today, a contractor ...whose name is now "Mud"...working for Progress Telecom misunderstood his instructions and disconnected all network cables from a router that serviced our Local Loop to Internap...Isn't there some sort of dubious award for this kind of behavior? Like... the IT (and non-lethal) equavalent of the Darwin Awards? :clown: That would not have been so bad, until our Quagga BGP daemon did not properly recognize the link going down and advertise this to our other routers...Just goes to show, it pays to advertise!To resolve, I gained access to the routers...by sitting down in front of the laser and being digitized and sucked into the network, Tron-style... and manually killed off the BGP daemon on the NOVA router ...by throwing his disc at it with great power and accuracy!

Long live the Admin!

Long live Terra-Tron! :yeah:

[Terra]

:sillylol:

We love you Bob!!! :vday2:

Thanks everyone for not beating us up over this, your understanding is very much appreciated! :ytout:

--
Terra
--Whom should have known better when walking in this morning and greeted everyone with a smile and exclaiming "Today is a good day" - Murphy was not amused--
FutureQuest

[Snarpy]

Is there any visual indication that you're subscribed to a forum? All I can see are my subscribed threads.Just click Options - subscribed forums are underneath new subscribed threads and above latest reputation received.

[cindik]

FQ outages are so rare that when I can't get through I start worrying about the whole state of Florida. Thanks for the great service! :)
Same here. I immediately started checking Orlando news.

[hobbes]

Thanks everyone for not beating us up over this, your understanding is very much appreciated! Hey, no problem. I just auto-forwarded all incoming client calls to the toll-free QuestLine.

[Jerome]

......by sitting down in front of the laser and being digitized and sucked into the network, Tron-style......by throwing his disc at it with great power and accuracy!

Long live the Admin!

Long live Terra-Tron! :yeah:

This was really good. :smile:

[DogAndPony]

Aw, shucks, folks... Just trying to do my part for morale. Kinda like a poor man's Bob Hope for the USO of web hosting. :smile:

[Randall]

Just click Options - subscribed forums are underneath new subscribed threads and above latest reputation received. OK, got ya. I had News & Announcements marked, but not Notices & Alerts...

Randall

[Andilinks]

I just subscribed to the rss feed:

http://www.aota.net/feeds/news.xml

As recently discussed here:

http://www.aota.net/forums/showthread.php?t=21362

You should check that out if you use a feed reader.

[Snarpy]

I've subscribed to both, but I always get the email before the feed. I guess I should adjust my feed settings or something. The email is so fast that the feed seems superfluous, so I haven't bothered.

[Matt]

It occurred to me when this happened that there was NO way for FQ to communicate with its users. The FQ network was down, the forums were down, and e-mail was down. I know of at least one hosting company that has set up an emergency announcement site off-site just for this sort of thing. In this particular case, it wasn't that bad since the network wasn't down for a very long time (though I saw about a 3.5 hour or more delay on e-mail working again properly), but it could have been (especially since ALL communication is through the internet).

So, I very much hope that FQ will set up some kind of emergency site where users can go when FQ "disappears" from the internet. -MAtt

[Andilinks]

...some kind of emergency site where users can go when FQ "disappears" from the internet. I thought that was a part of the Colorado strategy because of its central location. Maybe I should go back and read those original announcements, it seems like it was a thousand years ago...

[Randall]

It is disconcerting when everything goes down like that. Happens more frequently with our NT hosts, but at least the phones still work. :wink:

Renting some rackspace at another facility might not be such a bad idea. Sorta like the Bat Signal. :winky:

Randall

[Andilinks]

Renting some rackspace at another facility might not be such a bad idea.Sounds like the perfect barter to me...

[Jeff]

If NS1 and NS2 are in Florida, what would be the effect of having NS3 in
1.) Colorado (central location) or
2.) California (to have nameservers closest to both east coast and west coast customers)?

Is geographic diversity for NS servers an advantage, or is simply being right on a major backbone the most logical?

[DogAndPony]

Is geographic diversity for NS servers an advantage, or is simply being right on a major backbone the most logical?I would think that in the case of a layered-redundant server like an NS3, geographic diversity would be less about what/who the server's close to than about how far from the affected servers it is.

I'm in favor of putting a status server in Colorado... And I'd also think that using a separate, unique domain for it -- with its NS living at a third-party provider -- would be a good idea.

It might also be good to enable that status server to send automated alert emails or text messages on request.

Fortunately, I only had one client notice the outage, but she was a bit freaked out. And speaking of eggs in one basket... Her panicked email couldn't get to me due to the outage... And I wouldn't have been able to reply. So I'm now thinking about my own offsite status and notification domain and server somwhere.

As incredibly trustworthy and reliable as the talented humans at FQ are, there are still those upstream monkeys to think about. :EG:

[cindik]

upstream monkeys

I saw that band. They opened for Terrycloth Lobster.

[DogAndPony]

I saw that band. They opened for Terrycloth Lobster.Oops... You forgot the link:

http://www.terryclothlobster.com/

:yeah: