[FQuest Alert] Network Outage

[Bob]

The FutureQuest network suffered an outage starting at approximately 10:35 AM ET. Technicians were alerted and working on the problem immediately.

At this time they are continuing to work and as soon as additional information is available we will post that here.

Our sincere apologies for this issue,
Bob

[hobbes]

Once again ... a request for an FQ update site hosted outside the FQ network. Phone systems also need to be upgraded...

[farlane]

Once again ... a request for an FQ update site hosted outside the FQ network. Phone systems also need to be upgraded...

echo ... echo ... echo ... :kewl:

I agree.

[phppete]

Once again ... a request for an FQ update site hosted outside the FQ network. Phone systems also need to be upgraded...

I agree, even a blospot page or something similar would do, smoke signals would be better than what we have now. As soon as I noticed the outage I read through some Nostradamus Quatrains to see if there was a cryptic clue as to what was happening. Ideally I would have had a bookmark to a non FQ update page.

[Terra]

The only thing that has changed to our core routing is that I was working on rate limiting UDP floods that are spawned from a site owners account which had been cracked via their XOOPS system...

The rate limiting has been pulled for now, and I will re implement it next weekend ...

On the subject of external status page, we are still looking for a viable host that we would be comfortable with putting an external DNS system with... Changes have to be made to our existing DNS system that would prevent our internal systems from going to the external source for lookups, and these are non-trivial updates...

Our phone systems will not be updated, as we do not provide support via phone... When an outage occurs, our work areas light up like Christmas trees and we are quickly on top of the problem... Our monitoring systems alert us much faster than a human picking up the phone to poke us adding to the stress of trying to focus on the problem at hand...

In conclusion, please be patient with us in the event of a network issue, as we are already feverishly scrambling behind the scenes to restore service in seconds or minutes, rather than what you expect from utility companies that may take hours or days...

--
Terra
sysAdmin
FutureQuest

[farlane]

I appreciate the fact that you are very fast and that added communication is pretty much a nuisance when dealing with a problem.

From the perspective of the site owner / reseller, it would be nice to have a brief message that we could use to respond to the requests we are receiving.

[Terra]

From the perspective of the site owner / reseller, it would be nice to have a brief message that we could use to respond to the requests we are receiving.
Understood, sometimes I don't realize that I'm not the only one in boiling hot water... :(

FQ update site hosted outside the FQ network.
Bob has a viable idea for this, and we are investigating this avenue now...

--
Terra
sysAdmin
FutureQuest

[hobbes]

Terra,

The issue is when an outage occurs, we have no idea how long the outage will be. With no outside source of information, the reason I call, is so I can get on the queue to at least talk with someone and get some news that I can pass on to the 30+ clients I have hosted at FQ (who are calling us to find out what's going on).

As noted in my reply to Bob's email, it took me: 15 minutes to get through on the phone. Between busy signals, dropped rings, and an odd ringing every 3-5 seconds, the experience was incredibly frustrating. This also happened during the last outage.

If FQ is serious about providing a business-class service (as has been my impression vs. hosting for hobbyists), then these issues need to be addressed vs. waiting for the next outage for us to rehash the same requests.

[kmc500]

Understood, sometimes I don't realize that I'm not the only one in boiling hot water...

Ditto what farlane and hobbes stated. We have 500 users in 13 different locations that could not get their email or work on their web sites, even though their internet connection was working fine.

We generally let our users think we know exactly what is going on when there is an issue, as it fosters some level of confidence, and so far we've been okay doing that since FQ is so seldom down.

It sure would be nice (and help pr internal IT confidence level) to have that brief, potentially changing message that Farlane mentioned.

Kevin C

[Kevin]

Hobbes, as Terra said we don't really provide phone support. When there is an outage like this we are not going to stop working on it to answer the phone and when the problem is solved we don't want a hundred voice mails to return all with the same answer.

You can expect that whenever our internet connections are down so is our voice mail phone line.

The notification of an outage and recovery will be done through a web interface (which hopefully will soon exist) but not the phone.

[hobbes]

Kevin - Bottom line, there needs to be a means for FQ clients/resellers to get information in the case of an FQ network outage. The current system is broken in this regard, and it sounds like I'm not the only looking for a fix. As much faith as I have on the FQ team, just knowing you're working on it is not good enough. Having a communication black hole is not a good business solution for you or us.

[cindik]

It would be great if there was just an outgoing-message-only number one could call that usually said "all systems are operating normally" but would occasionally say "we are working on a problem we hope to have resolved by 0000 UCT".

Single message recorded, many customers reassured.

[Judy Doherty]

Hobbes, as Terra said we don't really provide phone support. When there is an outage like this we are not going to stop working on it to answer the phone and when the problem is solved we don't want a hundred voice mails to return all with the same answer.

You can expect that whenever our internet connections are down so is our voice mail phone line.

The notification of an outage and recovery will be done through a web interface (which hopefully will soon exist) but not the phone.

I was able to get through to customer support during the outage via email using my gmail email account - Bob answered right away.

I was also going to suggest using Peter's site - you guys at FQ recommended him to me a while back because I needed an additional site that is <horrors> on an MS server. I have been using him for that for a few years now and have been quite happy with his service. He has similar service to FQ and I bet he would let you do that. www.commkal.com

[Kevin]

I actually thought of the recorded phone message idea a while back however there are several problems...

1) Availability. If we use a land line only one person can call it at a time so most people would get busy signals during a major problem. If we use VOIP it isn't going to work during an outage anyways.

2) Cost. A single business phone line is ~$50/month. It would get even more expensive if we made it toll free. The costs for multiple lines go up from there.

3) Updates. Normally there isn't anyone at the DataCenter. The recorded message wouldn't be updated until I actually drove in and updated it manually. My time would be better spent at my desk working on the problem.

[shatty925]

personally, I don't want or need phone support. however, i do like that it sounds like you are using voip (yes?) and not a traditional phone system.

i would like someplace I can visit for info when there is a problem. it is a bit unnerving to not be able to access any fq site.

[Kevin]

Yes, the QuestLine voice mail number is in fact VOIP. This allows us to record your messages directly onto a computer where any of us can listen to them through email or a web interface and it allows us to return calls from our remote offices with the correct CallerID number and on the companies dime.

The old way was much more kludgey. It involved things like a plain old answering machine, a web cam, and monthly submitting of phone bills to get reimbursed.

[ElvisFrog]

As a reseller I want to let you know that I think you guys to a GREAT job and when there is a system outage I don't hesitate telling my clients that it will be down for less than an hour as with the history of the FQ Team it rarely takes longer. Most of the time it is back just as my clients notice it being down.

A play by play of events is always useful to the tech community but I don’t think is necessary. A simple status page with the acknowledgement of an issue and maybe an eta would suffice. One thought is for resellers to sign-up to an internal off site email system with an alternative email address that would send out a broadcast email of an outage. Heck it could just be a Gmail account.

Again, I can’t stress enough how comfortable I am with NOT knowing what is going on. Though I feel that stress of unknown I have NO doubt that FQ is working as fast and effectively as possible to resolve the issue and that it is being handles with great efficiency than any other organization of its type.

So in closing. Good Job Guys! :clapper:

[Buck]

3) Updates. Normally there isn't anyone at the DataCenter. The recorded message wouldn't be updated until I actually drove in and updated it manually. My time would be better spent at my desk working on the problem.

Our local phone company offers us a Voice-Mail announcement line for $7 a month. It's just that, a voice-mail only phone line. Our Ski Club uses it as their hotline. Call it, listen to the current message, hang up. To change the message, we only have to call the number, dial a certain sequence, and we can record a new message anytime.

A nice, cheap, easy solution to a problem like this.

[Andilinks]

There are cheap easy solutions to the problem which get discussed every time this happens and every time they are rejected.

I'm losing faith that FQ is up to the job of hosting if they cannot address the simple task of providing alternate communication when these increasingly common outages occur.

I'm beginning to sense a downward spiral, I hate to see this happen.

[Bob]

Hello Andi,

As Terra and Kevin have both posted earlier in this thread we are working on an External Notification Solution.

We have been actively looking for solutions for some time, it is not something we have simply rejected, but sometimes the trees get missed for the forest, or is that the forest gets missed for the trees?, in any event maybe we were looking too hard :dopey:

A post by one of the site owners gave me any idea for a simple solution and we are working to provide that in the event a FutureQuest wide situation occurs again.

As soon as the details are available we will post them in an Announcement.

-Bob

[Wassercrats]

I noticed an outage last night, I think a little before midnight. My tracert showed a delay of over 1000 ms in the last hop.

[Bob]

FutureQuest externally hosted Network Status Page announcement.
http://www.aota.net/forums/showthread.php?postid=157746#post157746

-Bob

[Andilinks]

Definitely a bite out of that 10 am bar but if I hadn't read this thread I probably wouldn't have noticed it. Thanks for fixing it quickly. :)

http://www.andifashion.com/graph.gif