[FQuest Emergency Maintenance] HUGO Server

[Bob]

The HUGO server appears to be experiencing a drive failure.

A Technician is on his way to the Data Center in preparation for a drive swap which will require taking the HUGO Server offline for this emergency maintenance.

We will post again immediately prior to taking the server offline.

Our apologies for any inconvenience this may cause,
Bob

[Bob]

Technicians are about to begin maintenance on the HUGO server and it will be taken offline for this procedure.

We will post once again when this has been completed.

Again our apologies for any inconvenience,
Bob

[Terra]

Overview:
Drive 1 is good, however it caused a couple resets on the SCSI bus
Drive 2 failed and needed to be replaced

The Apache engines on HUGO will be offline for approximately 35 minutes as I don't fully trust Drive 1, and want to remove as much loading as possible to better increase the chance of getting Drive 2 synched to Drive 1 before Drive 1 potentially fails as well...

Only the Apache engines will be offline, all other services on HUGO will be operational...

--
Terra
sysAdmin
FutureQuest

[Terra]

We are going ahead and replacing Drive 1 as a precaution and should have HUGO back in operation shortly...

--
Terra
sysAdmin
FutureQuest

[Bob]

The HUGO server emergency maintenance has been completed and it has been returned to Full Service as of this time.

Again our apologies and also our thanks for your patience and understanding,
Bob

[TVB]

Thanks for your quick work, folks!:bow:

[JRepici]

Thanks guys.

Thanks for the quick fix as well as the transparency about what was going on.

You'z guyz is great!

-djr

P.S. How old was the restore? Might we have lost any of today's data?

[Terra]

How old was the restore?
Backups are run nightly, so whatever changed between 6:00am and 2:54pm would have not been able to be restored...

Might we have lost any of today's data?
Yes, if Drive 1 had crashed before the replacement Drive 2 was fully mirrored... Fortunately we were able to head off such an unfortunate event by rescuing the (current live) data from Drive 1...

The probability of having both drives in a RAID-1 mirror fail in a short time range (of each other) is astronomical, however it has happened to us in the past... I should have played the lottery that day! ;)

--
Terra
sysAdmin
FutureQuest

[JRepici]

Terra,

Thanks. That's great to know.

We've just been reminded again that there's a difference between a typical hosting service and FutureQuest.

-djr

[Randall]

Wha ... who ... whup?

Dang. Missed all the excitement.

Randall