[FQuest Alert] ZOOMER Emergency Repairs

[Terra]

ZOOMER is suffering from a failing hard drive, and efforts are underway to get this server back under control and prepped for a drive swap...

Ok - update, the server has punted out the bad drive from the RAID array and loads are coming down quickly...

We will post a few minutes before we reboot the server...

--
Terra
sysAdmin
FutureQuest

[Terra]

ZOOMER is going offline for drive replacement procedures...

--
Terra
sysAdmin
FutureQuest

[Terra]

ZOOMER is back online and the RAID partitions are rebuilding...

Our apologies are any inconvenience this has caused on such a beautiful Sunday morning...

--
Terra
sysAdmin
FutureQuest

[LightGuide]

That was quick. Last time I yanked my hard drive, it took ten hours, not ten minutes.

However, this raises an opportunity for a question about how restoration works on FQ servers in a circumstance like this.

For instance, we made massive changes, site-wide, including many back-end scripting fixes and upgrades, late into the night last night.

Yet all of the changes appear to be present and accounted for after the ZOOMER repairs.

I was dreading possibly having to go and redo half of what we'd done; yet this was not the case.

[Terra]

The Speed, Power, and Flexibility of RAID-1... :QTcylon:

Yes, it doubles the cost, but when it counts - it is priceless...

See, it even surprised and amazed you... You can't buy that kind of marketing _anywhere_...

The end result == another satisfied client...

--
Terra
--where stuff is just simply expected to work--
FutureQuest

[Jeff]

And a great argument for having RAID in our next home system too :)

--laptops with RAID?

[LightGuide]

Thanks, Terra.

I never knew it was quite so quick in a failure situation.

etLux
http://www.killsbugsdead.com/

[GregJ]

Terra,

Would you mind giving your views on the various RAID levels? Particularly your views about RAID 0+1 versus RAID-5. My understanding (very limited) is that RAID-5 has parity checking and 0+1 does not. I take it from the discussion above that FQ uses RAID-1. Also, is there a difference between 0+1 and 1+0 (I vaguely recall a reference to a difference)?

And while I'm at it, your preferred method to handling a drive failue it not to hot swap because of Linux instability. Is this a problem if you have a hardware controller managing the entire RAID?

For low performance requirement systems, do you have any thoughts on the IDE/ATA raid cards now being sold? I want data reliability over speed and need to keep costs down.

Thanks for your efforts at teaching the unwashed masses...

-g

[Andilinks]

I do anticipate Terra's answer to Greg's question, I too am interested in the "Redundant Array of Independent/Inexpensive Disks. But in the mean time I did find an ExtremeTech article dated 4/18 titled "RAID Goes Mainstream, But Is It Safe?"

http://www.extremetech.com/article2/0,3973,1034025,00.asp

Probably a good primer as we await Terra's response.

Andi

[Terra]

Wow GregJ, you sure are the curious one... :P

Would you mind giving your views on the various RAID levels?
Sure, I'll see what I can do...

Particularly your views about RAID 0+1 versus RAID-5. My understanding (very limited) is that RAID-5 has parity checking and 0+1 does not
It all comes down to what type of traffic is going to be sent to the array and it's overall duty...

RAID-1 == fast but less storage capacity due to requiring 2x amount of hard drives... If properly optimized, reads (seeks) can come from either of the drives sometimes doubling performance...

RAID-5 == slow (parity is expensive) but more storage capacity as capacity is measured as the sum of all drives - 1...

I take it from the discussion above that FQ uses RAID-1.
In the Community Servers, we use RAID-1
On the server that drives these forums, we use RAID-10 (RAID 1+0)

We do not currently use RAID-5 in any of our servers...

Also, is there a difference between 0+1 and 1+0 (I vaguely recall a reference to a difference)?
Yes, there is a difference:

A good article that I dug up on this can be found at:
http://www.ofb.net/~jheiss/raid10/

And while I'm at it, your preferred method to handling a drive failue it not to hot swap because of Linux instability. Is this a problem if you have a hardware controller managing the entire RAID?
Well, I wouldn't say Linux instability, but more of playing it safe... The other issue is that the Linux SCSI driver may not disconnect the drive from the bus if it is active or the driver is trying to issue scsi ABORT methods...

Hardware RAID usually handles hardware disconnects better than Software RAID does but then again it all depends on the underlying SCSI driver...

As it stands, doing cold swap RAID repairs is just a matter of policy and I feel it is the safest approach to take for our servers and well being of the data these servers hold...

For low performance requirement systems, do you have any thoughts on the IDE/ATA raid cards now being sold? I want data reliability over speed and need to keep costs down.
Well, unfortunately I cannot comment much on 'low' performance systems... I don't really work within that realm... :P

My IDE RAID card of choice are 3Ware controllers, which are high performance...

You must ask yourself, if you were to lose all of your data in the blink of an eye - how ticked off would you be? How much is your data worth to you? If you run backups, how much is losing a day/week/weeks worth of work to you?

I find that losing data is pretty much the same as breaking up with your girlfriend (or boyfriend) and then trying to get back together - it never really is the same no matter how much effort one may put into rebuilding the relationship...

Thanks for your efforts at teaching the unwashed masses...
I can recommend either Caress or Irish Spring bath soap... :QTbarb:

--
Terra
-- http://www.Aota.net/4F/100bil.wav --
FutureQuest

[Syneryder]

I find that losing data is pretty much the same as breaking up with your girlfriend (or boyfriend)
Wow, not only can Terra explain difficult hardware topics like RAID succinctly and clearly, but he's also got a bright future writing the Relationships For Geeks™ book series... I'm impressed! :D

[Mandi]

I was going to say that this is what I love about FQ community . . . it's not so much that I can't get you all to understand how I might feel about my partner . . . . it's that I can't get my partner to understand how I feel about my data!

[Deb]

I'm just trying to get him to agree to NOT test his theory with me! :QTtongue:

[GregJ]

Thank yew veddy much.RAID-5 == slow I took a gander at the 3Ware 7500-x performance specs and noted that RAID-5 is about 1/2 as fast as RAID-0. I gather that a RAID-10 would have the same performance speeds as RAID-0 and have the benefits of mirroring also.

If I read it right, the RAID-10 would require 4 drives at a minimum. Because of the mirroring, drives would need to be added in pairs. There was discussion on the vendor site saying that the drives should be all alike (drive configuration and speed), preferably the same vendor & model.

More questions (ever the curious one ;) )
Is it enough that each mirrored pair be alike, or do all of the drives in the array need to be alike?

What about upgrading the drives in the array? How does one go about that? Do you have to replace all of the drives at once (backup then restore) or is it ok to just replace mirrored pairs.

I remember on the Big Iron I used to work on, when we wanted to upgrade drives, we could add in new drives to the group and turn (physically & logically) turn it on. Later we could tell the system to 'drain' one of the old drives and have it migrate the data off of the drive so we could remove it from service. When it was empty, we could logically turn it off, then physically remove the drive. All without shutting down the system.

Are there equivalent functions in the RAID Management software that can accomplish similar actions? It would be nice to be able to upgrade drives in an array without having to take the system down.

-g
# smell better already

[Kevin]

Originally posted by GregJ:
More questions (ever the curious one ;) )
Is it enough that each mirrored pair be alike, or do all of the drives in the array need to be alike?
If you use drives that are different they will all be downgraded to the size of the smallest drive. Meaning that any extra space on the larger drive will be ignored. The RAID vendors recommend the same manufacturer/model drives because they are all slightly different sizes. IOW a 40GB Maxtor and a 40GB Seagate are not the exact same size.

What about upgrading the drives in the array? How does one go about that? Do you have to replace all of the drives at once (backup then restore) or is it ok to just replace mirrored pairs.
In order to get a size increase you would have to replace all of the drives or add more drives. The best way to do that is to have 2 arrays in parallel and migrate the data over.

I remember on the Big Iron I used to work on, when we wanted to upgrade drives, we could add in new drives to the group and turn (physically & logically) turn it on. Later we could tell the system to 'drain' one of the old drives and have it migrate the data off of the drive so we could remove it from service. When it was empty, we could logically turn it off, then physically remove the drive. All without shutting down the system.

Are there equivalent functions in the RAID Management software that can accomplish similar actions? It would be nice to be able to upgrade drives in an array without having to take the system down.
It depends on the RAID system.

[Randall]

The info in this thread is much appreciated. I've read a little about the various RAID configs, but reinforcement is always a good thing.

My hypothetical Computer for People with Too Much Money® would require a RAID 1+0, no question -- performance and safety are what it's built around (well, in my head at any rate).

Randall

# We're taking orders now, don't delay!