SSD or RAID?

[Randall]

Despite my aversion to using RAID for a system drive, I'd be willing to consider it for our data files as long as we have redundant backups. We have lots of database activity and whatnot, and even after we upgrade to gigabit ethernet I expect the hard drives will still be the main bottleneck.

This review (http://www.notebookreview.com/default.asp?newsID=4387) of a Crucial SSD has me intrigued, though. Scroll down to the end and make note of the HD Tach test results.

The drives in our office fall squarely in the neighborhood of that 7200rpm HDD they used for comparison -- ie, half the average read speed of the Crucial drive, which is even more pathetic when you recognize that the HDD's average is propped up by faster throughput (still well below 99MB/s) at the beginning of the disk. And a seek time of 0.5ms is impossible to beat.

So the question: Is a RAID 10 system of equal or lesser cost competitive with this SSD? We would need four drives (not especially large ones for our purposes -- a 32GB SSD would be out of the question otherwise) in an external Firewire 800 or eSATA enclosure, because the PC has room for only one additional drive.

(Some of you will recall that we already have a 3ware RAID card that we're using as a mere IDE controller replacement, but it can't do RAID 10 -- only got 2 ports.)

In either case we would be using a PCI controller card, since our server machine doesn't have onboard SATA or Firewire.

Randall

[Kevin]

I really would love to try an SSD sometime but I just can't justify spending that much money for that small of a storage device. I do have hopes that they will become more cost effective in the future but right now I think they are only for special cases where you just plain can't handle the seek rates of even a 15k RPM disk.

[Randall]

I really would love to try an SSD sometime but I just can't justify spending that much money for that small of a storage device. That had been my thinking too. But when I considered that our data and network-based applications -- the programs that slow us down the most -- currently top out at 14GB, a 32GB SSD doesn't sound so unreasonable. By the time we fill it up, there will be bigger drives at lower price points.

And I have to admit that I'm a sucker for anything that replaces spinning platters with flash RAM. :dopey: The only part I'm unsure about is the write latency. I do have hopes that they will become more cost effective in the future but right now I think they are only for special cases where you just plain can't handle the seek rates of even a 15k RPM disk. I could be wrong, but I suspect that a 15Krpm drive would be a little noisy for an office environment.

There's another interesting storage technology out there, but it's even more expensive than a flash drive: the DDR SSD (http://www.hyperossystems.co.uk/07042003/hardware.htm). Limited to 16GB (there's a vague reference to a 32GB model, but I don't see it in their product listing); you have to use RAID for higher capacities.

And yeah, $2,390 seems kinda steep for 16 gigs. The RAM itself is the least expensive component. :blah:

Randall

[Jeff]

I could be wrong, but I suspect that a 15Krpm drive would be a little noisy for an office environment.
I think you'd have to be pretty darn sensitive to notice just a couple 15k drives... (and not have any air conditioning, forced air heating, etc.) - they're quieter than my typical tower computer fans

Then again, I'm sitting 12" from the 8 7.2k and 2 15k drives. I like the hum to let me know they're on. If you did do raid 10 and go to all the expense of an enclosure, controller etc, I say go for as many drives as possible to really boost the performance (unless you don't have a pcie/x slot available so are constrained anyway.)

[Kevin]

It depends on what kind of 15k drive.

I just bought a 36GB Seagate Cheetah 15k.4 disk (not that expensive on eBay) and it makes slightly less noise than most 7200 RPM IDE/SATA disks. On the other hand the old model Fujitsu 15k RPM disk it replaced was horribly loud. In fact the Fujitsu is so loud I doubt I will even use it for anything else.

[Jeff]

Yea, the 15k drives I'm sitting next to are cheetah 15k.5 sas drives - paid about $200 each for them and they seem plenty quiet to me. My previous atlas 15k2 has more of a whine on spin up.

[Kevin]

The SCSI ones are only around $60-$80 on eBay.

[Randall]

If you did do raid 10 and go to all the expense of an enclosure, controller etc, I say go for as many drives as possible to really boost the performance (unless you don't have a pcie/x slot available so are constrained anyway.) Plain-vanilla PCI, I'm afraid.

If upgrading to a machine with PCIe and SATA would have a cost benefit compared to building a RAID system with the existing equipment, that might be a consideration.

But my question remains: Can a RAID 10 setup compete with the Crucial SSD on price and performance? It looks like a SCSI RAID enclosure would cost more than the SSD all by itself, which limits me to SATA. No Cheetahs for you.

SATA II drives can be had for about $40. Four-bay eSATA Enclosures seem to start at $200 -- I see lots of CFI enclosures, but are they any good? PCI eSATA cards that support 3.0GB/s seem few and far between, but I see prices around $50. So roughly $410 for the complete setup, unless I've missed something. The Crucial SSD would be $600 plus the SATA controller.

I'm out of my depth on RAID systems, so I'm looking for some insight on their limitations. The SSD is one piece of hardware, and I think I know what to expect from it.

Oh, and one other thing: 32GB flash chips are on the way (http://arstechnica.com/news.ars/post/20080529-intel-intros-32gb-flash-chips-targets-solid-state-drives.html), so prices could drop in the near future. Or not.

Randall

[TigerLilly]

While you are making this decision don't forget to take into consideration how you will destroy the drive when it becomes obsolete. :wink:

[Kevin]

I wouldn't actually suggest SCSI hardware RAID. It is usually only needed when doing parity calculations like with RAID5. The rest of the time software RAID is just as good or good enough.

In fact my fancy new 15k RPM SCSI disk is connected to an ancient Adaptec 2940UW controller that is capped at 40MB/sec across all devices and it is still faster than a 7200 RPM IDE disk that benched at ~60MB/sec. The reason is that for what I am doing with that disk the latency of the disk seeking is way more important than the sequential read or write speed.

Even though I have never actually gotten to play with one I have no doubt that an SSD would be a bigger improvement but the huge improvement of the 15k RPMs is good enough for me and way cheaper (I already had the SCSI controller).

Also btw, IMHO large capacity SSDs haven't been around long enough for me to feel safe putting important data on them. I know they aren't going to have a head crash or a bad motor or something like that but they can still get fried or zapped. So I would probably still want to use them in a redundant RAID configuration if they were going to hold anything important.

[Jeff]

But my question remains: Can a RAID 10 setup compete with the Crucial SSD on price and performance? It looks like a SCSI RAID enclosure would cost more than the SSD all by itself, which limits me to SATA. No Cheetahs for you.
With eight 7.2k sata drives I'd be all for it -- I like what I get
http://islanddesign.com/temp/Adaptec-31205-RAID-10-array.jpg

I'm not sure if I'd go to all the trouble for only 4 (meaning striping only 2 and I'm assuming about half that performance) if you don't need lots of space that it can provide -- I think it would be on-par, but with only 4 sata drives you're pretty close to the performance of a single 15k cheetah but with higher latency.

[Randall]

With eight 7.2k sata drives I'd be all for it -- I like what I get Eek. Does it take that many drives to get high data rates out of a RAID 10?

Eight drives ... the performance is impressive, but I notice that it doesn't improve on the seek time the way I thought it would. Like Kevin, I think that's going to be more of a factor than data throughput (though I won't really know until I try an SSD or a Cheetah). And with double the number of drives and a bigger case, you're probably erasing its price advantage over a (small) SSD.

I'd also worry about heat output and/or fan noise (it'll be sitting near Boss #2's desk) and power consumption (our rent just went up with the new lease we signed this week).

As the review pointed out, SSDs run hotter than we've been led to believe, so I have to wonder just how energy efficient they really are -- might not be much better than a 7.2k drive, or even a Cheetah. But eight drives (or even four) could possibly be an issue.

Randall

[Kevin]

The seek time on hard drives is a function of the mechanical parts moving to the right spot on the disk. First the heads have to move to the right track then you have to wait for the platter to spin to put the right sector under the head. The higher the RPMs the faster the second part is plus I bet they put faster and more expensive seekers in there too. The latency of the seeking has nothing at all to do with the transfer rate benchmarks.

With SSDs the latency comes from the electronics looking up data from a specific address and then translating it into whatever protocol the SSD speaks. The latency should be lower on SSDs because they have no moving parts adding physical latency.

Unfortunately RAID rarely affects latency. It may occasionally allow the disks to multi-task better but that is only when things happen just right. The only performance boost you get from RAID is increased transfer rates.

If seeking is really the problem you have two choices: lower latency storage devices or do less seeking.

If you can rearrange the data so that things are more sequential then there will be less seeking and the transfer rate becomes more important than the seek rate. Also, if you can prevent or reduce filesystem fragmentation that will have a similar effect.

[Randall]

Unfortunately RAID rarely affects latency. It may occasionally allow the disks to multi-task better but that is only when things happen just right. The only performance boost you get from RAID is increased transfer rates. Ah, I thought the multitasking aspect would be more effective than that. If seeking is really the problem you have two choices: lower latency storage devices or do less seeking.

If you can rearrange the data so that things are more sequential then there will be less seeking and the transfer rate becomes more important than the seek rate. Also, if you can prevent or reduce filesystem fragmentation that will have a similar effect. Unfortunately the databases are all proprietary, so there's precious little I can do about them. Sometimes the files can be cleaned up for better performance, but that's about it.

Defragging may be worth pursuing -- one downside to cloning drives with True Image or Ghost is that you copy all of the fragmentation with it.

The databases aren't the only problem. Waiting (and waiting) for applications to launch wastes a lot of our time, and I'm sure latency is a factor there too. All those DLLs and stuff to load.

So I imagine that an SSD RAID with 0.5ms seek times would be insanely fast if you could afford it, at least on read performance. A DDR SSD RAID ... best not to think about such a thing, or we'll blow our technology budget for sure. :dopey:

Randall

[PaulKroll]

Several things:

If the SSD is mostly being read, great, but keep in mind that SSDs can wear out if written to a lot. Now, "a lot" gets really hard to define, since even a small write of 4K might be writing a 2 meg block of SSD (because that may be the minimum block size for the device). There are different types of flash RAM too, and there's an order of magnitude difference in write lifetime between them.

If you haven't defragmented the database files... ever... then do nothing until you do that. Really, a fragmented database file is a nightmare. They should be periodically defragmented, at the filesystem level, even if that means that you need to take them offline... after doing a backup... Many databases also have a "defragment" of their internal structure, and if it's available, that's probably needed TO, but not instead of, filesystem level defrag.

On Windows I defrag with JKDefrag (http://www.kessels.com/JkDefrag/), mostly because I can script it to death and it's open source.

You already have, I take it, added all the RAM the computer can handle right? It's So, So cheap these days...

Should we actually be talking about this specific request, about SSD versus RAID, or should we be talking about "we've got this set of circumstances and we've done this and what else can we do to speed it up?" Because that might lead to solutions outside of the box, to drag corporatespeak into this.

Also, love the "Noble, stoic" pic. :) I expect above it to see "Randall. Mission Impossible IV."

[Randall]

If the SSD is mostly being read, great, but keep in mind that SSDs can wear out if written to a lot. Now, "a lot" gets really hard to define, since even a small write of 4K might be writing a 2 meg block of SSD (because that may be the minimum block size for the device). Wear-leveling techniques should be mature enough by now that we'd probably fill the disk before it started to wear out.

Seems there are some more expensive SSDs can write as fast as they read, so I need to do more research in that area...

In the meantime, I'll see what JKDefrag can do for the HD.

Randall

[Jeff]

Since we're only talking about 32 GB, would a RAM drive be a viable option to really increase performance by an order of magnitude?

[Randall]

Since we're only talking about 32 GB, would a RAM drive be a viable option to really increase performance by an order of magnitude? The two I've seen only hold 16GB, so you need to gang them up in RAID to match higher-capacity SSDs. And one of them was going for $2,390 unpopulated. Aiyeesh.

But the concept does have its appeal. Makes me wonder if someone will write (open source?) software to turn a RAM-packed PC into a DRAM SSD.

Probably would still cost a fortune for the hardware, though. :blah:

Randall

[PaulKroll]

if someone will write (open source?) software to turn a RAM-packed PC into a DRAM SSD
Hmmm... you know, I'd bet it'd be relatively easy for someone to write software to make SEVERAL PCs into a DRAM based SSD... the same way they wrote memcached. In fact, there's probably a lot of code from memcached that would be appropriate to such an endeavor.

Then again it may all be moot in two years, they way the SSD folks are pushing.

Edit: Under Linux, you could raid hard drives and partition only the first section of the discs, which is A) the fastest part, and B) cuts down seek time tremendously. But you waste most of the drives in doing this.
Edit 2 Electric Boogaloo: Not counting video cards I get $1,764 for 4 machines with 8 gig of RAM each, including cases and processors, so heck... (they'd have to boot via ethernet/no hard drives in those costs, but still less than the price you quoted for the known system.)

[Jeff]

Hope so on the SSD front -- with slim laptops and devices, maybe they'll finally be the commodity to push it to the next level. But there's a long way to go from the current generation "up to 100MB/s (read), 35MB/s (write)" vs. 15k hard drives at 1/4 the price delivering up to 125 Mb/s vs. Ram speeds -- looking at a supermicro motherboard now which advertises "Up to 21.3 GB/s Memory Bandwidth (w/ 4+ DDR2800 Modules)" Quite a gap there...

Though products like gigabyte's iram never took off to offer a easy ram drive, possibly too limited in capacity & flexibility at their pricepoint.

[PaulKroll]

Yeah, those SSD's ain't gonna fight RAM anytime soon. :) But hard drives, on the low end? Those are gonna go away.

[Kevin]

Of course the big difference between SSD and RAM is that SSDs don't forget everything when you power them off.

[PaulKroll]

I feel like we've come full circle. :)

[Jeff]

Is there a way to do an rsync-like backup every x-minutes that wouldn't collide with active user read/write requests? I would think with the huge performance advantage over physical and flash drives now, it would be possible to run a very frequent backup and still have a huge performance margin.

> But hard drives, on the low end? Those are gonna go away.

Agreed. It feels odd carrying a laptop around knowing there is a physical head skimming the surface of a disk... the thought of a microdrive spinning in my digital camera years ago also seemed odd... at the same time it still amazes me that physically moving a head is faster than current flash technology... hopefully that will turn around any month.

[Kevin]

For that you may as well just use a really big disk cache.

[Jeff]

just use a really big disk cache.
Does such a thing exist in the 32 GB range? Would linux simply handle such a task (keeping cache fresh, ahead of demand etc.) automagically with buffers/cache using excess RAM as efficiently as writing to a ram drive and handling backup in the background?

I thought about getting an areca raid controller which offered 2 GB of onboard cache, but that was the max I've seen in the realm of affordability.

[Kevin]

In Linux the available RAM is used for a disk cache however it isn't psychic. If you read data that is in the cache it will read really fast. If you read data that isn't in the cache it will not be. Data is generally only in the cache if it was read or written recently.

On the writing side of things the cache will cache writes to an extent. With journaling filesystems like ext3 that extent is usually until the journal fills up which is usually only 32MB. You can tweak the journal size to be much bigger but the downside there is that when the journal does fill up the disk will pretty much just stop while it flushes that data out.

The advantage of the cache on a RAID controller like the Areca is that assuming you have the battery backup option you can use the less safe write cache mode where it caches all disk writes and can recover them if there is a hard crash. Without the battery doing this could really damage a journaling filesystem.

[Jeff]

That's another question I had -- why do raid controllers need a secondary battery to allow you to enable write cache (without warning message) if the server/workstation is powered by a UPS anyway?

[Kevin]

#1 The RAID controller has no way of knowing about your UPS
#2 The UPS can fail or the computer's power supply can fail
#3 Someone may just hit the power button or trip over the power cable
#4 The UPS>computer communications may not be setup so the computer might just keep running until the UPS is drained
#5 They are covering their butts.

[Bruce]

#5 They are covering their butts.#6 The operating system could crash, leaving the system fully powered, but the data completely inaccessable. :shocked:

[Jeff]

#6 The operating system could crash, leaving the system fully powered, but the data completely inaccessable. :shocked:
Ah, I take it the fear with a huge cache is that the OS rebooting after an OS crash would power cycle the raid controller before it had a chance to write the data in cache to the drives, and the onboard battery would maintain power and possibly prevent the re-initialization of the controller on system reboot?

[Bruce]

Ah, I take it the fear with a huge cache is that the OS rebooting after an OS crash would power cycle the raid controller before it had a chance to write the data in cache to the drives, and the onboard battery would maintain power and possibly prevent the re-initialization of the controller on system reboot?Essentially correct, yes. It's just one more boundary case to cover safely.

[Randall]

Of course the big difference between SSD and RAM is that SSDs don't forget everything when you power them off. The DRAM SSDs have onboard batteries for that reason (and flash or HD backup options). I still can't understand why they cost so much, even without the RAM.

Anyway, when are we finally going to get MRAM? They've been promising it as the ultimate RAM chip for more than a decade, the replacement for flash and DRAM. Superfast storage, non-volatile RAM ... what's not to like? It feels odd carrying a laptop around knowing there is a physical head skimming the surface of a disk... the thought of a microdrive spinning in my digital camera years ago also seemed odd... at the same time it still amazes me that physically moving a head is faster than current flash technology... hopefully that will turn around any month. I just upgraded my laptop's HDD, but I'm waiting semi-patiently for the day when SSDs start catching up in cost per GB and I can turn my Mac into an oversized iPod Nano. :clown:

Randall