Why RAID 10 doesn’t help on EBS
August 24th, 2008
Russell over at UKD1 did some benchmarking of Amazon’s new Elastic Block Store (I’m not loving the name, but the product certainly does look compelling), using a single device, and various numbers of drives in RAID 0 (striping), RAID 5 (striping with parity), and RAID 10 (striping across mirrors).
His results show a single drive maxing out at just under 65MB/s, RAID 0 hitting the ceiling at 110MB/s, RAID 5 maxxing out about 60MB/s, and RAID 10 “F2″ at under 55MB/s.
He then writes,
EBS. However, I would have expected RAID 10 F2 to have similar performance to RAID 0, rather than the results seen.
First of all, after glancing at this “F2″ variant of RAID 10, I’ve got to say that I haven’t bothered to fully grok it. I’ll give the benefit of the doubt that it’s performant. He shows the command-line he used, and it clearly has 3 component volumes in it, and even if I understood how one might use F2, 3 volumes in a RAID 10 seems clearly wrong.
But assuming that the system is intelligent enough to make sense of it…
I also haven’t ever mounted an EBS volume on the EC2 instance I’ve set up.
But one thing is obvious as to why the RAID 10 F2 volume performed poorly… it’s obviously bandwidth constrained, probably by a single gigabit ethernet connection. Software mirroring requires that data is written twice, once to each set of disks. When the disks are on separate controllers, they’re not competing on the same storage bus. Even when they’re on the same controller, but it’s SCSI or FibreChannel or SATA or whatever it may be — the bus has higher throughput than the disks do. You won’t saturate a U320 SCSI bus with a pair of disks, the disks are the bottleneck.
I’m quite impressed by the 110MB/s achieved through striping… now imagine that for every write, you’re clogging the pipe with 2 copies… what would you expect the capacity for each set of data to be? Strangely enough… 55MB/s.
QED
Amazon’s description of EBS refers to it as “highly available, highly reliable volumes”, which I take to mean fault tolerant - I’d assume RAID 1,5,6, or Z (Sun’s ZFS Raid) - which would make the mirroring portion of RAID 10 unnecessary because Amazon’s already taking care of the redundancy, so I don’t see this as a downside.
Entry Filed under: Reference, The Geek Factor
5 Comments Add your own
1. Jeff | August 26th, 2008 at 11:39 am
So perhaps its setup like this:
Single Spindle - 1 host(obvious)
RAID 0 - 1 host
Raid 3/5 - Two Servers(Disk 1 2 in first host, Disk 3+ on second host)
RAID 10 - Two Servers(Disk 1 2 3 in first host, Disk 4 5 6 on second host)
Look at the bandwidth your getting. It screams at gigabit as the culprit, no?
I’d even go far as to say a modified GNBD scheme is in play…http://sourceware.org/cluster/gnbd/
Thoughts?
2. James | August 29th, 2008 at 1:22 pm
It’s important to know that it is only a WRITE test.
Maybe with Raid 10 will have some gains in READ only.
3. James | August 29th, 2008 at 1:40 pm
I miss one thing:
Even running this test in 4 EBS units in parallel I got this (each):
1073741824 bytes (1,1 GB) copied, 37,6084 seconds, 28,6 MB/s
1073741824 bytes (1,1 GB) copied, 38,6781 seconds, 27,8 MB/s
1073741824 bytes (1,1 GB) copied, 38,6271 seconds, 27,8 MB/s
1073741824 bytes (1,1 GB) copied, 39,148 seconds, 27,4 MB/s
There is a limitation of ~108MB/s. Let’s calculate:
In a gigabit connection the max transfer I can get is:
1024Mbits/s / 8 = 128MB/s
~108MB/s is the throughput.
Okie.. it’s not a mystery.
But we need to take in consideration more scenarios:
To sequential WRITEs:
With 2 EBS in RAID 0 you achieve this! There’s no need to any other configuration.
To non-sequential WRITEs (the most common scenario):
Need to do tests… I believe that 2 EBS will don’t archive 108MB/s.
To sequential READs:
Need to do tests.
To non-sequential READs:
Need to do tests.
4. NevDull | August 29th, 2008 at 3:32 pm
@Jeff,
To be honest, I’ve not read much about the actual implementation details of much of what goes on behind the APIs. It seems to me that things are a little skimpy on details because Amazon wants it to “not matter” — stop worrying about the details, and instead worry about how to get business done. S3 had limited appeal to me because it and SimpleDB required redesigning solutions specifically for Amazon. EBS, allowing far more flexibility to use existing solutions simply migrated to the cloud, I found particularly interesting.
Multicast between EC2 instances, and global traffic management are the next two things I’d love to see. Multicast for drop-in clustering solutions (including distributed, peer-to-peer synced caches), and global traffic management to allow both traffic distribution and seamless failover without a hacked-together add-a-bottleneck proxies.
@James,
My primary point was that host-based mirroring is fairly pointless, and fundamentally harmful from a performance perspective. Russell seemed to consider it a mystery that his assumption that RAID10 was slower, because he’s been told that it’s the fastest. If parity’s being done on the storage side, it’s wasteful to do it on the host side. The ability to arbitrarily snapshot should realistically meet most peoples’ data protection needs, and anyone for whom it doesn’t is going to be the “it’s mine to touch, feel, power, and cool” type of business.
Please do continue to share your findings, though. I’ve got a lot of other things in my head at the moment or I’d try playing myself.
If you’re having fun, want to see how ZFS / RAID-Z/Z2 does on EBS?
5. Russell Smith | September 18th, 2008 at 7:08 am
@devnull Your right 3 volumes in a RAID 10 seems odd, but it works - mdadm is pretty clever. 55mb / s makes sense also now you point out the obvious, these benchmarks were done quickly!
@James I will be doing some different benchmarks when I get some free time, to check for random read / write performance - I have a feeling other RAID levels may help.
@jeff yes, it looks very much like its hitting a gigabit network speed limit
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
Trackback this post | Subscribe to the comments via RSS Feed