@James I will be doing some different benchmarks when I get some free time, to check for random read / write performance - I have a feeling other RAID levels may help.

@jeff yes, it looks very much like its hitting a gigabit network speed limit

To be honest, I’ve not read much about the actual implementation details of much of what goes on behind the APIs. It seems to me that things are a little skimpy on details because Amazon wants it to “not matter” — stop worrying about the details, and instead worry about how to get business done. S3 had limited appeal to me because it and SimpleDB required redesigning solutions specifically for Amazon. EBS, allowing far more flexibility to use existing solutions simply migrated to the cloud, I found particularly interesting.

Multicast between EC2 instances, and global traffic management are the next two things I’d love to see. Multicast for drop-in clustering solutions (including distributed, peer-to-peer synced caches), and global traffic management to allow both traffic distribution and seamless failover without a hacked-together add-a-bottleneck proxies.

@James,
My primary point was that host-based mirroring is fairly pointless, and fundamentally harmful from a performance perspective. Russell seemed to consider it a mystery that his assumption that RAID10 was slower, because he’s been told that it’s the fastest. If parity’s being done on the storage side, it’s wasteful to do it on the host side. The ability to arbitrarily snapshot should realistically meet most peoples’ data protection needs, and anyone for whom it doesn’t is going to be the “it’s mine to touch, feel, power, and cool” type of business.

Please do continue to share your findings, though. I’ve got a lot of other things in my head at the moment or I’d try playing myself.

If you’re having fun, want to see how ZFS / RAID-Z/Z2 does on EBS?

Even running this test in 4 EBS units in parallel I got this (each):
1073741824 bytes (1,1 GB) copied, 37,6084 seconds, 28,6 MB/s
1073741824 bytes (1,1 GB) copied, 38,6781 seconds, 27,8 MB/s
1073741824 bytes (1,1 GB) copied, 38,6271 seconds, 27,8 MB/s
1073741824 bytes (1,1 GB) copied, 39,148 seconds, 27,4 MB/s

There is a limitation of ~108MB/s. Let’s calculate:
In a gigabit connection the max transfer I can get is:
1024Mbits/s / 8 = 128MB/s
~108MB/s is the throughput.

Okie.. it’s not a mystery.
But we need to take in consideration more scenarios:

To sequential WRITEs:
With 2 EBS in RAID 0 you achieve this! There’s no need to any other configuration.

To non-sequential WRITEs (the most common scenario):
Need to do tests… I believe that 2 EBS will don’t archive 108MB/s.

To sequential READs:
Need to do tests.

To non-sequential READs:
Need to do tests.

Single Spindle - 1 host(obvious)
RAID 0 - 1 host

Raid 3/5 - Two Servers(Disk 1 2 in first host, Disk 3+ on second host)

RAID 10 - Two Servers(Disk 1 2 3 in first host, Disk 4 5 6 on second host)

Look at the bandwidth your getting. It screams at gigabit as the culprit, no?

I’d even go far as to say a modified GNBD scheme is in play…http://sourceware.org/cluster/gnbd/

Thoughts?