I dont know many applications that really need even 10MB of data written sync() to disk and returning in 0.1s. That’s a *HUGE* amount of bandwidth to continually deal with, even if you only get 1 request every 5 seconds. Very very few applications require this design (there are some that do, but most LAMP installs running a website with a mysql back end DO NOT).

Why doesnt anyone talk about random seeks per second? I do most of my bonnies with -s 0 and -n 512:1000:0:512 -b to ignore the total throughput and see how fast it can write 512*1024 1K files to disk – SYNC (so we dont get buffering help, since mysql doesnt use it, yes i know inno has its own buffering management) – but that gives me AN IDEA of what the drive seek and random write performance is.

Random reads isnt as critical to me, more ram will help with that, which is why it’s important to keep your database files smaller than your free cache ram – wasted space in databases means you cant hold most of your DB in cache, and you’ll be going to disk to retrieve data instead. Bad design.

Writes, esp sync writes, are the costly thing (though EC2 instances seem to be tuned to have *HUGE* buffers, ive seen 800mb before! tho sync() writes wont use that) – so make sure whatever you are using sync() for NEEDS to survive a crash/reboot, cuz that’s what sync is for. If you can design around async writes, and avoid full ACID db compliance, do it. (memcached sped up my one customer’s front page load by 2x and reduced cpu usage by 5x and disk usage by 10x).

The real world performance of these technologies is not the synthetic case, and there is compression going on. So if you hand a compression engine a 3 gigabyte burst of nulls, that you can compress down to 8 bytes.

If the benchmarks don’t represent real-world data they’re not useful for measuring performance.

Amazon’s EBS does not run nearly this fast on real data unless you’re very concerned with manipulating files that are filled with zeroes.

@James I will be doing some different benchmarks when I get some free time, to check for random read / write performance – I have a feeling other RAID levels may help.

@jeff yes, it looks very much like its hitting a gigabit network speed limit

To be honest, I’ve not read much about the actual implementation details of much of what goes on behind the APIs. It seems to me that things are a little skimpy on details because Amazon wants it to “not matter” — stop worrying about the details, and instead worry about how to get business done. S3 had limited appeal to me because it and SimpleDB required redesigning solutions specifically for Amazon. EBS, allowing far more flexibility to use existing solutions simply migrated to the cloud, I found particularly interesting.

Multicast between EC2 instances, and global traffic management are the next two things I’d love to see. Multicast for drop-in clustering solutions (including distributed, peer-to-peer synced caches), and global traffic management to allow both traffic distribution and seamless failover without a hacked-together add-a-bottleneck proxies.

@James,
My primary point was that host-based mirroring is fairly pointless, and fundamentally harmful from a performance perspective. Russell seemed to consider it a mystery that his assumption that RAID10 was slower, because he’s been told that it’s the fastest. If parity’s being done on the storage side, it’s wasteful to do it on the host side. The ability to arbitrarily snapshot should realistically meet most peoples’ data protection needs, and anyone for whom it doesn’t is going to be the “it’s mine to touch, feel, power, and cool” type of business.

Please do continue to share your findings, though. I’ve got a lot of other things in my head at the moment or I’d try playing myself.

If you’re having fun, want to see how ZFS / RAID-Z/Z2 does on EBS?

Even running this test in 4 EBS units in parallel I got this (each):
1073741824 bytes (1,1 GB) copied, 37,6084 seconds, 28,6 MB/s
1073741824 bytes (1,1 GB) copied, 38,6781 seconds, 27,8 MB/s
1073741824 bytes (1,1 GB) copied, 38,6271 seconds, 27,8 MB/s
1073741824 bytes (1,1 GB) copied, 39,148 seconds, 27,4 MB/s

There is a limitation of ~108MB/s. Let’s calculate:
In a gigabit connection the max transfer I can get is:
1024Mbits/s / 8 = 128MB/s
~108MB/s is the throughput.

Okie.. it’s not a mystery.
But we need to take in consideration more scenarios:

To sequential WRITEs:
With 2 EBS in RAID 0 you achieve this! There’s no need to any other configuration.

To non-sequential WRITEs (the most common scenario):
Need to do tests… I believe that 2 EBS will don’t archive 108MB/s.

To sequential READs:
Need to do tests.

To non-sequential READs:
Need to do tests.

Single Spindle – 1 host(obvious)
RAID 0 – 1 host

Raid 3/5 – Two Servers(Disk 1 2 in first host, Disk 3+ on second host)

RAID 10 – Two Servers(Disk 1 2 3 in first host, Disk 4 5 6 on second host)

Look at the bandwidth your getting. It screams at gigabit as the culprit, no?

I’d even go far as to say a modified GNBD scheme is in play…http://sourceware.org/cluster/gnbd/

Thoughts?