IOPs, Bandwidth, Throughput…

Storage Contention

I was given a chart from a vendor that shows raw throughput of a disk array they suggested I purchase. This chart was a comparison between one array versus a list of others in terms of throughput.  In an attempt to objectify this storage contention idea that keeps getting thrown in my face, I looked up the definitions of IOPS, throughput, and Bandwidth and their relationships in a storage environment.  Here’s what I learned:

  • IOPS = IO’s Per Second which is 1 / (average latency in ms + average seek time in ms) also based on block size
  • Throughput = Amount of data the network can handle (3Gb, 6Gb, 12Gb SAS)
  • Bandwidth = Network speed of the storage device (7.2k, 10k, 15k RPM drives)

Being somewhat of a gear head, I see these terms correlate closely to automotive concepts of horsepower, torque, and vehicle function.  We Americans are known to purchase horsepower off-the-lot; however, we drive torque when choosing our vehicles. Likewise, many computer guys purchase throughput – 10k vs. 15k SAS drives when our applications actually “drive” IOPS.  It is rare that a user’s speed perception and end-user experience is based solely on throughput. A high-horsepower vehicle may under-perform when compared to a lower-horsepower yet higher low-end torque vehicle at the green light. So, why is it that sales people and application vendors focus solely on drive RPM speed bandwidth and storage network throughput?

NASCAR news: “School bus wins”

Choosing the “correct” drives and array depends on the application; likewise, choosing the “correct” race car depends on the track.  I once witnessed a Mazda Miata MX-5 beat a 911 Porsche Turbo, Ford GT, and a Ferrari 430 on a certain track just north of Dallas.  I’ve also benchmarked an $80 7.2k SATA drive outperform a $600 15k SAS Cheetah drive.  But looking at speed in only this way is like concluding that only yellow school buses can win a race. While it is true that a single school bus can transport more passengers between two points faster than any sports car or motorcycle, there’s more to it than just the interface speed.  My point is to prove that raw throughput and bandwidth should not be the sole metric used when building and sizing your storage array to your application.

An OLTP database will generally have to move small blocks (4k) around the disk. Formatting a drive with a large block-size for this application would hinder IOPS performance in this environment. Surely a 300HP Hayabusa drag bike would lose on a twisty track against an SV650 that can lean 170 degrees in turns, has good brakes, and has high low-end torque from the 90HP 650cc V-Twin engine. If we are storing large imaging files, we would choose to run at much larger block sizes (128k) on slower, inexpensive large capacity drives. Therefore we would probably hit the bandwidth limit before we hit the IOPS limit.

The drive/array metric measured in Gb refers to raw throughput. This is size of the pipe to the array itself and should be divided in 10bit-frames when transferred on-the-wire. Storage network throughput could be a factor depending on the number of hosts connected to it. I look at this in the same way an Ethernet switch is connected and cascaded on a network.  A single computer with a 10GbE can’t push more than the 1Gb cascaded interface without exceeding the bandwidth of the pipe.

Loud Datacenters?

From the video above, you’ll find that environment is a factor as well. It is well known and easily proved that SAS will outperform SATA with transactional workloads but only until the IOPs get above a certain value. Other than that, there is really no difference between SAS and SATA performance on drives of the same basic specifications.

My point here is that we can split hairs on this all day long.  What it really comes down to is a general understanding of what we actually need in order to accomplish our basic mission.


Vendors:  I implore you to describe the storage requirements of your applications in terms of IOPS and block sizes in addition to capacity.  The bandwidth and throughput requirements can easily be back-calculated.  Charts you show me are meaningless if you’re always going to suggest buying a cutting-edge SAN as a panacea to my future storage woes. I know that I can “win” with a lot less if it is configured properly.

So, like with the Mazda Miata: please stick to describing the track and leave the equipment selection to us.



2 Comments on “IOPs, Bandwidth, Throughput…”

  1. Tim says:

    Great post, man. We liberally use “performance” in the industry to describe slow systems, but it’s worth looking into where the latency actually resides. Well done.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s