Demonstrations of iolatency, the Linux ftrace version.

Here's a busy system doing over 4k disk IOPS:

# ./iolatency
Tracing block I/O. Output every 1 seconds. Ctrl-C to end.

  >=(ms) .. <(ms)   : I/O      |Distribution                          |
       0 -> 1       : 4381     |######################################|
       1 -> 2       : 9        |#                                     |
       2 -> 4       : 5        |#                                     |
       4 -> 8       : 0        |                                      |
       8 -> 16      : 1        |#                                     |

  >=(ms) .. <(ms)   : I/O      |Distribution                          |
       0 -> 1       : 4053     |######################################|
       1 -> 2       : 18       |#                                     |
       2 -> 4       : 9        |#                                     |
       4 -> 8       : 2        |#                                     |
       8 -> 16      : 1        |#                                     |
      16 -> 32      : 1        |#                                     |

  >=(ms) .. <(ms)   : I/O      |Distribution                          |
       0 -> 1       : 4658     |######################################|
       1 -> 2       : 9        |#                                     |
       2 -> 4       : 2        |#                                     |

  >=(ms) .. <(ms)   : I/O      |Distribution                          |
       0 -> 1       : 4298     |######################################|
       1 -> 2       : 17       |#                                     |
       2 -> 4       : 10       |#                                     |
       4 -> 8       : 1        |#                                     |
       8 -> 16      : 1        |#                                     |
^C
Ending tracing...

Disk I/O latency is usually between 0 and 1 milliseconds, as this system uses
SSDs. There are occasional outliers, up to the 16->32 ms range.

Identifying outliers like these is difficult from iostat(1) alone, which at
the same time reported:

# iostat 1
[...]
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.53    0.00    1.05   46.84    0.53   51.05

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00   28.00     0.00   112.00     8.00     0.02    0.71    0.00    0.71   0.29   0.80
xvdb              0.00     0.00 2134.00    0.00 18768.00     0.00    17.59     0.51    0.24    0.24    0.00   0.23  50.00
xvdc              0.00     0.00 2088.00    0.00 18504.00     0.00    17.72     0.47    0.22    0.22    0.00   0.22  46.40
md0               0.00     0.00 4222.00    0.00 37256.00     0.00    17.65     0.00    0.00    0.00    0.00   0.00   0.00

I/O latency ("await") averages 0.24 and 0.22 ms for our busy disks, but this
output doesn't show that occasionally is much higher.

To get more information on these I/O, try the iosnoop(8) tool.
