FIO是测试IOPS的非常好的工具，用来对硬件进行压力测试和验证，支持13种不同的I/O引擎，
包括:sync,mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio 等等。&
fio 官网地址：http://freshmeat.net/projects/fio/&
一，FIO安装&
wget http://brick.kernel.dk/snaps/fio-2.0.7.tar.gz&
yum install libaio-devel&
tar -zxvf fio-2.0.7.tar.gz&
cd fio-2.0.7&
make install&
二，随机读测试：&
fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=16k -size=200G&
-numjobs=10 -runtime=1000 -group_reporting -name=mytest&
filename=/dev/sdb1 测试文件名称，通常选择需要测试的盘的data目录。&
direct=1 测试过程绕过机器自带的buffer。使测试结果更真实。&
rw=randwrite 测试随机写的I/O&
rw=randrw 测试随机写和读的I/O&
bs=16k 单次io的块文件大小为16k&
bsrange=512-2048 同上，提定数据块的大小范围&
size=5g 本次的测试文件大小为5g，以每次4k的io进行测试。&
numjobs=30 本次的测试线程为30.&
runtime=1000 测试时间为1000秒，如果不写则一直将5g文件分4k每次写完为止。&
ioengine=psync io引擎使用pync方式&
rwmixwrite=30 在混合读写的模式下，写占30%&
group_reporting 关于显示结果的，汇总每个进程的信息。&
lockmem=1g 只使用1g内存进行测试。&
zero_buffers 用0初始化系统buffer。&
nrfiles=8 每个进程生成文件的数量。&
fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=mytest&
fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=mytest&
fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=mytest&
混合随机读写：&
fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=mytest -ioscheduler=noop&
三，实际测试范例：&
[root@localhost ~]# fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30&
-runtime=100 -group_reporting -name=mytest1&
mytest1: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1&
mytest1: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1&
fio 2.0.7&
Starting 30 threads&
Jobs: 1 (f=1): [________________m_____________] [3.5% done] [K /s] [423 /190 iops] [eta 48m:20s] s]&
mytest1: (groupid=0, jobs=30): err= 0: pid=23802&
read&: io=1853.4MB,&bw=18967KB/s, iops=1185&, runt=100058msec&
clat (usec): min=60 , max=871116 , avg=25227.91, stdev=31653.46&
lat (usec): min=60 , max=871117 , avg=25228.08, stdev=31653.46&
clat percentiles (msec):&
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 8],&
| 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 15], 60.00th=[ 19],&
| 70.00th=[ 26], 80.00th=[ 37], 90.00th=[ 57], 95.00th=[ 79],&
| 99.00th=[ 151], 99.50th=[ 202], 99.90th=[ 338], 99.95th=[ 383],&
| 99.99th=[ 523]&
bw (KB/s) : min= 26, max= 1944, per=3.36%, avg=636.84, stdev=189.15&
write: io=803600KB,&bw=8031.4KB/s, iops=501&, runt=100058msec&
clat (usec): min=52 , max=9302 , avg=146.25, stdev=299.17&
lat (usec): min=52 , max=9303 , avg=147.19, stdev=299.17&
clat percentiles (usec):&
| 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 68], 20.00th=[ 74],&
| 30.00th=[ 84], 40.00th=[ 87], 50.00th=[ 89], 60.00th=[ 90],&
| 70.00th=[ 92], 80.00th=[ 97], 90.00th=[ 120], 95.00th=[ 370],&
| 99.00th=[ 1688], 99.50th=[ 2128], 99.90th=[ 3088], 99.95th=[ 3696],&
| 99.99th=[ 5216]&
bw (KB/s) : min= 20, max= 1117, per=3.37%, avg=270.27, stdev=133.27&
lat (usec) : 100=24.32%, 250=3.83%, 500=0.33%, 750=0.28%, %&
lat (msec) : 2=0.64%, 4=3.08%, 10=20.67%, 20=19.90%, 50=17.91%&
lat (msec) : 100=6.87%, 250=1.70%, 500=0.19%, 750=0.01%, %&
cpu : usr=1.70%, sys=2.41%, ctx=5237835, majf=0, minf=6344162&
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &=64=0.0%&
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &=64=0.0%&
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &=64=0.0%&
issued : total=r=118612/w=50225/d=0, short=r=0/w=0/d=0&
Run status group 0 (all jobs):&
READ: io=1853.4MB, aggrb=18966KB/s, minb=18966KB/s, maxb=18966KB/s, mint=100058msec, maxt=100058msec&
WRITE: io=803600KB, aggrb=8031KB/s, minb=8031KB/s, maxb=8031KB/s, mint=100058msec, maxt=100058msec&
Disk stats (read/write):&
sdb: ios=24, merge=0/0, ticks=0, in_queue=2998169, util=99.77%&
主要查看以上红色字体部分的iops(read/write)&
**磁盘阵列吞吐量与IOPS两大瓶颈分析**
　　吞吐量主要取决于阵列的构架，光纤通道的大小(现在阵列一般都是光纤阵列，至于SCSI这样的SSA阵列，我们不讨论)以及硬盘的个数。阵列的构架与每个阵列不同而不同，他们也都存在内部带宽(类似于pc的系统总线)，不过一般情况下，内部带宽都设计的很充足，不是瓶颈的所在。
　　光纤通道的影响还是比较大的，如数据仓库环境中，对数据的流量要求很大，而一块2Gb的光纤卡，所77能支撑的最大流量应当是2Gb/8(小B)=250MB/s(大B)的实际流量，当4块光纤卡才能达到1GB/s的实际流量，所以数据仓库环境可以考虑换4Gb的光纤卡。
　　最后说一下硬盘的限制，这里是最重要的，当前面的瓶颈不再存在的时候，就要看硬盘的个数了，我下面列一下不同的硬盘所能支撑的流量大小：
　　10 K rpm 15 K rpm ATA
　　——— ——— ———
　　10M/s 13M/s 8M/s
　　那么，假定一个阵列有120块15K rpm的光纤硬盘，那么硬盘上最大的可以支撑的流量为120*13=1560MB/s，如果是2Gb的光纤卡，可能需要6块才能够，而4Gb的光纤卡，3-4块就够了。
　　决定IOPS的主要取决与阵列的算法，cache命中率，以及磁盘个数。阵列的算法因为不同的阵列不同而不同，如我们最近遇到在hds usp上面，可能因为ldev(lun)存在队列或者资源限制，而单个ldev的iops就上不去，所以，在使用这个存储之前，有必要了解这个存储的一些算法规则与限制。
　　cache的命中率取决于数据的分布，cache size的大小，数据访问的规则，以及cache的算法，如果完整的讨论下来，这里将变得很复杂，可以有一天好讨论了。我这里只强调一个cache的命中率，如果一个阵列，读cache的命中率越高越好，一般表示它可以支持更多的IOPS，为什么这么说呢?这个就与我们下面要讨论的硬盘IOPS有关系了。
　　硬盘的限制，每个物理硬盘能处理的IOPS是有限制的，如
　　10 K rpm 15 K rpm ATA
　　——— ——— ———
　　100 150 50
　　同样，如果一个阵列有120块15K rpm的光纤硬盘，那么，它能撑的最大IOPS为120*150=18000，这个为硬件限制的理论值，如果超过这个值，硬盘的响应可能会变的非常缓慢而不能正常提供业务。
　　在raid5与raid10上，读iops没有差别，但是，相同的业务写iops，最终落在磁盘上的iops是有差别的，而我们评估的却正是磁盘的IOPS，如果达到了磁盘的限制，性能肯定是上不去了。
　　那我们假定一个case，业务的iops是10000，读cache命中率是30%，读iops为60%，写iops为40%，磁盘个数为120，那么分别计算在raid5与raid10的情况下，每个磁盘的iops为多少。
　　raid5:
　　单块盘的iops = (1.3)*0.6 + 4 * ())/120
　　= (4200 + 1
　　这里的1.3)*0.6表示是读的iops，比例是0.6，除掉cache命中，实际只有4200个iops
　　而4 * () 表示写的iops，因为每一个写，在raid5中，实际发生了4个io，所以写的iops为16000个
　　为了考虑raid5在写操作的时候，那2个读操作也可能发生命中，所以更精确的计算为：
　　单块盘的iops = (1.3)*0.6 + 2 * ()*(1-0.3) + 2 * ())/120
　　= (4200 + 5600 +
　　计算出来单个盘的iops为148个，基本达到磁盘极限
　　raid10
　　单块盘的iops = (1.3)*0.6 + 2 * ())/120
　　= (4200 +
　　可以看到，因为raid10对于一个写操作，只发生2次io，所以，同样的压力，同样的磁盘，每个盘的iops只有102个，还远远低于磁盘的极限iops。
　　在一个实际的case中，一个恢复压力很大的standby(这里主要是写，而且是小io的写)，采用了raid5的方案，发现性能很差，通过分析，每个磁盘的iops在高峰时期，快达到200了，导致响应速度巨慢无比。后来改造成raid10，就避免了这个性能问题，每个磁盘的iops降到100左右。
测试结果详细分析参考：http://tobert.github.io/post/-fio-output-explained.html
Fio Output Explained
Previously, I blogged about setting up my&. Now that it's
up and running, I've started exploring the fio benchmarking tool.
&- the Flexible IO Tester is an application written by&, who may be better known as the maintainer of the Linux kernel's block IO subsystem. It resembles the older ffsb tool in a few ways, but doesn't seem to have any relation to it. As power tools go, it's capable of generating pretty much arbitrary
load. The tradeoff is that it's difficult to learn and that's exactly what I've been doing.
Here's a section-by-section breakdown of the default output. I'll look at other output options in future posts. The data displayed is from a Samsung 840 Pro SSD.
The explanation for each section can be found below the output text.
read : io=10240MB, bw=63317KB/s, iops=15829, runt=165607msec
The first line is pretty easy to read. fio did a total of 10GB of IO at 63.317MB/s for a total of 15829 IOPS (at the default 4k block size), and ran for 2 minutes and 45 seconds.
The first latency metric you'll see is the 'slat' or submission latency. It is pretty much what it sounds like, meaning &how long did it take to submit this IO to the kernel for processing?&
slat (usec): min=3, max=335, avg= 9.73, stdev= 5.76
I originally thought that submission latency would be useless for tuning, but the numbers below changed my mind. 269usec or 1/4 of a millisecond seems to be noise, but check it out. I haven't tuned anything yet, so I suspect that changing the scheduler and
telling the kernel it's not a rotating device will help.
Here are some more examples from the other devices:
slat (usec): min=3, max=335, avg= 9.73, stdev= 5.76 (SATA SSD)
slat (usec): min=5, max=68,
avg=26.21, stdev= 5.97 (SAS 7200)
slat (usec): min=5, max=63,
avg=25.86, stdev= 6.12 (SATA 7200)
slat (usec): min=3, max=269, avg= 9.78, stdev= 2.85 (SATA SSD)
slat (usec): min=6, max=66,
avg=27.74, stdev= 6.12 (MDRAID0/SAS)
clat (usec): min=1, max=18600, avg=51.29, stdev=16.79
Next up is completion latency. This is the time that passes between submission to the kernel and when the IO is complete, not including submission latency. In older versions of fio, this was the best metric for approximating application-level latency.
lat (usec): min=44, max=18627, avg=61.33, stdev=17.91
From what I can see, the 'lat' metric is fairly new. It's not documented in the man page or docs. Looking at the C code, it seems that this metric starts the moment the IO struct is created in fio and is completed right after clat, making this the one that
best represents what applications will experience. This is the one that I will graph.
clat percentiles (usec):
45], 10.00th=[
45], 20.00th=[
| 30.00th=[
47], 40.00th=[
47], 50.00th=[
49], 60.00th=[
| 70.00th=[
53], 80.00th=[
56], 90.00th=[
60], 95.00th=[
| 99.00th=[
78], 99.50th=[
81], 99.90th=[
94], 99.95th=[
| 99.99th=[
Completion latency percentiles are fairly self-explanatory and probably the most useful bit of info in the output. I looked at the source code and this is not slat + it is tracked in its own struct.
The buckets are configurable in the config file. In the terse output, this is 20 fields of %f=%d;%f=%d;... which makes parsing more fun than it should be.
For comparison, here's the same section from a 7200 RPM SAS drive running the exact same load.
clat percentiles (usec):
1.00th=[ 3952],
5.00th=[ 5792], 10.00th=[ 7200], 20.00th=[ 8896],
| 30.00th=[10304], 40.00th=[11456], 50.00th=[12608], 60.00th=[13760],
| 70.00th=[15168], 80.00th=[16768], 90.00th=[18816], 95.00th=[20608],
| 99.00th=[23424], 99.50th=[24192], 99.90th=[26752], 99.95th=[28032],
| 99.99th=[30080]
/s): min=52536, max=75504, per=67.14%, avg=63316.81, stdev=4057.09
Bandwidth is pretty self-explanatory except for the per= part. The docs say it's meant for testing a single device with multiple workloads, so you can see how much of the IO was consumed by each process. When fio is run against multiple devices, as I did for
this output, it doesn't provide much meaning but is amusing when SSDs are mixed with spinning rust.
And here's the SAS drive again with 0.36% of the total IO out of 4 devices being tested.
251, per=0.36%, avg=154.84, stdev=18.29
lat (usec) :
20=0.01%, 50=51.41%
lat (usec) : 100=48.53%, 250=0.06%, 500=0.01%, %
lat (msec) :
The latency distribution section took me a couple passes to understand. This is one series of metrics. Instead of using the same units for all three lines, the third line switches to milliseconds to keep the text width under control. Read the last line as 2000,
, and 20,000usec and it makes more sense.
As this is a latency distribution, it's saying that 51.41% of requests took less than 50usec, 48.53% took less than 100usec and so on.
lat (msec) : 4=1.07%, 10=27.04%, 20=65.43%, 50=6.46%, 100=0.01%
In case you were thinking of parsing this madness with a quick script, you might want to know that the lat section will omit entries and whole lines if there is no data. For example, the SAS drive I've been referencing didn't manage to do any IO faster than
a millisecond, so this is the only line.
: usr=5.32%, sys=21.95%, ctx=2829095, majf=0, minf=21
Here's the user/system CPU percentages followed by context switches then major and minor&. Since the test
is&&to use direct IO, there should be very few page faults.
: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &=64=0.0%
Fio has an iodepth setting that controls how many IOs it issues to the OS at any given time. This is entirely application-side, meaning it is not the same thing as the device's IO queue. In this case, iodepth was set to 1 so the IO depth was always 1 100% of
: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &=64=0.0%
: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &=64=0.0%
submit and complete represent the number of submitted IOs at a time by fio and the number completed at a time. In the case of the thrashing test used to generate this output, the iodepth is at the default value of 1, so 100% of IOs were submitted 1 at a time
placing the results in the 1-4 bucket. Basically these only matter if iodepth is greater than 1.
These will get much more interesting when I get around to testing the various schedulers.
: total=r=2621440/w=0/d=0, short=r=0/w=0/d=0
The number of IOs issued. Something is weird here since this was a 50/50 read/write load, so there should have been an equal number of writes. I suspect having&&enabled
is making fio count all IOs as reads.
If you see short IOs in a direct IO test something has probably gone wrong. The reference I found in the&&indicates that this happens at EOF and likely end of device.
: target=0, window=0, percentile=100.00%, depth=1
Fio can be&&with a latency target, which will cause it to adjust
throughput until it can consistently hit the configured latency target. I haven't messed with this much yet. In time or size-based tests, this line will always look the same. All four of these values represent the configuration settings&,&,&,
Run status group 0 (all jobs):
fio supports grouping different tests for aggregation. For example, I can have one config for SSDs and HDDs mixed in the same file, but set up groups to report the IO separately. I'm not doing this for now, but future configs will need this functionality.
MIXED: io=12497MB, aggrb=42653KB/s, minb=277KB/s, maxb=41711KB/s, mint=300000msec, maxt=300012msec
And finally, the total throughput and time. io= indicates the amount of IO done in total. It will be variable for timed tests and should match the&parameter
for sized tests. aggrb is the aggregate bandwidth across all processes / devices. minb/maxb show minimum/maximum observed bandwidth. mint/maxt show the shortest & longest times for tests. Similar to the io= parameter, these should match the&&parameter
for time-based tests and will vary in size-based tests.
Since I ran this test with&&enabled, we only see
a line for MIXED. If it's disabled there will be separate lines for READ and WRITE.
Simple, right? I'll be spending a lot more time with fio for the next few weeks and will post more examples of configs, output, and graphing code.
参考知识库
* 以上用户言论只代表其个人观点，不代表CSDN网站的观点或立场
访问：180929次
积分：2459
积分：2459
排名：第13555名
转载：344篇
(2)(4)(7)(8)(8)(4)(1)(5)(6)(15)(11)(1)(3)(7)(14)(2)(4)(6)(17)(9)(10)(6)(23)(15)(20)(21)(25)(45)(43)(4)1775人阅读
运维（20）
FIO是测试IOPS的非常好的工具，用来对硬件进行压力测试和验证，支持13种不同的I/O引擎，包括:sync,mmap, libaio, posixaio, SG v3, splice, null, network, syslet,guasi, solarisaio 等等。
fio 官网地址：
fio -filename=/dev/sdb1 -direct=1 -iodepth1 -thread -rw=randread -ioengine=psync -bs=16k -size=200G -numjobs=10-runtime=1000 -group_reporting -name=mytest
filename=/dev/sdb1&&&&&& 测试文件名称，通常选择需要测试的盘的data目录。
direct=1&&&&&&&&&&&&&&&& 测试过程绕过机器自带的buffer。使测试结果更真实。
rw=randwrite&&&&&&&&&&&& 测试随机写的I/O
rw=randrw&&&&&&&&&&&&&&& 测试随机写和读的I/O
bs=16k&&&&&&&&&&&&&&&&&& 单次io的块文件大小为16k
bsrange=512-2048&&&&&&&& 同上，提定数据块的大小范围
size=5g&&&本次的测试文件大小为5g，以每次4k的io进行测试。
numjobs=30&&&&&&&&&&&&&& 本次的测试线程为30.
runtime=1000&&&&&&&&&&&& 测试时间为1000秒，如果不写则一直将5g文件分4k每次写完为止。
ioengine=psync&&&&&&&&&& io引擎使用pync方式
rwmixwrite=30&&&&&&&&&&& 在混合读写的模式下，写占30%
group_reporting&&&&&&&&& 关于显示结果的，汇总每个进程的信息。
这个工具最大的特点是使用简单，支持的文件操作非常多, 可以覆盖到我们能见到的文件使用方式:
sync：Basic read(2) or write(2) I/O. fseek(2) is used to position the I/O location.
psync：Basic pread(2) or pwrite(2) I/O.
vsync: Basic readv(2) or writev(2) I/O. Will emulate queuing by coalescing adjacents IOs into a single submission.
libaio: Linux native asynchronous I/O.
posixaio: glibc POSIX asynchronous I/O using aio_read(3) and aio_write(3).
mmap: File is memory mapped with mmap(2) and data copied using memcpy(3).
splice： splice(2) is used to transfer the data and vmsplice(2) to transfer data from user-space to the kernel.
syslet-rw： Use the syslet system calls to make regular read/write asynchronous.
sg：SCSI generic sg v3 I/O.
net ： Transfer over the network. filename must be set appropriately to `host/port’ regardless of data direction. If receiving,
only the port argument is used.
netsplice： Like net, but uses splice(2) and vmsplice(2) to map data and send/receive.
guasi The GUASI I/O engine is the Generic Userspace Asynchronous Syscall Interface approach to asycnronous I/O.
lockmem=1g&&&&&&&&&&&&&& 只使用1g内存进行测试。
zero_buffers&&&&&&&&&&&& 用0初始化系统buffer。
nrfiles=8&&&&&&&&&&&&&&& 每个进程生成文件的数量。
fio -filename=/dev/sdb1 -direct=1 -iodepth1 -thread -rw=read -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000-group_reporting -name=mytest
fio -filename=/dev/sdb1 -direct=1 -iodepth1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=200G -numjobs=30-runtime=1000 -group_reporting -name=mytest
fio -filename=/dev/sdb1 -direct=1 -iodepth1 -thread -rw=write -ioengine=psync -bs=16k -size=200G -numjobs=30-runtime=1000 -group_reporting -name=mytest
混合随机读写：
fio -filename=/dev/sdb1 -direct=1 -iodepth1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G-numjobs=30 -runtime=100 -group_reporting -name=mytest -ioscheduler=noop
翻译原文来源
fio，又称为Flexible IO Tester，是Jens Axboe编写的应用程序。Jens是Linux Kernel中block IO subsystem的维护者。fio从多个方面来看类似于更古老的ffsb工具，但他们之间似乎没有任何关系。作为一个强大的工具，fio可以产生足够多的任意类型的负载 (arbitrary load)。作为权衡，fio不容易学习，这就是这篇文章的目的。
configraw output
以下是一个50/50读写的垃圾处理IO (Trashing IO)负载，对于多个盘进行的读写。
5分钟，磁盘垃圾处理测试，50/50读写在每个盘上产生相同数量的随机读和写IO，为每个盘产生测试数据。
#&a&5&minute&disk&thrashing&benchmark&&#&generates&equal&amounts&of&random&read&and&write&IO&on&every&drive&&#&will&generate&metrics&for&each&drive&&[global]&&ioengine=libaio&&direct=1&&unified_rw_reporting=1&&rw=randrw&&time_based=1&&runtime=300s&&&&&strong&#&Seagate&7200RPM&SAS&512G&ST9500430SS&(sdb)&/strong&&&[/dev/disk/by-path/pci-.0-sas-0xf96d9-lun-0]&&write_lat_log=7200RPMSAS-0xf96d9&&&&&strong&#&Seagate&7200RPM&Enterprise&SATA&1TB&STNS&(sdg)&/strong&&&[/dev/disk/by-id/wwn-0x29dd]&&write_lat_log=7200RPMEnterpriseSATA-0x29dd&&&&&strong&#&Samsung&840&Pro&128GB&(on&a&3G&SATA&port)&(sdd)&/strong&&&[/dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S1ANNSADB05219A]&&write_lat_log=SSDSATA-S1ANNSADB05219A&&&&#&2x&SAS&drives&with&GPT&partition&&&MDRAID0&(sdi1&+&sdc1)&&&strong&#&Seagate&7200RPM&SAS&512G&ST9500430SS&/strong&&&[/dev/disk/by-id/md-uuid-6bb71ed6:eaf0b7:0afe758d]&&write_lat_log=7200RPMSAS-MDRAID0&&
以上链接可打开上述fio配置的原始输出。
接下来按照每个部分分析输出内容。这里显示的数据是Samsung 840 Pro SSD，其他盘的数据稍后再深入研究。
对于每一个部分的描述在输出文字下面。
read&:&io=10240MB,&bw=63317KB/s,&iops=15829,&runt=165607msec&&
第一行很容易读懂。fio做了10GB的IO，速率63.317MB/s，总IOPS 15829 (默认4k block size)，运行了2分钟45秒。
你看到的第一个延迟(Latency)数据是slat，或称为submission latency。这个值和他的名字很相像，代表“盘需要多久将IO提交到kernel做处理?”。
slat&(usec):&min=3,&max=335,&avg=&9.73,&stdev=&5.76&&
我起初认为submission latency对于性能调试没有用，但是下面的数据让我改变了观点。269usec或1/4 ms看起来是噪音(noise)，需要关注一下。我还没有做任何调试，所以我猜测改变scheduler以告诉kernel这不是机械硬盘会有效果。
以下是从其他盘上得到的更多例子。
slat&(usec):&min=3,&max=335,&avg=&9.73,&stdev=&5.76&(SATA&SSD)&&slat&(usec):&min=5,&max=68,&&avg=26.21,&stdev=&5.97&(SAS&7200)&&slat&(usec):&min=5,&max=63,&&avg=25.86,&stdev=&6.12&(SATA&7200)&&slat&(usec):&min=3,&max=269,&avg=&9.78,&stdev=&2.85&(SATA&SSD)&&slat&(usec):&min=6,&max=66,&&avg=27.74,&stdev=&6.12&(MDRAID0/SAS)&&&&clat&(usec):&min=1,&max=18600,&avg=51.29,&stdev=16.79&&
接下来是completion latency。这是命令提交到kernel到IO做完之间的时间，不包括submission latency。在老版本的fio中，这是估计应用级延迟的最好指标。
lat (usec): min=44, max=18627, avg=61.33, stdev=17.91
在我看来，'lat'是一个新的指标，在man或者文档中都没有描述。分析C代码，似乎这个值是从IO结构体创建时刻开始，直到紧接着clat完成，这个算法最好地表现出了应用程序的行为。
clat&percentiles&(usec):&&&&&&&|&&1.00th=[&&&42],&&5.00th=[&&&45],&10.00th=[&&&45],&20.00th=[&&&46],&&&&&&&|&30.00th=[&&&47],&40.00th=[&&&47],&50.00th=[&&&49],&60.00th=[&&&51],&&&&&&&|&70.00th=[&&&53],&80.00th=[&&&56],&90.00th=[&&&60],&95.00th=[&&&67],&&&&&&&|&99.00th=[&&&78],&99.50th=[&&&81],&99.90th=[&&&94],&99.95th=[&&101],&&&&&&&|&99.99th=[&&112]&&
Completion latency百分数的解释一目了然，可能是输出信息中最有用的部分。我看了代码，这不是slat+clat，而是用了单独的结构体记录。
这个列表可以在config文件中配置。在精简输出模式下有20个这样的格式，%f=%d; %f=%d;... 解析这样的输出格式会很有趣。
作为比较，这里列出一个7200RPM SAS硬盘运行完全相同的负载的统一部分数据。
Seagate 7200RPM SAS 512G ST9500430SS
clat&percentiles&(usec):&&&&&&&|&&1.00th=[&3952],&&5.00th=[&5792],&10.00th=[&7200],&20.00th=[&8896],&&&&&&&|&30.00th=[10304],&40.00th=[11456],&50.00th=[12608],&60.00th=[13760],&&&&&&&|&70.00th=[15168],&80.00th=[16768],&90.00th=[18816],&95.00th=[20608],&&&&&&&|&99.00th=[23424],&99.50th=[24192],&99.90th=[26752],&99.95th=[28032],&&&&&&&|&99.99th=[30080]&&&&&&&&bw&(KB&&/s):&min=52536,&max=75504,&per=67.14%,&avg=63316.81,&stdev=4057.09&&
带宽(bandwidth)的意思显而易见，而per=part就不是很好理解。文档上说这个值是指在单个盘上跑多个负载，可以用来看每个进程消耗了多少IO。对于我这样把fio跑在多个盘的情况，这个值意义不大。但由于SSD和机械硬盘混合使用，这个值挺有趣。
下面是另一个SAS硬盘，占测试的所有4个盘总IO的0.36%。
bw&(KB&&/s):&min=&&&71,&max=&&251,&per=0.36%,&avg=154.84,&stdev=18.29&&&&lat&(usec)&:&&&2=&0.01%,&&&4=0.01%,&&10=0.01%,&&&20=0.01%,&50=51.41%&&lat&(usec)&:&100=48.53%,&250=0.06%,&500=0.01%,&%&&lat&(msec)&:&&&2=&0.01%,&&&4=0.01%,&&10=0.01%,&&&20=0.01%&&
latency分布部分我看了几遍才理解。这是一组数据。与三行使用一样的单位不同，第三行使用了毫秒(ms)，使得文本宽度可控。把第三行读成, 1微秒(us)就更清晰了。
这组数据表示latency的分布，说明了51.41%的request延迟小于50微秒，48.53%的延迟小于100微秒(但是大于50微秒)，以此类推。
lat&(msec)&:&4=1.07%,&10=27.04%,&20=65.43%,&50=6.46%,&100=0.01%&&
如果想用快速脚本解析这些繁琐的数据，你可能需要知道，最后一部分会省略那些没有数据的项。例如，我使用的SAS盘没有IO可以在1毫秒中完成，所以只有一行。
cpu&&&&&&&&&&:&usr=5.32%,&sys=21.95%,&ctx=2829095,&majf=0,&minf=21&&
这是用户/系统CPU占用率，进程上下文切换(context switch)次数，主要和次要(major and minor)页面错误数量(page faults)。由于测试是配置成使用直接IO，page faults数量应该极少。
IO&depths&&&&:&1=100.0%,&2=0.0%,&4=0.0%,&8=0.0%,&16=0.0%,&32=0.0%,&&=64=0.0%&&
Fio有一个iodepth设置，用来控制同一时刻发送给OS多少个IO。这完全是纯应用层面的行为，和盘的IO queue不是一回事。这里iodepth设成1，所以IO depth在全部时间都是1。
submit&&&&:&0=0.0%,&4=100.0%,&8=0.0%,&16=0.0%,&32=0.0%,&64=0.0%,&&=64=0.0%&&complete&&:&0=0.0%,&4=100.0%,&8=0.0%,&16=0.0%,&32=0.0%,&64=0.0%,&&=64=0.0%&&
submit和complete代表同一时间段内fio发送上去和已完成的IO数量。对于产生这个输出的垃圾回收测试用例来说，iodepth是默认值1，所以100%的IO在同一时刻发送1次，放在1-4栏位里。通常来说，只有iodepth大于1才需要关注这一部分数据。
我会找时间测试多种调度策略，这些数据会变得更有趣。
issued&&&&:&total=r=2621440/w=0/d=0,&short=r=0/w=0/d=0&&
发送的IO数量。这里出现了奇怪的现象，因为这是50/50的读写负载，照道理应该有相同数量的write。我猜测把unified_rw_reporting打开是的fio把所有的IO都认为是read。
如果你在直接IO测试是看到了IO值很低，那么可能是出问题了。我在Linux kernel中找到参考说这种现象发生在文件末尾EOL或可能是设备的尾端。
latency&&&:&target=0,&window=0,&percentile=100.00%,&depth=1&&
Fio可以配置一个延迟目标值，这个值可以调节吞吐量直到达到预设的延迟目标。我还没有太多深入了解这部分。在基于时间或和容量的测试中，这行通常看起来一样。四个值分别代表预设的latency_target, latency_window, latency_percentile和iodepth。
Run&status&group&0&(all&jobs):&&
Fio支持把不同的测试聚合。例如，我可以用一个配置文件混合包含SSD和HDD，但是设置分组(group)把IO单独汇总。我现在还没涉及这个功能，但未来会用到。
MIXED:&io=12497MB,&aggrb=42653KB/s,&minb=277KB/s,&maxb=41711KB/s,&mint=300000msec,&maxt=300012msec&&
最后，汇总输出吞吐量和时间。io=表示总共完成的IO数量。在基于时间的测试中这是一个变量，在基于容量的测试中，这个值能匹配size参数。aggrb是所有进程/设备的汇总带宽。minb/maxb表示测量到的最小/最大带宽。mint/maxt表示测试的最短和最长耗时。和io=参数类似，时间值对于基于时间的测试应该能匹配runtime参数，对于基于容量的测试是一个变量。
由于我设置了unified_rw_reporting参数运行测试，所以只看到MIXED一行。如果禁用这个参数，对于读和写会有单独的行。
够简单吧？我未来的几周会花更多的时间研究fio，我会发布更多关于配置，输出和图表代码的例子。
参考知识库
* 以上用户言论只代表其个人观点，不代表CSDN网站的观点或立场
访问：46506次
排名：千里之外
原创：11篇
转载：65篇
(2)(1)(1)(2)(2)(4)(14)(2)(8)(1)(1)(2)(1)(2)(1)(3)(6)(8)(6)(2)(8)