fio存儲性能壓測

時間 2020-05-28

原文原文鏈接

1、環境說明：

這次演示是dellr620 物理服務器，系統爲centos7.6 x86_64位
硬件相關的信息以下：mysql

[root@mysql-redis scripts]# sh hw_msg1.sh 
+--------------------------------------------------------------+
| This Machine's Hyper-Threading is Enabled(recommend disable) |
+--------------------------------------------------------------+
Systembit    :    64
MEM info     :    20*4096 MB
Disk_totle   :    
Pro_SN_name  :    Product Name: PowerEdge R620 Serial Number: 5RV2962
System name  :    \S
Board_SN name:    Product Name: Serial Number:
CPU model    :    32  Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
CPU_phy_num  :    2
CPU_core_num :    8
CPU_logic_num:    16
Netcard info :    Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
+++++++++++++++++++++++++++++++++++
<<raid informations>>
 Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 893.75 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
--
Virtual Drive Information:
Virtual Drive: 1 (Target Id: 1)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 223.0 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
+++++++++++++++++++++++++++++++++++
<<Virtual Drive Information>>
Device Id: 3
PD Type: SATA
Raw Size: 894.252 GB [0x6fc81ab0 Sectors]
Inquiry Data: 56AB609UKCDU        TOSHIBA Q300.                           SAFM12.3
Device Speed: 6.0Gb/s 
Drive Temperature : N/A
Device Id: 0
PD Type: SATA
Raw Size: 223.570 GB [0x1bf244b0 Sectors]
Inquiry Data: PHDV723500E8240AGN  INTEL SSDSC2BB240G7                     N2010112
Device Speed: 6.0Gb/s 
Drive Temperature :25C (77.00 F)

2、安裝fio

1.redhat linux系統直接yum 安裝
yum install fio.x86_64
centos7.6系統上直接yum 安裝的話，默認的版本是fio.3.7linux

2.源碼編譯安裝
爲了測試異步io，須要安裝libaio ios

yum install libaio-devel

wget  -P /data/soft http://github.com/axboe/fio/archive/fio-3.7.tar.gz
tar xf /data/soft/fio.3.7.tar.gz -C /usr/local/
cd /usr/local/fio.3.7
./configure
make && make install

3、重要參數介紹

Fio各個版本的參數不盡相同，這裏只列出平常較爲廣發使用的重要參數
filename=/dev/sdb1 或者/data/soft/test1 ：指定測試硬盤名稱或者測試文件名稱（切忌選擇系統設備測試，會致使系統奔潰的）
direct=1: 測試過程繞過機器自帶的緩衝區，使得測試的結果更準確。
rw=randwrite: 隨機寫。 rw可取值：randwrite/randread/read/write/randrw/trim/randtrim/trimwrite ;分別是隨機寫，隨機讀，順序讀，順序寫，混合隨機讀寫，
bs=16k : 單次I/O塊大小爲16KB,單次I/O大小對IOPS的影響比較大，通常來講要想獲得最大的IOPS，bs越小越好。固然，這也和文件系統最小快的大小有關。通常linux下的文件分區最小快的大小爲512B,1KB,2KB,4KB,8KB等能夠根據文件系統類型進行選擇。
size=5g : 測試文件大小爲5GB
ioengine=libaio: I/O引擎使用libaio,libaio是異步的I/O引擎，一次提交一批I/O,而後等待這批I/O完成，這種方式減小了交互的次數，效率更好。FIO 至少支持13種不通的I/O引擎。sync,mmap,libaio,network等
iodepth=4 : I/O隊列深度，主要是根據設備的並行度來調整。一般有兩種I/O訪問方式。也就是同步I/O，和異步I/O.同步I/O一次只能發送一個I/o請求，等待內核完成才能夠返回。這對於單個線程來講iodepth老是小於1.想要iodepth大於1，能夠經過多線程併發執行作到。異步I/O一次提交多個I/O請求。等I/O完成或者間隔一段時間收割一次。iodepth通常用在異步I/O模型（libaio）,用於指定一次發起多少個I/O請求。Flash存儲通常有多個並行的單元。支持多個I/O併發執行。較合適異步的I/O
-numjobs=4: 使用4個線程對硬盤設備進行壓測對於Raid設備，如raid1,raid5或者並行高的設備，能夠考慮適當加大測試線程數-numjobs來測試。git

runtime=1000: 測試時間爲1000s.和參數size共同起做用，屬於雙限制，即達到任何一個限制都中止測試。github

rwmixread=70: 在混合讀寫模式下，讀佔70%
group_reporting: 設置顯示結果，指定group_reporting 將彙總每一個進程的信息。
-name iops_randwrite：改測試命名爲 iops_randwrite
--output TestResult.log: 測試結果指定輸出到文件TestResult.log 中redis

4、測試io性能

可使用fio模擬測試各類不一樣類型的io請求，入隨機寫，順序寫，隨機讀，順序讀。混合隨機讀寫sql

4.1隨機寫：

time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite   --output TestResult.log

命令介紹：
採用4個線程對 /data/soft/test1文件作持續時間爲60s，隊列深度爲1，塊大小爲4kb的direct y異步隨機寫（libaio）測試，該測試命名爲iops_randwrite.輸出結果不安4個jobs展現，而是按照group 組彙總，這樣得出的在此壓力下今後文件系統的隨機寫的IOPS。對於Raid設備，如raid1,raid5或者並行高的設備，能夠考慮適當加大測試線程數-numjobs來測試。
--output TestResult.log 輸出到指定的日誌centos

下面是隨機寫具體執行過程：服務器

[root@mysql-redis soft]# time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite 
iops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 4 processes
iops_randwrite: Laying out IO file (1 file / 5120MiB)
Jobs: 4 (f=4): [w(4)][100.0%][r=0KiB/s,w=68.2MiB/s][r=0,w=17.5k IOPS][eta 00m:00s]
iops_randwrite: (groupid=0, jobs=4): err= 0: pid=13505: Thu May 21 18:50:33 2020
  write: IOPS=39.2k, BW=153MiB/s (160MB/s)(9182MiB/60001msec)
    slat (usec): min=6, max=27813, avg=17.42, stdev=36.21
    clat (nsec): min=1177, max=123217k, avg=82183.17, stdev=323538.76
     lat (usec): min=53, max=123230, avg=99.84, stdev=325.69
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   51], 10.00th=[   57], 20.00th=[   62],
     | 30.00th=[   65], 40.00th=[   70], 50.00th=[   75], 60.00th=[   79],
     | 70.00th=[   84], 80.00th=[   91], 90.00th=[  103], 95.00th=[  117],
     | 99.00th=[  161], 99.50th=[  206], 99.90th=[  330], 99.95th=[  619],
     | 99.99th=[14615]
   bw (  KiB/s): min=17536, max=45304, per=25.12%, avg=39354.01, stdev=5145.44, samples=476
   iops        : min= 4384, max=11326, avg=9838.50, stdev=1286.36, samples=476
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=4.22%
  lat (usec)   : 100=84.07%, 250=11.50%, 500=0.12%, 750=0.03%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=4.33%, sys=19.34%, ctx=2824889, majf=0, minf=1185
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2350467,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=9182MiB (9628MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=0/2349456, merge=0/0, ticks=0/167181, in_queue=166516, util=96.46%

real    1m0.706s
user    0m11.167s
sys 0m47.017s

4.2隨機讀：

time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread

關於命令參數說明：
命令介紹：採用2個線程對 /data/soft/test1文件作持續時間爲60s，隊列深度爲1，塊大小爲4kb的direct y異步隨機讀（libaio）測試，該測試命名爲iops_randread.輸出結果不安2個jobs展現，而是按照group 組彙總，這樣得出的在此壓力下今後文件系統的隨機讀的IOPS多線程

下面是隨機讀具體執行過程：

[root@mysql-redis39 ~]# time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread 
iops_randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [r(2)][100.0%][r=36.7MiB/s,w=0KiB/s][r=9389,w=0 IOPS][eta 00m:00s]
iops_randread: (groupid=0, jobs=2): err= 0: pid=24566: Fri May 22 06:21:46 2020
   read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec)
    slat (usec): min=7, max=1480, avg=23.09, stdev= 8.09
    clat (usec): min=8, max=123984, avg=185.97, stdev=570.65
     lat (usec): min=136, max=123999, avg=209.50, stdev=570.70
    clat percentiles (usec):
     |  1.00th=[  139],  5.00th=[  141], 10.00th=[  145], 20.00th=[  169],
     | 30.00th=[  174], 40.00th=[  178], 50.00th=[  182], 60.00th=[  188],
     | 70.00th=[  198], 80.00th=[  204], 90.00th=[  212], 95.00th=[  219],
     | 99.00th=[  255], 99.50th=[  273], 99.90th=[  322], 99.95th=[  537],
     | 99.99th=[ 1549]
   bw (  KiB/s): min=13824, max=19616, per=49.99%, avg=18713.87, stdev=1060.66, samples=238
   iops        : min= 3456, max= 4904, avg=4678.43, stdev=265.16, samples=238
  lat (usec)   : 10=0.01%, 100=0.01%, 250=98.87%, 500=1.08%, 750=0.02%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 20=0.01%, 250=0.01%
  cpu          : usr=4.11%, sys=15.44%, ctx=561546, majf=0, minf=498
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=561503,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=36.6MiB/s (38.3MB/s), 36.6MiB/s-36.6MiB/s (38.3MB/s-38.3MB/s), io=2193MiB (2300MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=560393/2, merge=0/0, ticks=102314/0, in_queue=101910, util=97.80%

real    1m0.706s
user    0m5.649s
sys 0m19.196s

以上須要關注的指標：
read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec) io=2193MiB (2300MB) 這些參數能夠了解到iops和通途量
lat (usec): min=136, max=123999, avg=209.50, stdev=570.70 這些參數也要注意：io延遲平均在209.5微妙。

4.3順序寫：

time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write

說明：採用4個線程對/data/soft/test3文件作持續60s，隊列深度爲1的塊大小爲1M的direct異步順序寫（libaio）壓力測試該測試命名爲bw_write.輸出結果不安4個job分別展現，而是按照group彙總展現。

[root@mysql-redis ~]# time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write  
bw_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 5 processes
bw_write: Laying out IO file (1 file / 5120MiB)
Jobs: 5 (f=5): [W(5)][100.0%][r=0KiB/s,w=184MiB/s][r=0,w=184 IOPS][eta 00m:00s]
bw_write: (groupid=0, jobs=5): err= 0: pid=26278: Fri May 22 06:44:38 2020
  write: IOPS=319, BW=320MiB/s (335MB/s)(18.7GiB/60009msec)
    slat (usec): min=56, max=303, avg=140.02, stdev=29.86
    clat (msec): min=3, max=1092, avg=15.49, stdev=67.68
     lat (msec): min=3, max=1093, avg=15.63, stdev=67.68

4.4順序讀：

time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read

[root@mysql-redis ~]# time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read 
bw_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=259MiB/s,w=0KiB/s][r=259,w=0 IOPS][eta 00m:00s]
bw_read: (groupid=0, jobs=2): err= 0: pid=26789: Fri May 22 06:51:01 2020
   read: IOPS=255, BW=256MiB/s (268MB/s)(10.0GiB/40034msec)
    slat (usec): min=95, max=1149, avg=245.64, stdev=42.23
    clat (msec): min=4, max=130, avg= 7.57, stdev= 2.98
     lat (msec): min=4, max=130, avg= 7.81, stdev= 2.98

4.5混合讀寫

time fio  -rw=randrw  -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw

命令介紹：
2個線程對/data/soft/test2 文件作持續時間爲60s，隊列深度爲1，塊大小爲16kb的direct異步混合讀寫的（libaio）壓力測試，讀寫比僞7:3，測試被命名爲iops_randrw

rwmixwrite=30 意思是在混合讀寫的模式下，寫佔30%。
-rwmixread=70 意思是在混合讀寫的模式下，讀佔70%。

混合讀寫具體輸出結果：

[root@mysql-redis ~]# time fio  -rw=randrw  -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw

iops_randrw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [m(2)][100.0%][r=62.0MiB/s,w=27.2MiB/s][r=4030,w=1740 IOPS][eta 00m:00s]
iops_randrw: (groupid=0, jobs=2): err= 0: pid=27095: Fri May 22 06:55:08 2020
   read: IOPS=3931, BW=61.4MiB/s (64.4MB/s)(3686MiB/60001msec)
    slat (usec): min=8, max=6353, avg=36.20, stdev=19.77
    clat (nsec): min=1648, max=123466k, avg=346175.56, stdev=725094.33
     lat (usec): min=12, max=123516, avg=382.99, stdev=725.55
    clat percentiles (usec):
     |  1.00th=[  215],  5.00th=[  241], 10.00th=[  255], 20.00th=[  277],
     | 30.00th=[  293], 40.00th=[  310], 50.00th=[  322], 60.00th=[  334],
     | 70.00th=[  351], 80.00th=[  375], 90.00th=[  412], 95.00th=[  449],
     | 99.00th=[ 1221], 99.50th=[ 1467], 99.90th=[ 1631], 99.95th=[ 1827],
     | 99.99th=[16712]
   bw (  KiB/s): min=23328, max=33792, per=49.97%, avg=31431.39, stdev=1649.11, samples=238
   iops        : min= 1458, max= 2112, avg=1964.44, stdev=103.06, samples=238
  write: IOPS=1686, BW=26.4MiB/s (27.6MB/s)(1581MiB/60001msec)
    slat (usec): min=12, max=2005, avg=42.06, stdev=16.09
    clat (usec): min=3, max=123270, avg=225.73, stdev=432.84
     lat (usec): min=86, max=123320, avg=268.44, stdev=433.42
    clat percentiles (usec):
     |  1.00th=[   78],  5.00th=[   85], 10.00th=[  100], 20.00th=[  129],
     | 30.00th=[  149], 40.00th=[  174], 50.00th=[  202], 60.00th=[  233],
     | 70.00th=[  265], 80.00th=[  306], 90.00th=[  359], 95.00th=[  408],
     | 99.00th=[  510], 99.50th=[ 1090], 99.90th=[ 1549], 99.95th=[ 1614],
     | 99.99th=[ 5080]
   bw (  KiB/s): min= 9152, max=15329, per=49.97%, avg=13484.37, stdev=892.91, samples=238
   iops        : min=  572, max=  958, avg=842.76, stdev=55.80, samples=238
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=3.06%
  lat (usec)   : 250=22.22%, 500=72.91%, 750=0.76%, 1000=0.05%
  lat (msec)   : 2=0.96%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=4.64%, sys=15.91%, ctx=346233, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=235884,101198,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=61.4MiB/s (64.4MB/s), 61.4MiB/s-61.4MiB/s (64.4MB/s-64.4MB/s), io=3686MiB (3865MB), run=60001-60001msec
  WRITE: bw=26.4MiB/s (27.6MB/s), 26.4MiB/s-26.4MiB/s (27.6MB/s-27.6MB/s), io=1581MiB (1658MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=274477/101294, merge=0/0, ticks=88860/19119, in_queue=107576, util=95.82%

real    1m0.690s
user    0m6.300s
sys 0m19.718s

4.6SSD性能測試配置文件

配置文件內容以下：

[root@mysql-redis scripts]# cat /data/soft/ssd-test.fio 
##2020-05-24
[global]
bs=4k
ioengine=libaio
iodepth=4
numjobs=1
group_reporting
size=5g
direct=1
runtime=60
directory=/data/soft
filename=ssd.test.file
#inflate-log=/data/soft/TestResult.log
#output=/data/soft/TestResult.log
[seq-read]
name=iops_seqread
rw=read
stonewall

[rand-read]
name=iops_randread
rw=randread
stonewall

[seq-write]
name=iops_seqwrite
rw=write
stonewall

[rand-write]
name=iops_randwrite
rw=randwrite
stonewall

[randrw]
name=iops_randrw
rwmixread=70
rw=randrw
stonewall

下面是fio測試指定配置文件輸出的內容：

[root@mysql-redis soft]# fio ssd-test.fio 
iops_seqread: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randread: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_seqwrite: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randwrite: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randrw: (g=4): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.7
Starting 5 processes
iops_seqread: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [_(4),m(1)][90.5%][r=47.8MiB/s,w=20.7MiB/s][r=12.2k,w=5308 IOPS][eta 00m:21s]     
iops_seqread: (groupid=0, jobs=1): err= 0: pid=2142: Sun May 24 20:06:20 2020
   read: IOPS=57.5k, BW=225MiB/s (236MB/s)(5120MiB/22782msec)
    slat (usec): min=3, max=1943, avg= 5.13, stdev= 2.21
    clat (usec): min=25, max=12467, avg=63.13, stdev=202.65
     lat (usec): min=40, max=12472, avg=68.41, stdev=202.72
    clat percentiles (usec):
     |  1.00th=[   45],  5.00th=[   53], 10.00th=[   55], 20.00th=[   56],
     | 30.00th=[   56], 40.00th=[   56], 50.00th=[   57], 60.00th=[   57],
     | 70.00th=[   57], 80.00th=[   58], 90.00th=[   59], 95.00th=[   62],
     | 99.00th=[   69], 99.50th=[   79], 99.90th=[ 2073], 99.95th=[ 5080],
     | 99.99th=[10814]
   bw (  KiB/s): min=43579, max=193812, per=75.80%, avg=174446.82, stdev=47419.87, samples=45
   iops        : min=10894, max=48453, avg=43611.36, stdev=11855.06, samples=45
  lat (usec)   : 50=3.39%, 100=96.25%, 250=0.08%, 500=0.07%, 750=0.04%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.06%, 4=0.04%, 10=0.05%, 20=0.01%
  cpu          : usr=19.83%, sys=39.29%, ctx=1222141, majf=0, minf=114
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_randread: (groupid=1, jobs=1): err= 0: pid=2170: Sun May 24 20:06:20 2020
   read: IOPS=21.7k, BW=84.9MiB/s (89.0MB/s)(5094MiB/60001msec)
    slat (usec): min=3, max=1068, avg=12.62, stdev= 6.14
    clat (usec): min=42, max=38568, avg=167.76, stdev=315.76
     lat (usec): min=62, max=38588, avg=180.75, stdev=316.07
    clat percentiles (usec):
     |  1.00th=[  130],  5.00th=[  135], 10.00th=[  137], 20.00th=[  141],
     | 30.00th=[  143], 40.00th=[  147], 50.00th=[  151], 60.00th=[  155],
     | 70.00th=[  161], 80.00th=[  172], 90.00th=[  192], 95.00th=[  206],
     | 99.00th=[  251], 99.50th=[  273], 99.90th=[ 3982], 99.95th=[ 7308],
     | 99.99th=[14091]
   bw (  KiB/s): min= 5397, max=74196, per=79.00%, avg=68682.74, stdev=15713.88, samples=119
   iops        : min= 1349, max=18549, avg=17170.32, stdev=3928.48, samples=119
  lat (usec)   : 50=0.01%, 100=0.02%, 250=98.94%, 500=0.82%, 750=0.02%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.03%, 4=0.04%, 10=0.07%, 20=0.02%, 50=0.01%
  cpu          : usr=11.52%, sys=37.46%, ctx=668232, majf=0, minf=319
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1304064,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_seqwrite: (groupid=2, jobs=1): err= 0: pid=2231: Sun May 24 20:06:20 2020
  write: IOPS=47.1k, BW=184MiB/s (193MB/s)(5120MiB/27848msec)
    slat (usec): min=4, max=139, avg= 7.50, stdev= 2.96
    clat (usec): min=27, max=15374, avg=76.14, stdev=278.53
     lat (usec): min=49, max=15381, avg=83.78, stdev=279.26
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   59], 10.00th=[   60], 20.00th=[   61],
     | 30.00th=[   62], 40.00th=[   64], 50.00th=[   65], 60.00th=[   67],
     | 70.00th=[   68], 80.00th=[   71], 90.00th=[   73], 95.00th=[   74],
     | 99.00th=[   81], 99.50th=[   89], 99.90th=[ 3326], 99.95th=[ 3392],
     | 99.99th=[15139]
   bw (  KiB/s): min= 3769, max=175262, per=80.83%, avg=152166.87, stdev=43009.10, samples=55
   iops        : min=  942, max=43815, avg=38041.35, stdev=10752.22, samples=55
  lat (usec)   : 50=0.24%, 100=99.43%, 250=0.07%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.21%, 10=0.01%, 20=0.03%
  cpu          : usr=11.79%, sys=43.48%, ctx=652945, majf=0, minf=95
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_randwrite: (groupid=3, jobs=1): err= 0: pid=2262: Sun May 24 20:06:20 2020
  write: IOPS=47.2k, BW=185MiB/s (194MB/s)(5120MiB/27743msec)
    slat (usec): min=4, max=415, avg= 7.49, stdev= 2.82
    clat (usec): min=26, max=21032, avg=75.63, stdev=276.91
     lat (usec): min=48, max=21058, avg=83.27, stdev=277.60
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   60], 10.00th=[   60], 20.00th=[   62],
     | 30.00th=[   62], 40.00th=[   64], 50.00th=[   65], 60.00th=[   67],
     | 70.00th=[   68], 80.00th=[   70], 90.00th=[   73], 95.00th=[   74],
     | 99.00th=[   80], 99.50th=[   86], 99.90th=[ 3326], 99.95th=[ 3392],
     | 99.99th=[15139]
   bw (  KiB/s): min= 3175, max=142274, per=68.21%, avg=128901.20, stdev=33671.41, samples=55
   iops        : min=  793, max=35568, avg=32224.95, stdev=8417.81, samples=55
  lat (usec)   : 50=0.21%, 100=99.50%, 250=0.04%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.20%, 10=0.01%, 20=0.02%, 50=0.01%
  cpu          : usr=12.97%, sys=43.05%, ctx=654916, majf=0, minf=258
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_randrw: (groupid=4, jobs=1): err= 0: pid=2292: Sun May 24 20:06:20 2020
   read: IOPS=11.3k, BW=44.3MiB/s (46.5MB/s)(2659MiB/60001msec)
    slat (usec): min=4, max=594, avg=13.08, stdev= 6.88
    clat (usec): min=61, max=45073, avg=244.45, stdev=441.39
     lat (usec): min=134, max=45115, avg=257.84, stdev=441.61
    clat percentiles (usec):
     |  1.00th=[  137],  5.00th=[  143], 10.00th=[  151], 20.00th=[  176],
     | 30.00th=[  190], 40.00th=[  202], 50.00th=[  212], 60.00th=[  221],
     | 70.00th=[  233], 80.00th=[  258], 90.00th=[  330], 95.00th=[  359],
     | 99.00th=[  474], 99.50th=[ 1270], 99.90th=[ 6521], 99.95th=[10028],
     | 99.99th=[17695]
   bw (  KiB/s): min= 3072, max=51824, per=99.92%, avg=45336.55, stdev=11250.66, samples=119
   iops        : min=  768, max=12958, avg=11334.12, stdev=2812.66, samples=119
  write: IOPS=4860, BW=18.0MiB/s (19.9MB/s)(1139MiB/60001msec)
    slat (usec): min=6, max=874, avg=16.94, stdev= 7.42
    clat (usec): min=40, max=31265, avg=192.80, stdev=413.19
     lat (usec): min=56, max=31306, avg=210.09, stdev=413.32
    clat percentiles (usec):
     |  1.00th=[   52],  5.00th=[   58], 10.00th=[   62], 20.00th=[   78],
     | 30.00th=[  145], 40.00th=[  169], 50.00th=[  188], 60.00th=[  202],
     | 70.00th=[  212], 80.00th=[  227], 90.00th=[  251], 95.00th=[  277],
     | 99.00th=[  619], 99.50th=[ 1205], 99.90th=[ 5997], 99.95th=[ 9634],
     | 99.99th=[16581]
   bw (  KiB/s): min= 1184, max=22176, per=99.94%, avg=19428.64, stdev=4847.41, samples=119
   iops        : min=  296, max= 5544, avg=4857.13, stdev=1211.85, samples=119
  lat (usec)   : 50=0.12%, 100=7.22%, 250=74.36%, 500=17.22%, 750=0.24%
  lat (usec)   : 1000=0.10%
  lat (msec)   : 2=0.52%, 4=0.06%, 10=0.11%, 20=0.04%, 50=0.01%
  cpu          : usr=7.88%, sys=29.46%, ctx=340101, majf=0, minf=374
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=680630,291634,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=5120MiB (5369MB), run=22782-22782msec

Run status group 1 (all jobs):
   READ: bw=84.9MiB/s (89.0MB/s), 84.9MiB/s-84.9MiB/s (89.0MB/s-89.0MB/s), io=5094MiB (5341MB), run=60001-60001msec

Run status group 2 (all jobs):
  WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5120MiB (5369MB), run=27848-27848msec

Run status group 3 (all jobs):
  WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5120MiB (5369MB), run=27743-27743msec

Run status group 4 (all jobs):
   READ: bw=44.3MiB/s (46.5MB/s), 44.3MiB/s-44.3MiB/s (46.5MB/s-46.5MB/s), io=2659MiB (2788MB), run=60001-60001msec
  WRITE: bw=18.0MiB/s (19.9MB/s), 18.0MiB/s-18.0MiB/s (19.9MB/s-19.9MB/s), io=1139MiB (1195MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=3294785/2912828, merge=0/3, ticks=430527/238028, in_queue=666582, util=99.09%
[root@mysql-redis soft]#

5、總結

關於fio測試硬盤的建議：
1、建議使用順序I/O和較大的blocksize 來測試設備的通吐量和延遲
2、建議使用隨機I/O和較小的blocksize來測試設備的IOPS和延遲
3、在配置numjobs和iodepth 測試底層存儲性能時，建議要深刻了解應用到底採用的是同步的io仍是異步的io（是多進程併發i/o請求仍是一次提交一批的i/o的請求）
提示：對fio壓測磁盤性能感興趣的話，能夠參考下ezfio(直接在github.com上搜索ezfio)
重要提示：fio是支持對裸設備直接進行I/O測試的，在測試裸設備時，必定要避開系統所在的裸設備，不然會致使系統奔潰。
例如：

[root@mysql-redis soft]# df -h / /boot 
文件系統        容量  已用  可用 已用% 掛載點
/dev/sdb5       207G  7.5G  200G    4% /
/dev/sdb1       297M  120M  177M   41% /boot

/dev/sdb /dev/sdb5 /dev/sdb1 這些設備是絕對禁止進行fio測試的，
同時在測試其餘的設備裸設備時，必定要提早肯定好此裸設備上是否有重要的數據，必定要重要的設備提早作好備份。

以上就是我這邊對fio測試硬盤的簡單學習總結，僅供參考，若有網友直接複製應用形成損失，本博文不負擔任何責任。
參考文檔：
http://www.ssdfans.com/?p=6087
https://linux.die.net/man/1/fio
https://blog.csdn.net/don_chiang709/article/details/92628623