iowait太高處理

網管告警：

告警主機：YiDHLWJKFZ-js-app-16 主機IP：192.168.***.*** 告警項目：system.cpu.util[,iowait] 告警時間：2019.02.25 15:18:48 告警等級：Warning 告警信息：Disk I/O is overloaded on YiDHLWJKFZ-js-app-16 問題詳情：CPU iowait time:20.14 % 當前狀態：PROBLEM:20.14 % 事件ID：1003377

 

top查看：（wa值爲17.7%）

[root@localhost vmuser]# top top - 15:31:31 up  1:17,  1 user,  load average: 3.56, 3.45, 3.40 Tasks: 232 total,   1 running, 231 sleeping,   0 stopped,   0 zombie Cpu(s): 1.0%us,  2.6%sy,  0.0%ni, 78.2%id, 17.7%wa,  0.2%hi,  0.3%si,  0.0%st Mem: 16467984k total, 2823808k used, 13644176k free, 331524k buffers Swap: 16383992k total, 0k used, 16383992k free, 565436k cached PID USER PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND 389 root      20   0     0    0    0 D  5.3  0.0   2:16.37 jbd2/sda3-8                                                                                               
 2483 postfix   20   0  104m  46m 2244 S  3.7  0.3   1:45.87 qmgr 1531 root      20   0  242m 3484  920 S  3.0  0.0   1:17.92 rsyslogd 23237 postfix   20   0 62740 3556 2616 S  1.7  0.0   0:00.05 local 2476 root      20   0 62360 2844 2128 S  1.3  0.0   0:44.87 master 23230 postfix   20   0 62584 2984 2256 S  1.3  0.0   0:00.05 cleanup 23238 root      20   0 62740 3564 2616 D  1.3  0.0   0:00.04 local 2488 postfix   20   0 62540 2980 2144 S  1.0  0.0   0:29.32 trivial-rewrite 23229 postfix   20   0 62584 2984 2256 S  1.0  0.0   0:00.06 cleanup 2482 postfix   20   0 62440 2900 2164 D  0.7  0.0   0:39.73 pickup 23231 postfix   20   0 62584 2988 2256 D  0.7  0.0   0:00.03 cleanup

iostat查看：（iowait值一直太高）

[root@localhost vmuser]# iostat  -x 2 5 Linux 2.6.32-71.el6.x86_64 (localhost.localdomain)      02/25/2019      _x86_64_        (8 CPU) avg-cpu:  %user   %nice %system %iowait  %steal   %idle 2.96    0.00    2.14   19.78    0.00   75.12 Device: rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util sda 12.58   361.59   28.73 1621.40   616.25 15618.31     9.84     3.72    2.25   0.53  86.86 sdb 307.71     0.01    6.02    0.00  1255.11     0.05   208.45     0.14   22.52  14.07   8.47 avg-cpu:  %user   %nice %system %iowait  %steal   %idle 0.84    0.00    2.12   21.38    0.00   75.66 Device: rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util sda 0.00   465.00   36.00 1641.00   288.00 16508.00    10.02     3.27    1.98   0.58  97.30 sdb 0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00 avg-cpu:  %user   %nice %system %iowait  %steal   %idle 1.44    0.00    4.49   15.09    0.00   78.97 Device: rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util sda 0.00  1167.00   18.00 4016.50   144.00 40720.00    10.13     3.61    0.88   0.22  87.95 sdb 0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00 avg-cpu:  %user   %nice %system %iowait  %steal   %idle 0.44    0.00    1.19   24.88    0.00   73.50 Device: rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util sda 0.00   351.00   22.50  879.00   180.00  9696.00    10.96     3.25    3.68   1.10  98.90 sdb 0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00 avg-cpu:  %user   %nice %system %iowait  %steal   %idle 0.87    0.00    2.31   20.07    0.00   76.75 Device: rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util sda 0.00   435.50   19.00 2666.50   152.00 24512.00     9.18     3.23    1.21   0.35  93.00 sdb 0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00 [root@localhost vmuser]#

 

iotop查看：

[root@localhost vmuser]# iotop Total DISK READ: 124.32 K/s | Total DISK WRITE: 6.69 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 389 be/3 root        0.00 B/s  236.99 K/s  0.00 % 92.26 % [jbd2/sda3-8] 2482 be/4 postfix   124.32 K/s    0.00 B/s  0.00 % 88.09 % pickup -l -t fifo -u 2483 be/4 postfix     0.00 B/s    0.00 B/s  0.00 %  6.21 % qmgr -l -t fifo -u 23557 be/4 postfix     0.00 B/s   58.28 K/s  0.00 %  6.02 % cleanup -z -t unix -u 1532 be/4 root        0.00 B/s   27.20 K/s  0.00 %  3.82 % rsyslogd -c 4
23564 be/4 postfix     0.00 B/s  158.28 K/s  0.00 %  1.79 % cleanup -z -t unix -u 23545 be/4 postfix     0.00 B/s  260.31 K/s  0.00 %  1.62 % local -t unix unix -u 23570 be/4 postfix     0.00 B/s   19.43 K/s  0.00 %  0.75 % bounce -z -t unix -u 23556 be/4 postfix     0.00 B/s   58.28 K/s  0.00 %  0.75 % cleanup -z -t unix -uth :/usr/local/sa/sa-agent/main/~ocal/sa/sa-agent com.transfar.sa.agent.main.Bootstrap 17 rt/4 root        0.00 B/s   54.39 B/s  0.00 %  0.67 % bounce -z -t unix -u 9
23578 rt/4 postfix     0.00 B/s   38.85 B/s  0.00 %  0.60 % bounce -z -t unix -u 19 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/5] 20 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/5] 21 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/6] 23 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/6] 23 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/6]]

關閉沒必要要的postfix進程

[root@localhost vmuser]# service postfix stop Shutting down postfix: [ OK ] [root@localhost vmuser]#