RPS和RFS網卡多隊列性能調優實踐
前言
爲了解決LVS ksoftirqd CPU使用率100%致使網卡軟中斷丟包，我和同事們一塊兒搜索了大量的資料去分析問題，特別是感謝美團技術團隊的分享幫助咱們快速梳理優化思路，最後明確瞭如何重構RPS和RFS網卡多隊列的優化腳本。我的認爲這是一個你們可能廣泛會遇到的問題，文章內的分析思路和解決方案未必是最優解，也歡迎各位分享本身的解決方法。html
RPS和RFS網卡多隊列性能調優實踐
更新歷史
2019年07月03日 - 初稿node
閱讀原文 - https://wsgzao.github.io/post...linux
擴展閱讀git
Redis 高負載下的中斷優化 - https://tech.meituan.com/2018...github
事故覆盤
咱們遇到的問題屬於計劃外的incident，現象是某產品用戶在線率忽然下降，LVS Master同時收到CPU High Load告警，檢查發現該節點出現網卡大量斷開重連和丟包狀況，應急切換到LVS Slave也出現上述問題，在排除掉流量異常和外部攻擊後選擇切換DNS到背後的Nginx Real Servers後服務逐步恢復。redis
覆盤核心緣由在於系統初始化時rps優化腳本沒有成功執行，這個腳本起初是由於早期DBA團隊遇到過CPU負載較高致使網卡異常，這個優化腳本也一直傳承至今，卻已經沒有人知道爲何添加。如今大多數服務器沒有執行成功而被你們一直所忽視顯然也是post check沒有作到位。在早期你們都停留在Bash Shell運維的階段，沒有專職的團隊來管理確實容易失控，好在如今能夠基於Ansible來作初始化和檢查，運維的壓力也減輕了一部分。docker
經過Google搜索相關知識的過程當中，咱們也發如今很多人都會遇到這樣相似的問題。好比這篇文章提到lvs/irqubuntu
lvs 的性能問題，軟中斷耗盡 CPU 單核後到達處理極限緩存
-
瓶頸現象：當壓力較大時，Lvs 服務器 CPU 的其中一個核使用率達到 100%（處理軟中斷）。bash
- 當 Lvs 服務器處理軟中斷的那個核使用率達到 100%，就到達系統處理上限。
- 佔用 CPU 的是進程 「ksoftirqd」，它未能使用到多核。
- 作雙網卡綁定，而後調試內核 SMP，中斷主要是來自網卡的，不是 LVS 自己。須要把 2 個網卡來的 IRQ 均衡在雙核 CPU 上面。
和華爲的工程師們在交換經驗的時候對方分享了一個關於RSS和RPS關係圖，以後的內容還會引用美團技術團隊的分析
咱們遇到的狀況是缺乏可用服務器資源選擇把用戶外部請求流量和Codis Cache Cluster內部流量臨時混在了同一個LVS上，雖然看上去CPU和traffic的總體壓力都不算高，可是CPU的處理壓力可能剛好集中在了和外網Bond1網卡相同的Core上最後引發了ksoftirqd軟中斷，而內網Bond0網卡就沒有監控到任何丟包。雖然咱們也有正常開啓irqbalance，但不清楚是否是由於受到cpupower performance和NUMA的影響最後也沒能阻止事故的發生，最終的優化方案主要是手動開啓RPS和RFS，大體步驟以下：
- set cpupower
cpupower frequency-set -g performance，CPU 優化建議使用 cpupower 設置 CPU Performance 模式 - activate rps/rfs by script: rps.sh
- double ring buffer size:
ethtool -G p1p1 [rx|tx] 4096, checkethtool -g p1p1 - double NAPI poll budget:
sysctl -w net.core.netdev_budget=600 - add zabbix monitor on net.if.in[eth0,errors,dropped,overruns]
#!/bin/bash
# chkconfig: 2345 90 60
### BEGIN INIT INFO
# Provides: rps
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: enable rps config for ubuntu
# Description: enabele rps which is a kernel tweak for network performance
### END INIT INFO
NAME=rps
DESC=rps
# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
# cpupower frequency-set -g performance
# activate rps/rfs by script: https://gist.github.com/wsgzao/18828f69147635f3e38a14690a633daf
# double ring buffer size: ethtool -G p1p1 [rx|tx] 4096, ethtool -g p1p1
# double NAPI poll budget: sysctl -w net.core.netdev_budget=600
rps() {
net_interface=`ip link show | grep "state UP" | awk '{print $2}' | egrep -v '^docker|^veth' | tr ":\n" " "`
for em in ${net_interface[@]}
do
rq_count=`ls /sys/class/net/$em/queues/rx-* -d | wc -l`
rps_flow_cnt_value=`expr 32768 / $rq_count`
for ((i=0; i< $rq_count; i++))
do
echo $rps_flow_cnt_value > /sys/class/net/$em/queues/rx-$i/rps_flow_cnt
done
flag=0
while [ -f /sys/class/net/$em/queues/rx-$flag/rps_cpus ]
do
echo `cat /sys/class/net/$em/queues/rx-$flag/rps_cpus | sed 's/0/f/g' ` > /sys/class/net/$em/queues/rx-$flag/rps_cpus
flag=$(($flag+1))
done
done
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
sysctl -p
}
check_rps() {
ni_list=`ip link show | grep "state UP" | awk '{print $2}' | egrep -v "^docker|^veth" | tr ":\n" " "`
for n in $ni_list
do
rx_queues=`ls /sys/class/net/$n/queues/ | grep "rx-[0-9]"`
for q in $rx_queues
do
rps_cpus=`cat /sys/class/net/$n/queues/$q/rps_cpus`
rps_flow_cnt=`cat /sys/class/net/$n/queues/$q/rps_flow_cnt`
echo "[$n]" $q "--> rps_cpus =" $rps_cpus ", rps_flow_cnt =" $rps_flow_cnt
done
done
rps_sock_flow_entries=`cat /proc/sys/net/core/rps_sock_flow_entries`
echo "rps_sock_flow_entries =" $rps_sock_flow_entries
}
case "$1" in
start)
echo -n "Starting $DESC: "
rps
check_rps
;;
stop)
echo -n "Stop is not supported. "
;;
restart|reload|force-reload)
echo -n "Restart is not supported. "
;;
status)
check_rps
;;
*)
echo "Usage: $0 [start|status]"
;;
esac
exit 0
Scaling in the Linux Networking Stack
瞭解RSS、RPS、RFS等基礎知識很重要
This document describes a set of complementary techniques in the Linux
networking stack to increase parallelism and improve performance for
multi-processor systems.
The following technologies are described:
- RSS: Receive Side Scaling
- RPS: Receive Packet Steering
- RFS: Receive Flow Steering
- Accelerated Receive Flow Steering
- XPS: Transmit Packet Steering
https://www.kernel.org/doc/Do...
RPS
Receive Packet Steering (RPS) is similar to RSS in that it is used to direct packets to specific CPUs for processing. However, RPS is implemented at the software level, and helps to prevent the hardware queue of a single network interface card from becoming a bottleneck in network traffic.
RPS has several advantages over hardware-based RSS:
- RPS can be used with any network interface card.
- It is easy to add software filters to RPS to deal with new protocols.
- RPS does not increase the hardware interrupt rate of the network device. However, it does introduce inter-processor interrupts.
RPS is configured per network device and receive queue, in the /sys/class/net/*device*/queues/*rx-queue*/rps_cpus file, where device is the name of the network device (such as eth0) and rx-queue is the name of the appropriate receive queue (such as rx-0).
The default value of the rps_cpus file is zero. This disables RPS, so the CPU that handles the network interrupt also processes the packet.
To enable RPS, configure the appropriate rps_cpus file with the CPUs that should process packets from the specified network device and receive queue.
The rps_cpus files use comma-delimited CPU bitmaps. Therefore, to allow a CPU to handle interrupts for the receive queue on an interface, set the value of their positions in the bitmap to 1. For example, to handle interrupts with CPUs 0, 1, 2, and 3, set the value of rps_cpus to 00001111 (1+2+4+8), or f (the hexadecimal value for 15).
For network devices with single transmit queues, best performance can be achieved by configuring RPS to use CPUs in the same memory domain. On non-NUMA systems, this means that all available CPUs can be used. If the network interrupt rate is extremely high, excluding the CPU that handles network interrupts may also improve performance.
For network devices with multiple queues, there is typically no benefit to configuring both RPS and RSS, as RSS is configured to map a CPU to each receive queue by default. However, RPS may still be beneficial if there are fewer hardware queues than CPUs, and RPS is configured to use CPUs in the same memory domain.
RFS
Receive Flow Steering (RFS) extends RPS behavior to increase the CPU cache hit rate and thereby reduce network latency. Where RPS forwards packets based solely on queue length, RFS uses the RPS backend to calculate the most appropriate CPU, then forwards packets based on the location of the application consuming the packet. This increases CPU cache efficiency.
RFS is disabled by default. To enable RFS, you must edit two files:
/proc/sys/net/core/rps_sock_flow_entries
Set the value of this file to the maximum expected number of concurrently active connections. We recommend a value of 32768 for moderate server loads. All values entered are rounded up to the nearest power of 2 in practice.
/sys/class/net/*device*/queues/*rx-queue*/rps_flow_cnt
Replace device with the name of the network device you wish to configure (for example, eth0), and rx-queue with the receive queue you wish to configure (for example, rx-0).
Set the value of this file to the value of rps_sock_flow_entries divided by N, where N is the number of receive queues on a device. For example, if rps_flow_entries is set to 32768 and there are 16 configured receive queues, rps_flow_cnt should be set to 2048. For single-queue devices, the value of rps_flow_cnt is the same as the value of rps_sock_flow_entries.
Data received from a single sender is not sent to more than one CPU. If the amount of data received from a single sender is greater than a single CPU can handle, configure a larger frame size to reduce the number of interrupts and therefore the amount of processing work for the CPU. Alternatively, consider NIC offload options or faster CPUs.
Consider using numactl or taskset in conjunction with RFS to pin applications to specific cores, sockets, or NUMA nodes. This can help prevent packets from being processed out of order.
瞭解接收數據包的流程
這裏摘取了美團技術團隊的分析，在此表示感謝
接收數據包是一個複雜的過程，涉及不少底層的技術細節，但大體須要如下幾個步驟：
- 網卡收到數據包。
- 將數據包從網卡硬件緩存轉移到服務器內存中。
- 通知內核處理。
- 通過 TCP/IP 協議逐層處理。
- 應用程序經過
read()從socket buffer讀取數據。
將網卡收到的數據包轉移到主機內存（NIC 與驅動交互）
NIC 在接收到數據包以後，首先須要將數據同步到內核中，這中間的橋樑是 rx ring buffer。它是由 NIC 和驅動程序共享的一片區域，事實上，rx ring buffer 存儲的並非實際的 packet 數據，而是一個描述符，這個描述符指向了它真正的存儲地址，具體流程以下：
- 驅動在內存中分配一片緩衝區用來接收數據包，叫作
sk_buffer； - 將上述緩衝區的地址和大小（即接收描述符），加入到
rx ring buffer。描述符中的緩衝區地址是 DMA 使用的物理地址； - 驅動通知網卡有一個新的描述符；
- 網卡從
rx ring buffer中取出描述符，從而獲知緩衝區的地址和大小； - 網卡收到新的數據包；
- 網卡將新數據包經過 DMA 直接寫到
sk_buffer中。
當驅動處理速度跟不上網卡收包速度時，驅動來不及分配緩衝區，NIC 接收到的數據包沒法及時寫到 sk_buffer，就會產生堆積，當 NIC 內部緩衝區寫滿後，就會丟棄部分數據，引發丟包。這部分丟包爲 rx_fifo_errors，在 /proc/net/dev 中體現爲 fifo 字段增加，在 ifconfig 中體現爲 overruns 指標增加。
通知系統內核處理（驅動與 Linux 內核交互）
這個時候，數據包已經被轉移到了 sk_buffer 中。前文提到，這是驅動程序在內存中分配的一片緩衝區，而且是經過 DMA 寫入的，這種方式不依賴 CPU 直接將數據寫到了內存中，意味着對內核來講，其實並不知道已經有新數據到了內存中。那麼如何讓內核知道有新數據進來了呢？答案就是中斷，經過中斷告訴內核有新數據進來了，並須要進行後續處理。
提到中斷，就涉及到硬中斷和軟中斷，首先須要簡單瞭解一下它們的區別：
- 硬中斷： 由硬件本身生成，具備隨機性，硬中斷被 CPU 接收後，觸發執行中斷處理程序。中斷處理程序只會處理關鍵性的、短期內能夠處理完的工做，剩餘耗時較長工做，會放到中斷以後，由軟中斷來完成。硬中斷也被稱爲上半部分。
- 軟中斷： 由硬中斷對應的中斷處理程序生成，每每是預先在代碼裏實現好的，不具備隨機性。（除此以外，也有應用程序觸發的軟中斷，與本文討論的網卡收包無關。）也被稱爲下半部分。
當 NIC 把數據包經過 DMA 複製到內核緩衝區 sk_buffer 後，NIC 當即發起一個硬件中斷。CPU 接收後，首先進入上半部分，網卡中斷對應的中斷處理程序是網卡驅動程序的一部分，以後由它發起軟中斷，進入下半部分，開始消費 sk_buffer 中的數據，交給內核協議棧處理。
經過中斷，可以快速及時地響應網卡數據請求，但若是數據量大，那麼會產生大量中斷請求，CPU 大部分時間都忙於處理中斷，效率很低。爲了解決這個問題，如今的內核及驅動都採用一種叫 NAPI（new API）的方式進行數據處理，其原理能夠簡單理解爲 中斷 + 輪詢，在數據量大時，一次中斷後經過輪詢接收必定數量包再返回，避免產生屢次中斷。
什麼是中斷？
因爲接收來自外圍硬件 (相對於 CPU 和內存) 的異步信號或者來自軟件的同步信號，而進行相應的硬件、軟件處理；發出這樣的信號稱爲進行中斷請求 (interrupt request, IRQ)
[](#硬中斷與軟中斷 "硬中斷與軟中斷?")硬中斷與軟中斷？
- 硬中斷：外圍硬件發給 CPU 或者內存的異步信號就稱之爲硬中斷
- 軟中斷：由軟件系統自己發給操做系統內核的中斷信號，稱之爲軟中斷。一般是由硬中斷處理程序或進程調度程序對操做系統內核的中斷，也就是咱們常說的系統調用 (System Call)
[](#硬中斷與軟中斷之區別與聯繫？ "硬中斷與軟中斷之區別與聯繫？")硬中斷與軟中斷之區別與聯繫？
- 硬中斷是有外設硬件發出的，須要有中斷控制器之參與。其過程是外設偵測到變化，告知中斷控制器，中斷控制器經過 CPU 或內存的中斷腳通知 CPU，而後硬件進行程序計數器及堆棧寄存器之現場保存工做（引起上下文切換），並根據中斷向量調用硬中斷處理程序進行中斷處理
- 軟中斷則一般是由硬中斷處理程序或者進程調度程序等軟件程序發出的中斷信號，無需中斷控制器之參與，直接以一個 CPU 指令之形式指示 CPU 進行程序計數器及堆棧寄存器之現場保存工做 (亦會引起上下文切換)，並調用相應的軟中斷處理程序進行中斷處理 (即咱們一般所言之系統調用)
- 硬中斷直接以硬件的方式引起，處理速度快。軟中斷以軟件指令之方式適合於對響應速度要求不是特別嚴格的場景
- 硬中斷經過設置 CPU 的屏蔽位可進行屏蔽，軟中斷則因爲是指令之方式給出，不能屏蔽
- 硬中斷髮生後，一般會在硬中斷處理程序中調用一個軟中斷來進行後續工做的處理
- 硬中斷和軟中斷均會引發上下文切換 (進程 / 線程之切換)，進程切換的過程是差很少的
查看中斷狀況
1.top 按下數字鍵 1
- hi : time spent servicing hardware interrupts
- si : time spent servicing software interrupts
2.mpstat -P ALL 2
- irq 爲硬中斷
- soft 爲軟中斷
mpstat使用介紹和輸出參數詳解 - https://wsgzao.github.io/post...
[root@localhost wangao]# top top - 19:17:57 up 7 days, 7:11, 1 user, load average: 0.00, 0.01, 0.05 Tasks: 338 total, 1 running, 337 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 2.0 si, 0.0 st %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st %Cpu2 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st %Cpu6 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st %Cpu8 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st %Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st %Cpu10 : 0.0 us, 0.0 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 1.3 si, 0.0 st %Cpu11 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st %Cpu12 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st %Cpu13 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st %Cpu14 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st %Cpu15 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st %Cpu16 : 0.0 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 2.7 si, 0.0 st %Cpu17 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st %Cpu18 : 0.3 us, 0.7 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st %Cpu19 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st %Cpu20 : 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st %Cpu21 : 0.0 us, 0.0 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st %Cpu22 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st %Cpu23 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st %Cpu24 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st %Cpu25 : 0.0 us, 0.0 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st %Cpu26 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st %Cpu27 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st %Cpu28 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st %Cpu29 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st %Cpu30 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st %Cpu31 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st KiB Mem : 65688788 total, 62127224 free, 1742128 used, 1819436 buff/cache KiB Swap: 66060284 total, 66060284 free, 0 used. 63291844 avail Mem [root@localhost wangao]# mpstat -P ALL 2 Linux 3.10.0-957.21.3.el7.x86_64 (localhost.localdomain) 07/04/2019 _x86_64_ (32 CPU) 07:18:35 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 07:18:37 PM all 0.02 0.00 0.02 0.00 0.00 2.14 0.00 0.00 0.00 97.83 07:18:37 PM 0 0.00 0.00 0.00 0.00 0.00 2.50 0.00 0.00 0.00 97.50 07:18:37 PM 1 0.00 0.00 0.00 0.00 0.00 4.41 0.00 0.00 0.00 95.59 07:18:37 PM 2 0.00 0.00 0.00 0.00 0.00 2.51 0.00 0.00 0.00 97.49 07:18:37 PM 3 0.00 0.00 0.00 0.00 0.00 2.96 0.00 0.00 0.00 97.04 07:18:37 PM 4 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99 07:18:37 PM 5 0.00 0.00 0.00 0.00 0.00 2.99 0.00 0.00 0.00 97.01 07:18:37 PM 6 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99 07:18:37 PM 7 0.00 0.00 0.00 0.00 0.00 2.99 0.00 0.00 0.00 97.01 07:18:37 PM 8 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50 07:18:37 PM 9 0.00 0.00 0.00 0.00 0.00 2.01 0.00 0.00 0.00 97.99 07:18:37 PM 10 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50 07:18:37 PM 11 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50 07:18:37 PM 12 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99 07:18:37 PM 13 0.00 0.00 0.00 0.00 0.00 2.49 0.00 0.00 0.00 97.51 07:18:37 PM 14 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99 07:18:37 PM 15 0.00 0.00 0.00 0.00 0.00 3.94 0.00 0.00 0.00 96.06 07:18:37 PM 16 0.00 0.00 0.00 0.00 0.00 2.50 0.00 0.00 0.00 97.50 07:18:37 PM 17 0.00 0.00 0.00 0.00 0.00 2.97 0.00 0.00 0.00 97.03 07:18:37 PM 18 0.00 0.00 0.00 0.00 0.00 2.02 0.00 0.00 0.00 97.98 07:18:37 PM 19 0.00 0.00 0.00 0.00 0.00 1.49 0.00 0.00 0.00 98.51 07:18:37 PM 20 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99 07:18:37 PM 21 0.00 0.00 0.00 0.00 0.00 2.50 0.00 0.00 0.00 97.50 07:18:37 PM 22 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50 07:18:37 PM 23 0.00 0.00 0.00 0.00 0.00 3.48 0.00 0.00 0.00 96.52 07:18:37 PM 24 0.00 0.00 0.00 0.00 0.00 0.51 0.00 0.00 0.00 99.49 07:18:37 PM 25 0.00 0.00 0.00 0.00 0.00 2.97 0.00 0.00 0.00 97.03 07:18:37 PM 26 0.00 0.00 0.00 0.00 0.00 1.99 0.00 0.00 0.00 98.01 07:18:37 PM 27 0.00 0.00 0.00 0.00 0.00 0.51 0.00 0.00 0.00 99.49 07:18:37 PM 28 0.00 0.00 0.00 0.00 0.00 1.51 0.00 0.00 0.00 98.49 07:18:37 PM 29 0.00 0.00 0.00 0.00 0.00 3.45 0.00 0.00 0.00 96.55 07:18:37 PM 30 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50 07:18:37 PM 31 0.00 0.00 0.00 0.00 0.00 2.97 0.00 0.00 0.00 97.03
優化建議和效果演示
- 查看網卡支持多隊列的狀況，
ethtool -l eth0 - 開啓 irqbalance 服務，讓系統自動調整網絡中斷在多個 CPU 核上的分配,
systemctl start irqbalance - 開啓 RPS/RFS 特性，軟件方式實現 CPU 均衡，接收包中斷的優化
查看RPS/RFS優化先後變化
# 優化前 [root@localhost wangao]# service rps status [em1] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [em1] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [em1] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [em1] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-10 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-11 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-12 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-13 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-14 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-15 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-16 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-17 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-18 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-19 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-20 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-21 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-22 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-23 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-24 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-25 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-26 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-27 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-28 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-29 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-30 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-31 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-4 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-5 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-6 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-7 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-8 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p1] rx-9 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-10 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-11 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-12 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-13 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-14 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-15 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-16 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-17 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-18 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-19 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-20 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-21 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-22 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-23 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-24 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-25 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-26 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-27 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-28 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-29 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-30 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-31 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-4 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-5 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-6 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-7 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-8 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [p1p2] rx-9 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-10 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-11 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-12 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-13 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-14 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-15 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-4 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-5 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-6 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-7 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-8 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 [bond0] rx-9 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0 rps_sock_flow_entries = 0 # 優化後 [root@localhost wangao]# service rps status [em1] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192 [em1] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192 [em1] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192 [em1] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192 [p1p1] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-10 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-11 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-12 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-13 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-14 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-15 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-16 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-17 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-18 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-19 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-20 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-21 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-22 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-23 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-24 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-25 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-26 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-27 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-28 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-29 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-30 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-31 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-4 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-5 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-6 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-7 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-8 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p1] rx-9 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-10 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-11 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-12 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-13 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-14 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-15 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-16 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-17 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-18 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-19 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-20 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-21 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-22 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-23 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-24 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-25 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-26 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-27 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-28 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-29 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-30 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-31 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-4 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-5 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-6 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-7 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-8 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [p1p2] rx-9 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024 [bond0] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-10 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-11 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-12 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-13 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-14 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-15 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-4 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-5 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-6 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-7 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-8 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 [bond0] rx-9 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048 rps_sock_flow_entries = 32768
查看軟中斷變化
# 優化前 [root@localhost wangao]# egrep 'cpu|p1p1' /proc/interrupts 68: 14737707 0 7843 0 571920 0 265157 0 0 0 1610 0 15028 0 314078 0 54904 0 449958 0 126160 0 338804 0 668880 0 122459 0 174803 0 26147 0 IR-PCI-MSI-edge p1p1-TxRx-0 69: 56960 0 115829 0 2669071 0 61425 0 25161 0 4573083 0 72772 0 30975 0 61705 0 189408 0 13134 0 212159 0 49345 0 104657 0 1475007 0 82145 0 IR-PCI-MSI-edge p1p1-TxRx-1 70: 73053 0 118657 0 95018 0 8645414 0 60962 0 83141 0 53948 0 7465 0 99615 0 119273 0 26191 0 14160 0 45703 0 335306 0 4014649 0 94692 0 IR-PCI-MSI-edge p1p1-TxRx-2 71: 46293 0 75419 0 57967 0 6088 0 44337 0 88437 0 22859 0 51370 0 45510 0 120119 0 53276 0 1105728 0 92556 0 7627687 0 374344 0 79394 0 IR-PCI-MSI-edge p1p1-TxRx-3 74: 55261 0 738486 0 14860 0 76202 0 173074 0 215494 0 6385081 0 2691319 0 167374 0 60585 0 17804 0 63837 0 1267656 0 25655 0 94509 0 115839 0 IR-PCI-MSI-edge p1p1-TxRx-4 75: 59182 0 58481 0 89362 0 49916 0 53663 0 40476 0 81847 0 48582 0 7661643 0 74951 0 81414 0 37247 0 58610 0 244719 0 1449948 0 53745 0 IR-PCI-MSI-edge p1p1-TxRx-5 76: 46993 0 290826 0 58661 0 31146 0 112260 0 51531 0 124002 0 47309 0 74370 0 5474439 0 8948212 0 97511 0 184151 0 141408 0 60222 0 19645 0 IR-PCI-MSI-edge p1p1-TxRx-6 77: 44716 0 30601 0 111528 0 61197 0 29883 0 1262486 0 44401 0 19157 0 55785 0 80069 0 143110 0 67065 0 8581759 0 75214 0 201489 0 116415 0 IR-PCI-MSI-edge p1p1-TxRx-7 78: 68542 0 46913 0 60848 0 61158 0 66591 0 8619198 0 1284598 0 68180 0 88156 0 3904413 0 32320 0 158036 0 74480 0 36877 0 91269 0 66078 0 IR-PCI-MSI-edge p1p1-TxRx-8 79: 45256 0 70914 0 57025 0 58231 0 70398 0 24195 0 29084 0 55756 0 8559005 0 91255 0 105623 0 66336 0 34652 0 161266 0 1555160 0 63201 0 IR-PCI-MSI-edge p1p1-TxRx-9 80: 81984 0 71120 0 8853871 0 4829736 0 38789 0 66915 0 37601 0 55277 0 52352 0 226681 0 64967 0 68163 0 100074 0 48442 0 106219 0 120024 0 IR-PCI-MSI-edge p1p1-TxRx-10 81: 69573 0 55072 0 98255 0 49468 0 11084926 0 50300 0 67870 0 91907 0 61667 0 69887 0 83711 0 80970 0 87310 0 47258 0 1438136 0 80672 0 IR-PCI-MSI-edge p1p1-TxRx-11 82: 41874 0 141338 0 40619 0 9095415 0 2744017 0 63918 0 203991 0 314563 0 1205451 0 124044 0 112602 0 308959 0 43454 0 128518 0 46659 0 80356 0 IR-PCI-MSI-edge p1p1-TxRx-12 83: 33080 0 8793595 0 53807 0 46637 0 62925 0 54755 0 72271 0 60842 0 93343 0 47785 0 82287 0 73409 0 745000 0 832425 0 102907 0 90783 0 IR-PCI-MSI-edge p1p1-TxRx-13 84: 52045 0 52518 0 83353 0 64534 0 32414 0 35972 0 45190 0 36925 0 24817 0 79178 0 1443964 0 3712758 0 9480150 0 109740 0 82856 0 58387 0 IR-PCI-MSI-edge p1p1-TxRx-14 85: 63117 0 22871 0 220711 0 75206 0 19498 0 70962 0 52117 0 97241 0 123345 0 67058 0 115891 0 7140767 0 1527076 0 230527 0 52956 0 36916 0 IR-PCI-MSI-edge p1p1-TxRx-15 86: 341182 0 25306 0 436494 0 79810 0 336125 0 24769 0 1015036 0 56332 0 996708 0 46704 0 44728 0 171661 0 73625 0 112345 0 595442 0 1249696 0 IR-PCI-MSI-edge p1p1-TxRx-16 87: 27329 0 55574 0 348849 0 43122 0 333589 0 68221 0 146119 0 264597 0 42572 0 84245 0 217853 0 446305 0 47318 0 202998 0 423085 0 961201 0 IR-PCI-MSI-edge p1p1-TxRx-17 88: 270648 0 46600 0 217253 0 31500 0 38469 0 3095476 0 84290 0 52591 0 26967 0 34871 0 38145 0 87283 0 67059 0 270541 0 210215 0 3993171 0 IR-PCI-MSI-edge p1p1-TxRx-18 89: 148447 0 48653 0 326406 0 53179 0 123863 0 66748 0 391198 0 159050 0 98765 0 54322 0 141506 0 173276 0 1397478 0 133225 0 78736 0 119593 0 IR-PCI-MSI-edge p1p1-TxRx-19 90: 56503 0 135502 0 687704 0 380045 0 469732 0 37274 0 281894 0 1552891 0 121443 0 47009 0 173177 0 650010 0 104931 0 131578 0 217383 0 633766 0 IR-PCI-MSI-edge p1p1-TxRx-20 91: 214823 0 61146 0 339177 0 49928 0 327002 0 64010 0 318068 0 541504 0 45756 0 20992 0 176829 0 57846 0 148528 0 1162846 0 94836 0 433683 0 IR-PCI-MSI-edge p1p1-TxRx-21 92: 106435 0 26816 0 555204 0 31009 0 391325 0 125873 0 290335 0 567300 0 40470 0 341570 0 61671 0 192984 0 221575 0 3245714 0 80733 0 596397 0 IR-PCI-MSI-edge p1p1-TxRx-22 93: 115951 0 70761 0 381334 0 143441 0 872170 0 118644 0 517816 0 60193 0 100746 0 41460 0 70866 0 102305 0 134298 0 79193 0 79264 0 1835037 0 IR-PCI-MSI-edge p1p1-TxRx-23 94: 146101 0 35816 0 733832 0 29613 0 313268 0 202986 0 340662 0 100681 0 1993051 0 335538 0 71931 0 23162 0 246612 0 556509 0 224855 0 559908 0 IR-PCI-MSI-edge p1p1-TxRx-24 95: 150668 0 36783 0 251091 0 36610 0 452604 0 118424 0 197907 0 471898 0 916471 0 25373 0 43497 0 170329 0 504482 0 504129 0 61751 0 349911 0 IR-PCI-MSI-edge p1p1-TxRx-25 96: 175244 0 85547 0 149382 0 25817 0 27945 0 55507 0 297689 0 1877839 0 72682 0 294648 0 34613 0 54712 0 184015 0 5097111 0 62440 0 893488 0 IR-PCI-MSI-edge p1p1-TxRx-26 97: 188255 0 79222 0 196770 0 93331 0 62542 0 197921 0 499815 0 1489952 0 685652 0 127194 0 39956 0 48082 0 46078 0 444262 0 89668 0 197128 0 IR-PCI-MSI-edge p1p1-TxRx-27 98: 113119 0 44914 0 418302 0 17345 0 51941 0 187466 0 37948 0 572466 0 54668 0 27553 0 580539 0 366022 0 2375605 0 49572 0 73136 0 1132828 0 IR-PCI-MSI-edge p1p1-TxRx-28 99: 94404 0 56968 0 407066 0 45791 0 302591 0 106122 0 261980 0 583169 0 32802 0 50997 0 47888 0 99704 0 129685 0 315057 0 38119 0 1544062 0 IR-PCI-MSI-edge p1p1-TxRx-29 100: 98958 0 178434 0 354708 0 111329 0 159026 0 156721 0 224555 0 188416 0 224497 0 30343 0 497511 0 121221 0 280827 0 354135 0 84543 0 1650501 0 IR-PCI-MSI-edge p1p1-TxRx-30 101: 199414 0 34177 0 506856 0 92717 0 246991 0 118922 0 123314 0 261694 0 145339 0 730269 0 397150 0 26237 0 541393 0 357194 0 174133 0 520104 0 IR-PCI-MSI-edge p1p1-TxRx-31 102: 5 0 0 0 0 0 130 0 0 0 4 0 5 0 2 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 57 0 IR-PCI-MSI-edge p1p1 # 優化後 [root@localhost wangao]# egrep 'cpu|enp175s0f0' /proc/interrupts 545: 0 0 0 0 0 0 0 0 78728441 66114965 52520832 41158901 65698825 55803106 20888551 23716200 0 0 0 0 0 0 0 0 41296940 19654307 42367407 68035795 31971316 73083489 109360661 60151057 IR-PCI-MSI-edge enp175s0f0-TxRx-0 546: 0 0 0 0 0 0 0 0 293334701 57784003 57947637 279907263 83003310 59328994 14582717 10594077 0 0 0 0 0 0 0 0 33844346 53425312 104463040 121835272 48115201 11875829 149517303 27888367 IR-PCI-MSI-edge enp175s0f0-TxRx-1 547: 0 0 0 0 0 0 0 0 65087834 74714070 70617455 50729188 50629957 53222216 29320338 43600286 0 0 0 0 0 0 0 0 78790098 21664768 53133919 104396382 28219169 98109898 106275780 6075631 IR-PCI-MSI-edge enp175s0f0-TxRx-2 548: 0 0 0 0 0 0 0 0 108290552 62484073 55417923 64030173 55771760 74127972 63180417 62512564 0 0 0 0 0 0 0 0 127950399 63170356 20177522 106969671 21966002 107295156 138414635 3139534 IR-PCI-MSI-edge enp175s0f0-TxRx-3 549: 0 0 0 0 0 0 0 0 175701987 134359768 118283034 93078181 56579608 43699849 83211249 72178065 0 0 0 0 0 0 0 0 67170073 21723454 56826677 103798934 14026848 39506854 136483492 11700838 IR-PCI-MSI-edge enp175s0f0-TxRx-4 550: 0 0 0 0 0 0 0 0 161298050 70517371 30581739 186350377 170422634 226218127 55471402 95255735 0 0 0 0 0 0 0 0 158450530 36931783 14121926 68432292 26086904 5561980 46129837 10039 IR-PCI-MSI-edge enp175s0f0-TxRx-5 551: 0 0 0 0 0 0 0 0 59810450 85668650 74748733 66956510 53509915 67475372 37312571 27187643 0 0 0 0 0 0 0 0 42998302 18434014 26932131 54190593 16723043 46515600 59603049 11897912 IR-PCI-MSI-edge enp175s0f0-TxRx-6 552: 0 0 0 0 0 0 0 0 37068253 137459908 126379700 132069647 44577185 92132779 18951865 48826631 0 0 0 0 0 0 0 0 168471567 18399908 9609939 156723119 45300251 47356473 71094409 13049049 IR-PCI-MSI-edge enp175s0f0-TxRx-7 553: 0 0 0 0 0 0 0 0 9148701 15294255 337628836 28379485 21729246 18839950 5383431 5064436 0 0 0 0 0 0 0 0 45773035 6898143 3359206 4497486 569915849 5649581 659950071 561102436 IR-PCI-MSI-edge enp175s0f0-TxRx-8 554: 0 0 0 0 0 0 0 0 45919755 648235162 90862075 82511196 121536855 775131101 5059996 2245523 0 0 0 0 0 0 0 0 39913983 123180793 4250458 117911846 13217963 4291001 72759615 11901382 IR-PCI-MSI-edge enp175s0f0-TxRx-9 555: 0 0 0 0 0 0 0 0 17252329 12804726 202960565 226950817 436490841 62135664 286074659 2405684 0 0 0 0 0 0 0 0 1937801 2774288 198336045 26598791 3834288 620409019 7956742 0 IR-PCI-MSI-edge enp175s0f0-TxRx-10 556: 0 0 0 0 0 0 0 0 2543207 1589549 378184536 5537639 2432570 10278492 370935530 2557214 0 0 0 0 0 0 0 0 2084807 2817750 3214825 3352447 1633147902 3067434 3704468 0 IR-PCI-MSI-edge enp175s0f0-TxRx-11 557: 0 0 0 0 0 0 0 0 35578276 274609699 253038113 4759061 62653759 2988138 24435129 2729652 0 0 0 0 0 0 0 0 1276874078 3470696 522229642 4623985 3795035 3779577 4484373 26368 IR-PCI-MSI-edge enp175s0f0-TxRx-12 558: 0 0 0 0 0 0 0 0 3282953 3151370 13102970 15841447 2290018 191758517 253118172 3139260 0 0 0 0 0 0 0 0 2245766 6169610 1545045398 30596340 4355187 163581051 4176977 0 IR-PCI-MSI-edge enp175s0f0-TxRx-13 559: 0 0 0 0 0 0 0 0 88746890 2665364 81746473 477713921 4323711 87193204 74776878 3650446 0 0 0 0 0 0 0 0 2540382 403859013 3537468 997197460 189897196 4592593 5480200 0 IR-PCI-MSI-edge enp175s0f0-TxRx-14 560: 0 0 0 0 0 0 0 0 6646487 5048477 4430038 8497796 18754540 225111260 231266854 1499236783 0 0 0 0 0 0 0 0 138682422 4968757 27628949 22916070 3702479 28557972 20003328 0 IR-PCI-MSI-edge enp175s0f0-TxRx-15 561: 0 0 0 0 0 0 0 0 60867 79129 72520 56561 47863 58682 37248 30413 0 0 0 0 0 0 0 0 31560 22821 36684 41662 23009 40489 40755 10139346 IR-PCI-MSI-edge enp175s0f0-TxRx-16 562: 0 0 0 0 0 0 0 0 704280 2485475 1162836 815515 762812 681074 684065 578558 0 0 0 0 0 0 0 0 384355 706857 392981 350292 339121 403073 201587 19983 IR-PCI-MSI-edge enp175s0f0-TxRx-17 563: 0 0 0 0 0 0 0 0 629722 2106933 1485145 1179751 579723 864401 771639 621684 0 0 0 0 0 0 0 0 357663 563356 465863 413412 351007 341288 208065 25611 IR-PCI-MSI-edge enp175s0f0-TxRx-18 564: 0 0 0 0 0 0 0 0 823889 835158 898125 1042075 479060 649698 488703 431781 0 0 0 0 0 0 0 0 3198859 368537 361450 268475 279292 335527 175329 21693 IR-PCI-MSI-edge enp175s0f0-TxRx-19 565: 0 0 0 0 0 0 0 0 676732 1450852 1388628 704602 644689 423264 404306 365596 0 0 0 0 0 0 0 0 3038526 391864 405727 263275 191222 248226 136003 21112 IR-PCI-MSI-edge enp175s0f0-TxRx-20 566: 0 0 0 0 0 0 0 0 566595 1175664 2031641 958319 714165 319767 349309 318957 0 0 0 0 0 0 0 0 2958683 218495 239817 214993 179935 191555 106269 22241 IR-PCI-MSI-edge enp175s0f0-TxRx-21 567: 0 0 0 0 0 0 0 0 832674 1176655 1652999 789321 736090 853359 748998 577413 0 0 0 0 0 0 0 0 498610 570634 823144 451361 350612 336046 200696 23342 IR-PCI-MSI-edge enp175s0f0-TxRx-22 568: 0 0 0 0 0 0 0 0 486851 946241 1375282 525162 582995 609945 696507 440446 0 0 0 0 0 0 0 0 2986687 390852 513487 317316 283314 320359 191080 21907 IR-PCI-MSI-edge enp175s0f0-TxRx-23 569: 0 0 0 0 0 0 0 0 51211317 79773036 82956204 46567489 42244838 62523416 34798214 31871168 0 0 0 0 0 0 0 0 89233552 34741833 45475248 56079776 44626874 112466556 127468577 26730942 IR-PCI-MSI-edge enp175s0f0-TxRx-24 570: 0 0 0 0 0 0 0 0 84554805 62689462 101496218 40518441 34792544 140921853 28422396 29428544 0 0 0 0 0 0 0 0 23860917 73164249 50084153 72011910 24525127 548979217 332764098 50908113 IR-PCI-MSI-edge enp175s0f0-TxRx-25 571: 0 0 0 0 0 0 0 0 69909137 55776777 94776131 63355410 42321451 74321693 45528982 50604702 0 0 0 0 0 0 0 0 56618040 31516738 58332108 69357737 20168247 58852410 137885013 28088790 IR-PCI-MSI-edge enp175s0f0-TxRx-26 572: 0 0 0 0 0 0 0 0 97305670 87729287 88586433 59983480 55828246 79305722 51064394 44409948 0 0 0 0 0 0 0 0 41248135 60491280 43010548 99696137 32513292 64373823 74114831 49645455 IR-PCI-MSI-edge enp175s0f0-TxRx-27 573: 0 0 0 0 0 0 0 0 72108897 76540055 69546704 69542158 46167801 57170600 48419068 58439907 0 0 0 0 0 0 0 0 68912196 27371642 58904860 82050006 79475074 107799031 109654468 81082896 IR-PCI-MSI-edge enp175s0f0-TxRx-28 574: 0 0 0 0 0 0 0 0 69014002 80647341 62399685 97216594 87591440 103529815 72314579 45731155 0 0 0 0 0 0 0 0 89557655 50919154 68754628 109583064 33891553 76526249 188283182 29596358 IR-PCI-MSI-edge enp175s0f0-TxRx-29 575: 0 0 0 0 0 0 0 0 70240109 84468547 61070305 57381753 122569467 76404910 51871561 47071056 0 0 0 0 0 0 0 0 97676445 22451436 44986642 93854970 51734947 141860090 97976950 63786565 IR-PCI-MSI-edge enp175s0f0-TxRx-30 576: 0 0 0 0 0 0 0 0 66395628 99188539 63355633 82424649 110498163 119343763 129927329 49811047 0 0 0 0 0 0 0 0 76096869 78511976 32199289 122850446 108523248 49641666 346741057 101551776 IR-PCI-MSI-edge enp175s0f0-TxRx-31 577: 0 0 0 0 0 0 0 0 307 422 349 309 268 317 222 167 0 0 0 0 0 0 0 0 139 126 215 193 95 182 231 7349 IR-PCI-MSI-edge enp175s0f0
參考文章
Redis 高負載下的中斷優化
網卡軟中斷太高問題優化總結
Linux 網絡協議棧收消息過程 - TCP Protocol Layer
Monitoring and Tuning the Linux Networking Stack: Receiving Data
Performance Tuning Guide
相關文章
- 1. netmap性能優化筆記：網卡多隊列
- 2. linux內核網絡中RPS/RFS原理
- 3. Java性能調優實踐
- 4. JVM性能調優實踐
- 5. 容器雲負載均衡之三：RSS、RPS、RFS和XPS調整
- 6. Solarflare低延遲萬兆網卡配置和性能調優實踐
- 7. Linux多隊列網卡
- 8. Unity3D_NGUI_性能優化實踐_CPU卡頓
- 9. Linux系統中RPS/RFS介紹
- 10. Linux內核中RPS/RFS代碼分析
- 更多相關文章...
- • C# 隊列（Queue） - C#教程
- • Thymeleaf項目實踐 - Thymeleaf 教程
- • TiDB 在摩拜單車在線數據業務的應用和實踐
- • 互聯網組織的未來：剖析GitHub員工的任性之源
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
