Linux System and Performance Monitoring(Network篇)

8.0 Network 監控介紹

在全部的子系統監控中,網絡是最困難的.這主要是因爲網絡概念很抽象.當監控系統上的網絡性能,這有太多因素.這些因素包括了延遲,衝突,擁擠和數據包丟失.

這個章節討論怎麼樣檢查Ethernet(譯註:網卡),IP,TCP的性能.

8.1 Ethernet Configuration Settings(譯註:網卡配置的設置)

除非很明確的指定,幾乎全部的網卡都是自適應網絡速度.當一個網絡中有不少不一樣的網絡設備時,會各自採用不一樣的速率和工做模式.

多數商業網絡都運行在100 或 1000BaseTX.使用ethtool 能夠肯定這個系統是處於那種速率.

如下的例子中,是一個有100BaseTX 網卡的系統,自動協商適應至10BaseTX 的狀況.

# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: Yes
Speed: 10Mb/s
Duplex: Half
Port: MII
PHYAD: 32
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: d
Current message level: 0×00000007 (7)
Link detected: yes

如下示範例子中,如何強制網卡速率調整至100BaseTX:

# ethtool -s eth0 speed 100 duplex full autoneg off

# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: No
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 32
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: pumbg
Wake-on: d
Current message level: 0×00000007 (7)
Link detected: yes

8.2 Monitoring Network Throughput(譯註:網絡吞吐量監控)

接口之間的同步並不意味着僅僅有帶寬問題.重要的是,如何管理並優化,這2臺主機之間的交換機,網線,或者路由器.測試網絡吞吐量最好的方式就是,在這2個系統之間互相發送數據傳輸並統計下來,好比延遲和速度.

8.2.0 使用iptraf 查看本地吞吐量

iptraf 工具(http://iptraf.seul.org),提供了每一個網卡吞吐量的儀表盤.

#iptraf -d eth0

Figure 1: Monitoring for Network Throughput

從輸出中可看到,該系統發送傳輸率(譯註:Outgoing rates)爲 61 mbps,這對於100 mbps網絡來講,有點慢.

8.2.1 使用netperf 查看終端吞吐量

不一樣於iptraf 被動的在本地監控流量,netperf 工具可讓管理員,執行更加可控的吞吐量監控.對於肯定從客戶端工做站到一個高負荷的服務器端(好比file 或web server),它們之間有多少吞吐量是很是有幫助的.netperf 工具運行的是client/server 模式.

完成一個基本可控吞吐量測試,首先netperf server 必須運行在服務器端系統上:

server# netserver
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC

netperf 工具可能須要進行多重採樣.多數基本測試就是一次標準的吞吐量測試.如下例子就是,一個LAN(譯註:局域網) 環境下,從client 上執行一次30秒的TCP 吞吐量採樣:

從輸出可看出,該網絡的吞吐量大體在89 mbps 左右.server(192.168.1.215) 與client 在同一LAN 中.這對於100 mbps網絡來講,性能很是好.

client# netperf -H 192.168.1.215 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.230 (192.168.1.230) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 16384 30.02 89.46

從LAN 切換到具有54G(譯註:Wireless-G是將來54Mbps無線網聯網標準)無線網絡路由器中,並在10 英尺範圍內測試時.該吞吐量就急劇的降低.在最大就爲54 MBits的可能下,筆記本電腦可實現總吞吐量就爲14 MBits.

client# netperf -H 192.168.1.215 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.215 (192.168.1.215) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 16384 30.10 14.09

若是在50英尺範圍內呢,則進一步會降低至5 MBits.

# netperf -H 192.168.1.215 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.215 (192.168.1.215) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 16384 30.64 5.05

若是從LAN 切換到互聯網上,則吞吐量跌至1 Mbits下了.

# netperf -H litemail.org -p 1500 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
litemail.org (72.249.104.14 8) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 16384 31.58 0.93

最後是一個××× 鏈接環境,這是全部網絡環境中最槽糕的吞吐量了.

# netperf -H 10.0.1.129 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.0.1.129 (10.0.1.129) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 16384 31.99 0.51

另外,netperf 能夠幫助測試每秒總計有多少的TCP 請求和響應數.經過創建單一TCP 鏈接並順序地發送多個請求/響應(ack 包來回在1個byte 大小).有點相似於RDBMS 程序在執行多個交易或者郵件服務器在同一個鏈接管道中發送郵件.

如下例子在30 秒的持續時間內,模擬TCP 請求/響應:

client# netperf -t TCP_RR -H 192.168.1.230 -l 30
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 192.168.1.230 (192.168.1.230) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec

16384 87380 1 1 30.00 4453.80
16384 87380

在輸出中看出,這個網絡支持的處理速率爲每秒4453 psh/ack(包大小爲1 byte).這實際上是理想狀態下,由於實際狀況時,多數requests(譯註:請求),特別是responses(譯註:響應),都大於1 byte.

現實狀況下,netperf 通常requests 默認使用2K大小,responses 默認使用32K大小:

client# netperf -t TCP_RR -H 192.168.1.230 -l 30 — -r 2048,32768
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.230 (192.168.1.230) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec

16384 87380 2048 32768 30.00 222.37
16384 87380

這個處理速率減小到了每秒222.

8.2.2 使用iperf 評估網絡效率

基於都是須要在2端檢查鏈接狀況下,iperf 和netperf 很類似.不一樣的是,iperf 更深刻的經過windows size和QOS 設備來檢查TCP/UDP 的效率狀況.這個工具,是給須要優化TCP/IP stacks以及測試這些stacks 效率的管理員們量身定作的.

iperf 做爲一個二進制程序,可運行在server 或者client 任一模式下.默認使用50001 端口.

首先啓動server 端(192.168.1.215):

server# iperf -s -D
Running Iperf Server as a daemon
The Iperf daemon process ID : 3655
————————————————————
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
————————————————————

在如下例子裏,一個無線網絡環境下,其中client 端重複運行iperf,用於測試網絡的吞吐量狀況.這個環境假定處於被充分利用狀態,不少主機都在下載ISO p_w_picpaths文件.

首先client 端鏈接到server 端(192.168.1.215),並在總計60秒時間內,每5秒進行一次帶寬測試的採樣.

client# iperf -c 192.168.1.215 -t 60 -i 5
————————————————————
Client connecting to 192.168.1.215, TCP port 5001
TCP window size: 25.6 KByte (default)
————————————————————
[ 3] local 192.168.224.150 port 51978 connected with
192.168.1.215 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 5.0 sec 6.22 MBytes 10.4 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 5.0-10.0 sec 6.05 MBytes 10.1 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 10.0-15.0 sec 5.55 MBytes 9.32 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 15.0-20.0 sec 5.19 MBytes 8.70 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 20.0-25.0 sec 4.95 MBytes 8.30 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 25.0-30.0 sec 5.21 MBytes 8.74 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 30.0-35.0 sec 2.55 MBytes 4.29 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 35.0-40.0 sec 5.87 MBytes 9.84 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 40.0-45.0 sec 5.69 MBytes 9.54 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 45.0-50.0 sec 5.64 MBytes 9.46 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 50.0-55.0 sec 4.55 MBytes 7.64 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 55.0-60.0 sec 4.47 MBytes 7.50 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 61.9 MBytes 8.66 Mbits/sec

這臺主機的其餘網絡傳輸,也會影響到這部分的帶寬採樣.因此能夠看到總計60秒時間內,都在4 - 10 MBits 上下起伏.

除了TCP 測試以外,iperf 的UDP 測試主要是評估包丟失和抖動.

接下來的iperf 測試,是在一樣的54Mbit G標準無線網絡中.在早期的示範例子中,目前的吞吐量只有9 Mbits.

# iperf -c 192.168.1.215 -b 10M
WARNING: option -b implies udp testing
————————————————————
Client connecting to 192.168.1.215, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 107 KByte (default)
————————————————————
[ 3] local 192.168.224.150 port 33589 connected with 192.168.1.215 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.8 MBytes 9.90 Mbits/sec
[ 3] Sent 8420 datagrams
[ 3] Server Report:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 6.50 MBytes 5.45 Mbits/sec 0.480 ms 3784/ 8419 (45%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order

從輸出中可看出,在嘗試傳輸10M 的數據時,實際上只產生了5.45M.卻有45% 的包丟失.

8.3 Individual Connections with tcptrace

tcptrace 工具提供了對於某一具體鏈接裏,詳細的TCP 相關信息.該工具使用libcap 來分析某一具體TCP sessions.該工具彙報的信息,有時很難在某一TCP stream被發現.這些信息

包括了有:

1,TCP Retransmissions(譯註:IP 轉播) - 全部數據大小被髮送所需的包總額
2,TCP Windows Sizes - 鏈接速度慢與小的windows sizes 有關
3,Total throughput of the connection - 鏈接的吞吐量
4,Connection duration - 鏈接的持續時間

8.3.1 案例學習 - 使用tcptrace

tcptrace 工具可能已經在部分Linux 發佈版中有安裝包了,該文做者經過網站,下載的是源碼安裝包:http://dag.wieers.com/rpm/packages /tcptrace.tcptrace 須要libcap 基於文件輸入方式使用.在tcptrace 沒有選項的狀況下,默認每一個惟一的鏈接過程都將被捕獲.

如下例子是,使用libcap 基於輸入文件爲bigstuff:

# tcptrace bigstuff
1 arg remaining, starting with ‘bigstuff’
Ostermann’s tcptrace — version 6.6.7 — Thu Nov 4, 2004

146108 packets seen, 145992 TCP packets traced
elapsed wallclock time: 0:00:01.634065, 89413 pkts/sec analyzed
trace file elapsed time: 0:09:20.358860
TCP connection info:
1: 192.168.1.60:pcanywherestat - 192.168.1.102:2571 (a2b) 404> 450< 2: 192.168.1.60:3356 - ftp.strongmail.net:21 (c2d) 35> 21< 3: 192.168.1.60:3825 - ftp.strongmail.net:65023 (e2f) 5> 4< (complete) 4: 192.168.1.102:1339 - 205.188.8.194:5190 (g2h) 6> 6< 5: 192.168.1.102:1490 - cs127.msg.mud.yahoo.com:5050 (i2j) 5> 5< 6: py-in-f111.google.com:993 - 192.168.1.102:3785 (k2l) 13> 14< 上面的輸出中,每一個鏈接都有對應的源主機和目的主機.tcptrace 使用-l 和-o 選項可查看某一鏈接更詳細的數據. 如下的結果,就是在bigstuff 文件中,#16 鏈接的相關統計數據: # tcptrace -l -o1 bigstuff 1 arg remaining, starting with ‘bigstuff’ Ostermann’s tcptrace — version 6.6.7 — Thu Nov 4, 2004 146108 packets seen, 145992 TCP packets traced elapsed wallclock time: 0:00:00.529361, 276008 pkts/sec analyzed trace file elapsed time: 0:09:20.358860 TCP connection info: 32 TCP connections traced: TCP connection 1: host a: 192.168.1.60:pcanywherestat host b: 192.168.1.102:2571 complete conn: no (SYNs: 0) (FINs: 0) first packet: Sun Jul 20 15:58:05.472983 2008 last packet: Sun Jul 20 16:00:04.564716 2008 elapsed time: 0:01:59.091733 total packets: 854 filename: bigstuff a->b: b->a:
total packets: 404 total packets: 450
ack pkts sent: 404 ack pkts sent: 450
pure acks sent: 13 pure acks sent: 320
sack pkts sent: 0 sack pkts sent: 0
dsack pkts sent: 0 dsack pkts sent: 0
max sack blks/ack: 0 max sack blks/ack: 0
unique bytes sent: 52608 unique bytes sent: 10624
actual data pkts: 391 actual data pkts: 130
actual data bytes: 52608 actual data bytes: 10624
rexmt data pkts: 0 rexmt data pkts: 0
rexmt data bytes: 0 rexmt data bytes: 0
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 0 outoforder pkts: 0
pushed data pkts: 391 pushed data pkts: 130
SYN/FIN pkts sent: 0/0 SYN/FIN pkts sent: 0/0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 0 bytes mss requested: 0 bytes
max segm size: 560 bytes max segm size: 176 bytes
min segm size: 48 bytes min segm size: 80 bytes
avg segm size: 134 bytes avg segm size: 81 bytes
max win adv: 19584 bytes max win adv: 65535 bytes
min win adv: 19584 bytes min win adv: 64287 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 19584 bytes avg win adv: 64949 bytes
initial window: 160 bytes initial window: 0 bytes
initial window: 2 pkts initial window: 0 pkts
ttl stream length: NA ttl stream length: NA
missed data: NA missed data: NA
truncated data: 36186 bytes truncated data: 5164 bytes
truncated packets: 391 pkts truncated packets: 130 pkts
data xmit time: 119.092 secs data xmit time: 116.954 secs
idletime max: 441267.1 ms idletime max: 441506.3 ms
throughput: 442 Bps throughput: 89 Bps

8.3.2 案例學習 - 計算轉播率

幾乎不可能肯定說哪一個鏈接會有嚴重不足的轉播問題,只是須要分析,使用tcptrace 工具能夠經過過濾機制和布爾表達式來找出出問題的鏈接.一個很繁忙的網絡中,會有不少的鏈接,幾乎全部的鏈接都會有轉播.找出其中最多的一個,這就是問題的關鍵.

下面的例子裏,tcptrace 將找出那些轉播大於100 segments(譯註:分段數)的鏈接:

# tcptrace -f’rexmit_segs>100′ bigstuff
Output filter: ((c_rexmit_segs>100)OR(s_rexmit_segs>100))
1 arg remaining, starting with ‘bigstuff’
Ostermann’s tcptrace — version 6.6.7 — Thu Nov 4, 2004

146108 packets seen, 145992 TCP packets traced
elapsed wallclock time: 0:00:00.687788, 212431 pkts/sec analyzed
trace file elapsed time: 0:09:20.358860
TCP connection info:
16: ftp.strongmail.net:65014 - 192.168.1.60:2158 (ae2af) 18695> 9817< 在這個輸出中,是#16 這個鏈接裏,超過了100 轉播.如今,使用如下命令查看關於這個鏈接的其餘信息: # tcptrace -l -o16 bigstuff arg remaining, starting with ‘bigstuff’ Ostermann’s tcptrace — version 6.6.7 — Thu Nov 4, 2004 146108 packets seen, 145992 TCP packets traced elapsed wallclock time: 0:00:01.355964, 107752 pkts/sec analyzed trace file elapsed time: 0:09:20.358860 TCP connection info: 32 TCP connections traced: ================================ TCP connection 16: host ae: ftp.strongmail.net:65014 host af: 192.168.1.60:2158 complete conn: no (SYNs: 0) (FINs: 1) first packet: Sun Jul 20 16:04:33.257606 2008 last packet: Sun Jul 20 16:07:22.317987 2008 elapsed time: 0:02:49.060381 total packets: 28512 filename: bigstuff ae->af: af->ae:

unique bytes sent: 25534744 unique bytes sent: 0
actual data pkts: 18695 actual data pkts: 0
actual data bytes: 25556632 actual data bytes: 0
rexmt data pkts: 1605 rexmt data pkts: 0
rexmt data bytes: 2188780 rexmt data bytes: 0

計算轉播率:
rexmt/actual * 100 = Retransmission rate

1605/18695* 100 = 8.5%

這個慢鏈接的緣由,就是由於它有8.5% 的轉播率.

8.3.3 案例學習 - 計算轉播時間

tcptrace 工具備一系列的模塊展現不一樣的數據,按照屬性,其中就有protocol(譯註:協議),port(譯註:端口),time等等.Slice module使得你可觀察在一段時間內的TCP 性能.你能夠在一系列的轉發過程當中,查看其餘性能數據,以肯定找出瓶頸.

如下例子示範了,tcptrace 是怎樣使用slice 模式的:

# tcptrace –xslice bigfile

以上命令會建立一個slice.dat 文件在如今的工做目錄中.這個文件內容,包含是每15秒間隔內轉播的相關信息:

# ls -l slice.dat
-rw-r–r– 1 root root 3430 Jul 10 22:50 slice.dat
# more slice.dat
date segs bytes rexsegs rexbytes new active
————— ——– ——– ——– ——– ——– ——–
22:19:41.913288 46 5672 0 0 1 1
22:19:56.913288 131 25688 0 0 0 1
22:20:11.913288 0 0 0 0 0 0
22:20:26.913288 5975 4871128 0 0 0 1
22:20:41.913288 31049 25307256 0 0 0 1
22:20:56.913288 23077 19123956 40 59452 0 1
22:21:11.913288 26357 21624373 5 7500 0 1
22:21:26.913288 20975 17248491 3 4500 12 13
22:21:41.913288 24234 19849503 10 15000 3 5
22:21:56.913288 27090 22269230 36 53999 0 2
22:22:11.913288 22295 18315923 9 12856 0 2
22:22:26.913288 8858 7304603 3 4500 0 1

8.4 結論

監控網絡性能由如下幾個部分組成:

1,檢查並肯定全部網卡都工做在正確的速率. 2,檢查每塊網卡的吞吐量,並確認其處於服務時的網絡速度. 3,監控網絡流量的類型,並肯定適當的流量優先級策略.