解決Zabbix某臺主機忽然頻繁告警"Zabbix agent on xxxxxx is unreachable for x minutes"

1、某臺主機忽然某一天頻繁告警zabbix agent不可達

查看zabbix agent日誌沒有發現異常

2、查看zabbix server日誌發現這臺主機的日誌有大量報錯信息"first network error"以及"another network error"

[root@zabbix_server etc]# cat /tmp/zabbix_server.log|grep 172.28.5.63|more

 27849:20191218:094413.077 Zabbix agent item "perf_counter[\2\250]" on host "172.28.5.63" failed: another network error, wait fo r 15 seconds 27848:20191218:094428.098 resuming Zabbix agent checks on host "172.28.5.63": connection restored 27837:20191218:094446.128 Zabbix agent item "net.if.in[Microsoft ISATAP Adapter #2]" on host "172.28.5.63" failed: first networ k error, wait for 15 seconds 27849:20191218:094504.088 Zabbix agent item "net.if.out[WAN Miniport (Network Monitor)-QoS Packet Scheduler-0000]" on host "172
.28.5.63" failed: another network error, wait for 15 seconds
 27845:20191218:094519.094 resuming Zabbix agent checks on host "172.28.5.63": connection restored 27836:20191218:094536.258 Zabbix agent item "net.if.in[Broadcom NetXtreme Gigabit Ethernet #4]" on host "172.28.5.63" failed: f irst network error, wait for 15 seconds 27846:20191218:094551.117 resuming Zabbix agent checks on host "172.28.5.63": connection restored 27843:20191218:094600.102 Zabbix agent item "net.if.out[Broadcom NetXtreme Gigabit Ethernet-WFP LightWeight Filter-0000]" on ho st "172.28.5.63" failed: first network error, wait for 15 seconds 27843:20191218:094615.127 resuming Zabbix agent checks on host "172.28.5.63": connection restored 27837:20191218:094623.818 Zabbix agent item "net.if.in[Broadcom NetXtreme Gigabit Ethernet #4-QoS Packet Scheduler-0000]" on ho st "172.28.5.63" failed: first network error, wait for 15 seconds 27847:20191218:094641.112 Zabbix agent item "net.if.in[WAN Miniport (SSTP)]" on host "172.28.5.63" failed: another network erro r, wait for 15 seconds 27845:20191218:094657.134 resuming Zabbix agent checks on host "172.28.5.63": connection restored 27834:20191218:094702.464 Zabbix agent item "vfs.fs.size[D:,free]" on host "172.28.5.63" failed: first network error, wait for 
15 seconds 27852:20191218:094720.139 resuming Zabbix agent checks on host "172.28.5.63": connection restored 27840:20191218:094723.709 Zabbix agent item "vm.memory.size[pavailable]" on host "172.28.5.63" failed: first network error, wai t for 15 seconds 27847:20191218:094738.149 resuming Zabbix agent checks on host "172.28.5.63": connection restored 27836:20191218:094802.499 Zabbix agent item "net.if.out[Broadcom NetXtreme Gigabit Ethernet #3]" on host "172.28.5.63" failed: first network error, wait for 15 seconds 27843:20191218:094818.149 resuming Zabbix agent checks on host "172.28.5.63": connection restored 27832:20191218:094825.129 Zabbix agent item "net.if.in[Broadcom NetXtreme Gigabit Ethernet #3-QoS Packet Scheduler-0000]" on ho st "172.28.5.63" failed: first network error, wait for 15 seconds 27851:20191218:094859.175 resuming Zabbix agent checks on host "172.28.5.63": connection restored 27832:20191218:094903.413 Zabbix agent item "vfs.fs.size[E:,free]" on host "172.28.5.63" failed: first network error, wait for 
15 seconds

3、查看主機TCP鏈接，發現存在大量的TIME_WAIT鏈接

 

4、 百度一下，具體是由於以下緣由

從系統啓動，Windows Vista 中、 在 Windows 7 中，Windows Server 2008 中和在 Windows Server 2008 R2 中的 497 天后未關閉 TIME_WAIT 狀態的全部 TCP/IP 端口

意思是說，系統啓動的497天之後，全部在"TIME_WAIT"狀態的TCP連接都不會被關閉。TCP端口逐漸被佔用完，不能建立新的TCP/IP鏈接

5、登陸主機查看系統運行時長

 

 正好的前天凌晨出現的頻繁告警

6、解決方案

一、重啓服務器，可是運行497天后，問題還會出現

二、下載微軟補丁

微軟官網公告地址

https://support.microsoft.com/zh-cn/help/2553549/all-the-tcp-ip-ports-that-are-in-a-time-wait-status-are-not-closed-aft

如今已經不能下載補丁包了，能夠使用window update來更新補丁