【錯誤彙總】zabbix 監控偶遇問題一記

時間 2019-11-05

原文原文鏈接

現象：
在zabbix web 端，JMX 顯示紅色，而且提示 cannot reslove 192.168.100.107,192.168.100.108......
當時第一時間想到的是，將server端的 /etc/zabbix/zabbix_server.conf 中 JavaGateway 後的 ip 只保留 192.168.100.107，而後重啓 zabbix-serverjava

但是JMX仍是顯示紅色，而且出現了「，(Connection refused): service:jmx:rmi:///jndi/rmi://192.168.100.107:9999/jmxrmi」的問題

解決方法：
到 192.168.100.107 下去nginx

[root@web1 ~]# /usr/local/tomcat/bin/shutdown.sh 
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/jdk1.8/jre
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Jul 07, 2019 10:48:46 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact [localhost:8005] (base port [8005] and offset [0]). Tomcat may not be running.
Jul 07, 2019 10:48:46 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Error stopping Catalina
java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:390)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:480)

[root@web1 ~]# !net
netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:9999            0.0.0.0:*               LISTEN      9629/java           
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      6387/nginx: master  
tcp        0      0 0.0.0.0:20048           0.0.0.0:*               LISTEN      15753/rpc.mountd    
tcp        0      0 0.0.0.0:37457           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:49366           0.0.0.0:*               LISTEN      14929/rpc.statd     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2993/sshd           
tcp        0      0 0.0.0.0:44247           0.0.0.0:*               LISTEN      9629/java           
tcp        0      0 0.0.0.0:34332           0.0.0.0:*               LISTEN      9629/java           
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      10386/zabbix_agentd 
tcp        0      0 0.0.0.0:10052           0.0.0.0:*               LISTEN      10320/java          
tcp6       0      0 :::43533                :::*                    LISTEN      -                   
tcp6       0      0 :::111                  :::*                    LISTEN      15627/rpcbind       
tcp6       0      0 :::20048                :::*                    LISTEN      15753/rpc.mountd    
tcp6       0      0 :::34298                :::*                    LISTEN      14929/rpc.statd     
tcp6       0      0 :::2049                 :::*                    LISTEN      -                   
tcp6       0      0 :::10050                :::*                    LISTEN      10386/zabbix_agentd

我發現，tomcat徹底關不掉，那麼只有 kill 掉了web

[root@web1 ~]# kill 9629
[root@web1 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      6387/nginx: master  
tcp        0      0 0.0.0.0:20048           0.0.0.0:*               LISTEN      15753/rpc.mountd    
tcp        0      0 0.0.0.0:37457           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:49366           0.0.0.0:*               LISTEN      14929/rpc.statd     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2993/sshd           
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      10386/zabbix_agentd 
tcp        0      0 0.0.0.0:10052           0.0.0.0:*               LISTEN      10320/java          
tcp6       0      0 :::43533                :::*                    LISTEN      -                   
tcp6       0      0 :::111                  :::*                    LISTEN      15627/rpcbind       
tcp6       0      0 :::20048                :::*                    LISTEN      15753/rpc.mountd    
tcp6       0      0 :::34298                :::*                    LISTEN      14929/rpc.statd     
tcp6       0      0 :::2049                 :::*                    LISTEN      -                   
tcp6       0      0 :::10050                :::*                    LISTEN      10386/zabbix_agentd

再重啓 tomcat，最好同時重啓 zabbix-java-gateway 和 zabbix-agentapache

[root@web1 ~]# /usr/local/tomcat/bin/startup.sh 
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/jdk1.8/jre
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@web1 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:41125           0.0.0.0:*               LISTEN      10506/java          
tcp        0      0 0.0.0.0:8009            0.0.0.0:*               LISTEN      10506/java          
tcp        0      0 0.0.0.0:9999            0.0.0.0:*               LISTEN      10506/java          
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      10506/java          
tcp        0      0 0.0.0.0:40496           0.0.0.0:*               LISTEN      10506/java          
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      6387/nginx: master  
tcp        0      0 0.0.0.0:20048           0.0.0.0:*               LISTEN      15753/rpc.mountd    
tcp        0      0 0.0.0.0:37457           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:49366           0.0.0.0:*               LISTEN      14929/rpc.statd     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2993/sshd           
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      10386/zabbix_agentd 
tcp        0      0 0.0.0.0:10052           0.0.0.0:*               LISTEN      10320/java          
tcp6       0      0 :::43533                :::*                    LISTEN      -                   
tcp6       0      0 :::111                  :::*                    LISTEN      15627/rpcbind       
tcp6       0      0 :::20048                :::*                    LISTEN      15753/rpc.mountd    
tcp6       0      0 :::34298                :::*                    LISTEN      14929/rpc.statd     
tcp6       0      0 :::2049                 :::*                    LISTEN      -                   
tcp6       0      0 :::10050                :::*                    LISTEN      10386/zabbix_agentd

一樣的方法，將其餘有問題的 tomcat 機器重啓，就應該能夠了
bootstrap