jmxtrans+influxdb+grafana監控zookeeper實戰

序

本文主要研究一下如何使用jmxtrans+influxdb+granfa監控zookeeper

配置zookeeper jmx

在conf目錄下新增zookeeper-env.sh，並使用chmod +x賦予執行權限，內容以下

JMXLOCALONLY=false
JMXDISABLE=false
JMXPORT=8999
JMXAUTH=false
JMXSSL=false
複製代碼

• 指定JMXPORT爲8999

搭建influxdb及grafana

這裏使用docker構建，以下

docker network create monitoring-network
docker run -d -p 8083:8083 -p 8086:8086 \
    --net monitoring-network  \
    --name influxdb  \
    influxdb:1.5.4
curl -POST http://192.168.99.100:8086/query --data-urlencode "q=CREATE DATABASE zookeeper"
docker run -d -p 3000:3000 --net monitoring-network --name grafana grafana/grafana:5.2.4
curl 'http://admin:admin@192.168.99.100:3000/api/datasources' -X POST \
-H 'Content-Type: application/json;charset=UTF-8' \
--data-binary '{"name":"influx","type":"influxdb","url":"http://192.168.99.100:8086", "access":"proxy","isDefault":true,"database":"zookeeper","user":"admin","password":"admin"}'
複製代碼

• 建立monitoring-network，讓docker中的influxdb與grafana連通，而後使用rest api建立zookeeper數據庫
• 建立grafana並指定net爲monitoring-network，而後使用rest api配置數據源

jmxtrans配置

• 下載jmxtrans

wget http://central.maven.org/maven2/org/jmxtrans/jmxtrans/270/jmxtrans-270-all.jar
複製代碼

• 下載執行腳本

wget -q https://raw.githubusercontent.com/jmxtrans/jmxtrans/master/jmxtrans/jmxtrans.sh
chmod +x jmxtrans.sh
複製代碼

• 配置zookeeper.json

{
  "servers": [
    {
      "port": 8999,
      "host": "localhost",
      "alias": "zk",
      "queries": [
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*",
          "attr": [
            "QuorumSize"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*",
          "attr": [
            "MaxClientCnxnsPerHost",
            "MaxRequestLatency",
            "AvgRequestLatency",
            "MinRequestLatency",
            "MaxSessionTimeout",
            "MinSessionTimeout",
            "OutstandingRequests",
            "PacketsReceived",
            "PacketsSent",
            "PendingRevalidationCount",
            "TickTime"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*,name3=InMemoryDataTree",
          "attr": [
            "NodeCount",
            "WatchCount"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*,name3=Connections,name4=*,name5=*",
          "resultAlias": "Clients",
          "attr": [
            "AvgLatency",
            "LastLatency",
            "MaxLatency",
            "MinLatency",
            "OutstandingRequests",
            "PacketsReceived",
            "PacketsSent"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=StandaloneServer_port*",
          "resultAlias": "Clients",
          "attr": [
            "AvgLatency",
            "LastLatency",
            "MaxLatency",
            "MinLatency",
            "OutstandingRequests",
            "PacketsReceived",
            "PacketsSent",
            "NumAliveConnections"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "java.lang:type=Memory",
          "resultAlias": "Memory",
          "attr": [
            "HeapMemoryUsage",
            "NonHeapMemoryUsage"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "java.lang:type=OperatingSystem",
          "resultAlias": "Process",
          "attr": [
            "OpenFileDescriptorCount",
            "ProcessCpuLoad"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "java.lang:type=Threading",
          "resultAlias": "Threading",
          "attr": [
            "ThreadCount"
          ]
        }
      ],
      "numQueryThreads": 2
    }
  ]
}
複製代碼

放在跟jmxtrans.sh腳本同一個目錄下便可自動加載

• 啓動

JAR_FILE=jmxtrans-270-all.jar ./jmxtrans.sh start
複製代碼

• 日誌輸出實例

2018-10-04 11:28:30 [main] level org.quartz.impl.StdSchedulerFactory [StdSchedulerFactory.java:1179] - Using default implementation for ThreadExecutor
2018-10-04 11:28:30 [main] level org.quartz.core.SchedulerSignalerImpl [SchedulerSignalerImpl.java:60] - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:229] - Quartz Scheduler v.1.8.6 created.
2018-10-04 11:28:30 [main] level org.quartz.simpl.RAMJobStore [RAMJobStore.java:139] - RAMJobStore initialized.
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:255] - Scheduler meta-data: Quartz Scheduler (v1.8.6) 'ServerScheduler' with instanceId 'Mars.local1538623710548'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

2018-10-04 11:28:30 [main] level org.quartz.impl.StdSchedulerFactory [StdSchedulerFactory.java:1324] - Quartz scheduler 'ServerScheduler' initialized from an externally opened InputStream.
2018-10-04 11:28:30 [main] level org.quartz.impl.StdSchedulerFactory [StdSchedulerFactory.java:1328] - Quartz scheduler version: 1.8.6
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:2267] - JobFactory set to: com.googlecode.jmxtrans.guice.GuiceJobFactory@263f04ca
2018-10-04 11:28:30 [main] level com.googlecode.jmxtrans.JmxTransformer [JmxTransformer.java:177] - Starting Jmxtrans on : .
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:519] - Scheduler ServerScheduler_$_Mars.local1538623710548 started.
2018-10-04 11:28:30 [main] level com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory [InfluxDbWriterFactory.java:121] - Result Tags to write set to: [ResultAttribute(name=typeName), ResultAttribute(name=objDomain), ResultAttribute(name=className), ResultAttribute(name=attributeName)]
2018-10-04 11:28:30 [main] level com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory [InfluxDbWriterFactory.java:102] - Connecting to url: http://192.168.99.100:8086/ as: username: admin
2018-10-04 11:28:31 [main] level com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory [InfluxDbWriterFactory.java:121] - Result Tags to write set to: [ResultAttribute(name=typeName), ResultAttribute(name=objDomain), ResultAttribute(name=className), ResultAttribute(name=attributeName)]
複製代碼

查詢influxdb數據

docker exec -it influxdb influx
Connected to http://localhost:8086 version 1.5.4
InfluxDB shell version: 1.5.4
> use zookeeper
Using database zookeeper
> show MEASUREMENTS
name: measurements
name
----
Clients
Memory
Process
Threading
> show series from "Memory"
key
---
Memory,attributeName=HeapMemoryUsage,className=sun.management.MemoryImpl,hostname=zk,objDomain=java.lang,typeName=type\=Memory
Memory,attributeName=NonHeapMemoryUsage,className=sun.management.MemoryImpl,hostname=zk,objDomain=java.lang,typeName=type\=Memory
複製代碼

• 以後經過select * from "Memory"能夠查看具體指標數據
• 查詢到有數據以後，就能夠在grafana上面進行配置可視化，而後進行監控

小結

對於zookeeper、kafka之類應用來講，內置了jmx，於是其監控能夠經過jmxtrans進行指標收集轉換，而後輸出到influxdb或者graphite或者prometheus等，最後經過grafana進行可視化及監控報警。

doc

• jmxtrans
• jmxtrans-270-all.jar
• How do i enable remote jmx with port in zookeeper zkServer.cmd
• Monitoring Apache Kafka with Grafana / InfluxDB via JMX