kafka監控指標項

監控配置

​ kafka基本分爲broker、producer、consumer三個子項，每一項的啓動都須要用到 $KAFKA_HOME/bin/kafka-run-class.sh 腳本，在該腳本中，存在如下語句：

if ...
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false"
fi
if ...
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi

​ 在啓動kafka的過程當中，只要指定 JMX_PORT 的值，便可對broker、producer、consumer進行監控。目前有兩種方法，

• 在 $KAFKA_HOME/bin/kafka-server-start.sh、 $KAFKA_HOME/bin/kafka-console-consumer.sh、 $KAFKA_HOME/bin/kafka-console-producer.sh 三個腳本中分別添加 $JMX_PORT=XXXX 語句，可是隻適用於使用console方式對topic進行使用的狀況。
• 修改$KAFKA_HOME/bin/kafka-run-class.sh 腳本中的上述語句，使其端口隨機變化，能夠經過 ps -ef |grep kafka 命令來獲取隨機的端口號，來進行監控

指標項來源

如下端口均隨機得到。

主機名	類別	端口號
192.168.20.10	broker	9183
192.168.20.10	producer	9108
192.168.20.10	consumer	9173

kafka監控項

OS監控項

objectName	指標項	說明
java.lang:type=OperatingSystem	FreePhysicalMemorySize	空閒物理內存
java.lang:type=OperatingSystem	SystemCpuLoad	系統CPU利用率
java.lang:type=OperatingSystem	ProcessCpuLoad	進程CPU利用率
java.lang:type=GarbageCollector, name=G1 Young Generation	CollectionCount	GC次數

broker指標

objectName	指標項	說明
kafka.server:type=BrokerTopicMetrics, name=BytesInPerSec	Count	每秒輸入的流量
kafka.server:type=BrokerTopicMetrics, name=BytesOutPerSec	Count	每秒輸出的流量
kafka.server:type=BrokerTopicMetrics, name=BytesRejectedPerSec	Count	每秒扔掉的流量
kafka.server:type=BrokerTopicMetrics, name=MessagesInPerSec	Count	每秒的消息寫入總量
kafka.server:type=BrokerTopicMetrics, name=FailedFetchRequestsPerSec	Count	當前機器每秒fetch請求失敗的數量
kafka.server:type=BrokerTopicMetrics, name=FailedProduceRequestsPerSec	Count	當前機器每秒produce請求失敗的數量
kafka.server:type=ReplicaManager, name=PartitionCount	Value	該broker上的partition的數量
kafka.server:type=ReplicaManager, name=LeaderCount	Value	Leader的replica的數量
kafka.network:type=RequestMetrics, name=TotalTimeMs,request=FetchConsumer	Count	一個請求FetchConsumer耗費的全部時間
kafka.network:type=RequestMetrics, name=TotalTimeMs,request=FetchFollower	Count	一個請求FetchFollower耗費的全部時間
kafka.network:type=RequestMetrics, name=TotalTimeMs,request=Produce	Count	一個請求Produce耗費的全部時間

producer以及topic指標

objectName	指標項	官網說明	譯文說明
kafka.producer:type=producer-metrics,client-id=console-producer(client-id會變化)	incoming-byte-rate	The average number of incoming bytes received per second from all servers.	producer每秒的平均寫入流量
kafka.producer:type=producer-metrics,client-id=console-producer(client-id會變化)	outgoing-byte-rate	The average number of outgoing bytes sent per second to all servers.	producer每秒的輸出流量
kafka.producer:type=producer-metrics,client-id=console-producer(client-id會變化)	request-rate	The average number of requests sent per second to the broker.	producer每秒發給broker的平均request次數
kafka.producer:type=producer-metrics,client-id=console-producer(client-id會變化)	response-rate	The average number of responses received per second from the broker.	producer每秒發給broker的平均response次數
kafka.producer:type=producer-metrics,client-id=console-producer(client-id會變化)	request-latency-avg	The average time taken for a fetch request.	一個fetch請求的平均時間
kafka.producer:type=producer-topic-metrics,client-id=console-producer,topic=testjmx(client-id和topic名稱會變化)	record-send-rate	The average number of records sent per second for a topic.	每秒從topic發送的平均記錄數
kafka.producer:type=producer-topic-metrics,client-id=console-producer,topic=testjmx(client-id和topic名稱會變化)	record-retry-total	The total number of retried record sends	重試發送的消息總數量
kafka.producer:type=producer-topic-metrics,client-id=console-producer,topic=testjmx(client-id和topic名稱會變化)	record-error-total	The total number of record sends that resulted in errors	發送錯誤的消息總數量

consumer指標

objectName	指標項	官網說明	說明
kafka.consumer:type=consumer-fetch-manager-metrics,client-id=consumer-1(client-id會變化)	records-lag-max	Number of messages the consumer lags behind the producer by. Published by the consumer, not broker.	由consumer提交的消息消費lag
kafka.consumer:type=consumer-fetch-manager-metrics,client-id=consumer-1(client-id會變化)	records-consumed-rate	The average number of records consumed per second	每秒平均消費的消息數量