《Apache kafka實戰》讀書筆記-kafka集羣監控工具
《Apache kafka實戰》讀書筆記-kafka集羣監控工具html
做者：尹正傑java
版權聲明：原創做品，謝絕轉載！不然將追究法律責任。node
如官網所述，Kafka使用基於yammer metrics的監控指標體系來統計broker端和clinets端的各類監控指標（metric）。說到yammer metrics，其官網過出了這樣的一句話：mysql
yammer metrics是一個java庫，它使得你可以對生產環境代碼所作之事具備無與倫比的洞察力。
固然，雖然該項目已經進化到3.x版本，但即便最新版本的kafka易燃在使用2.2版本的yammer metrics。目前，Kafka默認提供了超多的監控指標，無一例外，用戶皆使用JMX接口訪問這些指標。JMX，即Java Management Extension，Java管理擴展，是一套爲各個應用程序，設備或系統等植入管理功能的框架。JMX自己是跨平臺的，所以具備高度的靈活性，能夠無縫集成進各類系統中。git
雖然有一小部分功能是以Shell腳本直接容許的Java類來提供的，可是Kafka集羣大部分的容許表現均可由JMX指標來表徵，所以搭建一套可以訪問JMX的監控框架或系統對於監控kafka集羣而言相當重要。github
Kafka的每一個監控指標都是以JMX MBean的形式定義的。雖然JMX規範再也不本書的討論範圍內，但這裏依然簡單介紹一些MBean的概念。MBean表明一個被管理的資源實例，他表示管理資源的Java對象。每一個MBean的管理接口以下：web
1>.屬性值;sql
2>.可以執行的操做；docker
3>.能發出的通知事件；
apache
4>.構建器；
對於kafka而言，咱們只關係上述列表中的第一項，即MBean的屬性值，它包含了Kafka各類監控指標的真實值。MBean須要註冊以後才能使用，在註冊時必須指定MBean的名稱。在kafka中，全部MBean的命名規範有着統一的的格式：「xxx.type=xxx,{attr}=xxx」。第一個等式中的xxx一般表示所屬的kafka組建，好比「kafka.server」,"kafka.producer","kafka.consumer"等；第二個等式的中的xxx和前面的attr一般表示該MBean的範圍，好比topic=yinzhengjie-kafka表示該MBean的做用範圍是名爲「yinzhengjie-kafka」的topic，即它是一個topic級別的MBean，表示該特topic下的某個性能指標。
對於Java用戶而言，訪問JMX端口最簡單直接方法就是使用JDK自帶的JConsole工具（能夠參考我以前的筆記：http://www.javashuo.com/article/p-uuoreltx-bt.html）。固然，你也能夠直接查看本篇博客的末尾，我使用了jmxTool工具的一個案例。
一.安裝Kafka Monitor
kafka Monitor 是LinkedIn公司開源的一個免費框架，用於在kafka集羣上執行真實的kafka系統測試並實時監控測試度量結果。該框架由LinkedIn的kafka團隊成員維護，維護的速度要比kafka-manager快一些。
如上所述，kafka monitor是在集羣上執行系統測試的，以便讓用戶對Kafka集羣的性能有大體的瞭解。他會自動建立一個監控topic，而後分配分區並觸發preferred leader選舉以確保每一個broker都至少是一個分區的leader。這樣作的好處在於能夠全面地測試集羣上全部broker的性能而無須用戶手動干預。
1>.安裝git工具包
[root@node101 ~]# yum -y install git Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile 10gen | 2.5 kB 00:00:00 base | 3.6 kB 00:00:00 centosplus | 3.4 kB 00:00:00 epel | 3.2 kB 00:00:00 extras | 3.4 kB 00:00:00 mysql-connectors-community | 2.5 kB 00:00:00 mysql-tools-community | 2.5 kB 00:00:00 mysql56-community | 2.5 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/2): epel/x86_64/updateinfo | 934 kB 00:00:00 (2/2): epel/x86_64/primary | 3.6 MB 00:00:01 epel 12720/12720 Resolving Dependencies --> Running transaction check ---> Package git.x86_64 0:1.8.3.1-14.el7_5 will be installed --> Processing Dependency: perl-Git = 1.8.3.1-14.el7_5 for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Term::ReadKey) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Git) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Error) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: libgnome-keyring.so.0()(64bit) for package: git-1.8.3.1-14.el7_5.x86_64 --> Running transaction check ---> Package libgnome-keyring.x86_64 0:3.12.0-1.el7 will be installed ---> Package perl-Error.noarch 1:0.17020-2.el7 will be installed ---> Package perl-Git.noarch 0:1.8.3.1-14.el7_5 will be installed ---> Package perl-TermReadKey.x86_64 0:2.30-20.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================== Installing: git x86_64 1.8.3.1-14.el7_5 updates 4.4 M Installing for dependencies: libgnome-keyring x86_64 3.12.0-1.el7 base 109 k perl-Error noarch 1:0.17020-2.el7 base 32 k perl-Git noarch 1.8.3.1-14.el7_5 updates 54 k perl-TermReadKey x86_64 2.30-20.el7 base 31 k Transaction Summary =================================================================================================================================================================================== Install 1 Package (+4 Dependent packages) Total download size: 4.6 M Installed size: 23 M Downloading packages: (1/5): perl-Error-0.17020-2.el7.noarch.rpm | 32 kB 00:00:00 (2/5): libgnome-keyring-3.12.0-1.el7.x86_64.rpm | 109 kB 00:00:00 (3/5): perl-TermReadKey-2.30-20.el7.x86_64.rpm | 31 kB 00:00:00 (4/5): perl-Git-1.8.3.1-14.el7_5.noarch.rpm | 54 kB 00:00:00 (5/5): git-1.8.3.1-14.el7_5.x86_64.rpm | 4.4 MB 00:00:00 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 19 MB/s | 4.6 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : 1:perl-Error-0.17020-2.el7.noarch 1/5 Installing : perl-TermReadKey-2.30-20.el7.x86_64 2/5 Installing : libgnome-keyring-3.12.0-1.el7.x86_64 3/5 Installing : git-1.8.3.1-14.el7_5.x86_64 4/5 Installing : perl-Git-1.8.3.1-14.el7_5.noarch 5/5 Verifying : 1:perl-Error-0.17020-2.el7.noarch 1/5 Verifying : git-1.8.3.1-14.el7_5.x86_64 2/5 Verifying : libgnome-keyring-3.12.0-1.el7.x86_64 3/5 Verifying : perl-Git-1.8.3.1-14.el7_5.noarch 4/5 Verifying : perl-TermReadKey-2.30-20.el7.x86_64 5/5 Installed: git.x86_64 0:1.8.3.1-14.el7_5 Dependency Installed: libgnome-keyring.x86_64 0:3.12.0-1.el7 perl-Error.noarch 1:0.17020-2.el7 perl-Git.noarch 0:1.8.3.1-14.el7_5 perl-TermReadKey.x86_64 0:2.30-20.el7 Complete! [root@node101 ~]#
2>.克隆kafka-monitor
[root@node101 ~]# cd /yinzhengjie/kafka-monitor/ [root@node101 kafka-monitor]# [root@node101 kafka-monitor]# ll total 0 [root@node101 kafka-monitor]# [root@node101 kafka-monitor]# git clone https://github.com/linkedin/kafka-monitor.git Cloning into 'kafka-monitor'... remote: Enumerating objects: 16, done. remote: Counting objects: 100% (16/16), done. remote: Compressing objects: 100% (12/12), done. remote: Total 1199 (delta 1), reused 7 (delta 0), pack-reused 1183 Receiving objects: 100% (1199/1199), 583.88 KiB | 248.00 KiB/s, done. Resolving deltas: 100% (479/479), done. [root@node101 kafka-monitor]# [root@node101 kafka-monitor]# ll total 0 drwxr-xr-x 11 root root 309 Dec 3 17:38 kafka-monitor [root@node101 kafka-monitor]#
3>.構建gradlew
[root@yinzhengjie kafka-monitor]# ll total 64 drwxr-xr-x 3 root root 129 Dec 3 17:35 bin drwxr-xr-x 7 root root 85 Dec 3 18:14 build -rw-r--r-- 1 root root 1797 Dec 3 17:35 build.gradle drwxr-xr-x 2 root root 50 Dec 3 17:35 checkstyle drwxr-xr-x 2 root root 130 Dec 3 17:35 config -rw-r--r-- 1 root root 906 Dec 3 17:35 CONTRIBUTING.md drwxr-xr-x 2 root root 90 Dec 3 17:35 docker drwxr-xr-x 3 root root 19 Dec 3 17:35 docs drwxr-xr-x 3 root root 20 Dec 3 17:35 gradle -rwxr-xr-x 1 root root 4971 Dec 3 17:35 gradlew -rw-r--r-- 1 root root 2404 Dec 3 17:35 gradlew.bat -rw-r--r-- 1 root root 28832 Dec 3 17:35 LICENSE -rw-r--r-- 1 root root 1371 Dec 3 17:35 NOTICE -rw-r--r-- 1 root root 6369 Dec 3 17:35 README.md -rw-r--r-- 1 root root 0 Dec 3 17:35 settings.gradle drwxr-xr-x 4 root root 28 Dec 3 17:35 src drwxr-xr-x 3 root root 32 Dec 3 17:35 webapp [root@yinzhengjie kafka-monitor]# ./gradlew jar
下面是構建時他會自動聯網下載一些maven依賴關係：
若是你安裝成功的話，說明問題不大， 可是你的Java環境若是沒有部署好的話，可能會拋異常以下：
..................(一系列消息被我省略) 18:12:29.045 [INFO] [org.gradle.api.internal.tasks.compile.JdkJavaCompiler] Compiling with JDK Java compiler API. 18:12:29.045 [DEBUG] [org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter] Removed task artifact state for {} from context. 18:12:29.045 [DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter] Finished executing task ':compileJava' 18:12:29.046 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] :compileJava FAILED 18:12:29.046 [INFO] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] :compileJava (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.06 secs. 18:12:29.046 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationWorkerRegistry] Worker root.1 completed (0 in use) 18:12:29.046 [DEBUG] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] Task worker [Thread[Daemon worker Thread 3,5,main]] finished, busy: 0.06 secs, idle: 0.0 secs 18:12:29.047 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 18:12:29.047 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception. 18:12:29.047 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong: 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Execution failed for task ':compileJava'. 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Could not find tools.jar. Please check that /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre contains a valid JDK installation. 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try: 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace. 18:12:29.048 [LIFECYCLE] [org.gradle.internal.buildevents.BuildResultLogger] 18:12:29.048 [LIFECYCLE] [org.gradle.internal.buildevents.BuildResultLogger] BUILD FAILED 18:12:29.048 [LIFECYCLE] [org.gradle.internal.buildevents.BuildResultLogger] 18:12:29.048 [LIFECYCLE] [org.gradle.internal.buildevents.BuildResultLogger] Total time: 1.174 secs 18:12:29.060 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for task history cache (/data/yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-monitor/.gradle/3.2.1/taskArtifacts) 18:12:29.060 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on task history cache (/data/yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-monitor/.gradle/3.2.1/taskArtifacts). 18:12:29.064 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs) 18:12:29.064 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs) 18:12:29.064 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.ResolutionResultsStoreFactory] Deleted 2 resolution results binary files in 0.001 secs 18:12:29.064 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache Plugin Resolution Cache (/root/.gradle/caches/3.2.1/plugin-resolution) was closed 0 times. 18:12:29.064 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache Generated Gradle JARs cache (/root/.gradle/caches/3.2.1/generated-gradle-jars) was closed 0 times. 18:12:29.064 [DEBUG] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] Stopping 0 compiler daemon(s). 18:12:29.065 [INFO] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] Stopped 0 compiler daemon(s). 18:12:29.065 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for artifact cache (/root/.gradle/caches/modules-2) 18:12:29.065 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache module-metadata.bin (/root/.gradle/caches/modules-2/metadata-2.23/module-metadata.bin) 18:12:29.065 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache artifact-at-repository.bin (/root/.gradle/caches/modules-2/metadata-2.23/artifact-at-repository.bin) 18:12:29.065 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache module-artifacts.bin (/root/.gradle/caches/modules-2/metadata-2.23/module-artifacts.bin) 18:12:29.065 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on artifact cache (/root/.gradle/caches/modules-2). 18:12:29.065 [DEBUG] [org.gradle.launcher.daemon.server.exec.ExecuteBuild] The daemon has finished executing the build. 18:12:29.275 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientInputForwarder] Dispatching close input message: org.gradle.launcher.daemon.protocol.CloseInput@1f6ddd9f 18:12:29.275 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 14: dispatching class org.gradle.launcher.daemon.protocol.CloseInput 18:12:29.276 [INFO] [org.gradle.launcher.daemon.client.DaemonClient] Received result Failure[value=org.gradle.initialization.ReportedException: org.gradle.internal.exceptions.LocationAwareException: Execution failed for task ':compileJava'.] from daemon DaemonInfo{pid=18961, address=[d4fe0de0-9944-4ad9-a617-eae681b59f33 port:42738, addresses:[/127.0.0.1]], state=Idle, lastBusy=1543831940300, context=DefaultDaemonContext[uid=16910369-6f16-4b04-b799-3e7b93a3a0e6,javaHome=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre,daemonRegistryDir=/root/.gradle/daemon,pid=18961,idleTimeout=10800000,daemonOpts=-XX:MaxPermSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xmx1024m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]} (build should be done). 18:12:29.276 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: dispatching class org.gradle.launcher.daemon.protocol.Finished 18:12:29.277 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: connection stop [root@yinzhengjie kafka-monitor]#
報錯分析：
根據上述報錯信息，提示找不到tools.jar。多是你的java是基於yum方式安裝的，不少工具包可能不全。
解決方案：
將你下載好的jdk中存在的tools.jar包拷貝過去便可。
[root@yinzhengjie ~]# find / -name tools.jar /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/lib/tools.jar [root@yinzhengjie ~]# [root@yinzhengjie ~]# cp /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/lib/tools.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/lib/ [root@yinzhengjie ~]# [root@yinzhengjie ~]# cp /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/lib/tools.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/ [root@yinzhengjie ~]# [root@yinzhengjie ~]# find / -name tools.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/lib/tools.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/tools.jar /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/lib/tools.jar [root@yinzhengjie ~]#
將tools.jar包拷貝到指定環境變量後，咱們開始執行如下操做：（你會發現變異成功啦！）
[root@yinzhengjie kafka-monitor]# ll total 64 drwxr-xr-x 3 root root 129 Dec 3 17:35 bin drwxr-xr-x 7 root root 85 Dec 3 18:14 build -rw-r--r-- 1 root root 1797 Dec 3 17:35 build.gradle drwxr-xr-x 2 root root 50 Dec 3 17:35 checkstyle drwxr-xr-x 2 root root 130 Dec 3 17:35 config -rw-r--r-- 1 root root 906 Dec 3 17:35 CONTRIBUTING.md drwxr-xr-x 2 root root 90 Dec 3 17:35 docker drwxr-xr-x 3 root root 19 Dec 3 17:35 docs drwxr-xr-x 3 root root 20 Dec 3 17:35 gradle -rwxr-xr-x 1 root root 4971 Dec 3 17:35 gradlew -rw-r--r-- 1 root root 2404 Dec 3 17:35 gradlew.bat -rw-r--r-- 1 root root 28832 Dec 3 17:35 LICENSE -rw-r--r-- 1 root root 1371 Dec 3 17:35 NOTICE -rw-r--r-- 1 root root 6369 Dec 3 17:35 README.md -rw-r--r-- 1 root root 0 Dec 3 17:35 settings.gradle drwxr-xr-x 4 root root 28 Dec 3 17:35 src drwxr-xr-x 3 root root 32 Dec 3 17:35 webapp [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# ./gradlew jar :compileJava UP-TO-DATE :processResources UP-TO-DATE :classes UP-TO-DATE :copyDependantLibs UP-TO-DATE :jar UP-TO-DATE BUILD SUCCESSFUL Total time: 1.073 secs [root@yinzhengjie kafka-monitor]#
4>.修改kafka-monitor.properties配置文件
[root@yinzhengjie kafka-monitor]# pwd /yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-monitor [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# cat config/kafka-monitor.properties | grep -v ^# | grep -v ^$ { "single-cluster-monitor": { "class.name": "com.linkedin.kmf.apps.SingleClusterMonitor", "topic": "kafka-monitor-topic", "zookeeper.connect": "10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181", #指定zookeeper地址 "bootstrap.servers": "10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092", #指定broker地址 "produce.record.delay.ms": 100, "topic-management.topicCreationEnabled": true, "topic-management.replicationFactor" : 1, "topic-management.partitionsToBrokersRatio" : 2.0, "topic-management.rebalance.interval.ms" : 600000, "topic-management.topicFactory.props": { }, "topic-management.topic.props": { "retention.ms": "3600000" }, "produce.producer.props": { "client.id": "kmf-client-id" }, "consume.latency.sla.ms": "20000", "consume.consumer.props": { } }, "jetty-service": { "class.name": "com.linkedin.kmf.services.JettyService", "jetty.port": 8000 }, "jolokia-service": { "class.name": "com.linkedin.kmf.services.JolokiaService" }, "reporter-service": { "class.name": "com.linkedin.kmf.services.DefaultMetricsReporterService", "report.interval.sec": 1, "report.metrics.list": [ "kmf:type=kafka-monitor:offline-runnable-count", "kmf.services:type=produce-service,name=*:produce-availability-avg", "kmf.services:type=consume-service,name=*:consume-availability-avg", "kmf.services:type=produce-service,name=*:records-produced-total", "kmf.services:type=consume-service,name=*:records-consumed-total", "kmf.services:type=consume-service,name=*:records-lost-total", "kmf.services:type=consume-service,name=*:records-duplicated-total", "kmf.services:type=consume-service,name=*:records-delay-ms-avg", "kmf.services:type=produce-service,name=*:records-produced-rate", "kmf.services:type=produce-service,name=*:produce-error-rate", "kmf.services:type=consume-service,name=*:consume-error-rate" ] } } [root@yinzhengjie kafka-monitor]#
5>.啓動kafka monitor服務
[root@yinzhengjie kafka-monitor]# pwd /yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-monitor [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# ll total 64 drwxr-xr-x 3 root root 129 Dec 3 17:35 bin drwxr-xr-x 7 root root 85 Dec 3 18:14 build -rw-r--r-- 1 root root 1797 Dec 3 17:35 build.gradle drwxr-xr-x 2 root root 50 Dec 3 17:35 checkstyle drwxr-xr-x 2 root root 130 Dec 3 19:26 config -rw-r--r-- 1 root root 906 Dec 3 17:35 CONTRIBUTING.md drwxr-xr-x 2 root root 90 Dec 3 17:35 docker drwxr-xr-x 3 root root 19 Dec 3 17:35 docs drwxr-xr-x 3 root root 20 Dec 3 17:35 gradle -rwxr-xr-x 1 root root 4971 Dec 3 17:35 gradlew -rw-r--r-- 1 root root 2404 Dec 3 17:35 gradlew.bat -rw-r--r-- 1 root root 28832 Dec 3 17:35 LICENSE drwxr-xr-x 2 root root 29 Dec 3 19:24 logs -rw-r--r-- 1 root root 1371 Dec 3 17:35 NOTICE -rw-r--r-- 1 root root 6369 Dec 3 17:35 README.md -rw-r--r-- 1 root root 0 Dec 3 17:35 settings.gradle drwxr-xr-x 4 root root 28 Dec 3 17:35 src drwxr-xr-x 3 root root 32 Dec 3 17:35 webapp [root@yinzhengjie kafka-monitor]# yinzhengjie [root@yinzhengjie kafka-monitor]# mkdir -pv /yinzhengjie/softwares/kafkasoft/kafka-monitor/logs mkdir: created directory ‘/yinzhengjie/softwares/kafkasoft/kafka-monitor/logs’ [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# bin/kafka-monitor-start.sh config/kafka-monitor.properties >> /yinzhengjie/softwares/kafkasoft/kafka-monitor/logs/kafka-monitor-`date +%F`.log & [1] 16888 [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# jobs [1]+ Running bin/kafka-monitor-start.sh config/kafka-monitor.properties >> /yinzhengjie/softwares/kafkasoft/kafka-monitor/logs/kafka-monitor-`date +%F`.log & [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# tail -1000f /yinzhengjie/softwares/kafkasoft/kafka-monitor/logs/kafka-monitor-2018-12-03.log | more [2018-12-03 19:30:12,160] INFO MultiClusterTopicManagementServiceConfig values: topic = kafka-monitor-topic topic-management.rebalance.interval.ms = 600000 (com.linkedin.kmf.services.configs.MultiClusterTopicManagementServiceConfig) [2018-12-03 19:30:12,240] INFO TopicManagementServiceConfig values: topic = kafka-monitor-topic topic-management.minPartitionNum = 1 topic-management.partitionsToBrokersRatio = 2.0 topic-management.replicationFactor = 1 topic-management.topicCreationEnabled = true topic-management.topicFactory.class.name = com.linkedin.kmf.topicfactory.DefaultTopicFactory zookeeper.connect = 10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181 (com.linkedin.kmf.services.configs.TopicManagementServiceConfig) [2018-12-03 19:30:12,244] INFO ProduceServiceConfig values: bootstrap.servers = 10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 produce.latency.percentile.granularity.ms = 1 produce.latency.percentile.max.ms = 5000 produce.partitioner.class = class com.linkedin.kmf.partitioner.NewKMPartitioner produce.producer.class = com.linkedin.kmf.producer.NewProducer produce.producer.id = kmf-producer produce.record.delay.ms = 100 produce.record.size.byte = 100 produce.sync = true produce.thread.num = 5 produce.treat.zero.throughput.as.unavailable = true topic = kafka-monitor-topic zookeeper.connect = 10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181 (com.linkedin.kmf.services.configs.ProduceServiceConfig) [2018-12-03 19:30:12,338] INFO single-cluster-monitor/ProduceService is initialized. (com.linkedin.kmf.services.ProduceService) [2018-12-03 19:30:12,347] INFO ConsumeServiceConfig values: bootstrap.servers = 10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 consume.consumer.class = com.linkedin.kmf.consumer.NewConsumer consume.latency.percentile.granularity.ms = 1 consume.latency.percentile.max.ms = 5000 consume.latency.sla.ms = 20000 topic = kafka-monitor-topic zookeeper.connect = 10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181 (com.linkedin.kmf.services.configs.ConsumeServiceConfig) --More--
6>.檢查服務是否啓動成功
[root@yinzhengjie ~]# netstat -untalp | grep 8000 tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 16888/java [root@yinzhengjie ~]# [root@yinzhengjie ~]# hostname -i 10.19.104.189 [root@yinzhengjie ~]#
咱們能夠經過webUI的方式訪問kafka-monitor服務：
Kafka Monitor做爲LinkedIn Kafka 團隊維護的開源框架，從系統測試的角度給出了kafka集羣的性能評測結果。從這個意義上來講，它不一樣於其餘的監控框架。且平臺目前提供的Web界面功能十分簡單，所以實際使用用戶數並非不少。不過鑑於其代碼貢獻者基本上都是Kafka社區團隊成員，其框架的發展仍是值得期待的。
二.安裝Kafka Offset Monitor
這款監控工具不推薦你們使用，由於這個項目已經不維護了啦。尤爲是kafka0.9.0.0之後的kafka版本，不建議使用，由於我用kafka0.10.2.1這個版本時，它是沒法監控到broker集羣的信息的！這一點很讓人蛋疼！使人欣喜的是，該項目又一個同名的「姊妹」項目，地址是：「https://github.com/Morningstar/kafka-offset-monitor」，感興趣的小夥伴能夠搗鼓一下。
我已經下載好相應的安裝包了，感興趣的小夥伴能夠直接拿去用，我給出百度雲連接:https://pan.baidu.com/s/1Qbgtxtdd_1B2hlqatA0GRQ 密碼:u991
[root@yinzhengjie kafka-offset-monitor]# ll total 54468 -rw-r--r-- 1 root root 52937155 Dec 3 20:00 KafkaOffsetMonitor-assembly-0.2.1.jar -rw-r--r-- 1 root root 2830336 Dec 3 20:38 offsetapp.db [root@yinzhengjie kafka-offset-monitor]# [root@yinzhengjie kafka-offset-monitor]# java -cp KafkaOffsetMonitor-assembly-0.2.1.jar:kafka-offset-monitor-another-db-reporter.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk 10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181 --port 1010 --refresh 10.seconds --retain 2.days serving resources from: jar:file:/data/yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-offset-monitor/KafkaOffsetMonitor-assembly-0.2.1.jar!/offsetapp SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. log4j:WARN No appenders could be found for logger (org.I0Itec.zkclient.ZkConnection). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. log4j:WARN No appenders could be found for logger (org.I0Itec.zkclient.ZkEventThread). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 2018-12-03 20:08:56.101:INFO:oejs.Server:jetty-7.x.y-SNAPSHOT 2018-12-03 20:08:56.132:INFO:oejsh.ContextHandler:started o.e.j.s.ServletContextHandler{/,jar:file:/data/yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-offset-monitor/KafkaOffsetMonitor-assembly-0.2.1.jar!/offsetapp} 2018-12-03 20:08:56.139:INFO:oejs.AbstractConnector:Started SocketConnector@0.0.0.0:1010
而後你在webUI訪問該主機的1010端口便可，若是你的kafka版本太高的話，估計夠嗆，你應該查不到相應的信息！
三.CruiseControl部署
咱們聊聊LinkedIn公司於2017年8月底開源的Kafka監控框架，即CruiseControl。它主要用於管理查大規模的Kafka集羣，提供的功能以下：
1>.實時監控資源使用率（broker，topic，分區）。
2>.提供多緯度的重平衡策略：
2.1>.機架感知（rack awareness）
2.2>.資源使用率再平衡（CPU，磁盤，網絡I/O）
2.3>.leader副本流量分配
2.4>.副本均勻分佈
3>.異常檢測以及實時告警。
4>.開箱即用的運維操做：
4.1>.增長broker節點
4.2>.去除broker節點
4.3>.集羣在平衡。
因爲該框架剛剛開源，有不少功能尚在完善和演進中。打算使用CruiseControl的用戶最好時刻關注該框架的發展，其官網地址是：https://github.com/linkedin/cruise-control。
接下來咱們掩飾一些如何安裝，配置和啓動CruiseControl的詳細步驟。
1>.下載源碼
[root@yinzhengjie kafka-cruiseControl]# ll total 0 [root@yinzhengjie kafka-cruiseControl]# [root@yinzhengjie kafka-cruiseControl]# git clone https://github.com/linkedin/cruise-control.git Cloning into 'cruise-control'... remote: Enumerating objects: 47, done. remote: Counting objects: 100% (47/47), done. remote: Compressing objects: 100% (32/32), done. remote: Total 7615 (delta 9), reused 16 (delta 0), pack-reused 7568 Receiving objects: 100% (7615/7615), 2.41 MiB | 735.00 KiB/s, done. Resolving deltas: 100% (3537/3537), done. [root@yinzhengjie kafka-cruiseControl]# [root@yinzhengjie kafka-cruiseControl]# ll total 4 drwxr-xr-x 12 root root 4096 Dec 4 10:45 cruise-control [root@yinzhengjie kafka-cruiseControl]#
2>.進行編譯
[root@yinzhengjie kafka-cruiseControl]# ll total 4 drwxr-xr-x 12 root root 4096 Dec 4 10:45 cruise-control [root@yinzhengjie kafka-cruiseControl]# [root@yinzhengjie kafka-cruiseControl]# cd cruise-control/ [root@yinzhengjie cruise-control]# ll total 68 -rw-r--r-- 1 root root 8257 Dec 4 10:45 build.gradle drwxr-xr-x 3 root root 35 Dec 4 10:45 buildSrc drwxr-xr-x 2 root root 50 Dec 4 10:45 checkstyle drwxr-xr-x 2 root root 80 Dec 4 10:45 config -rw-r--r-- 1 root root 1079 Dec 4 10:45 CONTRIBUTING.md drwxr-xr-x 3 root root 16 Dec 4 10:45 cruise-control drwxr-xr-x 3 root root 16 Dec 4 10:45 cruise-control-core drwxr-xr-x 3 root root 16 Dec 4 10:45 cruise-control-metrics-reporter drwxr-xr-x 3 root root 19 Dec 4 10:45 docs drwxr-xr-x 3 root root 47 Dec 4 10:45 gradle -rw-r--r-- 1 root root 86 Dec 4 10:45 gradle.properties -rwxr-xr-x 1 root root 5299 Dec 4 10:45 gradlew -rw-r--r-- 1 root root 2260 Dec 4 10:45 gradlew.bat -rwxr-xr-x 1 root root 5129 Dec 4 10:45 kafka-cruise-control-start.sh -rw-r--r-- 1 root root 1332 Dec 4 10:45 LICENSE -rw-r--r-- 1 root root 304 Dec 4 10:45 NOTICE -rw-r--r-- 1 root root 11476 Dec 4 10:45 README.md -rw-r--r-- 1 root root 153 Dec 4 10:45 semantic-build-versioning.gradle -rw-r--r-- 1 root root 911 Dec 4 10:45 settings.gradle [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ./gradlew clean jar Downloading https://services.gradle.org/distributions/gradle-4.8-all.zip ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... Unzipping /root/.gradle/wrapper/dists/gradle-4.8-all/41c9mvq2w0645xx6nsenaerjl/gradle-4.8-all.zip to /root/.gradle/wrapper/dists/gradle-4.8-all/41c9mvq2w0645xx6nsenaerjl Set executable permissions for: /root/.gradle/wrapper/dists/gradle-4.8-all/41c9mvq2w0645xx6nsenaerjl/gradle-4.8/bin/gradle Welcome to Gradle 4.8! Here are the highlights of this release: - Dependency locking - Maven Publish and Ivy Publish plugins improved and marked stable - Incremental annotation processing enhancements - APIs to configure tasks at creation time For more details see https://docs.gradle.org/4.8/release-notes.html To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.8/userguide/gradle_daemon.html. Daemon will be stopped at the end of the build stopping after processing Download https://jcenter.bintray.com/org/apache/httpcomponents/fluent-hc/4.5.2/fluent-hc-4.5.2.pom Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-extractor-gradle/4.7.3/build-info-extractor-gradle-4.7.3.pom Download https://jcenter.bintray.com/org/ajoberstar/gradle-git/1.2.0/gradle-git-1.2.0.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcomponents-client/4.5.2/httpcomponents-client-4.5.2.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/project/7/project-7.pom Download https://jcenter.bintray.com/org/apache/apache/13/apache-13.pom Download https://jcenter.bintray.com/org/ajoberstar/grgit/1.3.0/grgit-1.3.0.pom Download https://jcenter.bintray.com/com/github/zafarkhaja/java-semver/0.9.0/java-semver-0.9.0.pom Download https://jcenter.bintray.com/org/sonatype/oss/oss-parent/9/oss-parent-9.pom Download https://jcenter.bintray.com/commons-logging/commons-logging/1.2/commons-logging-1.2.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.pom Download https://jcenter.bintray.com/org/apache/commons/commons-parent/34/commons-parent-34.pom Download https://jcenter.bintray.com/commons-lang/commons-lang/2.4/commons-lang-2.4.pom Download https://jcenter.bintray.com/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.pom Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-extractor/2.11.2/build-info-extractor-2.11.2.pom Download https://jcenter.bintray.com/org/apache/apache/7/apache-7.pom Download https://jcenter.bintray.com/commons-io/commons-io/2.2/commons-io-2.2.pom Download https://jcenter.bintray.com/org/apache/commons/commons-parent/24/commons-parent-24.pom Download https://jcenter.bintray.com/org/apache/apache/9/apache-9.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.usocket-jna/0.0.7/jsch.agentproxy.usocket-jna-0.0.7.pom Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit/3.7.0.201502260915-r/org.eclipse.jgit-3.7.0.201502260915-r.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.usocket-nc/0.0.7/jsch.agentproxy.usocket-nc-0.0.7.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy/0.0.7/jsch.agentproxy-0.0.7.pom Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit-parent/3.7.0.201502260915-r/org.eclipse.jgit-parent-3.7.0.201502260915-r.pom Download https://jcenter.bintray.com/org/sonatype/oss/oss-parent/6/oss-parent-6.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.pom Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit.ui/3.7.0.201502260915-r/org.eclipse.jgit.ui-3.7.0.201502260915-r.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.sshagent/0.0.7/jsch.agentproxy.sshagent-0.0.7.pom Download https://jcenter.bintray.com/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.pom Download https://jcenter.bintray.com/org/slf4j/slf4j-parent/1.7.7/slf4j-parent-1.7.7.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.pom Download https://jcenter.bintray.com/commons-codec/commons-codec/1.9/commons-codec-1.9.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcomponents-core/4.4.4/httpcomponents-core-4.4.4.pom Download https://jcenter.bintray.com/org/apache/commons/commons-parent/32/commons-parent-32.pom Download https://jcenter.bintray.com/com/google/guava/guava/18.0/guava-18.0.pom Download https://jcenter.bintray.com/com/thoughtworks/xstream/xstream/1.4.10/xstream-1.4.10.pom Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-client/2.11.2/build-info-client-2.11.2.pom Download https://jcenter.bintray.com/com/google/guava/guava-parent/18.0/guava-parent-18.0.pom Download https://jcenter.bintray.com/com/thoughtworks/xstream/xstream-parent/1.4.10/xstream-parent-1.4.10.pom Download https://jcenter.bintray.com/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.pom Download https://jcenter.bintray.com/com/jcraft/jsch/0.1.50/jsch-0.1.50.pom Download https://jcenter.bintray.com/org/sonatype/oss/oss-parent/5/oss-parent-5.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.core/0.0.7/jsch.agentproxy.core-0.0.7.pom Download https://jcenter.bintray.com/net/java/dev/jna/platform/3.4.0/platform-3.4.0.pom Download https://jcenter.bintray.com/net/java/dev/jna/jna/3.4.0/jna-3.4.0.pom Download https://jcenter.bintray.com/xmlpull/xmlpull/1.1.3.1/xmlpull-1.1.3.1.pom Download https://jcenter.bintray.com/xpp3/xpp3_min/1.1.4c/xpp3_min-1.1.4c.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.4.5/httpcore-4.4.5.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcomponents-core/4.4.5/httpcomponents-core-4.4.5.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-core/2.9.5/jackson-core-2.9.5.pom Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-api/2.11.2/build-info-api-2.11.2.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/jackson-base/2.9.5/jackson-base-2.9.5.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/jackson-bom/2.9.5/jackson-bom-2.9.5.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/jackson-parent/2.9.1/jackson-parent-2.9.1.pom Download https://jcenter.bintray.com/com/fasterxml/oss-parent/30/oss-parent-30.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-databind/2.9.5/jackson-databind-2.9.5.pom Download https://jcenter.bintray.com/org/apache/commons/commons-compress/1.16/commons-compress-1.16.pom Download https://jcenter.bintray.com/org/apache/commons/commons-parent/43/commons-parent-43.pom Download https://jcenter.bintray.com/org/apache/apache/18/apache-18.pom Download https://jcenter.bintray.com/org/objenesis/objenesis/2.6/objenesis-2.6.pom Download https://jcenter.bintray.com/org/objenesis/objenesis-parent/2.6/objenesis-parent-2.6.pom Generating JAR file 'gradle-api-4.8.jar' Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-extractor-gradle/4.7.3/build-info-extractor-gradle-4.7.3.jar Download https://jcenter.bintray.com/org/apache/httpcomponents/fluent-hc/4.5.2/fluent-hc-4.5.2.jar Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit.ui/3.7.0.201502260915-r/org.eclipse.jgit.ui-3.7.0.201502260915-r.jar Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-client/2.11.2/build-info-client-2.11.2.jar Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-extractor/2.11.2/build-info-extractor-2.11.2.jar Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-api/2.11.2/build-info-api-2.11.2.jar Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit/3.7.0.201502260915-r/org.eclipse.jgit-3.7.0.201502260915-r.jar Download https://jcenter.bintray.com/commons-logging/commons-logging/1.2/commons-logging-1.2.jar Download https://jcenter.bintray.com/org/ajoberstar/grgit/1.3.0/grgit-1.3.0.jar Download https://jcenter.bintray.com/commons-io/commons-io/2.2/commons-io-2.2.jar Download https://jcenter.bintray.com/org/ajoberstar/gradle-git/1.2.0/gradle-git-1.2.0.jar Download https://jcenter.bintray.com/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.sshagent/0.0.7/jsch.agentproxy.sshagent-0.0.7.jar Download https://jcenter.bintray.com/commons-lang/commons-lang/2.4/commons-lang-2.4.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.usocket-jna/0.0.7/jsch.agentproxy.usocket-jna-0.0.7.jar Download https://jcenter.bintray.com/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.usocket-nc/0.0.7/jsch.agentproxy.usocket-nc-0.0.7.jar Download https://jcenter.bintray.com/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.4.5/httpcore-4.4.5.jar Download https://jcenter.bintray.com/commons-codec/commons-codec/1.9/commons-codec-1.9.jar Download https://jcenter.bintray.com/com/google/guava/guava/18.0/guava-18.0.jar Download https://jcenter.bintray.com/com/thoughtworks/xstream/xstream/1.4.10/xstream-1.4.10.jar Download https://jcenter.bintray.com/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.core/0.0.7/jsch.agentproxy.core-0.0.7.jar Download https://jcenter.bintray.com/com/jcraft/jsch/0.1.50/jsch-0.1.50.jar Download https://jcenter.bintray.com/net/java/dev/jna/platform/3.4.0/platform-3.4.0.jar Download https://jcenter.bintray.com/xmlpull/xmlpull/1.1.3.1/xmlpull-1.1.3.1.jar Download https://jcenter.bintray.com/net/java/dev/jna/jna/3.4.0/jna-3.4.0.jar Download https://jcenter.bintray.com/xpp3/xpp3_min/1.1.4c/xpp3_min-1.1.4c.jar Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-databind/2.9.5/jackson-databind-2.9.5.jar Download https://jcenter.bintray.com/org/objenesis/objenesis/2.6/objenesis-2.6.jar Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-core/2.9.5/jackson-core-2.9.5.jar Download https://jcenter.bintray.com/org/apache/commons/commons-compress/1.16/commons-compress-1.16.jar Download https://jcenter.bintray.com/com/github/zafarkhaja/java-semver/0.9.0/java-semver-0.9.0.jar > Task :buildSrc:jar :jar: No valid plugin descriptors were found in META-INF/gradle-plugins Download https://plugins.gradle.org/m2/gradle/plugin/net/vivin/gradle-semantic-build-versioning/4.0.0/gradle-semantic-build-versioning-4.0.0.pom Download https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit/4.8.0.201706111038-r/org.eclipse.jgit-4.8.0.201706111038-r.pom Download https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit-parent/4.8.0.201706111038-r/org.eclipse.jgit-parent-4.8.0.201706111038-r.pom Download https://plugins.gradle.org/m2/com/jcraft/jsch/0.1.54/jsch-0.1.54.pom Download https://plugins.gradle.org/m2/com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.pom Download https://plugins.gradle.org/m2/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.pom Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.pom Download https://plugins.gradle.org/m2/org/slf4j/slf4j-parent/1.7.2/slf4j-parent-1.7.2.pom Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpcomponents-client/4.3.6/httpcomponents-client-4.3.6.pom Download https://plugins.gradle.org/m2/commons-codec/commons-codec/1.6/commons-codec-1.6.pom Download https://plugins.gradle.org/m2/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.pom Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.pom Download https://plugins.gradle.org/m2/org/apache/commons/commons-parent/28/commons-parent-28.pom Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpcomponents-core/4.3.3/httpcomponents-core-4.3.3.pom Download https://plugins.gradle.org/m2/org/apache/commons/commons-parent/22/commons-parent-22.pom Download https://plugins.gradle.org/m2/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar Download https://plugins.gradle.org/m2/com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.jar Download https://plugins.gradle.org/m2/gradle/plugin/net/vivin/gradle-semantic-build-versioning/4.0.0/gradle-semantic-build-versioning-4.0.0.jar Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.jar Download https://plugins.gradle.org/m2/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.jar Download https://plugins.gradle.org/m2/com/jcraft/jsch/0.1.54/jsch-0.1.54.jar Download https://plugins.gradle.org/m2/commons-codec/commons-codec/1.6/commons-codec-1.6.jar Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar Download https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit/4.8.0.201706111038-r/org.eclipse.jgit-4.8.0.201706111038-r.jar Download https://repo.maven.apache.org/maven2/junit/junit/4.12/junit-4.12.pom Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.pom Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/39/commons-parent-39.pom Download https://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom Download https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.pom Download https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-parent/1.3/hamcrest-parent-1.3.pom Download https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar Download https://repo.maven.apache.org/maven2/junit/junit/4.12/junit-4.12.jar Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.jar > Task :cruise-control-core:compileJava Note: /data/yinzhengjie/softwares/kafkasoft/kafka-cruiseControl/cruise-control/cruise-control-core/src/main/java/com/linkedin/cruisecontrol/common/config/ConfigDef.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.11/0.11.0.2/kafka_2.11-0.11.0.2.pom Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/0.11.0.2/kafka-clients-0.11.0.2.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.pom Download https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.pom Download https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.10/zookeeper-3.4.10.pom Download https://repo.maven.apache.org/maven2/net/sf/jopt-simple/jopt-simple/5.0.3/jopt-simple-5.0.3.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.11.11/scala-library-2.11.11.pom Download https://repo.maven.apache.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.pom Download https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.pom Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.11/0.11.0.2/kafka_2.11-0.11.0.2.jar Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/0.11.0.2/kafka-clients-0.11.0.2.jar Download https://repo.maven.apache.org/maven2/net/sf/jopt-simple/jopt-simple/5.0.3/jopt-simple-5.0.3.jar Download https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.11.11/scala-library-2.11.11.jar Download https://repo.maven.apache.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar Download https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.jar Download https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.10/zookeeper-3.4.10.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-server/9.4.6.v20170531/jetty-server-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.pom Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom Download https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-core/3.2.2/metrics-core-3.2.2.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-servlet/9.4.6.v20170531/jetty-servlet-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-project/9.4.6.v20170531/jetty-project-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom Download https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-parent/3.2.2/metrics-parent-3.2.2.pom Download https://repo.maven.apache.org/maven2/log4j/log4j/1.2.16/log4j-1.2.16.pom Download https://repo.maven.apache.org/maven2/jline/jline/0.9.94/jline-0.9.94.pom Download https://repo.maven.apache.org/maven2/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.pom Download https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.pom Download https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.6.1/slf4j-parent-1.6.1.pom Download https://repo.maven.apache.org/maven2/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/9.4.6.v20170531/jetty-io-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-http/9.4.6.v20170531/jetty-http-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/net/java/jvnet-parent/3/jvnet-parent-3.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-security/9.4.6.v20170531/jetty-security-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/io/netty/netty/3.10.5.Final/netty-3.10.5.Final.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/9.4.6.v20170531/jetty-util-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/io/netty/netty/3.10.5.Final/netty-3.10.5.Final.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-security/9.4.6.v20170531/jetty-security-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-servlet/9.4.6.v20170531/jetty-servlet-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-server/9.4.6.v20170531/jetty-server-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/jline/jline/0.9.94/jline-0.9.94.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-http/9.4.6.v20170531/jetty-http-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-core/3.2.2/metrics-core-3.2.2.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/9.4.6.v20170531/jetty-io-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/9.4.6.v20170531/jetty-util-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-compiler/2.11.11/scala-compiler-2.11.11.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/modules/scala-xml_2.11/1.0.5/scala-xml_2.11-1.0.5.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.11.11/scala-reflect-2.11.11.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.11.11/scala-reflect-2.11.11.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-compiler/2.11.11/scala-compiler-2.11.11.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/modules/scala-xml_2.11/1.0.5/scala-xml_2.11-1.0.5.jar Download https://repo.maven.apache.org/maven2/com/typesafe/zinc/zinc/0.3.15/zinc-0.3.15.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.6/scala-library-2.10.6.pom Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/compiler-interface/0.13.15/compiler-interface-0.13.15.pom Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/incremental-compiler/0.13.15/incremental-compiler-0.13.15.pom Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/sbt-interface/0.13.15/sbt-interface-0.13.15.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-compiler/2.10.6/scala-compiler-2.10.6.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.10.6/scala-reflect-2.10.6.pom Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/incremental-compiler/0.13.15/incremental-compiler-0.13.15.jar Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/sbt-interface/0.13.15/sbt-interface-0.13.15.jar Download https://repo.maven.apache.org/maven2/com/typesafe/zinc/zinc/0.3.15/zinc-0.3.15.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.6/scala-library-2.10.6.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.10.6/scala-reflect-2.10.6.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-compiler/2.10.6/scala-compiler-2.10.6.jar Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/compiler-interface/0.13.15/compiler-interface-0.13.15-sources.jar > Task :cruise-control:compileScala Pruning sources from previous analysis, due to incompatible CompileSetup. ………
在構建的過程當中，你可能會遇到以下的報錯信息：
若是遇到上面的報錯信息時，不要慌，根據報錯信息提示缺乏javac這個包，咱們能夠自行去oracle官網下載相應的jdk源碼包，將對應的javac包拷貝到指定目錄中便可。我服務器端已經安裝好了對應的jdk源碼包，因而我作了以下操做：
[root@yinzhengjie cruise-control]# find / -name javac /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/bin/javac [root@yinzhengjie cruise-control]# cp /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/bin/javac /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/bin [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# find / -name javac /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/bin/javac /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/bin/javac [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ./gradlew clean jar To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.8/userguide/gradle_daemon.html. Daemon will be stopped at the end of the build stopping after processing > Task :cruise-control-core:compileJava Note: /data/yinzhengjie/softwares/kafkasoft/kafka-cruiseControl/cruise-control/cruise-control-core/src/main/java/com/linkedin/cruisecontrol/common/config/ConfigDef.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :cruise-control:compileScala Pruning sources from previous analysis, due to incompatible CompileSetup. Note: /data/yinzhengjie/softwares/kafkasoft/kafka-cruiseControl/cruise-control/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/analyzer/AnalyzerUtils.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Download https://repo.maven.apache.org/maven2/commons-io/commons-io/2.5/commons-io-2.5.pom Download https://repo.maven.apache.org/maven2/org/easymock/easymock/3.4/easymock-3.4.pom Download https://repo.maven.apache.org/maven2/org/easymock/easymock-parent/3.4/easymock-parent-3.4.pom Download https://jcenter.bintray.com/com/linkedin/kafka/clients/li-apache-kafka-clients/0.0.12/li-apache-kafka-clients-0.0.12.pom Download https://repo.maven.apache.org/maven2/org/objenesis/objenesis/2.2/objenesis-2.2.pom Download https://repo.maven.apache.org/maven2/org/objenesis/objenesis-parent/2.2/objenesis-parent-2.2.pom Download https://repo.maven.apache.org/maven2/commons-io/commons-io/2.5/commons-io-2.5.jar Download https://repo.maven.apache.org/maven2/org/easymock/easymock/3.4/easymock-3.4.jar Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/0.11.0.2/kafka-clients-0.11.0.2-test.jar Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.11/0.11.0.2/kafka_2.11-0.11.0.2-test.jar Download https://jcenter.bintray.com/com/linkedin/kafka/clients/li-apache-kafka-clients/0.0.12/li-apache-kafka-clients-0.0.12.jar Download https://repo.maven.apache.org/maven2/org/objenesis/objenesis/2.2/objenesis-2.2.jar Download https://jcenter.bintray.com/com/linkedin/kafka/clients/li-apache-kafka-clients/0.0.12/li-apache-kafka-clients-0.0.12-tests.jar Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 38s 10 actionable tasks: 10 executed [root@yinzhengjie cruise-control]#
3>.修改broker配置文件
構建成功後，咱們能夠在項目目錄下的cruise-control-metrics-reporter/build/libs下會生成一個jar包，咱們將這個jar包拷貝到broker安裝目錄的libs目錄下，具體操做以下：
[root@yinzhengjie cruise-control]# ll total 68 -rw-r--r-- 1 root root 8257 Dec 4 10:45 build.gradle drwxr-xr-x 5 root root 61 Dec 4 10:47 buildSrc drwxr-xr-x 2 root root 50 Dec 4 10:45 checkstyle drwxr-xr-x 2 root root 80 Dec 4 10:45 config -rw-r--r-- 1 root root 1079 Dec 4 10:45 CONTRIBUTING.md drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-core drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-metrics-reporter drwxr-xr-x 3 root root 19 Dec 4 10:45 docs drwxr-xr-x 3 root root 47 Dec 4 10:45 gradle -rw-r--r-- 1 root root 86 Dec 4 10:45 gradle.properties -rwxr-xr-x 1 root root 5299 Dec 4 10:45 gradlew -rw-r--r-- 1 root root 2260 Dec 4 10:45 gradlew.bat -rwxr-xr-x 1 root root 5129 Dec 4 10:45 kafka-cruise-control-start.sh -rw-r--r-- 1 root root 1332 Dec 4 10:45 LICENSE -rw-r--r-- 1 root root 304 Dec 4 10:45 NOTICE -rw-r--r-- 1 root root 11476 Dec 4 10:45 README.md -rw-r--r-- 1 root root 153 Dec 4 10:45 semantic-build-versioning.gradle -rw-r--r-- 1 root root 911 Dec 4 10:45 settings.gradle [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ll cruise-control-metrics-reporter/build/libs/ total 36 -rw-r--r-- 1 root root 36547 Dec 4 11:00 cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# scp cruise-control-metrics-reporter/build/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 10.1.2.118:/soft/kafka/libs/ root@10.1.2.118's password: cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 100% 36KB 35.7KB/s 00:00 [root@yinzhengjie cruise-control]#
注意，上面的操做須要在每臺broker都須要作一樣的操做，以後須要在每臺broker端的配置文件「server.properties」中新增下列行，保存以後啓動broker：
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
4>.修改cruisecontrol項目目錄下的conf/cruisecontrol.properties文件（根據你實際生產環境自定義相應的參數便可）
[root@yinzhengjie cruise-control]# pwd /yinzhengjie/softwares/kafkasoft/kafka-cruiseControl/cruise-control [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ll total 68 -rw-r--r-- 1 root root 8257 Dec 4 10:45 build.gradle drwxr-xr-x 5 root root 61 Dec 4 10:47 buildSrc drwxr-xr-x 2 root root 50 Dec 4 10:45 checkstyle drwxr-xr-x 2 root root 80 Dec 4 10:45 config -rw-r--r-- 1 root root 1079 Dec 4 10:45 CONTRIBUTING.md drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-core drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-metrics-reporter drwxr-xr-x 3 root root 19 Dec 4 10:45 docs drwxr-xr-x 3 root root 47 Dec 4 10:45 gradle -rw-r--r-- 1 root root 86 Dec 4 10:45 gradle.properties -rwxr-xr-x 1 root root 5299 Dec 4 10:45 gradlew -rw-r--r-- 1 root root 2260 Dec 4 10:45 gradlew.bat -rwxr-xr-x 1 root root 5129 Dec 4 10:45 kafka-cruise-control-start.sh -rw-r--r-- 1 root root 1332 Dec 4 10:45 LICENSE -rw-r--r-- 1 root root 304 Dec 4 10:45 NOTICE -rw-r--r-- 1 root root 11476 Dec 4 10:45 README.md -rw-r--r-- 1 root root 153 Dec 4 10:45 semantic-build-versioning.gradle -rw-r--r-- 1 root root 911 Dec 4 10:45 settings.gradle [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# cat config/cruisecontrol.properties | grep -v ^# | grep bootstrap.servers bootstrap.servers=10.1.2.118:9092,10.1.2.117:9092,10.1.2.116:9092,10.1.2.115:9092,10.1.2.114:9092 [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# cat config/cruisecontrol.properties | grep -v ^# | grep zookeeper.connect zookeeper.connect=10.1.2.114:2181,10.1.2.115:2181,10.1.2.116:2181,10.1.2.117:2181,10.1.2.118:2181/kafka_yinzhengjie_cluster1 [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# cat config/cruisecontrol.properties | grep -v ^# | grep webserver.http.port webserver.http.port=7070 [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# cat config/cruisecontrol.properties | grep -v ^# | grep webserver.http.cors.origin webserver.http.cors.origin=http://localhost:7070/ [root@yinzhengjie cruise-control]#
[root@node118 ~]# cat `which xcall.sh ` #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判斷用戶是否傳參 if [ $# -lt 1 ];then echo "請輸入參數" exit fi #獲取用戶輸入的命令 cmd=$@ for (( i=114;i<=118;i++ )) do #使終端變綠色 tput setaf 2 echo ============= node${i}.aggrx : $cmd ============ #使終端變回原來的顏色，即白灰色 tput setaf 7 #遠程執行命令 ssh node${i}.aggrx $cmd #判斷命令是否執行成功 if [ $? == 0 ];then echo "命令執行成功" fi done [root@node118 ~]#
[root@node118 ~]# cat `which xkafka.sh ` #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判斷用戶是否傳參 if [ $# -ne 1 ];then echo "無效參數，用法爲: $0 {start|stop}" exit fi #獲取用戶輸入的命令 cmd=$1 for (( i=114 ; i<=118 ; i++ )) ; do tput setaf 2 echo ========== node${i}.aggrx $cmd ================ tput setaf 9 case $cmd in start) ssh node${i}.aggrx "source /etc/profile ; kafka-server-start.sh -daemon /soft/kafka/config/server.properties" echo node${i}.aggrx "服務已啓動" ;; stop) ssh node${i}.aggrx "source /etc/profile ; kafka-server-stop.sh" echo node${i}.aggrx "服務已中止" ;; *) echo "無效參數，用法爲: $0 {start|stop}" exit ;; esac done [root@node118 ~]#
[root@node118 ~]# xcall.sh jps ============= node114.aggrx : jps ============ 29930 QuorumPeerMain 1900 Jps 命令執行成功 ============= node115.aggrx : jps ============ 788 QuorumPeerMain 5023 Jps 命令執行成功 ============= node116.aggrx : jps ============ 19335 QuorumPeerMain 23516 Jps 命令執行成功 ============= node117.aggrx : jps ============ 26770 Jps 22639 QuorumPeerMain 命令執行成功 ============= node118.aggrx : jps ============ 20017 QuorumPeerMain 24949 Jps 30056 Bootstrap 命令執行成功 [root@node118 ~]# [root@node118 ~]# [root@node118 ~]# xkafka.sh start ========== node114.aggrx start ================ node114.aggrx 服務已啓動 ========== node115.aggrx start ================ node115.aggrx 服務已啓動 ========== node116.aggrx start ================ node116.aggrx 服務已啓動 ========== node117.aggrx start ================ node117.aggrx 服務已啓動 ========== node118.aggrx start ================ node118.aggrx 服務已啓動 [root@node118 ~]# [root@node118 ~]# [root@node118 ~]# xcall.sh jps ============= node114.aggrx : jps ============ 2256 Kafka 2480 Jps 29930 QuorumPeerMain 命令執行成功 ============= node115.aggrx : jps ============ 788 QuorumPeerMain 5588 Jps 5372 Kafka 命令執行成功 ============= node116.aggrx : jps ============ 19335 QuorumPeerMain 23865 Kafka 24079 Jps 命令執行成功 ============= node117.aggrx : jps ============ 27173 Kafka 27387 Jps 22639 QuorumPeerMain 命令執行成功 ============= node118.aggrx : jps ============ 20017 QuorumPeerMain 25556 Jps 25317 Kafka 30056 Bootstrap 命令執行成功 [root@node118 ~]#
[root@node118 ~]# xcall.sh lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============= node114.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 2256 root mem REG 253,0 36547 100826940 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 2256 root 13r REG 253,0 36547 100826940 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令執行成功 ============= node115.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 5372 root mem REG 253,0 36547 67503111 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 5372 root 13r REG 253,0 36547 67503111 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令執行成功 ============= node116.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 23865 root mem REG 253,0 36547 1046 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 23865 root 13r REG 253,0 36547 1046 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令執行成功 ============= node117.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 27173 root mem REG 253,0 36547 83946551 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 27173 root 13r REG 253,0 36547 83946551 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令執行成功 ============= node118.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 25317 root mem REG 253,0 36547 59044906 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 25317 root 13r REG 253,0 36547 59044906 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令執行成功 [root@node118 ~]#
5>.啓動cruisecontrol服務
[root@yinzhengjie cruise-control]# ll total 68 -rw-r--r-- 1 root root 8257 Dec 4 10:45 build.gradle drwxr-xr-x 5 root root 61 Dec 4 10:47 buildSrc drwxr-xr-x 2 root root 50 Dec 4 10:45 checkstyle drwxr-xr-x 2 root root 80 Dec 4 11:44 config -rw-r--r-- 1 root root 1079 Dec 4 10:45 CONTRIBUTING.md drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-core drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-metrics-reporter drwxr-xr-x 3 root root 19 Dec 4 10:45 docs drwxr-xr-x 3 root root 47 Dec 4 10:45 gradle -rw-r--r-- 1 root root 86 Dec 4 10:45 gradle.properties -rwxr-xr-x 1 root root 5299 Dec 4 10:45 gradlew -rw-r--r-- 1 root root 2260 Dec 4 10:45 gradlew.bat -rwxr-xr-x 1 root root 5129 Dec 4 10:45 kafka-cruise-control-start.sh -rw-r--r-- 1 root root 1332 Dec 4 10:45 LICENSE -rw-r--r-- 1 root root 304 Dec 4 10:45 NOTICE -rw-r--r-- 1 root root 11476 Dec 4 10:45 README.md -rw-r--r-- 1 root root 153 Dec 4 10:45 semantic-build-versioning.gradle -rw-r--r-- 1 root root 911 Dec 4 10:45 settings.gradle [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ./gradlew jar copyDependantLibs To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.8/userguide/gradle_daemon.html. Daemon will be stopped at the end of the build stopping after processing Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.54/bcpkix-jdk15on-1.54.pom Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcprov-jdk15on/1.54/bcprov-jdk15on-1.54.pom Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.54/bcpkix-jdk15on-1.54.jar Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcprov-jdk15on/1.54/bcprov-jdk15on-1.54.jar Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 1m 59s 11 actionable tasks: 4 executed, 7 up-to-date [root@yinzhengjie cruise-control]#
[root@yinzhengjie cruise-control]# ./kafka-cruise-control-start.sh config/cruisecontrol.properties [2018-12-04 12:27:10,878] INFO KafkaCruiseControlConfig values: anomaly.detection.allow.capacity.estimation = true anomaly.detection.goals = [com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal] anomaly.detection.interval.ms = 10000 anomaly.notifier.class = class com.linkedin.kafka.cruisecontrol.detector.notifier.SelfHealingNotifier bootstrap.servers = [10.1.2.118:9092, 10.1.2.117:9092, 10.1.2.116:9092, 10.1.2.115:9092, 10.1.2.114:9092] broker.capacity.config.resolver.class = class com.linkedin.kafka.cruisecontrol.config.BrokerCapacityConfigFileResolver broker.metric.sample.aggregator.completeness.cache.size = 5 broker.metrics.window.ms = 300000 client.id = kafka-cruise-control completed.user.task.retention.time.ms = 21600000 connections.max.idle.ms = 540000 cpu.balance.threshold = 1.1 cpu.capacity.threshold = 0.8 cpu.low.utilization.threshold = 0.0 default.goals = [com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal] disk.balance.threshold = 1.1 disk.capacity.threshold = 0.8 disk.low.utilization.threshold = 0.0 execution.progress.check.interval.ms = 10000 failed.brokers.zk.path = /CruiseControlBrokerList follower.network.inbound.weight.for.cpu.util = 0.3 goals = [com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.kafkaassigner.KafkaAssignerDiskUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.kafkaassigner.KafkaAssignerEvenRackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.PreferredLeaderElectionGoal] hard.goals = [com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal] leader.network.inbound.weight.for.cpu.util = 0.6 leader.network.outbound.weight.for.cpu.util = 0.1 linear.regression.model.cpu.util.bucket.size = 5 linear.regression.model.min.num.cpu.util.buckets = 5 linear.regression.model.required.samples.per.bucket = 100 max.active.user.tasks = 5 max.allowed.extrapolations.per.broker = 5 max.allowed.extrapolations.per.partition = 5 max.cached.completed.user.tasks = 100 max.replicas.per.broker = 10000 metadata.max.age.ms = 55000 metric.anomaly.finder.class = [com.linkedin.kafka.cruisecontrol.detector.KafkaMetricAnomalyFinder] metric.sampler.class = class com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler metric.sampler.partition.assignor.class = class com.linkedin.kafka.cruisecontrol.monitor.sampling.DefaultMetricSamplerPartitionAssignor metric.sampling.interval.ms = 120000 min.samples.per.broker.metrics.window = 1 min.samples.per.partition.metrics.window = 1 min.valid.partition.ratio = 0.995 network.inbound.balance.threshold = 1.1 network.inbound.capacity.threshold = 0.8 network.inbound.low.utilization.threshold = 0.0 network.outbound.balance.threshold = 1.1 network.outbound.capacity.threshold = 0.8 network.outbound.low.utilization.threshold = 0.0 num.broker.metrics.windows = 20 num.cached.recent.anomaly.states = 10 num.concurrent.leader.movements = 1000 num.concurrent.partition.movements.per.broker = 10 num.metric.fetchers = 1 num.partition.metrics.windows = 1 num.proposal.precompute.threads = 1 partition.metric.sample.aggregator.completeness.cache.size = 5 partition.metrics.window.ms = 300000 proposal.expiration.ms = 60000 receive.buffer.bytes = 32768 reconnect.backoff.ms = 50 replica.count.balance.threshold = 1.1 request.timeout.ms = 30000 sample.store.class = class com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS topics.excluded.from.partition.movement = use.linear.regression.model = false webserver.accesslog.enabled = true webserver.accesslog.path = access.log webserver.accesslog.retention.days = 14 webserver.api.urlprefix = /kafkacruisecontrol/* webserver.http.address = 0.0.0.0 webserver.http.cors.allowmethods = OPTIONS,GET,POST webserver.http.cors.enabled = false webserver.http.cors.exposeheaders = User-Task-ID webserver.http.cors.origin = http://localhost:7070/ webserver.http.port = 7070 webserver.request.maxBlockTimeMs = 10000 webserver.session.maxExpiryTimeMs = 60000 webserver.session.path = / webserver.ui.diskpath = ./cruise-control-ui/dist/ webserver.ui.urlprefix = /* zookeeper.connect = 10.1.2.114:2181,10.1.2.115:2181,10.1.2.116:2181,10.1.2.117:2181,10.1.2.118:2181/kafka_yinzhengjie_cluster1/ (com.linkedin.kafka.cruisecontrol.config.KafkaCruiseControlConfig) [2018-12-04 12:27:11,102] INFO ConsumerConfig values: auto.commit.interval.ms = 5000 auto.offset.reset = latest bootstrap.servers = [10.1.2.118:9092, 10.1.2.117:9092, 10.1.2.116:9092, 10.1.2.115:9092, 10.1.2.114:9092] check.crcs = true client.id = CruiseControlMetricsReporterSampler-9001457965431329813-consumer--1607149276 connections.max.idle.ms = 540000 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = CruiseControlMetricsReporterSampler-9001457965431329813 heartbeat.interval.ms = 3000 interceptor.classes = null internal.leave.group.on.close = true isolation.level = read_uncommitted key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 2147483647 max.poll.records = 2147483647 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 305000 retry.backoff.ms = 100 sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 session.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class com.linkedin.kafka.cruisecontrol.metricsreporter.metric.MetricSerde (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'anomaly.detection.interval.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'network.outbound.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'disk.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'metric.anomaly.percentile.lower.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'replica.count.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'min.monitored.partition.percentage' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'cpu.low.utilization.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'num.partition.metrics.windows' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'execution.progress.check.interval.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'capacity.config.file' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'webserver.http.cors.enabled' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'partition.metric.sample.store.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'network.outbound.capacity.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'cpu.capacity.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'webserver.ui.urlprefix' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'proposal.expiration.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'goals' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'webserver.api.urlprefix' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'metric.sampler.partition.assignor.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'webserver.http.cors.origin' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'metric.sampler.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'hard.goals' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'self.healing.enabled' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'broker.metrics.window.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'cpu.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'metric.anomaly.finder.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'max.cached.completed.user.tasks' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'num.sample.loading.threads' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'zookeeper.connect' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'disk.capacity.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'webserver.http.port' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'partition.metrics.window.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'disk.low.utilization.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'network.inbound.low.utilization.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'network.inbound.capacity.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'webserver.session.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'failed.brokers.zk.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'webserver.accesslog.enabled' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'num.concurrent.partition.movements.per.broker' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'webserver.ui.diskpath' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'completed.user.task.retention.time.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'metric.reporter.topic.pattern' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'min.samples.per.broker.metrics.window' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'num.broker.metrics.windows' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'webserver.accesslog.retention.days' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'metric.sampling.interval.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'min.samples.per.partition.metrics.window' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'anomaly.notifier.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'webserver.http.cors.exposeheaders' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'anomaly.detection.goals' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'network.outbound.low.utilization.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'num.proposal.precompute.threads' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'webserver.http.address' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'metric.anomaly.analyzer.metrics' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'network.inbound.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'webserver.session.maxExpiryTimeMs' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'webserver.accesslog.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'webserver.request.maxBlockTimeMs' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'metric.anomaly.percentile.upper.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'sample.store.topic.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'max.replicas.per.broker' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'broker.metric.sample.store.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'webserver.http.cors.allowmethods' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'num.metric.fetchers' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'max.active.user.tasks' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'sample.store.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'default.goals' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,159] INFO Kafka version : 0.11.0.2 (org.apache.kafka.common.utils.AppInfoParser) [2018-12-04 12:27:11,159] INFO Kafka commitId : 73be1e1168f91ee2 (org.apache.kafka.common.utils.AppInfoParser) Exception in thread "main" java.lang.IllegalStateException: Cruise Control cannot find sampling topic matches __CruiseControlMetrics in the target cluster. at com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler.configure(CruiseControlMetricsReporterSampler.java:203) at com.linkedin.kafka.cruisecontrol.config.KafkaCruiseControlConfig.getConfiguredInstance(KafkaCruiseControlConfig.java:1065) at com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcherManager.<init>(MetricFetcherManager.java:129) at com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcherManager.<init>(MetricFetcherManager.java:69) at com.linkedin.kafka.cruisecontrol.monitor.task.LoadMonitorTaskRunner.<init>(LoadMonitorTaskRunner.java:72) at com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.<init>(LoadMonitor.java:152) at com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.<init>(LoadMonitor.java:110) at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.<init>(KafkaCruiseControl.java:89) at com.linkedin.kafka.cruisecontrol.async.AsyncKafkaCruiseControl.<init>(AsyncKafkaCruiseControl.java:64) at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlMain.main(KafkaCruiseControlMain.java:71) [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]#
啓動的時候報錯了，這就有點小尷尬了，問題還在排除中......
四.Kafka Manager開源監控
我接觸的第一款kafka監控工具，配置簡單，大師不支持告警，詳情請參考：http://www.javashuo.com/article/p-hdbfcibx-w.html。
五.Kafka Eagle
我接觸的第二款Kafka監控工具，配置也簡單，想必Kafka Manager更優秀一點，支持告警功能，詳情請參考：http://www.javashuo.com/article/p-vaiifrqn-hr.html。
六.JmxTool
嚴格的來講，JmxTools不是一個框架，而是kafka社區默認提供的工具，用於實時查看JMX監控指標。若是用戶一時找不到合適的工具來監測JMX指標，能夠考慮使用「臨時救急」。打開一個終端並進入Kafka安裝目錄bin下，輸入命令「kafka-run-class.sh kafka.tools.JmxTool」即可以獲得JmxTool工具的幫助信息：
[root@yinzhengjie cruise-control]# kafka-run-class.sh kafka.tools.JmxTool Dump JMX values to standard output. Option Description ------ ----------- --attributes <String: name> #指定 CSV格式的屬性，若是不指定該參數則默認讀取全部屬性。查詢屬性的白名單。這個是逗號分隔的列表。若是沒有屬性指定全部對象將被詢問。 --date-format <String: format> # 設置時間格式，好比設置爲 YYYY-MM-ss,則輸出時間戳即爲2018-12-01這樣的格式 --help #查詢幫助信息 --jmx-url <String: service-url> #指定要連接的JMX接口，默認格式是service.jmx.rmi:///jndi/rmi://:<JMX端口>/jmxrmi --object-name <String: name> #指定JMX的Mbean名稱 --reporting-interval <Integer: ms> #設置實時監測時間間隔，默認每2秒打印一次JMX指標值。 [root@yinzhengjie cruise-control]#
假設咱們要實時監控JMX MBean的消息入站速率。咱們想要每一個3秒輸出一次該MBean過去15分鐘的平均值，則能夠執行下面的命令：
[root@yinzhengjie cruise-control]# kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec --jmx-url service:jmx:rmi:///jndi/rmi://10.1.3.116:12345/jmxrmi --date-format "YYYY-MM-dd HH:mm:ss" --attributes FifteenMinuteRate --reporting-interval 3000 Trying to connect to JMX url: service:jmx:rmi:///jndi/rmi://10.1.3.116:12345/jmxrmi. "time","kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec:FifteenMinuteRate" 2018-12-04 13:21:25,1.0616180583446175E7 2018-12-04 13:21:28,1.0610205773377854E7 2018-12-04 13:21:31,1.0610205773377854E7 2018-12-04 13:21:34,1.0605810471779421E7 2018-12-04 13:21:37,1.0599882204038633E7 2018-12-04 13:21:40,1.0599882204038633E7 2018-12-04 13:21:43,1.059681434484897E7 2018-12-04 13:21:46,1.059681434484897E7 2018-12-04 13:21:49,1.0591635691401184E7 2018-12-04 13:21:52,1.058678141082473E7 2018-12-04 13:21:55,1.058678141082473E7 2018-12-04 13:21:58,1.0581291456978852E7 2018-12-04 13:22:01,1.0581291456978852E7
相關文章
- 1. 《Apache Kafka實戰》讀書筆記-調優Kafka集羣
- 2. 《apache kafka 實戰》讀書筆記
- 3. 《Apache Kafka 實戰》讀書筆記-認識Apache Kafka
- 4. 《Apache kafka實戰》讀書筆記-管理Kafka集羣安全之ACL篇
- 5. 《Apache kafka實戰》-讀書筆記2 kafka新版本功能與集羣規劃
- 6. 《Apache kafka實戰》讀書筆記-1 kafka簡介
- 7. kafka監控工具kafka-manager
- 8. 【Kafka】- 監控工具kafka-manager
- 9. 《Apache Kafka實戰》閱讀筆記
- 10. Kafka學習筆記--《Apache Kafka實戰》
- 更多相關文章...
- • Swarm 集羣管理 - Docker教程
- • jQuery Mobile 工具欄 - jQuery Mobile 教程
- • Tomcat學習筆記（史上最全tomcat學習筆記）
- • PHP開發工具
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. vs2019運行opencv圖片顯示代碼時，窗口亂碼
- 2. app自動化 - 元素定位不到？別慌，看完你就能解決
- 3. 在Win8下用cisco ××× Client連接時報Reason 422錯誤的解決方法
- 4. eclipse快速補全代碼
- 5. Eclipse中Java/Html/Css/Jsp/JavaScript等代碼的格式化
- 6. idea+spring boot +mabitys(wanglezapin)+mysql (1)
- 7. 勒索病毒發生變種 新文件名將帶有「.UIWIX」後綴
- 8. 【原創】Python 源文件編碼解讀
- 9. iOS9企業部署分發問題深入瞭解與解決
- 10. 安裝pytorch報錯CondaHTTPError：******
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. 《Apache Kafka實戰》讀書筆記-調優Kafka集羣
- 2. 《apache kafka 實戰》讀書筆記
- 3. 《Apache Kafka 實戰》讀書筆記-認識Apache Kafka
- 4. 《Apache kafka實戰》讀書筆記-管理Kafka集羣安全之ACL篇
- 5. 《Apache kafka實戰》-讀書筆記2 kafka新版本功能與集羣規劃
- 6. 《Apache kafka實戰》讀書筆記-1 kafka簡介
- 7. kafka監控工具kafka-manager
- 8. 【Kafka】- 監控工具kafka-manager
- 9. 《Apache Kafka實戰》閱讀筆記
- 10. Kafka學習筆記--《Apache Kafka實戰》