Maxwell入門

1 下載tar包

• Download binary distro: https://github.com/zendesk/maxwell/releases/download/v1.19.4/maxwell-1.19.4.tar.gz

2 解壓

　　tar -zvxf maxwell-1.19.4.tar.gz

3 修改配置

　　cd maxwell-1.19.4

　　cp config.properties.example config.properties

根據本身mysql以及kafka配置　　

# tl;dr config
log_level=info

producer=kafka
kafka.bootstrap.servers=hdp1:9092,hdp2:9092,hdp3:9092,hdp4:9092

# mysql login info
host=127.0.0.1
user=maxwell
password=pwd123456

 

4 修改mysql的binlog配置  根據本身機器配置

vi /etc/my.cnf

$ vi my.cnf

[mysqld]

server_id=1

log-bin=master

binlog_format=row

 

5 建立同步用戶

mysql> GRANT ALL on maxwell.* to 'maxwell'@'%' identified by 'XXXXXX';
mysql> GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'%';

mysql>flush privileges;

6 重啓mysql查看bin log是否修改爲功

service mysqld restart

show variables like '%binlog%'

 

7 測試一下maxwell 

[root@hdp1 /mnt/software/maxwell-1.19.4]#bin/maxwell --user='maxwell' --password='pwd123456' --host='127.0.0.1' --producer=stdout 

#插入數據 根據本身mysql狀況測試

mysql> INSERT INTO `cmcc_offset` (`topics`, `partId`, `offsets`, `groupId`, `create_time`, `updater`, `update_time`, `is_delete`)

 成功收到數據

 

8 單獨的kafka producer配置

單獨copy指定版本kafka的client的jar包到lib目錄下， 我本身的kafka版本是1.1.0的

 

 

 

啓動kafka的producer命令，須要單獨制定kafka的version

bin/maxwell --user='maxwell' --password='pwd123456' --host='127.0.0.1'  --producer=kafka --kafka_version=1.1.0 --kafka.bootstrap.servers=hdp1:9092,hdp2:9092,hdp3:9092,hdp4:9092 --kafka_topic=maxwell

 

遇到問題  Kafka的K大寫了，改爲小寫就OK了

java.lang.RuntimeException: Unknown producer type: Kafka
at com.zendesk.maxwell.MaxwellContext.getProducer(MaxwellContext.java:372) ~[maxwell-1.19.4.jar:1.19.4]
at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:186) ~[maxwell-1.19.4.jar:1.19.4]
at com.zendesk.maxwell.Maxwell.start(Maxwell.java:156) ~[maxwell-1.19.4.jar:1.19.4]
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:246) ~[maxwell-1.19.4.jar:1.19.4]
01:08:02,127 INFO TaskManager - Stopped all tasks
java.lang.RuntimeException: Unknown producer type: Kafka
at com.zendesk.maxwell.MaxwellContext.getProducer(MaxwellContext.java:372)
at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:186)
at com.zendesk.maxwell.Maxwell.start(Maxwell.java:156)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:246)
[root@hdp1 /mnt/software/maxwell-1.19.4]#xcall.sh jps

 

其餘細節詳情請看官網http://maxwells-daemon.io/quickstart/