使用canal分析binlog(一) 入門

canal介紹

canal是應阿里巴巴存在杭州和美國的雙機房部署，存在跨機房同步的業務需求而提出的。早期，阿里巴巴B2B公司由於存在杭州和美國雙機房部署，存在跨機房同步的業務需求。不過早期的數據庫同步業務，主要是基於trigger的方式獲取增量變動，不過從2010年開始，阿里系公司開始逐步的嘗試基於數據庫的日誌解析，獲取增量變動進行同步，由此衍生出了增量訂閱&消費的業務，今後開啓了一段新紀元。ps. 目前內部使用的同步，已經支持mysql5.x和oracle部分版本的日誌解析。

基於日誌增量訂閱&消費支持的業務：
數據庫鏡像
數據庫實時備份
多級索引 (賣家和買家各自分庫索引)
search build
業務cache刷新
價格變化等重要業務消息

keyword：數據庫同步，增量訂閱&消費。

https://github.com/alibaba/canal/wiki

http://www.bubuko.com/infodetail-1268551.html

教程 http://blog.csdn.net/hackerwin7/article/details/37923607

http://liyonghui160com.iteye.com/blog/2176066

開啓mysql binlog

http://blog.csdn.net/aitangyong/article/details/53114633

大數據量堆疊

http://blog.csdn.net/zhanlanmg/article/details/51213631 

 mysql多個slave同一serverId衝突

http://www.penglixun.com/tech/database/mysql_multi_slave_same_serverid.html

 

照着官方的例子修改canal.properties和example/instance.properties，server端啓動起來並不難

在現有的spring boot項目的pom增長了canal引用後，項目沒法啓動

錯誤信息以下：

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/Users/Administrator/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.5/log4j-slf4j-impl-2.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/Users/Administrator/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Exception in thread "main" java.lang.IllegalArgumentException: LoggerFactory is not a Logback LoggerContext but Logback is on the classpath. Either remove Logback or the competing implementation (class org.apache.logging.slf4j.Log4jLoggerFactory loaded from file:/C:/Users/Administrator/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.5/log4j-slf4j-impl-2.5.jar). If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml Object of class [org.apache.logging.slf4j.Log4jLoggerFactory] must be an instance of class ch.qos.logback.classic.LoggerContext
at org.springframework.util.Assert.isInstanceOf(Assert.java:346)
at org.springframework.boot.logging.logback.LogbackLoggingSystem.getLoggerContext(LogbackLoggingSystem.java:221)
at org.springframework.boot.logging.logback.LogbackLoggingSystem.getLogger(LogbackLoggingSystem.java:213)
at org.springframework.boot.logging.logback.LogbackLoggingSystem.beforeInitialize(LogbackLoggingSystem.java:98)
at org.springframework.boot.logging.LoggingApplicationListener.onApplicationStartedEvent(LoggingApplicationListener.java:215)
at org.springframework.boot.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:197)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:163)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:136)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:119)
at org.springframework.boot.context.event.EventPublishingRunListener.publishEvent(EventPublishingRunListener.java:111)
at org.springframework.boot.context.event.EventPublishingRunListener.started(EventPublishingRunListener.java:60)
at org.springframework.boot.SpringApplicationRunListeners.started(SpringApplicationRunListeners.java:48)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1191)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1180)
at com.paybox.sync.App.main(App.java:32)

 

錯誤產生緣由： 現有項目使用的日誌組件是slf4j+log4j2，而canal使用的日誌組件是slf4j+logback，log4j和logback衝突致使系統發生異常沒法啓動。

查看canal的依賴 http://mvnrepository.com/artifact/com.alibaba.otter/canal.common/1.0.22

 解決方法：

修改pom文件

　　　　<dependency>
            <groupId>com.alibaba.otter</groupId>
            <artifactId>canal.client</artifactId>
            <version>${canal.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-core</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-classic</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.springframework</groupId>
                    <artifactId>spring</artifactId>
                </exclusion>
            </exclusions>
        </dependency>    

 

項目中原來已經配置過log4j2和slf4j，只要刪除canal對logback的依賴便可。

然後又發現項目啓動發生錯誤，由於canal還依賴spring2.5.6，如今項目使用的是spring boot4.2.5，我就直接把canal對spring的依賴給排除了，項目能正常運行，不過這樣會致使什麼還不清楚。