使用GoldenGate完成MySQL到MySQL的同步

時間 2020-02-09

原文原文鏈接

（一）基礎環境配置mysql

	源庫	目標庫
操做系統版本	CentOS Linux release 7.4	CentOS Linux release 7.4
IP地址	192.168.10.11	192.168.10.12
數據庫版本	社區版 5.7.24	社區版 5.7.24
須要同步的數據庫	testdb（全部表）	testdb_repl
GoldenGate版本	OGG-12.2.0.2	OGG-12.2.0.2

說明： sql

1.oracle GoldenGate下載地址：https://edelivery.oracle.com/ 免費註冊一個帳號便可登入，直接搜索「goldengate for non oracle database」，找到想要的版本下載便可。數據庫

2.通過測試，使用OGG-11.2.0.1同步數據庫會報錯:服務器

ERROR OGG-00146 VAM function VAMRead returned unexpected result: error 600 - VAM Client Report <CAUSE OF FAILURE : Sanity Check Failed for events
WHEN FAILED : While reading log event from binary log
WHERE FAILED : MySQLBinLog Reader Module
CONTEXT OF FAILURE : No Information Available!>.oracle

更換爲OGG-12.2.0.2後解決，故本文采用的Ogg版本是12.2.0.2。app

（二）MySQL數據庫初始信息tcp

	源庫	目標庫
MySQL安裝位置（basedir）	/usr/local/mysql	/usr/local/mysql
數據存放位置（datadir）	/mysql/data	/mysql/data
二進制日誌位置（log_bin）	/mysql/binlog	/mysql/binlog

（三）MySQL數據庫開啓二進制日誌ide

要使用ogg抽取數據，必須開啓二進制日誌。在mysql的配置文件/etc/my.cnf中配置二進制日誌,重啓數據庫生效。測試

[mysqld]

# binlog配置
server_id = 1
binlog_format=ROW
log_bin=/mysql/binlog/master-bin
sync_binlog=1 
expire_logs_days=1

（四）源端和目標端安裝配置GoldenGate（源端和目標端都要執行）ui

（4.1）源端和目標端建立安裝ogg，實際上ogg並不須要安裝，解壓就能夠使用

建立ogg安裝路徑

[root@masterdb ~]# mkdir /ogg

將安裝包拷貝到該路徑下解壓：

[root@masterdb ogg]# ls
ggs_Linux_x64_MySQL_64bit.tar
[root@masterdb ogg]# tar -xvf ggs_Linux_x64_MySQL_64bit.tar 
./
./mysql_checklist.sql
...略
./libicuuc.so.56.1
[root@masterdb ogg]# ls
bcpfmt.tpl              emsclnt                        libggnnzitp.so        mysql.txt
bcrypt.txt              extract                        libggparam.so         notices.txt
cachefiledump           freeBSD.txt                    libggperf.so          oggerr
checkprm                ggcmd                          libggrepo.so          prvtclkm.plb
chkpt_mysql_create.sql  ggMessage.dat                  libicudata.so.56      replicat
convchk                 ggparam.dat                    libicudata.so.56.1    retrace
convprm                 ggsci                          libicui18n.so.56      reverse
db2cntl.tpl             ggs_Linux_x64_MySQL_64bit.tar  libicui18n.so.56.1    server
defgen                  help.txt                       libicuuc.so.56        sqlldr.tpl
demo_mysql_create.sql   keygen                         libicuuc.so.56.1      tcperrs
demo_mysql_insert.sql   lib                            libmysqlclient.so.18  ucharset.h
demo_mysql_load.sql     lib12                          libxerces-c-3.1.so    UserExitExamples
demo_mysql_misc.sql     libantlr3c.so                  logdump               usrdecs.h
dirout                  libdb-6.1.so                   mgr                   zlib.txt
dirwww                  libgglog.so                    mysql_checklist.sql

（4.2）使用ggsci建立ogg相關路徑

[root@masterdb ogg]# pwd
/ogg
[root@masterdb ogg]# ./ggsci

Oracle GoldenGate Command Interpreter for MySQL
Version 12.2.0.2.2 OGGCORE_12.2.0.2.0_PLATFORMS_170630.0419
Linux, x64, 64bit (optimized), MySQL Enterprise on Jun 30 2017 06:32:17
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.



GGSCI (masterdb) 1> create subdirs

Creating subdirectories under current directory /ogg

Parameter files                /ogg/dirprm: created
Report files                   /ogg/dirrpt: created
Checkpoint files               /ogg/dirchk: created
Process status files           /ogg/dirpcs: created
SQL script files               /ogg/dirsql: created
Database definitions files     /ogg/dirdef: created
Extract data files             /ogg/dirdat: created
Temporary files                /ogg/dirtmp: created
Credential store files         /ogg/dircrd: created
Masterkey wallet files         /ogg/dirwlt: created
Dump files                     /ogg/dirdmp: created

（五）源端配置GoldenGate（源端執行）

（5.1）建立數據庫用戶ogg，該用戶用來同步數據

mysql> GRANT ALL PRIVILEGES ON `testdb`.* TO 'ogg'@'%'IDENTIFIED BY 'ogg';

（5.2）配置管理進程mgr

GGSCI (masterdb) 1> edit param mgr
port 7809  
dynamicportlist 7840-7939  
purgeoldextracts /ogg/dirdat/*,usecheckpoints, minkeepdays 1 GGSCI (masterdb) 2> start mgr Manager started. GGSCI (masterdb) 3> info all Program Status Group Lag at Chkpt Time Since Chkpt MANAGER RUNNING

(5.3)配置捕獲進程exta

GGSCI (masterdb) 5> edit param exta
extract exta  
sourcedb testdb@192.168.10.11:3306 userid ogg password ogg  
exttrail /ogg/dirdat/ea
discardfile  /ogg/dirrpt/exta.dsc,append  
TranLogOptions AltLogDest /mysql/binlog/master-bin.index 
table testdb.*; 

GGSCI (masterdb) 10> add extract exta,tranlog,begin now  
EXTRACT added.

GGSCI (masterdb) 11> add exttrail /ogg/dirdat/ea,extract exta  
EXTTRAIL added.

(5.4)配置投遞進程dmpa

GGSCI (masterdb) 15> edit param dmpa 
extract dmpa 
passthru  
sourcedb testdb@192.168.10.11:3306 userid ogg password ogg  
rmthost  192.168.10.12,mgrport 7809,compress  
rmttrail  /ogg/dirdat/da  
dynamicresolution  
numfiles 3000  
table testdb.*; 

GGSCI (masterdb) 16> add extract dmpa ,exttrailsource /ogg/dirdat/ea  
EXTRACT added.

GGSCI (masterdb) 17> add rmttrail /ogg/dirdat/da,extract dmpa 
RMTTRAIL added.

(5.5)建立表定義文件，傳送到目標端

（5.5.1）建立表定義文件

GGSCI (masterdb) 23> edit param defgen  
defsfile ./dirdef/testdb.def  
sourcedb testdb@192.168.10.11:3306 userid ogg, password ogg  
table testdb.*;  

GGSCI (masterdb) 26> exit

（5.5.2）生成表定義

[root@masterdb ogg]# ./defgen paramfile ./dirprm/defgen.prm  

***********************************************************************
        Oracle GoldenGate Table Definition Generator for MySQL
      Version 12.2.0.2.2 OGGCORE_12.2.0.2.0_PLATFORMS_170630.0419
 Linux, x64, 64bit (optimized), MySQL Enterprise on Jun 30 2017 07:27:58
 
Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.


                    Starting at 2020-02-08 18:15:40
***********************************************************************

Operating System Version:
Linux
Version #1 SMP Tue Aug 22 21:09:27 UTC 2017, Release 3.10.0-693.el7.x86_64
Node: masterdb
Machine: x86_64
                         soft limit   hard limit
Address Space Size   :    unlimited    unlimited
Heap Size            :    unlimited    unlimited
File Size            :    unlimited    unlimited
CPU Time             :    unlimited    unlimited

Process id: 2510

***********************************************************************
**            Running with the following parameters                  **
***********************************************************************
defsfile ./dirdef/testdb.def
sourcedb testdb@192.168.10.11:3306 userid ogg, password ***
table testdb.*;
Expanding wildcard table specification testdb.*:

Retrieving definition for testdb.test01.
Retrieving definition for testdb.test02.
Retrieving definition for testdb.test03.


Definitions generated for 3 tables in ./dirdef/testdb.def.

（5.5.3）傳送到目標端

[root@masterdb ogg]# scp /ogg/dirdef/testdb.def 192.168.10.12:/ogg/dirdef/
root@192.168.10.12's password: 
testdb.def                            100% 1785     1.9MB/s   00:00

（六）目標端配置GoldenGate（目標端執行）

（6.1）建立數據庫用戶ogg，該用戶用來同步數據

mysql> GRANT ALL PRIVILEGES ON `testdb_repl`.* TO 'ogg'@'%'IDENTIFIED BY 'ogg';

（6.2）配置管理進程mgr

GGSCI (slavedb) 3> edit param mgr
port 7809  
dynamicportlist 7840-7939  
purgeoldextracts /ogg/dirdat/*,usecheckpoints, minkeepdays 1 GGSCI (slavedb) 5> start mgr Manager started. GGSCI (slavedb) 6> info all Program Status Group Lag at Chkpt Time Since Chkpt MANAGER RUNNING

（6.3）配置檢查表

GGSCI (slavedb) 9> dblogin sourcedb testdb_repl@192.168.10.12:3306 userid ogg password ogg 
Successfully logged into database.

GGSCI (slavedb DBLOGIN as ogg) 10> add checkpointtable testdb_repl.checkpoint  

Successfully created checkpoint table testdb_repl.checkpoint.

GGSCI (slavedb DBLOGIN as ogg) 11> edit params ./GLOBALS 
checkpointtable testdb_repl.checkpoint

（6.4）配置應用進程repa

GGSCI (slavedb DBLOGIN as ogg) 13> edit params repa

replicat repa 
targetdb testdb_repl@192.168.10.12:3306 userid ogg password ogg  
handlecollisions  
sourcedefs /ogg/dirdef/testdb.def  
discardfile /ogg/dirrpt/repa.dsc,purge  
map testdb.*, target testdb_repl.*;

GGSCI (slavedb DBLOGIN as ogg) 16> add replicat repa,exttrail /ogg/dirdat/da,checkpointtable testdb_repl.checkpoint
REPLICAT added.

（七）導入初始化數據

將源庫須要同步的表導出，而後導入到目標數據庫

# 源庫導出，並傳到目標數據庫服務器上
[root@masterdb ~]# mysqldump -uroot -p123456 -h192.168.10.11  --single-transaction --flush-logs --master-data=2 testdb  > testdb.sql
mysqldump: [Warning] Using a password on the command line interface can be insecure.

[root@masterdb ~]# ls
anaconda-ks.cfg  testdb.sql  wgetlog-02-05-20-22:27.log  wgetlog-02-05-20-22:28.log  wget.sh

[root@masterdb ~]# scp testdb.sql root@192.168.10.12:/root
root@192.168.10.12's password: 
testdb.sql                                                                                100%   15MB  68.9MB/s   00:00

# 目標數據庫執行導入
[root@slavedb ~]# mysql -uroot -p123456 testdb_repl < testdb.sql 
mysql: [Warning] Using a password on the command line interface can be insecure.

（八）開啓ogg同步進程

(8.1)源端開啓捕獲進程（源端執行）

因爲數據庫數據一直在變化，因此對於導出的testdb.sql文件，咱們須要記錄二進制日誌文件的log_file和log_pos

[root@slavedb ~]# cat testdb.sql |grep "CHANGE MASTER TO MASTER_LOG_FILE"
-- CHANGE MASTER TO MASTER_LOG_FILE='master-bin.000005', MASTER_LOG_POS=154;

而後從導出時的binlog開始抽取

GGSCI (masterdb) 5> alter extract exta,vam,lognum 5,logpos 154
EXTRACT altered.


GGSCI (masterdb) 6> start exta

Sending START request to MANAGER ...
EXTRACT EXTA starting

（8.2）開啓投遞進程（源端執行）

GGSCI (masterdb) 19> start dmpa

Sending START request to MANAGER ...
EXTRACT DMPA starting

（8.3）開啓應用進程（目標端執行）

GGSCI (slavedb DBLOGIN as ogg) 18> start repa

Sending START request to MANAGER ...
REPLICAT REPA starting

（九）確認同步狀態

（9.1）使用info all查看進程是否運行正常

全部進程狀態爲「running」表明正常

源端：

GGSCI (masterdb) 8> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     RUNNING     DMPA        00:00:00      00:00:08    
EXTRACT     RUNNING     EXTA        00:00:00      00:00:04

目標端：

GGSCI (slavedb DBLOGIN as ogg) 23> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           
REPLICAT    RUNNING     REPA        01:14:40      00:00:00

（9.2）源端手動插入數據，查看目標端是否同步

源端執行：

insert into test03 values(1,'a');

目標端查看數據是否同步過來：

mysql> use testdb_repl ;
Database changed

mysql> select * from test03;
+-----+------+
| id3 | name |
+-----+------+
|   1 | a    |
+-----+------+
1 row in set (0.04 sec)

【完】