ORACLE GOLDEN GATE oracle同步數據至kafka

一.服務器信息

ip		軟件版本	ogg版本	軟件包	操做系統版本	OGG安裝路徑
10.1.50.52	源	oracle11.2.0.4	12.2.0.1.1	V100692-01.zip	centos6.9	/data/ogg
10.0.71.102	目標	kafka0.10	12.2.0.1.160823	123010_ggs_Adapters_Linux_x64.zip	centos7.3	/data/ogg

 

 

 

二.源端安裝OGG

1.解壓V100692-01.zip

unzip V100692-01.zip

這個時候會看到fbo_ggs_Linux_x64_shiphome文件夾

2.咱們採用靜默安裝的方式進行安裝（注意要使用oracle用戶來進行安裝，oggcore.rsp配置須要根據實際須要進行修改，也可以使用圖形界面進行安裝）

/usr/local/software/fbo_ggs_Linux_x64_shiphome/Disk1/runInstaller -silent -responseFile /usr/local/software/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp

三.源端OGG配置

1.開啓主庫歸檔日誌、補充日誌及force logging

alter database add supplemental log data;

alter database force logging;

alter system set enable_goldengate_replication=true scope=both;

2.建立ogg的管理用戶

create user ogg identified by password account unlock;

grant connect,resource to ogg;

grant select any dictionary to ogg;

grant select any table to ogg;

grant execute on utl_file to ogg;

grant restricted session to ogg;

grant create table,create sequence to ogg;

alter table DCSDBA.YH_ITF_INVENTORY_TRANSACTION add supplemental log data (all) columns;--若是kafka須要接收到完整的修改信息就須要開啓補全日誌

3.登陸ogg並建立目錄

/data/ogg/ggsci

create subdirs

4.配置源端manager進程

 

edit params mgr

port 1357
autostart er *
autorestart er *

5.啓動 manager並確認狀態

start mgr

info all 查看MANAGER進程爲running狀態

6.配置加密

ENCRYPT PASSWORD password ENCRYPTKEY DEFAULT

7.配置數據抽取進程

 

①加一個extract，名字隨便取

 

add extract hnwmsprd,tranlog,begin now

 

②爲這個ext指定他把遠程文件放到遠程機的哪一個目錄

 

ADD RMTTRAIL /data/ogg/dirdat/tt, EXTRACT hnwmsprd

 

③編輯配置文件

edit params hnwmsprd

 

extract hnwmsprd

 

setenv (NLS_LANG="AMERICAN_AMERICA.UTF8")

setenv (ORACLE_SID="hnwmsprd") 

userid ogg,password AACAAAAAAAAAAAKALILFJIXIECMGKFAHRBJHGDGBNJLBOAUC,ENCRYPTKEY DEFAULT

rmthost 10.0.71.102, mgrport 1357

rmttrail /data/ogg/dirdat/tt

discardfile /data/ogg/dirrpt/trail.dsc,append,megabytes 100

gettruncates

GETUPDATEBEFORES
NOCOMPRESSDELETES
NOCOMPRESSUPDATES

TRANLOGOPTIONS MINEFROMACTIVEDG

table DCSDBA.YH_ITF_INVENTORY_TRANSACTION;

 

④啓動進程

start hnwmsprd

 

四.OGG目標端搭建

目標端搭建只需解壓軟件便可

 

unzip 123010_ggs_Adapters_Linux_x64.zip

 

解壓出來ggs_Adapters_Linux_x64.tar文件

將文件移動到/data/ogg目錄下解壓

mv ggs_Adapters_Linux_x64.tar /data/ogg
tar -xvf ggs_Adapters_Linux_x64.tar

五.OGG目標端配置

1.登陸ogg並建立目錄

/data/ogg/ggsci

create subdirs

2.配置源端manager進程

edit params mgr

port 1357
autostart er *
autorestart er *

3.啓動manager並確認狀態

start mgr

info all 查看MANAGER進程爲running狀態

4.添加複製進程

add replicat rkafka,exttrail dirdat/tt,begin now

edit params rkafka

 

 

REPLICAT rkafka

TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 10000

MAP DCSDBA.YH_ITF_INVENTORY_TRANSACTION, TARGET DCSDBA.YH_ITF_INVENTORY_TRANSACTION;

5.修改kafka.props（注意：kafka.props要放在/data/ogg/dirprm路徑下）

gg.handlerlist = kafkahandler                                                                                                                           

gg.handler.kafkahandler.type = kafka                                                                                                             

gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties                     

gg.handler.kafkahandler.TopicName =hnwmsprd                                                                                          

gg.handler.kafkahandler.format =json                                                                                                           

gg.handler.kafkahandler.SchemaTopicName=ogg                                                                              

gg.handler.kafkahandler.BlockingSend =true                                                                                                

gg.handler.kafkahandler.includeTokens=false                                                                                               

gg.handler.kafkahandler.mode =tx                                                                                                                 

goldengate.userexit.timestamp=utc                                                                                                                  

goldengate.userexit.writers=javawriter                                                                                                            

javawriter.stats.display=TRUE                                                                                                                          

javawriter.stats.full=TRUE                                                                                                                                 

gg.log=log4j                                                                                                                                                          

gg.log.level=INFO                                                                                                                                               

gg.report.time=30sec                                                                                                                                          

gg.classpath=dirprm/:/usr/local/kafka/libs/*:                                                            

javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar 

以上配置文件根據實際配置進行修改

6.編輯custom_kafka_producer.properties文件（注意：custom_kafka_producer.properties要放在/data/ogg/dirprm路徑下）

#bootstrap.servers=host:port
bootstrap.servers=10.0.71.102:9092
acks=1
#compression.type=gzip
reconnect.backoff.ms=1000

value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
# 100KB per partition
batch.size=102400
linger.ms=10000

7.啓動複製進程

start rkafka

 

六.驗證

目標端執行

./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hnwmsprd

源端更新DCSDBA.YH_ITF_INVENTORY_TRANSACTION表。

 

 

 若是是在adg做爲源端須要在源抽取進程上添加參數

TRANLOGOPTIONS MINEFROMACTIVEDG  

並在主庫開啓ddl，能夠參考下面這位大神的配置

http://czmmiao.iteye.com/blog/1609823