OGG TO HDFS,及Kerberose驗證方法

1、準備工做

     一、源端ORACLE RDBMS 12c,及Oracle GoldenGate 12c for Oracle。安裝須要有圖形界面，須要準備好VNC或其餘。

     二、目的端Oracle GoldenGate 12 for Big Data 12 ,HADOOP 2.6

2、安裝步驟。

     一、源端ORACLE數據庫安裝及GoldenGate安裝都要用到./runInstaller。ORACLE安裝忽略， 本文安裝在/data1/oracle/。OGG FOR ORACLE安裝在/data/oracle/oggora。

     二、源端 ORACLE須要作的配置。

$sqlplus / as sysdba SQL>SELECT log_mode FROM v$database;

看看是不是歸檔模式，若不是要作以下操做。

SQL> alter database mount; Database altered. SQL> alter database archivelog; Database altered. SQL> alter database open; SQL> alter database add supplemental log data; Database altered. SQL> alter database force logging; Database altered. SQL>SELECT log_mode FROM v$database; LOG_MODE ------------ ARCHIVELOG

開啓ORACLE中相關OGG的參數

SQL> alter system set enable_goldengate_replication=true sid='*' scope=both; System altered.

建立ogg表空間，用戶，並賦予DBA權值，否則，後面可能沒法執行

SQL> CREATE TABLESPACE OGG DATAFILE '/data1/oracle/app/oracle/oradata/orcl/ogg.dbf' size 10M autoextend on; SQL> create user ogg identified by ogg default tablespace ogg temporary tablespace temp; SQL> EXEC DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE('OGGUSER');    SQL> grant connect, dba to ogg; Grant succeeded.

三、源端GOLDENGATE要作配置

     安裝後OGG FOR ORACLE後，默認已經create subdirs了，而且已經配置啓動mgr。

    運行    oggora$./ggsci進行入ogg命令行

GGSCI (TEST-BDD-057) 1>edit params mgr PORT 7809 userid ogg,password ogg purgeoldextracts /data1/oracle/oggora/*,usecheckpoints accessrule,prog server,allow

重啓MGR

GGSCI (TEST-BDD-057) 3>  stop mgr Manager process is required by other GGS processes. Are you sure you want to stop it (y/n)?y Sending STOP request to MANAGER ... Request processed. Manager stopped. GGSCI (TEST-BDD-057) 4> start mgr                                                        Manager started. GGSCI (TEST-BDD-057) 5> info mgr detail  Manager is running (IP port host01.example.com.7809, Process ID 52948). GGSCI (TEST-BDD-057) 6>

四、須要配置目標端MGR，才能回過來進行源端的EXTRACT的下一步操做。所以，開始安裝目標端的OGG FOR BIGDATA。

$mkdir ogg $tar xvpf ../ggs_Adapters_Linux_x64.tar $./ggsci GGSCI (NM-ITC-NF8460M3-303-016) 1> create subdirs GGSCI (NM-ITC-NF8460M3-303-016) 2> edit params mgr PORT 17809 DYNAMICPORTLIST 17810-17820 AUTORESTART REPLICAT *, RETRIES 5, WAITMINUTES 3, RESETMINUTES 60 PURGEOLDEXTRACTS ./dirdat/*, usecheckpoints, minkeepdays 1 LAGREPORTHOURS 1 LAGINFOMINUTES 30 GGSCI (NM-ITC-NF8460M3-303-016) 3> start manager Manager started.

 五、回到源端進行初始化EXTRACT，並在目標端生成文件。

$vi dirprm/ini_ext.prm SOURCEISTABLE userid ogg,password ogg RMTHOST 10.142.78.36, MGRPORT 17809 RMTFILE /usr/bdusr01/ogg/dirdat/initld,megabytes 2,purge table ggtest.*; -bash-4.1$  ./extract paramfile dirprm/ini_ext.prm reportfile dirrpt/ini_ext.rpt

六、切到目標端，查看是否生成文件

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$ ls dirdat initld

七、在源端配置extract。

$./ggsci Oracle GoldenGate Command Interpreter for Oracle Version 12.2.0.1.1 OGGCORE_12.2.0.1.0_PLATFORMS_151211.1401_FBO Linux, x64, 64bit (optimized), Oracle 12c on Dec 12 2015 02:56:48 Operating system character set identified as UTF-8. Copyright (C) 1995, 2015, Oracle and/or its affiliates. All rights reserved. GGSCI (TEST-BDD-057) 1> edit params ggext extract ggext userid ogg,password ogg --RMTHOSTOPTIONS RMTHOST sandbox, MGRPORT 7839 RMTFILE  /usr/bdusr01/ogg/dirdat/or, MEGABYTES 2, PURGE DDL include objname ogg.* TABLE ogg.*;   GGSCI (TEST-BDD-057) 2> dblogin userid ogg,password ogg Successfully logged into database. GGSCI (TEST-BDD-057) 3> register extract GGEXT database 2016-10-20 15:37:21  INFO    OGG-02003  Extract GGEXT successfully registered with database at SCN 17151616. GGSCI (TEST-BDD-057) 4> add extract ggext, INTEGRATED TRANLOG, BEGIN NOW EXTRACT (Integrated) added.

八、在目標端，編輯hdfs.props。

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$ vi dirprm/hdfs.props gg.handlerlist=hdfs gg.handler.hdfs.type=hdfs gg.handler.hdfs.includeTokens=true gg.handler.hdfs.maxFileSize=256m gg.handler.hdfs.rootFilePath=/data/hjpt/test/jl gg.handler.hdfs.fileRollInterval=0 gg.handler.hdfs.inactivityRollInterval=0 gg.handler.hdfs.fileSuffix=.txt gg.handler.hdfs.partitionByTable=true gg.handler.hdfs.rollOnMetadataChange=true #gg.handler.hdfs.authType=none gg.handler.hdfs.authType=kerberos gg.handler.hdfs.kerberosKeytabFile=/usr/bdusr01/bdusr01.keytab gg.handler.hdfs.kerberosPrincipal=krbtgt/HADOOP.CHINATELECOM.CN@HADOOP.CHINATELECOM.CN gg.handler.hdfs.format=delimitedtext gg.handler.hdfs.format.includeColumnNames=false gg.handler.hdfs.mode=tx gg.handler.hdfs.format.includeOpTimestamp=true gg.handler.hdfs.format.includeCurrentTimestamp=true gg.handler.hdfs.format.insertOpKey=I gg.handler.hdfs.format.updateOpKey=U gg.handler.hdfs.format.deleteOpKey=D gg.handler.hdfs.format.truncateOpKey=T gg.handler.hdfs.format.encoding=UTF-8 gg.handler.hdfs.format.fieldDelimiter=CDATA[\u0001] #gg.handler.hdfs.format.fieldDelimiter='|' gg.handler.hdfs.format.lineDelimiter=CDATA[\n] gg.handler.hdfs.format.includeTableName=true gg.handler.hdfs.format.keyValueDelimiter=CDATA[=] #gg.handler.hdfs.format.kevValuePairDelimiter=CDATA[,] gg.handler.hdfs.format.pkUpdateHandling=update gg.handler.hdfs.format.nullValueRepresentation= gg.handler.hdfs.format.missingValueRepresentation=CDATA[] gg.handler.hdfs.format.includePosition=false goldengate.userexit.timestamp=utc goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec gg.classpath=/etc/hadoop/conf.empty:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop/lib/* javawriter.bootoptions=-Xmx8200m –Xms8192m -Djava.class.path=ggjava/ggjava.jar

九、在目標端配置JAVA_HOME,LD_LIBRARY_PATH，必須要JAVA 1.7，否則後面運行可能出錯。

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64 export LD_LIBRARY_PATH=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/amd64/libjsig.so:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/amd64/server/libjvm.so

十、在目標端初始化hdfs

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$./replicat paramfile dirprm/irhdfs.prm reportfile dirrpt/ini_rhdfs.rpt [bdusr01@NM-ITC-NF8460M3-303-016 ogg]$ hadoop dfs -ls /data/hjpt/test/jl Found 2 items ...

十、在源端啓動extract

GGSCI (TEST-BDD-057) 6>start extract ggext Sending START request to MANAGER ... EXTRACT GGEXT starting

十一、在目標端配置REPLICAT。

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$ vi dirprm/rhdfs.prm REPLICAT rhdfs setenv HADOOP_COMMON_LIB_NATIVE_DIR=/usr/lib/hadoop/lib/native DDL include all TARGETDB LIBFILE libggjava.so SET property=dirprm/hdfs.props REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 10000 MAP ogg.*, TARGET bdtest.*; [bdusr01@NM-ITC-NF8460M3-303-016 ogg]$./ggsci GGSCI (NM-ITC-NF8460M3-303-016) 1> add replicat rhdfs, exttrail dirdat/or REPLICAT added. GGSCI (NM-ITC-NF8460M3-303-016) 2> start replicat rhdfs Sending START request to MANAGER ... REPLICAT RHDFS starting GGSCI (NM-ITC-NF8460M3-303-016) 3>

十二、目標端HIVE建表

hive> CREATE EXTERNAL TABLE BDTEST.TEST_TAB_1  (tran_flag string, tab_name string, tran_time_utc timestamp, tran_time_loc string,something string, something1 string,     > PK_ID INT, RND_STR VARCHAR(10),USE_DATE string,RND_STR_1 string, ACC_DATE string)     > stored as textfile location '/data/hjpt/test/jl'; OK Time taken: 0.327 seconds hive> select * from BDTEST.TEST_TAB_1; ...

1三、在源端插入數據，再到目標端查看變化，便可。

 

參考文檔：

http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/goldengate/12c/OGG12c_Integrated_Replicat/index.html

http://www.tuicool.com/articles/IzqAZnI