Oracle數據庫遷移到PostgreSQL/EDB初探

因爲某些非技術方面的緣由，咱們也在搞一些開源數據庫引入，替換商業數據庫，因而瞄上了PostgreSQL。

PostgreSQL自己的技術不在這裏作介紹，雖然國內PostgreSQL沒有Mysql那麼流行，可是搜索一下，仍是可以找到很多PostgreSQL中文的一些技術資料， 其中PG的官方文檔是最詳盡且最有助於學習PG的：

最新的9.4版本的online doc：http://www.postgresql.org/docs/9.4/static/index.html 

山東瀚高科技提供的中文翻譯文檔：http://www.highgo.com.cn/docs/docs90cn/

Oracle遷移到PostgreSQL：

比較流行的遷移工具 ora2pg：  http://ora2pg.darold.net/config.html#testing

大體原理：

該工具使用perl寫的， 主要遷移功能都由ora2pg.conf 這個參數文件控制，該參數文件定義源庫Oracle和目標庫的DSN，經過DBD::Oracle Perl 、 DBD::Pg Perl 的module鏈接到源庫和目標庫， 按照用戶制定的參數，導出源庫Oracle的相關對象結構（也能夠連數據一塊兒導），導入到目標庫PG。 導入能夠是在線的（導出腳本不落地直接導入PG），也能夠先生成導出腳本到本地，手動修改複覈後，再手動使用psql導入PG。 

ora2pg命令行各參數使用也比較清晰明瞭， 支持各類導出導入選項，好比你能夠只導某些schema或者對象，或者排除某些schema或者對象不導， 也能夠什麼都不作，只對源庫作一個遷移PG的分析，出具一份報告等等。

具體使用方法，仍是本身去實際嘗試會比較好。 

使用Ora2pg的方案將Oracle遷移至PG， 遇到的問題多少與源Oracle數據庫有多少與PG不兼容的東西成正比。下面是咱們遇到的問題簡單總結：

應用程序裏面sqlmap.xml 人工REVIEW時的問題發現：

Oracle	PostgreSQL
dual表	沒有dual表，能夠直接select 一、select user、select xxx
時間函數（sysdate)	current_date current_time current_timestamp
trunc	trunc/date trunc（時間截斷函數）
sys_guid()	有相似sys_guid的函數uuid_generate_v4，但須要安裝，create extension "uuid-oosp".
nvl	PG可使用COALESCE
where rownum < ...	select row_number() over() , * from XXXX或者limit

PG導入ora2pg產生的遷移腳本時發現的問題：

tables	DEFAULT Sys_guid(); session# serial# 字段	DEFAULT uuid_generate_v4; 去掉#
partitions	父表在tables中已建立，建立子表時，因爲大小寫問題提示找不到父表。	表名稱區分大小寫，繼承父表時，大小寫改成與父表相同。增長或減小分區要修改觸發器函數。
synonyms	轉換過來的語句相似以下： CREATE VIEW public.tablename AS SELECT * FROM owner.tablename SECURITY DEFINER;	PG中沒有同義詞，自動建立爲視圖，轉換過來的視圖名稱與存在的表名相同，須要修改視圖名稱。 SECURITY DEFINER不能加到建立語句的後面，可經過受權來控制權限。
packages	一、PG中沒有v$session.(oracle原來的packages裏面大多有這個） select sid, serial# into v_sid, v_serial# from v$session 二、有些轉換過來的語句順序不正確，須要重構。 三、packages自動轉換爲function,且function會建立到以原來packages命名的schema下	ora2pg會把Oracle裏面的package header轉換爲同名的schema
procedures		一、dbms_output.put_line('成功插入數據！');-->RAISE NOTICE '%',('成功插入數據！'); 二、 dbms_output.put_line(sqlerrm) --- RAISE NOTICE '%', sqlerrm; 三、表的子查詢必須包圍在圓括弧裏而且必須賦予一個別名 四、start with connect by 遞歸查詢在PG中WITH RECURSIVE
views	一、字符類型問題 二、遞歸查詢沒有轉換成功 三、外鏈接中的「+」號沒有轉換 四、DECODE函數須要重構 五、COALESCE 函數返回值類型不匹配	一、字符類型須要調整。 二、start with connect by 遞歸查詢在PG中WITH RECURSIVE 三、 (+)這樣的外鏈接寫法須要調整爲SQL標準的 table1 [LEFT|RIGHT|FULL] OUTER JOIN table2 ON (...); 四、DECODE函數須要重構成(case when some_column = 'some_value' then 'some_other_value' when ... then ... else 'some_default_value' end ) as some_column; 五、COALESCE 函數返回值類型不匹配、須要類型轉換。

等等。

能夠看到直接將Oracle遷移到PostgreSQL，不管是應用層，仍是數據庫層面， 都仍是有不少改寫工做要作的。 

因爲有上述諸多問題，咱們研究了一下號稱爲替換Oracle而生的EnterpriseDB（PostgreSQL的商業版）：

http://www.enterprisedb.com/products-services-training/products/documentation/enterpriseedition

EnterpriseDB的安裝指引：

http://www.enterprisedb.com/docs/en/9.4/instguide/toc.html

EnterpriseDB的官方文檔：

http://www.enterprisedb.com/docs/en/9.4/eeguide/toc.html

EnterpriseDB的Oracle遷移PG的工具Migration Toolkit使用指引：

http://www.enterprisedb.com/docs/en/9.4/migrate/toc.html

工具使用比較簡單，簡單配置一些參數接口， 遷移過程發現的問題相比ora2pg少了不少，但仍是有一些：

例如EDB中沒有function sys_connect_by_path函數、 oracle trigger語法裏面的referencing new as new old as old等。

結論：

雖然EDB的Migration Toolkit這個遷移工具自己沒有像ora2pg那麼多靈活的遷移選項組合， 可是其功能基本知足遷移需求，並且重要的是因爲EDB在PostgreSQL的基礎上，外面包了一層，實現了不少Oracle的概念，例如前面提到的dual表、同義詞、pkg、內建函數，甚至Oracle的分區表語法能夠直接在EDB運行。因此，一樣的源Oracle數據庫，咱們發現90%左右的均可以直接在目標EnterpriseDB運行了，只須要修改一少部分。 

具體EnterpriseDB的兼容Oracle的清單，參考

http://www.enterprisedb.com/products-services-training/products/postgres-plus-advanced-server?quicktabs_advanceservertab=3#quicktabs-advanceservertab

因此，從節省成本與運維穩定兩方面平衡考慮， 對於新建系統直接採用PostgreSQL，對於已有系統遷移到PostgreSQL，使用EnterpriseDB也不失爲一個過渡期的較好選擇。 

附錄：

ora2pg --help

Usage: ora2pg [-dhpqv --estimate_cost --dump_as_html] [--option value]

    -a | --allow str  : coma separated list of objects to allow from export.

Can be used with SHOW_COLUMN too.

    -b | --basedir dir: Used to set the default output directory, where files

resulting from exports will be stored.

    -c | --conf file  : Used to set an alternate configuration file than the

default /etc/ora2pg/ora2pg.conf.

    -d | --debug      : Enable verbose output.

    -e | --exclude str: coma separated list of objects to exclude from export.

Can be used with SHOW_COLUMN too.

    -h | --help       : Print this short help.

    -i | --input file : File containing Oracle PL/SQL code to convert with

no Oracle database connection initiated.

    -j | --jobs num   : number of parallel process to send data to PostgreSQL.

    -J | --copies num : number of parallel connection to extract data from Oracle.

    -l | --log file   : Used to set a log file. Default is stdout.

    -L | --limit num  : number of tuples extracted from Oracle and stored in

memory before writing, default: 10000.

    -n | --namespace schema : Used to set the Oracle schema to extract from.

    -o | --out file   : Used to set the path to the output file where SQL will

be written. Default: output.sql in running directory.

    -p | --plsql      : Enable PLSQL to PLPSQL code conversion.

    -P | --parallel num: Number of parallel tables to extract at the same time.

    -q | --quiet      : disable progress bar.

    -s | --source DSN : Allow to set the Oracle DBI datasource.

    -t | --type export: Used to set the export type. It will override the one

given in the configuration file (TYPE).

    -u | --user name  : Used to set the Oracle database connection user.

    -v | --version    : Show Ora2Pg Version and exit.

    -w | --password pwd : Used to set the password of the Oracle database user.

    --forceowner: if set to 1 force ora2pg to set tables and sequences owner

 like in Oracle database. If the value is set to a username this

 one will be used as the objects owner. By default it's the user

 used to connect to the Pg database that will be the owner.

    --nls_lang code: use this to set the Oracle NLS_LANG client encoding.

    --client_encoding code: Use this to set the PostgreSQL client encoding.

    --view_as_table str: coma separated list of view to export as table.

    --estimate_cost   : activate the migration cost evalution with SHOW_REPORT

    --cost_unit_value minutes: number of minutes for a cost evalution unit.

 default: 5 minutes, correspond to a migration conducted by a

 PostgreSQL expert. Set it to 10 if this is your first migration.

   --dump_as_html     : force ora2pg to dump report in HTML, used only with

                        SHOW_REPORT. Default is to dump report as simple text.

   --init_project NAME: initialise a typical ora2pg project tree. Top directory

                        will be created under project base dir.

   --project_base DIR : define the base dir for ora2pg project trees. Default

                        is current directory.

See full documentation at http://ora2pg.darold.net/ for more help or see manpage with 'man ora2pg'.

EDB的Migration Toolkit：

runMTK.sh -help

Running EnterpriseDB Migration Toolkit (Build 48.0.1) ...

EnterpriseDB Migration Toolkit (Build 48.0.1)

Usage: runMTK [-options] SCHEMA

If no option is specified, the complete schema will be imported.

where options include:

-helpDisplay the application command-line usage.

-versionDisplay the application version information.

-verbose [on|off] Display application log messages on standard output (default: on).

-schemaOnlyImport the schema object definitions only.

-dataOnlyImport the table data only. When -tables is in place, it imports data only for the selected tables. Note: If there are any FK constraints defined on target tables, use -truncLoad option along with this option.

-sourcedbtype db_type The -sourcedbtype option specifies the source database type. db_type may be one of the following values: mysql, oracle, sqlserver, sybase, postgresql, enterprisedb. db_type is case-insensitive. By default, db_type is oracle.

-targetdbtype db_type The -targetdbtype option specifies the target database type. db_type may be one of the following values: oracle, sqlserver, postgresql, enterprisedb. db_type is case-insensitive. By default, db_type is enterprisedb.

-allTablesImport all tables.

-tables LISTImport comma-separated list of tables.

-constraintsImport the table constraints.

-indexesImport the table indexes.

-triggersImport the table triggers.

-allViewsImport all Views.

-views LISTImport comma-separated list of Views.

-allProcsImport all stored procedures.

-procs LISTImport comma-separated list of stored procedures.

-allFuncsImport all functions.

-funcs LISTImport comma-separated list of functions.

-allPackagesImport all packages.

-packages LIST Import comma-separated list of packages.

-allSequencesImport all sequences.

-sequences LIST Import comma-separated list of sequences.

-targetSchema NAME Name of the target schema (default: target schema is named after source schema).

-allDBLinksImport all Database Links.

-allSynonymsIt enables the migration of all public and private synonyms from an Oracle database to an Advanced Server database.  If a synonym with the same name already exists in the target database, the existing synonym will be replaced with the migrated version.

-allPublicSynonymsIt enables the migration of all public synonyms from an Oracle database to an Advanced Server database.  If a synonym with the same name already exists in the target database, the existing synonym will be replaced with the migrated version.

-allPrivateSynonymsIt enables the migration of all private synonyms from an Oracle database to an Advanced Server database.  If a synonym with the same name already exists in the target database, the existing synonym will be replaced with the migrated version.

-dropSchema [true|false] Drop the schema if it already exists in the target database (default: false).

-truncLoadIt disables any constraints on target table and truncates the data from the table before importing new data. This option can only be used with -dataOnly.

-safeModeTransfer data in safe mode using plain SQL statements.

-copyDelimiterSpecify a single character to be used as delimiter in copy command when loading table data. Default is \t

-batchSizeSpecify the Batch Size to be used by the bulk inserts. Valid values are  1-1000, default batch size is 1000, reduce if you run into Out of Memory exception

-cpBatchSize Specify the Batch Size in MB, to be used in the Copy Command. Valid value is > 0, default batch size is 8 MB

-fetchSize Specify fetch size in terms of number of rows should be fetched in result set at a time. This option can be used when tables contain millions of rows and you want to avoid out of memory errors.

-filterPropThe properties file that contains table where clause.

-skipFKConstSkip migration of FK constraints.

-skipCKConstSkip migration of Check constraints.

-ignoreCheckConstFilterBy default MTK does not migrate Check constraints and Default clauses from Sybase, use this option to turn off this filter.

-fastCopyBypass WAL logging to perform the COPY operation in an optimized way, default disabled.

-customColTypeMapping LISTUse custom type mapping represented by a semi-colon separated list, where each entry is specified using COL_NAME_REG_EXPR=TYPE pair. e.g. .*ID=INTEGER

-customColTypeMappingFile PROP_FILEThe custom type mapping represented by a properties file, where each entry is specified using COL_NAME_REG_EXPR=TYPE pair. e.g. .*ID=INTEGER

-offlineMigration [PATH] This performs offline migration and saves the DDL/DML scripts in files for a later execution. By default the script files will be saved under user home folder, if required follow -offlineMigration option with a custom path. 

-logDir LOG_PATH Specify a custom path to save the log file. By default, on Linux the logs will be saved under folder $HOME/.enterprisedb/migration-toolkit/logs. In case of Windows logs will be saved under folder %HOMEDRIVE%%HOMEPATH%\.enterprisedb\migration-toolkit\logs.

-copyViaDBLinkOra This option can be used to copy data using dblink_ora COPY commad. This option can only be used in Oracle to EnterpriseDB migration mode.

-singleDataFileUse single SQL file for offline data storage for all tables. This option cannot be used in COPY format.

-allUsers Import all users and roles from the source database.

-users LIST Import the selected users/roles from the source database. LIST is a comma-separated list of user/role names e.g. -users MTK,SAMPLE

-allRules Import all rules from the source database.

-rules LIST Import the selected rules from the source database. LIST is a comma-separated list of rule names e.g. -rules high_sal_emp,low_sal_emp

-allGroups Import all groups from the source database.

-groups LIST Import the selected groups from the source database. LIST is a comma-separated list of group names e.g. -groups acct_emp,mkt_emp

-allDomains Import all domain, enumeration and composite types from the source database.

-domains LIST Import the selected domain, enumeration and composite types from the source database. LIST is a comma-separated list of domain names e.g. -domains d_email,d_dob, mood

-objecttypesImport the user-defined object types.

-replaceNullChar <CHAR> If null character is part of a column value, the data migration fails over JDBC protocol. This option can be used to replace null character with a user-specified character.

-importPartitionAsTable [LIST] Use this option to import Oracle Partitioned table as a normal table in EnterpriseDB. To apply the rule on a selected set of tables, follow the option by a comma-separated list of table names.

-enableConstBeforeDataLoad Use this option to re-enable constraints (and triggers) before data load. This is useful in the scenario when the migrated table is mapped to a partition table in EnterpriseDB.

-checkFunctionBodies [true|false] When set to false, it disables validation of the function body during function creation, this is to avoid errors if function contains forward references. Applicable when target database is Postgres/EnterpriseDB, default is true.

-retryCount VALUESpecify the number of re-attempts performed by MTK to migrate objects that failed due to cross-schema dependencies. The VALUE parameter should be greater than 0, default is 2.

-analyze It invokes ANALYZE operation against a target Postgres or Postgres Plus Advanced Server database. The ANALYZE collects statistics for the migrated tables that are utilized for efficient query plans.

-vacuumAnalyze It invokes VACUUM and ANALYZE operations against a target Postgres or Postgres Plus Advanced Server database. The VACUUM reclaims dead tuple storage whereas ANALYZE collects statistics for the migrated tables that are utilized for efficient query plans.

-loaderCount VALUESpecify the number of jobs (threads) to perform data load in parallel. The VALUE parameter should be greater than 0, default is 1.

-logFileSize VALUEIt represents the maximum file size limit (in MB) before rotating to a new log file, defaults to 50MB.

-logFileCount VALUEIt represents the number of files to maintain in log file rotation history, defaults to 20. Specify a value of zero to disable log file rotation.

-useOraCaseIt preserves the identifier case while migrating from Oracle, except for functions, procedures and packages unless identifier names are given in quotes. 

-logBadSQLIt saves the DDL scripts for the objects that fail to migrate, in a .sql file in log folder.

-targetDBVersionIt represents the major.minor version of the target database. This option is applicable for offline migration mode and is used to validate certain migration options as per target db version [default is 9.4 for EnterpriseDB database].

Database Connection Information:

The application will read the connectivity information for the source and target database servers from toolkit.properties file.

Refer to MTK readme document for more information.