Oracle11g在Windows和Linux下imp導入表，exp導出表，sqluldr2導出表，sqlldr導入表

Windows(Win10)

打開cmd

首先輸入sqlplus,依次輸入用戶名、口令

C:\Users\hasee>sqlplus SQL*Plus: Release 11.2.0.1.0 Production on 星期三 3月 13 16:55:46 2019 Copyright (c) 1982, 2010, Oracle. All rights reserved. 請輸入用戶名: scott 輸入口令: 鏈接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options

鏈接數據庫管理員

[oracle@CentOS7One ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.1.0 Production on 星期一 3月 18 14:28:40 2019 Copyright (c) 1982, 2009, Oracle. All rights reserved. 已鏈接到空閒例程。

1.exp導出表

退出oracle，將scott用戶下orcl數據庫的buy_cnt_c1表導出到e:/opt/oracle_output/daochu.dmp

C:\Users\hasee>exp scott/tiger@orcl file=e:\opt\\oracle_output\daochu.dmp tables=(buy_cnt_c1) Export: Release 11.2.0.1.0 - Production on 星期三 3月 13 16:58:30 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 鏈接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options 已導出 ZHS16GBK 字符集和 AL16UTF16 NCHAR 字符集 即將導出指定的表經過常規路徑... . . 正在導出表 BUY_CNT_C1導出了 8293 行 成功終止導出, 沒有出現警告。

8293條數據 22ms

8492033條數據18s65ms

2.imp導入表

導入前要將原來的表刪除

drop table buy_cnt_c1;

退出oracle，將e:/opt/oracle_output/daochu.dmp導入到scott用戶下orcl數據庫

C:\Users\hasee>imp scott/tiger@orcl file=e:\opt\\oracle_output\daochu.dmp Import: Release 11.2.0.1.0 - Production on 星期三 3月 13 17:03:29 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 鏈接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options 經由常規路徑由 EXPORT:V11.02.00 建立的導出文件 已經完成 ZHS16GBK 字符集和 AL16UTF16 NCHAR 字符集中的導入 . 正在將 SCOTT 的對象導入到 SCOTT . 正在將 SCOTT 的對象導入到 SCOTT . . 正在導入表 "BUY_CNT_C1"導入了        8293 行 成功終止導入, 沒有出現警告。

8293條數據  18ms

8492033條數據 1m19s8ms

 3.sqluldr2導出表

1.導出表

E:\bigdata\sqluldr2_win_jb51>sqluldr264 scott/tiger@127.0.0.1/orcl query="select * from buy_cnt_c1" head=yes file=e:\opt\oracle_output\tmp001.txt 0 rows exported at 2019-03-13 17:40:46, size 0 MB. 8293 rows exported at 2019-03-13 17:40:46, size 0 MB. output file e:\opt\oracle_output\tmp001.txt closed at 8293 rows, size 0 MB.

8293條數據  23ms

8492033條數據17s55ms

 

2.指定分隔符

field對字段進行分隔，record對記錄進行分隔。

ESCF是escape from，指定哪些形式的數據須要被轉義；

ESCAPE是轉義前綴

ESCT是escape to,轉義成目標字符串

\r=0x0d \n=0x0a |=0x7c ,=0x2c, \t=0x09, :=0x3a, #=0x23, "=0x22 '=0x27

sqluldr264 test/test@127.0.0.1/orcl query="select * from test" head=yes file=e:\opt\oracle_output\hex_test.txt log=e:\opt\oracle_output\hex.log  field=0x7c  record=0x0d ESCF=0x7c0x0d ESCAPE='\' ESCT=i

表信息以下

第一行「|a」字段裏含有咱們的字段分隔符，用ESCAPE便可替換

第五行含有0x0d，即換行符，不會影響咱們的結果，sqluldr會將它當作字符串

導出結果爲

ID|NAME 1|'ia
2|sanq 3|jieba 4|wuren 5|0x0d

 

4.sqlldr導入數據

首先建立表

DROP TABLE "SCOTT"."BUY_CNT_C1"; CREATE TABLE "SCOTT"."BUY_CNT_C1" ( "T_INDEX" NUMBER(19) NULL , "INNET_TIME" NUMBER(19) NULL , "UP_DATA_AMOUNT_TOTAL" FLOAT(126) NULL , "DOWN_DATA_AMOUNT_TOTAL" FLOAT(126) NULL , "PAY_CNT" NUMBER(19) NULL , "BUY_CNT" NUMBER(19) NULL , "MONTH_FEE" FLOAT(126) NULL , "CALL_DURATION" NUMBER(19) NULL , "AGE_RANGE_ID" FLOAT(126) NULL , "DAY_ACT_NUM_MEAN" FLOAT(126) NULL ) LOGGING NOCOMPRESS NOCACHE ;

編寫ctl文件

tmp_002.ctl

load data infile * into table BUY_CNT_TEST ( "T_INDEX" char terminated  by   ',', "INNET_TIME" char terminated   by   ',', "UP_DATA_AMOUNT_TOTAL" char terminated   by   ',', "DOWN_DATA_AMOUNT_TOTAL" char terminated   by   ',', "PAY_CNT" char terminated   by   ',', "BUY_CNT"  char terminated   by   ',', "MONTH_FEE" char terminated   by   ',', "CALL_DURATION"  char terminated   by   ',', "AGE_RANGE_ID" char terminated   by   ',', "DAY_ACT_NUM_MEAN" char terminated   by   ',' )

cmd輸入

E:\bigdata\sqluldr2_win_jb51>sqlldr userid=scott/tiger control=e:/opt/oracle_ctl/tmp002.ctl log=e:/opt/oracle_ctl/tmp001.log data=e:/opt/oracle_ctl/tmp001.csv rows=64 SQL*Loader: Release 11.2.0.1.0 - Production on 星期四 3月 14 10:50:46 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 達到提交點 - 邏輯記錄計數 64 達到提交點 - 邏輯記錄計數 128 達到提交點 - 邏輯記錄計數 192 達到提交點 - 邏輯記錄計數 256 
...
 達到提交點 - 邏輯記錄計數 8192 達到提交點 - 邏輯記錄計數 8256 達到提交點 - 邏輯記錄計數 8294

8294條數據 28ms

8492033條數據 2m6s12ms

Linux(CentOS7)

1.exp導出表

[oracle@CentOS7One ~]$ exp scott/tiger file=output/daochu.dmp tables='(buy_cnt_c1)' log=output/log/daochudmp.log Export: Release 11.2.0.1.0 - Production on 星期三 3月 20 09:12:08 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 鏈接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options 已導出 AL32UTF8 字符集和 AL16UTF16 NCHAR 字符集 服務器使用 ZHS16GBK 字符集 (可能的字符集轉換) 即將導出指定的表經過常規路徑... . . 正在導出表 BUY_CNT_C1導出了 8492032 行 成功終止導出, 沒有出現警告。

8293條數據 2s

8492033條數據60s

2.imp導入表

導入前要將原來的表刪除

drop table buy_cnt_c1;

[oracle@CentOS7One ~]$ imp scott/tiger@orcl11g file=input/daoru.cmp Import: Release 11.2.0.1.0 - Production on 星期三 3月 20 10:42:51 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 鏈接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options 經由常規路徑由 EXPORT:V11.02.00 建立的導出文件 已經完成 AL32UTF8 字符集和 AL16UTF16 NCHAR 字符集中的導入 導入服務器使用 ZHS16GBK 字符集 (可能的字符集轉換) . 正在將 SCOTT 的對象導入到 SCOTT . 正在將 SCOTT 的對象導入到 SCOTT . . 正在導入表 "BUY_CNT_C1"導入了     8492032 行 成功終止導入, 沒有出現警告。

8293條數據 48ms

8492032條數據61s

3.sqluldr2導出數據

sqluldr2_linux64_10204.bin user=scott/tiger@orcl11g query="select * from buy_cnt_c1" head=yes file=output/tmp002.csv log=output/log/tmp002.log

8293條數據 小於1s

8492032條數據 5min17s

4.sqlldr導入數據

ctl文件同上,先建立表

[oracle@CentOS7One ~]$ sqlldr userid=scott/tiger control=input/tmp002.ctl log=input/log/tmp002.log data=input/tmp001.csv rows=64 SQL*Loader: Release 11.2.0.1.0 - Production on 星期二 3月 19 16:46:56 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 達到提交點 - 邏輯記錄計數 64 達到提交點 - 邏輯記錄計數 128 達到提交點 - 邏輯記錄計數 192 達到提交點 - 邏輯記錄計數 256 達到提交點 - 邏輯記錄計數 320 ... 達到提交點 - 邏輯記錄計數 8192 達到提交點 - 邏輯記錄計數 8256 達到提交點 - 邏輯記錄計數 8294

8294條數據  47ms

8492033數據  5m27s26ms

 

測試

編寫測試腳本，統計腳本命令執行時間

Windows腳本以下

@echo off set /a startMS=%time:~9,2% set /a startS=%time:~6,2% set /a startM=%time:~3,2%
echo %time% ::寫你的命令 set /a endMS=%time:~9,2% set /a endS=%time:~6,2% set /a endM=%time:~3,2%
echo %time% set /a diffMS_=%endMS%-%startMS% set /a diffS_=%endS%-%startS% set /a diffM_=%endM%-%startM%
echo cost:%diffM_% %diffS_% %diffMS_% pause

 

Linux腳本以下

start_time=`date --date='0 days ago' "+%Y-%m-%d %H:%M:%S"` #this is your shell script #寫你的命令

 ############## finish_time=`date --date='0 days ago' "+%Y-%m-%d %H:%M:%S"` duration=$(($(($(date +%s -d "$finish_time")-$(date +%s -d "$start_time"))))) echo "this shell script execution duration: $duration"

 

CentOS7部署在虛擬機上,因此配置較Windows差不少

8294條數據（小數據量）

 8492033條數據（大數據量）

業務實操1

中移動3300萬數據的csv(6.03GB)使用sqlldr導入到oracle數據庫

表寬61列，建表語句以下，字段長度建議設置255以上

create table T_CJYX_DZXX ( id               VARCHAR2(50), version VARCHAR2(15), metacategory VARCHAR2(100), indicator VARCHAR2(20), iscity VARCHAR2(20), parentid VARCHAR2(24), name VARCHAR2(255), code VARCHAR2(255), nameabbrpy VARCHAR2(255), hierarchy VARCHAR2(20), alias VARCHAR2(255), alias2 VARCHAR2(255), alias3 VARCHAR2(255), alias4 VARCHAR2(255), postcode VARCHAR2(255), cover VARCHAR2(25), longitude VARCHAR2(50), latitude VARCHAR2(26), rgt VARCHAR2(20), lft VARCHAR2(20), status VARCHAR2(25), comments VARCHAR2(255), detailaddress VARCHAR2(255), logdate VARCHAR2(50), areaid VARCHAR2(28), isdefine VARCHAR2(22), faultstatus VARCHAR2(25), restoretime VARCHAR2(50), faultdescription VARCHAR2(2555), deviceusage VARCHAR2(25), faultdetailcode VARCHAR2(20), floortype VARCHAR2(25), faultcount VARCHAR2(20), createdate VARCHAR2(50), auditdate VARCHAR2(50), createuser VARCHAR2(255), audituser VARCHAR2(255), addresstype VARCHAR2(25), marketgrade VARCHAR2(25), iscoordinate VARCHAR2(20), ttmigrate VARCHAR2(25), gjgx VARCHAR2(25), accesstype VARCHAR2(25), shape VARCHAR2(128), fillcolor VARCHAR2(255), fillwidth VARCHAR2(22), fillalpha VARCHAR2(21), fillstyle VARCHAR2(255), fillbordercolor VARCHAR2(255), area VARCHAR2(23), areaunit VARCHAR2(22), createuserphone VARCHAR2(255), modifyuser VARCHAR2(255), modifydate VARCHAR2(50), gcstatus VARCHAR2(25), pmsprjcode VARCHAR2(255), pmsprjname VARCHAR2(255), deleteuser VARCHAR2(255), deletedate VARCHAR2(50), afterdeleteuser VARCHAR2(255), afterdeletedate VARCHAR2(50) ) tablespace TBS_AHJZH_DATA pctfree 10 initrans 1 maxtrans 255 storage ( initial 64 next 8 minextents 1 maxextents unlimited );

ctl文件以下

LOAD DATA INFILE * INTO TABLE T_CJYX_DZXX REPLACE FIELDS TERMINATED BY '$' TRAILING NULLCOLS ( id, version, metacategory, indicator, iscity, parentid, name, code, nameabbrpy, hierarchy, alias, alias2, alias3, alias4, postcode, cover, longitude, latitude, rgt, lft, status, comments, detailaddress, logdate, areaid, isdefine, faultstatus, restoretime, faultdescription, deviceusage, faultdetailcode, floortype, faultcount, createdate, auditdate, createuser, audituser, addresstype, marketgrade, iscoordinate, ttmigrate, gjgx, accesstype, shape, fillcolor, fillwidth, fillalpha, fillstyle, fillbordercolor, area, areaunit, createuserphone, modifyuser, modifydate, gcstatus, pmsprjcode, pmsprjname, deleteuser, deletedate, afterdeleteuser, afterdeletedate ) 

 

運行日誌以下，耗時約14小時

SQL*Loader: Release 11.2.0.1.0 - Production on 星期三 4月 3 08:46:55 2019 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. 控制文件: E:\opt\srcbigdata\di_70002_20190325.ctl 數據文件: E:\opt\srcbigdata\di_70002_20190325.csv 錯誤文件: E:\opt\srcbigdata\di_70002_20190325.bad 廢棄文件: 未做指定 (可廢棄全部記錄) 要加載的數: ALL 要跳過的數: 0 容許的錯誤: 50 綁定數組: 64 行, 最大 256000 字節 繼續: 未做指定 所用路徑: 常規 表 T_CJYX_DZXX,已加載從每一個邏輯記錄 插入選項對此表 REPLACE 生效 TRAILING NULLCOLS 選項生效 列名 位置 長度 停止 包裝數據類型 ------------------------------ ---------- ----- ---- ---- --------------------- ID FIRST * $ CHARACTER VERSION NEXT * $ CHARACTER METACATEGORY NEXT * $ CHARACTER INDICATOR NEXT * $ CHARACTER ISCITY NEXT * $ CHARACTER PARENTID NEXT * $ CHARACTER NAME NEXT * $ CHARACTER CODE NEXT * $ CHARACTER NAMEABBRPY NEXT * $ CHARACTER HIERARCHY NEXT * $ CHARACTER ALIAS NEXT * $ CHARACTER ALIAS2 NEXT * $ CHARACTER ALIAS3 NEXT * $ CHARACTER ALIAS4 NEXT * $ CHARACTER POSTCODE NEXT * $ CHARACTER COVER NEXT * $ CHARACTER LONGITUDE NEXT * $ CHARACTER LATITUDE NEXT * $ CHARACTER RGT NEXT * $ CHARACTER LFT NEXT * $ CHARACTER STATUS NEXT * $ CHARACTER COMMENTS NEXT * $ CHARACTER DETAILADDRESS NEXT * $ CHARACTER LOGDATE NEXT * $ CHARACTER AREAID NEXT * $ CHARACTER ISDEFINE NEXT * $ CHARACTER FAULTSTATUS NEXT * $ CHARACTER RESTORETIME NEXT * $ CHARACTER FAULTDESCRIPTION NEXT * $ CHARACTER DEVICEUSAGE NEXT * $ CHARACTER FAULTDETAILCODE NEXT * $ CHARACTER FLOORTYPE NEXT * $ CHARACTER FAULTCOUNT NEXT * $ CHARACTER CREATEDATE NEXT * $ CHARACTER AUDITDATE NEXT * $ CHARACTER CREATEUSER NEXT * $ CHARACTER AUDITUSER NEXT * $ CHARACTER ADDRESSTYPE NEXT * $ CHARACTER MARKETGRADE NEXT * $ CHARACTER ISCOORDINATE NEXT * $ CHARACTER TTMIGRATE NEXT * $ CHARACTER GJGX NEXT * $ CHARACTER ACCESSTYPE NEXT * $ CHARACTER SHAPE NEXT * $ CHARACTER FILLCOLOR NEXT * $ CHARACTER FILLWIDTH NEXT * $ CHARACTER FILLALPHA NEXT * $ CHARACTER FILLSTYLE NEXT * $ CHARACTER FILLBORDERCOLOR NEXT * $ CHARACTER AREA NEXT * $ CHARACTER AREAUNIT NEXT * $ CHARACTER CREATEUSERPHONE NEXT * $ CHARACTER MODIFYUSER NEXT * $ CHARACTER MODIFYDATE NEXT * $ CHARACTER GCSTATUS NEXT * $ CHARACTER PMSPRJCODE NEXT * $ CHARACTER PMSPRJNAME NEXT * $ CHARACTER DELETEUSER NEXT * $ CHARACTER DELETEDATE NEXT * $ CHARACTER AFTERDELETEUSER NEXT * $ CHARACTER AFTERDELETEDATE NEXT * $ CHARACTER ROWS 參數所用的值已從 64 更改成 16 表 T_CJYX_DZXX: 33543432 行 加載成功。 因爲數據錯誤, 0 行 沒有加載。 因爲全部 WHEN 子句失敗, 0 行 沒有加載。 因爲全部字段都爲空的, 0 行 沒有加載。 爲綁定數組分配的空間: 251808 字節 (16 行) 讀取 緩衝區字節數: 1048576 跳過的邏輯記錄總數: 0 讀取的邏輯記錄總數: 33543432 拒絕的邏輯記錄總數: 0 廢棄的邏輯記錄總數: 0 從 星期三 4月 03 08:46:55 2019 開始運行 在 星期三 4月 03 22:48:09 2019 處運行結束 通過時間爲: 14: 01: 14.33 CPU 時間爲: 00: 18: 19.89

業務實操2

建表語句

create table T_CJYX_HOMECOUNT_BACKUP ( acyc_id VARCHAR2(50), address_id VARCHAR2(50), address_name VARCHAR2(200), address_level VARCHAR2(50), check_type VARCHAR2(50), check_target_num VARCHAR2(50), check_value VARCHAR2(50), target_phone VARCHAR2(4000), notarget_phone VARCHAR2(4000), parent_id VARCHAR2(50), bcyc_id VARCHAR2(50) )

 

從服務器導入16G數據，表寬十一列，一億八千萬條記錄

控制文件di_00121_20190427.ctl以下

LOAD DATA INFILE * INTO TABLE T_CJYX_HOMECOUNT_BACKUP REPLACE FIELDS TERMINATED BY '$' TRAILING NULLCOLS ( acyc_id CHAR(40000), address_id CHAR(40000), address_name CHAR(40000), address_level CHAR(40000), check_type CHAR(40000), check_target_num CHAR(40000), check_value CHAR(40000), target_phone CHAR(40000), notarget_phone CHAR(40000), parent_id CHAR(40000), bcyc_id CHAR(40000) ) 

運行腳本

這種模式導入速度很是慢，十五小時只能導入三千萬條，可是此方案安全穩定

sqlldr userid=username/password@//10.243.5.16:1521/itgrept control=E:\opt\srcbigdata2\di_00121_20190427.ctl log=E:\opt\srcbigdata2\di_00121_20190427.log data=E:\opt\srcbigdata2\di_00121_20190427.dat

參數調優後的腳本,十五小時能夠導入一億條數據

sqlldr userid=username/password@//10.243.5.16:1521/itgrept control=E:\opt\srcbigdata2\di_00121_20190427.ctl log=E:\opt\srcbigdata2\di_00121_20190427_gbk.log data=E:\opt\srcbigdata2\di_00121_20190427_gbk.dat rows=10000 readsize=20680000 bindsize=20680000

最佳性能的狀況下，

報錯ORA-03113：通訊通道的文件結尾
緣由：sqlORA-03113 鏈接到數據庫的網絡中斷
1.SYSTEM中審計表aud$在數據庫編程過程當中，對SQL語句的編譯過程，資源佔用會較大
2.SYSAUX則進行的是AWR快照，也會進行佔用較多的空間
解決方案：以sysdba的身份清除審計表，最好的方法是弄一個本地表空間進行導入

報錯：
SQL*Loader-604:  試圖提交時出錯
ORA-03135: 鏈接失去聯繫

 

實際上腳本運行中途會由於各類緣由中斷，使用append方式向表追加

di_00121_20190427_append.ctl以下

LOAD DATA INFILE * INTO TABLE T_CJYX_HOMECOUNT_BACKUP APPEND FIELDS TERMINATED BY '$' TRAILING NULLCOLS ( acyc_id CHAR(40000), address_id CHAR(40000), address_name CHAR(40000), address_level CHAR(40000), check_type CHAR(40000), check_target_num CHAR(40000), check_value CHAR(40000), target_phone CHAR(40000), notarget_phone CHAR(40000), parent_id CHAR(40000), bcyc_id CHAR(40000) ) 

在108238266行中斷

從108238266行繼續導入

運行腳本以下

sqlldr userid=username/password@//10.243.5.16:1521/itgrept control=E:\opt\srcbigdata2\di_00121_20190427_append.ctl log=E:\opt\srcbigdata2\di_00121_20190427_gbk_append.log data=E:\opt\srcbigdata2\di_00121_20190427_gbk.dat rows=100 readsize=20680000 bindsize=20680000 skip=108238266

耗時大約四十一小時

總結

exp導出和imp導入的效率要大於sqluldr2和sqlldr；

小數據量體現不出來exp和imp的優點，效率和sqluldr2和sqlldr差很少；

大數據量exp和imp明顯優於sqluldr2和sqlldr；

硬件配置高會提升導入導出效率；

不一樣數據庫遷移建議使用sqluldr2和sqlldr,支持csv,txt等數據格式的導入導出；