Oracle sqlldr導入之「MAXIMUM ERROR COUNT EXCEEDED」

    昨天看到一個同事在經過PL/SQL Developer工具把文本數據往oracle表;有兩個文本；一個有30萬條記錄；一個7萬多條記錄。
在導入到過程當中；出現錯誤記錄還須要點擊確認。不過使用黑科技（屏幕精靈）自動點擊。單一個7萬多記錄的文本須要10分鐘左右的時間。

    看到這種狀況；我熱心腸爆發；我幫你來提快導入速度。我想到兩種方案；
    1：oracle的sqlldr命令
    2：oracle的外部表；
    因爲文本存在錯誤記錄；就選擇1.
    
文本格式以下：存在有76760條記錄

[oracle@oracle234 ~]$ wc -l lottu.txt               
76761 lottu.txt
[oracle@oracle234 ~]$ head lottu.txt
stat_user_stay_info.rowkey,stat_user_stay_info.appkey,stat_user_stay_info.phone_softversion,stat_user_stay_info.dim_type,stat_user_stay_info.dim_code,stat_user_stay_info.time_peroid,stat_user_stay_info.stat_date,stat_user_stay_info.indicator,stat_user_stay_info.stat_time,stat_user_stay_info.value
3a00997_7c34d20170108,307A5C626E6C2F6472636E6E6A2F736460656473,2.14.0,cpid,blf1298_12243_001,1,20170105,stay3day,20170109102339,1
3a00997_bf86b20170108,307A5C626E6C2F6472636E6E6A2F736460656473,2.13.0,cpid,blp1375_13621_001,1,20170105,stay3day,20170109102339,7
3a00e87_4b11a20170126,337A5C626E6C2F6472636E6E6A2F736460656473,1.4.0,cpid,all,1,20170123,stay3day,20170127095931,6
3a0129a_6575220170118,307A5C626E6C2F6460726E742F716D7472,all,cpid,bsf1389_10917_001,1,20170116,stay2day,20170119094145,1
3a0183b_5764a20170202,307A5C626E6C2F6472636E6E6A2F736460656473,1.91,cpid,blf1298_12523_001,1,20170128,stay5day,20170203094327,1
3a01b9b_54b4720170123,307A5C626E6C2F6472636E6E6A2F736460656473,2.13.0,cpid,blp1375_13641_001,1,20170122,stay1day,20170124102457,3
3a0230d_7464120170126,307A5C626E6C2F6460726E742F606F65736E686569646D716473,all,cpid,bsp1405_13363_001,1,20170122,stay4day,20170127100446,18
3a02bed_3ea3320170206,307A5C626E6C2F6472636E6E6A2F736460656473,2.15.0,cpid,blp1375_14217_001,1,20170130,stay7day,20170207135438,1
3a03fe3_4c5fe20170119,307A5C21626E6C2F6472776865646E21,all,cpid,bvf1328_10885_001,1,20170116,stay3day,20170120093733,1   

導入表結果以下：

SQL> desc STAT_USER_STAY_INFO1;
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 JOBID                                              VARCHAR2(64)
 APPKEY                                    NOT NULL VARCHAR2(200)
 PHONE_SOFTVERSION                         NOT NULL VARCHAR2(32)
 DIM_TYPE                                  NOT NULL VARCHAR2(64)
 DIM_CODE                                  NOT NULL VARCHAR2(64)
 TIME_PEROID                               NOT NULL VARCHAR2(4)
 STAT_DATE                                 NOT NULL VARCHAR2(500)
 INDICATOR                                 NOT NULL VARCHAR2(200)
 STAT_TIME                                          VARCHAR2(500)
 VALUE                                     NOT NULL NUMBER

執行sqlldr命令；但結果呢？只導入5萬條記錄；結果出乎意料。

sqlldr 'lottu/li0924' control=/home/oracle/stay_info.ctl log=/home/oracle/stay_info.log bad=/home/oracle/stay_info.bad

查看log文件;因爲篇幅的問題;只取關鍵部分。

.......
Record 55076: Rejected - Error on table STAT_USER_STAY_INFO1, column DIM_CODE.
ORA-01400: cannot insert NULL into ("LOTTU"."STAT_USER_STAY_INFO1"."DIM_CODE")

MAXIMUM ERROR COUNT EXCEEDED - Above statistics reflect partial run.

Table STAT_USER_STAY_INFO1:
  55025 Rows successfully loaded.        
  51 Rows not loaded due to data errors.
  0 Rows not loaded because all WHEN clauses were failed.
  0 Rows not loaded because all fields were null.


Space allocated for bind array:                 165120 bytes(64 rows)
Read   buffer bytes: 1048576

Total logical records skipped:          0
Total logical records read:         55105
Total logical records rejected:        51
Total logical records discarded:        0

Run began on Fri Feb 24 10:51:02 2017
Run ended on Fri Feb 24 10:51:09 2017

Elapsed time was:     00:00:06.87
CPU time was:         00:00:00.46  

  日誌提示；只導入了「55025 Rows successfully loaded. 」；加上拒絕的51條；這跟76761條記錄遠遠不夠。
日誌記錄是不會騙人的；進入表中查看確實是55025條記錄

SQL> select count(*) from STAT_USER_STAY_INFO1;

  COUNT(*)
----------
     55025

  奇了怪；明明76761條記錄；爲何oracle只承認5萬條記錄 。還有2萬多條記錄爲何不承認呢？
其實oracle給了提示；就在剛剛那個log文件中。只是我忽略這樣一句話"MAXIMUM ERROR COUNT EXCEEDED - Above statistics reflect partial run."。
這個意思是說；超過最大所容限錯誤數。
既然說得這麼清楚了；咱們在看看sqlldr命令

[oracle@oracle234 ~]$ sqlldr

SQL*Loader: Release 11.2.0.1.0 - Production on Fri Feb 24 11:00:08 2017

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.


Usage: SQLLDR keyword=value [,keyword=value,...]

Valid Keywords:

    userid -- ORACLE username/password           
   control -- control file name                  
       log -- log file name                      
       bad -- bad file name                      
      data -- data file name                     
   discard -- discard file name                  
discardmax -- number of discards to allow          (Default all)
      skip -- number of logical records to skip    (Default 0)
      load -- number of logical records to load    (Default all)
    errors -- number of errors to allow            (Default 50)
      rows -- number of rows in conventional path bind array or between direct path data saves
               (Default: Conventional path 64, Direct path all)
  bindsize -- size of conventional path bind array in bytes  (Default 256000)
    silent -- suppress messages during run (header,feedback,errors,discards,partitions)
    direct -- use direct path                      (Default FALSE)
   parfile -- parameter file: name of file that contains parameter specifications
  parallel -- do parallel load                     (Default FALSE)
      file -- file to allocate extents from      
skip_unusable_indexes -- disallow/allow unusable indexes or index partitions  (Default FALSE)
skip_index_maintenance -- do not maintain indexes, mark affected indexes as unusable  (Default FALSE)
commit_discontinued -- commit loaded rows when load is discontinued  (Default FALSE)
  readsize -- size of read buffer                  (Default 1048576)
external_table -- use external table for load; NOT_USED, GENERATE_ONLY, EXECUTE  (Default NOT_USED)
columnarrayrows -- number of rows for direct path column array  (Default 5000)
streamsize -- size of direct path stream buffer in bytes  (Default 256000)
multithreading -- use multithreading in direct path  
 resumable -- enable or disable resumable for current session  (Default FALSE)
resumable_name -- text string to help identify resumable statement
resumable_timeout -- wait time (in seconds) for RESUMABLE  (Default 7200)
date_cache -- size (in entries) of date conversion cache  (Default 1000)
no_index_errors -- abort load on any index errors  (Default FALSE)

PLEASE NOTE: Command-line parameters may be specified either by
position or by keywords.  An example of the former case is 'sqlldr
scott/tiger foo'; an example of the latter is 'sqlldr control=foo
userid=scott/tiger'.  One may specify parameters by position before
but not after parameters specified by keywords.  For example,
'sqlldr scott/tiger control=foo logfile=log' is allowed, but
'sqlldr scott/tiger control=foo log' is not, even though the
position of the parameter 'log' is correct. 

其中有一行內容「    errors -- number of errors to allow            (Default 50)」  
因此說上面出現問題就不奇怪了。

最後將sqlldr命令加errors參數便可。

sqlldr 'lottu/li0924' control=/home/oracle/stay_info.ctl log=/home/oracle/stay_info.log bad=/home/oracle/stay_info.bad errors=1000

整個過程20秒處理完成。相比之下；我那同事果斷選擇這種方法。