ORA-16401: archivelog rejected by RFS

ORA-16401: archivelog rejected by RFS

無線出單系統郵件告警10.111.20.1

 

1.     報錯

SYS > ! oerr ora 16041

16041, 00000, "Remote File Server fatal error"

// *Cause:  The Remote File Server (RFS) process at a redo transport

//          destination encountered a fatal error and was no longer

//          available. Further redo transmission to this destination may not be

//          possible.

// *Action: Refer to the appropriate RFS trace file located at the redo

//          transport destination for details regarding the error encountered

//          and take corrective action.

遠程傳輸錯誤，RFS服務器，與參數fal_server,fal_client參數有關

 

 

2.     搜索百度

一般此問題是因爲主庫的fal_client參數與備庫的log_archive_dest_n中的service參數不匹配形成的。

 

 

3.     查詢參數

Primary

--查詢DG成員

SQL> show parameter config

log_archive_config          rac,dgwless,wxcdrac 

--減小本身自己，可能存在兩個DG成員

 

--查詢主庫db_unique_name

SQL> show parameter db_unique_name

db_unique_name                wxcdrac

 

--查詢主庫歸檔參數

SQL> show parameter archive

NAME                 VALUE

---------------------------------------------------

log_archive_dest_2   service=dgwless  LGWR ASYNC va

                     lid_for=(online_logfiles,prima

                     ry_role) db_unique_name=dgwless

                                                

log_archive_dest_3   SERVICE=RAC LGWR ASYNC VALID_F

                     OR=(ONLINE_LOGFILES,PRIMARY_RO

                     LE) db_unique_name=rac

 

--查詢歸檔線程參數狀態，以上歸檔線程3對應的參數能夠過濾，由於此DG可能已被廢棄

--除非是人爲或者誤操做置爲defer，不然基本判斷問題，排除此DG                 

log_archive_dest_state_2             ENABLE

log_archive_dest_state_3             DEFER

 

--查詢主庫fal參數

SQL> show parameter fal

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

fal_client                           string      wxcdrac2

fal_server                           string      dgwless

 

--經過主庫參數，查詢備庫IP地址

SQL> host tnsping dgwless

Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.135.101.23)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dgwless)))

OK (90 msec)

 

--鏈接備庫後，查詢驗證主備關係

SQL> select dbid,name, database_role,open_mode,FLASHBACK_ON  from v$database;

 

       DBID NAME            DATABASE_ROL OPEN_MODE  FLASHBACK_ON

----------- --------------- ------------ ---------- ------------------

 2283108100 RAC             PRIMARY      READ WRITE NO

 

SQL> select dbid,name, database_role,open_mode,FLASHBACK_ON  from v$database;

 

       DBID NAME            DATABASE_ROL OPEN_MODE  FLASHBACK_ON

----------- --------------- ------------ ---------- ------------------

 2283108100 RAC             PHYSICAL STANDBY   MOUNTED    NO

                           

 

--standby

SQL> show parameter fal

 

NAME          VALUE

------------- ------------------------------

fal_client    dgwless

fal_server    wxcdrac  --發現與主庫fal_client參數不一樣

 

 

--查詢備庫歸檔參數

log_archive_dest_1   location=/picclife/app/oracle/archdg/                   

log_archive_dest_2   SERVICE=WXCDRAC LGWR ASYNC VAL  --發現tns也與主庫fal_client不一樣

                     ID_FOR=(ONLINE_LOGFILES,PRIMAR

                     Y_ROLE) db_unique_name=wxcdrac

                                               

--主庫fal_client 在11g已被棄用，按照道理將主庫fal_client與備庫log_archvie_service不一樣，無異常，查詢版本，

當前10G                                      

Enterprise Edition Release 10.2.0.3.0

 

--查詢主庫集羣狀態

wxcdrac2:/home/oracle$ export ORACLE_SID=+ASM1

wxcdrac2:/home/oracle$ cat .profile 

export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1

export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs_1

 

wxcdrac2:/home/oracle$ cd $ORA_CRS_HOME/bin

wxcdrac2:/picclife/app/oracle/product/10.2.0/crs_1/bin$

 

wxcdrac2:/picclife/app/oracle/product/10.2.0/crs_1/bin$ crs_stat -t

Name           Type           Target    State     Host       

------------------------------------------------------------

ora.wxcdrac.db application    ONLINE    ONLINE    wxcdrac1   

ora....c1.inst application    ONLINE    ONLINE    wxcdrac1   

ora....c2.inst application    ONLINE    ONLINE    wxcdrac2   

ora....SM1.asm application    ONLINE    ONLINE    wxcdrac1   

ora....C1.lsnr application    ONLINE    ONLINE    wxcdrac1   

ora....ac1.gsd application    ONLINE    ONLINE    wxcdrac1   

ora....ac1.ons application    ONLINE    ONLINE    wxcdrac1   

ora....ac1.vip application    ONLINE    ONLINE    wxcdrac1   

ora....SM2.asm application    ONLINE    ONLINE    wxcdrac2   

ora....C2.lsnr application    ONLINE    ONLINE    wxcdrac2   

ora....ac2.gsd application    ONLINE    ONLINE    wxcdrac2   

ora....ac2.ons application    ONLINE    ONLINE    wxcdrac2   

ora....ac2.vip application    ONLINE    ONLINE    wxcdrac2

 

4.     查詢日誌

--備庫日誌

Wed Aug  8 01:59:19 2018

Redo Shipping Client Connected as PUBLIC

-- Connected User is Valid

RFS[2586]: Assigned to RFS process 29818962

RFS[2586]: Identified database type as 'physical standby'

Primary database is in MAXIMUM PERFORMANCE mode

Primary database is in MAXIMUM PERFORMANCE mode

RFS[2586]: Successfully opened standby log 15: '/picclife/app/oracle/oradata/dgwless/std02_1.log'

Wed Aug  8 02:04:42 2018

RFS[2579]: Archivelog thread 2 sequence 3661 cannot be reused

Wed Aug  8 02:04:42 2018

Errors in file /picclife/app/oracle/admin/dgwless/udump/dgwless_rfs_13238468.trc:

ORA-16401: archivelog rejected by RFS

Redo Shipping Client Connected as PUBLIC

-- Connected User is Valid

 

--查詢trace文件

KCRRSNPS

Identified standby redo log 12 for implicit mid-log reconnect

*** 2018-08-08 02:04:42.324

*** 2018-08-08 02:04:42.324 61287 kcrr.c

RFS[2579]: Archivelog thread 2 sequence 3661 cannot be reused

This problem can occur as the result of multiple primary and/or

standby database(s) trying to archive to this standby database.

This is not allowed, since successfully completed archivelogs

would be overwritten.

ORA-16401: archivelog rejected by RFS

*** 2018-08-08 02:13:43.562

Dispatching RFS shutdown notification

 

以上備庫查詢獲得的信息是，歸檔接受出現網絡問題，但不明確

 

查詢主庫Alert日誌

--查詢郵件告警所在的數據庫Alert日誌

SQL> show parameter background_dump_dest

 

NAME                  VALUE

--------------------- ------------------------------

background_dump_dest  /picclife/app/oracle/admin/wxcdrac/bdump

 

Wed Aug  8 01:02:57 2018

LNS: Standby redo logfile selected for thread 1 sequence 3993 for destination LOG_ARCHIVE_DEST_2

Wed Aug  8 01:03:32 2018

Starting control autobackup

Control autobackup written to SBT_TAPE device

        comment 'API Version 2.0,MMS Version 10.0.0.116',

        media 'V_89405_737884'

        handle 'c-2283108100-20180808-01'

Wed Aug  8 01:08:19 2018

Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc2_10682458.trc:

ORA-16401: archivelog rejected by RFS

Wed Aug  8 01:11:58 2018

ARC0: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (3135)

ARC0: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned

Wed Aug  8 01:11:58 2018

Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc0_11337924.trc:

ORA-03135: connection lost contact

FAL[server, ARC0]: Error 3135 creating remote archivelog file 'dgwless'

FAL[server, ARC0]: FAL archive failed, see trace file.

Wed Aug  8 01:11:58 2018

Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc0_11337924.trc:

ORA-16055: FAL request rejected

ARCH: FAL archive failed. Archiver continuing

Wed Aug  8 01:11:58 2018

ORACLE Instance wxcdrac1 - Archival Error. Archiver continuing.

 

*** 2018-08-08 01:08:19.693

Error 16401 creating standby archive log file at host 'dgwless'

ORA-16401: archivelog rejected by RFS 

--從以上的信息仍是未明確發現因爲參數問題致使的報錯                    

 

5.     搜索MOS

1183143.1

ORA-16401: ARCHIVELOG REJECTED BY RFS error occurs when incorrect naming convention for fal_client parameter is set.

Version 10.2.0.4 to 11.1.0.7 [Release 10.2 to 11.1]

配置主備不一致的fal_client時會發生此錯誤

 

***** Primary *****

log_archive_config DG_CONFIG=(p1ncm1,p1ncm2)

log_archive_dest_1 location=/u99/ORACLE/p1ncm1/archive valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=p1ncm1

log_archive_dest_2 service=p1ncm2_dgfal.csm.fub.com optional reopen=60 valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=p1ncm2

fal_client p1ncm1_fc.csm.fub.com

fal_server p1ncm2_fs.csm.fub.com

 

********* Standby ********

log_archive_config DG_CONFIG=(p1ncm1,p1ncm2)

log_archive_dest_1 location=/u99/ORACLE/p1ncm2/archive valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=p1ncm2

log_archive_dest_2 service=p1ncm1_dgfal.csm.fub.com optional reopen=60 valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=p1ncm1

fal_client p1ncm2_fc.csm.fub.com

fal_server p1ncm1_fs.csm.fub.com

 

 

SOLUTION

 standby

 alter system set fal_client=p1ncm2_dgfal.csm.fub.com scope=both sid='*';

 

6.     處理

 

-修改主庫的fal_client參數，與備庫保持一致

--修改前須要進行驗證

SQL> host tnsping wxcdrac2

SQL> host tnsping wxcdrac

 

SQL> alter system set fal_client=wxcdrac;

 --第二次，在其它客戶遇到一樣報錯，問題是tnsping 主庫fal_client不通，端口錯誤。

 

7.     補充說明

 MOS ID 288074.1

Versions 11.2, 12.1  對於11g12c而言，極可能此消息是一個提示做用

 

Error:  ORA-16401 archive log rejected by Remote File Server (RFS)

---------------------------------------------------------------------------

Cause:  An attempt was made to re-archive an existing archive log.

Action: See alert log and trace file for more details. No action is necessary.

    This is an informational statement provided to record the event for

    diagnostic purposes.

Versions 9.2, 10.1, 10.2, 11.1

 

Error:  ORA-16401 archivelog rejected by RFS  對於10g而言，此報錯，極可能爲fal_client參數配置錯誤

---------------------------------------------------------------------------

Cause:  An attempt was made to re-archive an existing archivelog. This usually

    happens because either a multiple primary database or standby

    database(s) or both are trying to archive to this standby database.

Action: See alert log and trace file for more details. No action is necessary;

    this is an informational statement provided to record the event for

    diagnostic purposes.