歸檔日誌滿處理（OS Audit file could not be created; failing after 6 retries）

早上同事說測試環境的數據庫連不上了。檢查alert日誌，發現報錯以下：

************************************************************************
ARC3: Error 19809 Creating archive log file to '/u01/app/oracle/fast_recovery_area/SGDSDB/archivelog/2017_05_31/o1_mf_1_4695_%u_.arc'
ARC1: Error 19809 Creating archive log file to '/u01/app/oracle/fast_recovery_area/SGDSDB/archivelog/2017_05_31/o1_mf_1_4694_%u_.arc'
Errors in file /u01/app/oracle/diag/rdbms/sgdsdb/sgdsdb/trace/sgdsdb_arc0_5237.trc:
ORA-19815: WARNING: db_recovery_file_dest_size of 104857600000 bytes is 100.00% used, and has 0 remaining bytes available.
Errors in file /u01/app/oracle/diag/rdbms/sgdsdb/sgdsdb/trace/sgdsdb_arc1_5239.trc:
ORA-19815: WARNING: db_recovery_file_dest_size of 104857600000 bytes is 100.00% used, and has 0 remaining bytes available.
************************************************************************
************************************************************************
You have following choices to free up space from recovery area:
You have following choices to free up space from recovery area:
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
   then consider changing RMAN ARCHIVELOG DELETION POLICY.
   then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN
2. Back up files to tertiary device such as tape using RMAN
   BACKUP RECOVERY AREA command.
   BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to
3. Add disk space and increase db_recovery_file_dest_size parameter to
   reflect the new space.
   reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating
4. Delete unnecessary files using RMAN DELETE command. If an operating
   system command was used to delete files, then use RMAN CROSSCHECK and
   system command was used to delete files, then use RMAN CROSSCHECK and   DELETE EXPIRED commands.

   DELETE EXPIRED commands.
************************************************************************

 

本覺得是很簡單的歸檔日誌滿形成的報錯，按照以往的操做過程，先把db_recovery_file_dest對應的好久以前的歸檔刪除，再等數據庫本身把沒有歸檔的日誌進行恢復。

結果刪除完歸檔後，發現仍然報錯。sqlplus和rman也沒法進入。

由於是測試環境，因此就直接將pmon進程幹掉，而後將數據庫啓動到nomount狀態，結果發現仍然報錯：

OS Audit file could not be created; failing after 6 retries
OS Audit file could not be created; failing after 6 retries
Wed May 31 10:57:32 2017
Performing implicit shutdown abort due to dead PMON
Shutting down instance (abort)

這個應該是相應的audit的目錄沒有健全，或者oracle的執行文件的權限不對。

首先檢查權限：

[oracle@dsdev trace]$ ll $ORACLE_HOME/bin/oracle
-rwsr-s--x. 1 oracle oinstall 239626731 Jul  8  2015 /u01/app/oracle/product/11.2/db_1/bin/oracle
[oracle@dsdev trace]$ 

權限是對的，接着檢查對應的目錄

[oracle@dsdev dbs]$ ls -l /u01/app/oracle/admin/sgdsdb/adump
ls: cannot access /u01/app/oracle/admin/sgdsdb/adump: No such file or directory
[oracle@dsdev dbs]$ 

發現audit_file_dest的目錄不存在，建立好以後，數據庫恢復正常。

隨後在rman中crosscheck archivelog all和delete expired copy 刪除過時和無效的歸檔日誌。

 

另外，在隨後的監聽啓動中發現

啓動監聽，報錯

[oracle@dsdev ~]$ lsnrctl start

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 31-MAY-2017 14:02:02

Copyright (c) 1991, 2009, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dsdev)(PORT=1521)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused

直接用lsnrctl status仍然報錯，去查看相應的log無具體提示。

經檢查發現是hosts裏面的127.0.0.1對應的localhost被修改掉，從新添加127.0.0.1 localhost後正常(這個和安裝時沒有設置localhost有殊途同歸之妙)。