RAC升級後，一個節點沒法鏈接數據庫，報ORA-12537: TNS:connection closed

RAC從11.2.0.3升級到11.2.0.4後，一個節點的Public IP、VIP沒法鏈接數據庫

SQL> CONN SYS/oracle@192.168.122.101:1521/pplus as sysdba
ERROR:
ORA-12537: TNS:connection closed

 

查看RAC的監聽日誌：

# cd $ORACLE_BASE/diag/tnslsnr/racdb2/listener/trace/

# vi listener.log

06-MAY-2019 12:13:52 * (CONNECT_DATA=(SERVICE_NAME=posplus)(CID=(PROGRAM=sqlplus)(HOST=racdb1)(USER=grid))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.122.100)(PORT=40081)) * establish * posplus * 12518
TNS-12518: TNS:listener could not hand off client connection
  TNS-12546: TNS:permission denied
    TNS-12560: TNS:protocol adapter error
     TNS-00516: Permission denied
        Linux Error: 13: Permission denied

首先想到GRID和Database的是oracle文件權限的問題，都有s權限，確認沒有問題

[grid@racdb1 ~]$ ll $ORACLE_HOME/bin/oracle

-rwsr-s--x 1 grid oinstall 209914479 Mar 25 23:39 /home/grid/app/11.2.0.4/grid/bin/oracle

[oracle@racdb1 ~]$ ll $ORACLE_HOME/bin/oracle

-rwsr-s--x 1 oracle asmadmin 239626641 Mar 26 01:36 /home/oracle/app/oracle/product/11.2.0.4/db_1/bin/oracle

 

而後再用GRID用戶去訪問Database的HOME目錄

[grid@racdb1 ~]$ cd $ORACLE_HOME/bin
-bash: cd: $ORACLE_HOME/bin: Permission denied

果真沒有權限

[grid@racdb2 ~]$ ll /home/
total 8
drwxr-xr-x. 7 root oinstall 4096 Mar 23 16:43 grid
drwx------. 8 oracle oinstall 4096 May 6 11:45 oracle

把/home/oracle的權限修改成

chmod g+rx,o+rx /home/oracle

 

[grid@racdb1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed May 8 10:41:06 2019

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> CONN SYS/oracle@192.168.122.111:1521/posplus as sysdba
Connected.
SQL> CONN SYS/oracle@192.168.122.101:1521/posplus as sysdba
Connected.
SQL> CONN SYS/oracle@192.168.122.110:1521/posplus as sysdba
Connected.
SQL> CONN SYS/oracle@192.168.122.120:1521/posplus as sysdba

至此問題已解決

 

-------------------------------------------------------------------------------

參考Oracle官方文檔：

ORA-12537 / ORA-12547 or TNS-12518 if Listener (including SCAN Listener) and Database are Owned by Different OS User (文檔 ID 1069517.1)

轉到底部

---

In this Document   Symptoms   Cause   Solution   References   APPLIES TO: Oracle Database - Enterprise Edition - Version 11.2.0.3 and later Oracle Database Configuration Assistant - Version 11.1.0.7 and later Oracle Net Services - Version 11.2.0.2 and later Information in this document applies to any platform. SYMPTOMS In environment where listener home (including SCAN listener which resides in GRID Infrastructure/ASM home) and database home are owned by different OS user, ORA-12537 could happen when connecting through listener, when creating database through DBCA, or when installing database software and creating a database in runInstaller. Job Role Separation is a typical example as SCAN and local grid home listener is owned differently than database. Error detail 12537, 00000, "TNS:connection closed" // *Cause: "End of file" condition has been reached; partner has disconnected. // *Action: None needed; this is an information message. Before the error is reported, connection could hang for a while, and a core file may also be generated. Screen output: SQL> conn system/xxx@OSES ERROR: ORA-12537: TNS:connection closed OR PRCR-1079 : Failed to start resource ora.db11.db CRS-5011: Check of resource db11 failed: details at .. CRS-2674: Start of ora.db11.db on racnode1 failed ORA-12546: TNS:permission denied listener_scann.log or listener.log TNS-12518: TNS:listener could not hand off client connection TNS-12546: TNS:permission denied  TNS-12560: TNS:protocol adapter error   TNS-00516: Permission denied    Linux Error: 13: Permission denied OR TNS-12518: TNS:listener could not hand off client connection TNS-12547: TNS:lost contact  TNS-12560: TNS:protocol adapter error   TNS-00517: Lost contact    Linux Error: 32: Broken pipe runInstaller error if installing INFO: Starting Output Reader Threads for process /ocw/grid/bin/kfod INFO: Parsing KFOD-00300: OCI error [-1] [OCI error] [ORA-12547: TNS:lost contact INFO: Parsing ] [12547] INFO: Parsing INFO: The process /ocw/grid/bin/kfod exited with code 1 .. SEVERE: [FATAL] [INS-30502] No ASM disk group found.    CAUSE: There were no disk groups managed by the ASM instance +ASM1. CAUSE 1. In environments where the listener is not run in the same ORACLE_HOME where the database resides, the listener owner (including SCAN listener) may not be able to access the oracle binary in the database home.  This is common in RAC or whenever a GRID_HOME and a database ORACLE_HOME are installed. As listener owner: $ ls -l $RDBMS_HOME/bin/oracle ls: /home/oracle/app/oracle/product/11.2/db/bin/oracle:  Permission denied 2. Oracle binary in database home has wrong permission: ls -l $RDBMS_HOME/bin/oracle -rwxr-x--x 1 oracle asmadmin 184286251 Aug  9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle The permission "-rwxr-x--x" is wrong as it's missing suid bit, oracle binary should have permission of 6751:  -rwsr-s--x 1 oracle asmadmin 184286251 Aug  9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle Note: If Job Role Separation is in place, the group will be <asmadmin>, otherwise it will be <oinstall> 3. File System for database home does not support setuid/suid or has nosetuid/nosuid set: mount| grep <mount_point_of_ORACLE_HOME> /home/oracle on /dev/dsk/diskoracle read/write/nosuid..   4. RDBMS_HOME/lib has wrong ownership/permission: As listener owner: $ ls -l $RDBMS_HOME/lib ls: /home/oracle/app/oracle/product/11.2/db/lib:  Permission denied   5. Another cause may be permissions on the RDBMS Directory structure or Home directory which needs to be accessed by the CRS user. Check that the RDBMS $ORACLE_HOME is set to 755. This can be seen from an OS trace such as strace or truss when using it to trace the CRS user running the "oracle" executable which fails with the "Permission denied" error. Also: a) Log in as the "GRID" user on each node, and issue the following (on each directory under the RDBMS Home) :-   [grid@orcl002:+ASM2 ~]$ ls -al /home/oracle/app/oracle ls: cannot open directory /u01/app/oracle: Permission denied ***NOTE: the Oracle directory has 700 for the permissions, which should be changed to 755:   Current (incorrect) --> [grid@orcl002:+ASM2 ~]$ ll /home/oracle/app drwx------. 8 oracle oinstall  4096 Oct 12 08:38 oracle   Should be (correct) --> [grid@orcl002:+ASM2 ~]$ ll /home/oracle/app drwxr-xr-x. 8 oracle oinstall  4096 Oct 12 08:38 oracle     b) Likewise the /product directory has 700 perms, so change to 755 --> [grid@orcl002:+ASM2 ~]$ ls -al /home/oracle/app/oracle/product ls: cannot open directory /u01/app/oracle/product: Permission denied Current (incorrect) --> [grid@orcl002:+ASM2 ~]$ ls -al /home/oracle/app/oracle drwx------.  3 oracle oinstall 4096 Oct  12 08:58 product   Should be (correct) --> [grid@orcl002:+ASM2 ~]$ ls -al /u01/app/oracle drwxr-xr-x.  3 oracle oinstall 4096 Oct  12 08:58 product c) Now a connection should work: $ sqlplus system/<pwd>@'(description=(address=(protocol=tcp)(host=orcl002)(port=1521))(connect_data=(server=dedicated)(service_name=ORCL)))' SQL>   SOLUTION Solution is to make sure file system for database home has setuid/suid set, database binary($RDBMS_HOME/bin/oracle) has correct ownership and permission, and listener owner is able to access database oracle binary (as listener owner, "ls -l $RDBMS_HOME/bin/oracle" will tell)If its Job Role Separation environment, please refer to the following document:Oracle® Grid Infrastructure Installation Guide11g Release 2 (11.2)Oracle ASM Job Role Separation Option with SYSASM