從新配置ocr voting

    因爲存儲空間不足，下線的數據庫須要把存儲空間騰出來，關閉集羣資源，主機工程師收回lun須要（包括ocr 和 voting data 磁盤組），新的應用須要上線須要新的數據庫，新的hitach存儲到位須要從新安裝數據庫，上次刪除gi 和database軟件都在，此次只須要從新配置便可。參考How to Deconfigure/Reconfigure(Rebuild OCR) or Deinstall Grid Infrastructure (文檔 ID 1377349.1)

  

last node  刪除配置
[root@qdcx-db2 /]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force -verbose
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.crf' on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'qdcx-db2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'qdcx-db2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'qdcx-db2' succeeded
CRS-2677: Stop of 'ora.crf' on 'qdcx-db2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'qdcx-db2'
CRS-2677: Stop of 'ora.gipcd' on 'qdcx-db2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'qdcx-db2'
CRS-2677: Stop of 'ora.gpnpd' on 'qdcx-db2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'qdcx-db2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'qdcx-db2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
Successfully deconfigured Oracle clusterware stack on this node

first node
[root@qdcx-db1 /]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force -verbose   -lastnode
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
CRS-5702: Resource 'ora.cssd' is already running on 'qdcx-db1'
CRS-4000: Command Start failed, or completed with errors.
CSS startup failed with return code 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-2673: Attempting to stop 'ora.ctssd' on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.asm' on 'qdcx-db1'
CRS-2677: Stop of 'ora.ctssd' on 'qdcx-db1' succeeded
CRS-2677: Stop of 'ora.asm' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.cssd' on 'qdcx-db1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'qdcx-db1'
CRS-2676: Start of 'ora.cssdmonitor' on 'qdcx-db1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'qdcx-db1'
CRS-2672: Attempting to start 'ora.diskmon' on 'qdcx-db1'
CRS-2676: Start of 'ora.diskmon' on 'qdcx-db1' succeeded
CRS-2676: Start of 'ora.cssd' on 'qdcx-db1' succeeded
This ASM diskgroup does not contain voting disks to be deleted
ASM de-configuration trace file location: /tmp/asmcadc_clean2018-07-16_04-34-19-PM.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2018-07-16_04-34-19-PM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'qdcx-db1'
CRS-2677: Stop of 'ora.ctssd' on 'qdcx-db1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.mdnsd' on 'qdcx-db1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.gipcd' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.gpnpd' on 'qdcx-db1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'qdcx-db1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
Successfully deconfigured Oracle clusterware stack on this node

從新配置
節點1
[root@qdcx-db1 /]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
This may take several minutes. Please wait ...
/oracle/app/crs/install/usm/V7x/powerpc/bin//udefacfsctl[75]: /oracle/app/crs/install/usm/V7x/powerpc/bin//udefacfsctl.bin: 0403-006 Execute permission denied.
ACFS-9361: Removing device 'acfsctl' failed with error code '32256'.
Unable to uninstall ADVM/ACFS

解決
chmod -R  755 /oracle/app/crs/install/usm/V7x/powerpc/bin

/oracle/app/crs/crs/install/rootcrs.pl -deconfig -force
/oracle/app/crs/root.sh

節點2
[root@qdcx-db2 /etc/oracle]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
****Unable to retrieve Oracle Clusterware home.
Start Oracle Clusterware stack and try again.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CLSU-00100: Operating System function: opendir failed with error data: 2
CLSU-00101: Operating System error message: No such file or directory
CLSU-00103: error location: scrsearch1
CLSU-00104: additional error information: cant open scr home dir scls_scr_getval
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
This may take several minutes. Please wait ...
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Successfully deconfigured Oracle clusterware stack on this node


[root@qdcx-db2 /etc/oracle]# /oracle/app/crs/root.sh
Performing root user operation for Oracle 11g 

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /oracle/app/crs

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'qdcx-db2'
CRS-2676: Start of 'ora.mdnsd' on 'qdcx-db2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'qdcx-db2'
CRS-2676: Start of 'ora.gpnpd' on 'qdcx-db2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'qdcx-db2'
CRS-2672: Attempting to start 'ora.gipcd' on 'qdcx-db2'
CRS-2676: Start of 'ora.cssdmonitor' on 'qdcx-db2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'qdcx-db2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'qdcx-db2'
CRS-2672: Attempting to start 'ora.diskmon' on 'qdcx-db2'
CRS-2676: Start of 'ora.diskmon' on 'qdcx-db2' succeeded
CRS-2676: Start of 'ora.cssd' on 'qdcx-db2' succeeded

Disk Group ocr creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/rhdisk6' matches no disks
ORA-15025: could not open disk "/dev/rhdisk6"
ORA-27041: unable to open file


Configuration of ASM ... failed
see asmca logs at /oracle/app/grid/cfgtoollogs/asmca for details
Did not succssfully configure and start ASM at /oracle/app/crs/crs/install/crsconfig_lib.pm line 6912.
/oracle/app/crs/perl/bin/perl -I/oracle/app/crs/perl/lib -I/oracle/app/crs/crs/install /oracle/app/crs/crs/install/rootcrs.pl execution failed

查看hdisk6 是否同一個lun
[root@qdcx-db1 /]# lscfg -vpl hdisk6
  hdisk6           U5802.001.RCH5723-P1-C2-T1-W50060E8016609092-L4000000000000  Hitachi Disk Array (Fibre)

        Manufacturer................HITACHI 
        Machine Type and Model......OPEN-V          
        Part Number.................. .....
        ROS Level and ID............37303036
        Serial Number...............50 16090
        EC Level....................
        FRU Number..................
        Device Specific.(Z0)........00000332EF000002
        Device Specific.(Z1)........0B37 AC ....
        Device Specific.(Z2)............
        Device Specific.(Z3).........
        Device Specific.(Z4)............
        Device Specific.(Z5)........ 
        Device Specific.(Z6)........


  PLATFORM SPECIFIC

  Name:  disk
    Node:  disk
    Device Type:  block


[root@qdcx-db2 /]# lscfg -vpl hdisk6
  hdisk6           U5802.001.RCH4330-P1-C2-T1-W50060E8016609092-L4000000000000  Hitachi Disk Array (Fibre)

        Manufacturer................HITACHI 
        Machine Type and Model......OPEN-V          
        Part Number.................. .....
        ROS Level and ID............37303036
        Serial Number...............50 16090
        EC Level....................
        FRU Number..................
        Device Specific.(Z0)........00000332EF000002
        Device Specific.(Z1)........0B37 AC ....
        Device Specific.(Z2)............
        Device Specific.(Z3).........
        Device Specific.(Z4)............
        Device Specific.(Z5)........ 
        Device Specific.(Z6)........


  PLATFORM SPECIFIC

  Name:  disk
    Node:  disk
    Device Type:  block

從Device Specific.(Z1)........0B37 AC可知是同一個lun
查看hdisk6磁盤屬性
[root@qdcx-db2 /]# ls -l /dev/rhdisk6
crw-rw----    1 grid     asmadmin     15,  5 Jul 16 14:39 /dev/rhdisk6
[+ASM2]@qdcx-db2[/home/grid]$lsattr -E -l hdisk6|grep reserve_policy
reserve_policy PR_exclusive          Reserve Policy                   True
磁盤屬性reserve_policy不對，系統管理員忘記修改，運行腳本修改
[root@qdcx-db2 /tmp]# cat chmodall
begin=18
end=22

i=$begin
while [ $i -le $end ];
do
        chmod 660 /dev/rhdisk$i
        chown grid:asmadmin /dev/rhdisk$i
        chdev -l hdisk$i -a reserve_policy=no_reserve
        chdev -l hdisk$i -a queue_depth=8
        i=`expr $i + 1`
done

ls -l /dev/rhdisk*
繼續運行root.sh 此次沒報錯