測試一體機ASM Disk online操做

環境：3臺虛擬機 RHEL 7.3 + Oracle RAC 11.2.0.4
問題現象：RAC運行正常，ASM磁盤組Normal冗餘，節點主機重啓，offline狀態的asm disk短期內能夠直接online。

在《測試一體機ASM failgroup的相關問題處理》以前的文章，描述了從新添加磁盤的場景，其實若是故障發現及時（默認3.6h內），是能夠直接online對應磁盤的，這時候現象相似以下：

SQL> select group_number, disk_number, name, path, failgroup, mode_status, voting_file  from v$asm_disk order by 1, 2;
GROUP_NUMBER DISK_NUMBER NAME                           PATH                                FAILGROUP                      MODE_STATUS    VO
------------ ----------- ------------------------------ ----------------------------------- ------------------------------ -------------- --
           0           0                                /dev/CELL01-data2                                                  ONLINE         N
           0           1                                /dev/CELL01-data1                                                  ONLINE         N
           0           2                                /dev/CELL01-crs1                                                   ONLINE         Y
           1           0 CRS_0000                                                           CRS_0000                       OFFLINE        N
           1           1 CRS_0001                       /dev/CELL02-crs2                    CRS_0001                       ONLINE         Y
           1           2 CRS_0002                       /dev/CELL03-crs3                    CRS_0002                       ONLINE         Y
           2           0 DATA_0000                      /dev/CELL03-data1                   CELL03                         ONLINE         N
           2           1 DATA_0001                      /dev/CELL03-data2                   CELL03                         ONLINE         N
           2           2 DATA_0002                      /dev/CELL02-data1                   CELL02                         ONLINE         N
           2           3 DATA_0003                      /dev/CELL02-data2                   CELL02                         ONLINE         N
           2           4 DATA_0004                                                          CELL01                         OFFLINE        N
           2           5 DATA_0005                                                          CELL01                         OFFLINE        N

12 rows selected.

這種狀況就能夠直接online對應磁盤:

SQL> alter diskgroup CRS online disk CRS_0000;

Diskgroup altered.

SQL> alter diskgroup DATA online disk DATA_0004,DATA_0005; 
alter diskgroup DATA online disk DATA_0004,DATA_0005
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15282: ASM disk "DATA_0005" is not visible cluster-wide
ORA-15282: ASM disk "DATA_0004" is not visible cluster-wide

直接online對應磁盤若是遇到上面這個報錯，由於有其餘節點沒有看到要online的磁盤，檢查其餘節點：

[root@db02 ~]# ls -l /dev/CELL*
lrwxrwxrwx 1 root root 3 Dec  2 21:16 /dev/CELL01-crs1 -> sdc
lrwxrwxrwx 1 root root 3 Dec  2 21:16 /dev/CELL02-crs2 -> sdi
lrwxrwxrwx 1 root root 3 Dec  2 21:16 /dev/CELL02-data1 -> sdj
lrwxrwxrwx 1 root root 3 Dec  2 21:16 /dev/CELL02-data2 -> sdk
lrwxrwxrwx 1 root root 3 Dec  2 21:16 /dev/CELL03-crs3 -> sdd
lrwxrwxrwx 1 root root 3 Dec  2 21:16 /dev/CELL03-data1 -> sdf
lrwxrwxrwx 1 root root 3 Dec  2 21:16 /dev/CELL03-data2 -> sdh

此時先要確認lsscsi底層正常（若是不正常先解決iscsi層的問題，我這裏是正常的）：

[root@db02 ~]# lsscsi
[1:0:0:0]    cd/dvd  VBOX     CD-ROM           1.0   /dev/sr0 
[2:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sda 
[3:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sdb 
[4:0:0:0]    disk    LIO-ORG  disk1            4.0   /dev/sdc 
[4:0:0:1]    disk    LIO-ORG  disk2            4.0   /dev/sde 
[4:0:0:2]    disk    LIO-ORG  disk3            4.0   /dev/sdg 
[5:0:0:0]    disk    LIO-ORG  disk1            4.0   /dev/sdd 
[5:0:0:1]    disk    LIO-ORG  disk2            4.0   /dev/sdf 
[5:0:0:2]    disk    LIO-ORG  disk3            4.0   /dev/sdh 
[6:0:0:0]    disk    LIO-ORG  disk1            4.0   /dev/sdi 
[6:0:0:1]    disk    LIO-ORG  disk2            4.0   /dev/sdj 
[6:0:0:2]    disk    LIO-ORG  disk3            4.0   /dev/sdk

從新刷新udev規則，確認全部磁盤識別正確：

[root@db02 ~]# udevadm control --reload
[root@db02 ~]# udevadm trigger
[root@db02 ~]# ls -l /dev/CELL*
lrwxrwxrwx 1 root root 3 Dec  2 21:18 /dev/CELL01-crs1 -> sdc
lrwxrwxrwx 1 root root 3 Dec  2 21:18 /dev/CELL01-data1 -> sde
lrwxrwxrwx 1 root root 3 Dec  2 21:18 /dev/CELL01-data2 -> sdg
lrwxrwxrwx 1 root root 3 Dec  2 21:18 /dev/CELL02-crs2 -> sdi
lrwxrwxrwx 1 root root 3 Dec  2 21:18 /dev/CELL02-data1 -> sdj
lrwxrwxrwx 1 root root 3 Dec  2 21:18 /dev/CELL02-data2 -> sdk
lrwxrwxrwx 1 root root 3 Dec  2 21:18 /dev/CELL03-crs3 -> sdd
lrwxrwxrwx 1 root root 3 Dec  2 21:18 /dev/CELL03-data1 -> sdf
lrwxrwxrwx 1 root root 3 Dec  2 21:18 /dev/CELL03-data2 -> sdh

而後再次進行online磁盤成功：

SQL> alter diskgroup DATA online disk DATA_0004,DATA_0005; 

Diskgroup altered.

這種能夠直接online disk的狀況就免去了ASM磁盤組rebalance的大量時間，因此這類問題發現後最好要及時聯繫工程師進行處理。