經過udev建立ASM共享磁盤（RAC）

時間 2020-07-06

標籤經過 udev 建立 asm 共享磁盤 rac 简体版

原文原文鏈接

OS：RedHat EL6.0 node

Oracle： Oracle 11gR2 sql

在Oracle 11gR2，構建RAC時能夠經過ASM建立asm disk，可是須要安裝asmlib相關軟件；對於RedHat EL6.0，Oracle的asm不能很好的支持，因此能夠經過udev來建立asm disk，就不須要安裝asmlib相關軟件。數據庫

1、查看系統環境 vim

內核版本：服務器

[root@rac01 rules.d]# uname -a oracle

Linux rac01 2.6.32-642.6.2.el6.x86_64 #1 SMP Wed Oct 26 06:52:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux app

系統用戶： dom

[root@rac01 rules.d]# id grid tcp

uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1300(dba),1200(asmadmin),1201(asmdba),1202(asmoper) ui

2、建立共享磁盤

在Windows server 2012R2 服務器上啓用iSCSI服務，並新建iSCSI共享磁盤

選擇有足夠容量的卷

給虛擬磁盤命名並劃分大小，而後配置iSCSI目標

命名訪問目標，添加訪問程序，選擇"IP地址類型"，輸入RAC 節點IP（將兩個節點的IP都添加上）

繼續下一步，不啓用驗證服務，確認建立，建立完成後會在管理頁面顯示出已經建立好的共享磁盤

3、配置共享存儲

在節點1和節點2上分別安裝iSCSI Initiator，以節點一爲例，主要涉及如下命令

yum -y install iscsi-initiator-utils

chkconfig iscsid on

chkconfig iscsi on

最後發現而且掛載iscsi target

[root@rac01 rules.d]# iscsiadm -m discovery -t sendtargets -p 192.168.2.70

192.168.2.70:3260,1 iqn.1991-05.com.microsoft:win-ct207djdp08-max-002-target

192.168.2.70:3260,1 iqn.1991-05.com.microsoft:win-ct207djdp08-test1-target

[root@rac01 rules.d]# iscsiadm -m node -T iqn.1991-05.com.microsoft:win-ct207djdp08-max-002-target --login

成功後，用fdisk -l命令能夠看到多出的共享磁盤，進行分區，分區結果以下圖

4、創建UDEV規則文件

查看磁盤UUID：

[root@rac01 rules.d]# /sbin/scsi_id -g -u -d /dev/sdb

360003ff44dc75adca44127e408e97955

建立UDEV配置文件：

[root@rac01 rules.d]# cd /etc/udev/rules.d/

[root@rac01 rules.d]# vim 99-oracle-asmdisk.rules

KERNEL=="sdb5", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sdb6", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk2", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sdb7", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk3", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sdb8", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk4", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sdb9", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk5", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sdb10", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk6", OWNER="grid", GROUP="asmadmin", MODE="0660"

5、啓動UDEV service

載入配置文件：

[root@rac01 rules.d]# udevadm control --reload-rules

啓動UDEV service:

[root@rac01 rules.d]# start_udev

查看asm disks:

[root@rac01 rules.d]# ls -l /dev/asm*

brw-rw---- 1 grid asmadmin 8, 21 Nov 25 11:22 /dev/asm_disk1

brw-rw---- 1 grid asmadmin 8, 22 Nov 25 11:22 /dev/asm_disk2

brw-rw---- 1 grid asmadmin 8, 23 Nov 25 11:22 /dev/asm_disk3

brw-rw---- 1 grid asmadmin 8, 24 Nov 25 11:22 /dev/asm_disk4

brw-rw---- 1 grid asmadmin 8, 25 Nov 25 11:22 /dev/asm_disk5

brw-rw---- 1 grid asmadmin 8, 26 Nov 25 11:22 /dev/asm_disk6

6、添加額外ASM磁盤

安裝完RAC環境後，忽然以爲以前配置的ASM磁盤不夠用，因而打算再添加一塊大容量的ASM磁盤；前面操做都無差別，這裏補充後續操做。

查看asm disks:

[root@rac01 rules.d]# ls -l /dev/asm*

brw-rw---- 1 grid asmadmin 8, 21 Nov 25 11:22 /dev/asm_disk1

brw-rw---- 1 grid asmadmin 8, 22 Nov 25 11:22 /dev/asm_disk2

brw-rw---- 1 grid asmadmin 8, 23 Nov 25 11:22 /dev/asm_disk3

brw-rw---- 1 grid asmadmin 8, 24 Nov 25 11:22 /dev/asm_disk4

brw-rw---- 1 grid asmadmin 8, 25 Nov 25 11:22 /dev/asm_disk5

brw-rw---- 1 grid asmadmin 8, 26 Nov 25 11:22 /dev/asm_disk6

brw-rw---- 1 grid asmadmin 8, 33 Nov 25 09:42 /dev/asm_disk7

在節點1中，使用grid用戶運行asmca:

成功彈出ASM設置界面，選擇Create

輸入磁盤組名，選擇External（外部冗餘）（由於已經存在常規冗餘磁盤組，爲了省事兒這裏選擇外部冗餘），選擇磁盤：

點擊OK

點擊OK回到初始界面，點擊右下角Exit，而後選擇Yes保留操做並退出

使用oracle 帳戶登陸數據庫並查詢ASM磁盤組信息：

[root@rac01 ~]# su - oracle

[oracle@rac01 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:09:54 2016

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select name,total_mb from v$asm_diskgroup;

NAME TOTAL_MB

------------------------------ ----------

DATA_DB 16378

DATA 6144

CY 102398

7、額外問題

完成上面的操做後，發現節點2出現了一些問題

[oracle@rac02 admin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:13:45 2016

Connected to an idle instance.

SQL> select name,total_mb from v$asm_diskgroup;

select name,total_mb from v$asm_diskgroup

ERROR at line 1:

ORA-01034: ORACLE not available

Process ID: 0

Session ID: 0 Serial number: 0

實例彷佛被關閉了，重啓實例後，發現新加的ASM磁盤組掛載異常

SQL> startup

ORACLE instance started.

Total System Global Area 1603411968 bytes

Fixed Size 2253664 bytes

Variable Size 1056967840 bytes

Database Buffers 536870912 bytes

Redo Buffers 7319552 bytes

Database mounted.

Database opened.

SQL> select name,total_mb from v$asm_diskgroup;

NAME TOTAL_MB

------------------------------ ----------

DATA_DB 16378

DATA 6144

CY 0

[grid@rac02 admin]$ ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 3

Total space (kbytes) : 262120

Used space (kbytes) : 3072

Available space (kbytes) : 259048

ID : 277519239

Device/File Name : +DATA

Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

手動掛載：

[grid@rac02 admin]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:30:30 2016

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup CY mount;

Diskgroup altered.

SQL> select GROUP_NUMBER,NAME,TYPE,ALLOCATION_UNIT_SIZE,STATE from v$asm_diskgroup;

GROUP_NUMBER NAME TYPE ALLOCATION_UNIT_SIZE STATE

------------ ------------------------------ ------ --------------------

1 DATA_DB NORMAL 1048576 MOUNTED

2 DATA NORMAL 1048576 MOUNTED

3 CY EXTERN 1048576 MOUNTED

再次查詢ASM磁盤組信息：

[oracle@rac02 admin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:40:10 2016

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select name,total_mb from v$asm_diskgroup;

NAME TOTAL_MB

------------------------------ ----------

DATA_DB 16378

DATA 6144

CY 102398

完成後發現CRS服務異常：

[grid@rac02 admin]$ crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4534: Cannot communicate with Event Manager

排查後發現是硬盤名稱發生了變化，修改完相關配置文件（vim /etc/udev/rules.d/99-oracle-asmdisk.rules），CRS服務正常：

[grid@rac02 ~]$ crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

而後發現由於以前的異常操做，致使監聽異常：

[grid@rac02 admin]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 25-NOV-2016 15:19:25

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))

STATUS of the LISTENER

------------------------

Alias LISTENER

Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production

Start Date 25-NOV-2016 15:11:22

Uptime 0 days 0 hr. 8 min. 2 sec

Trace Level off

Security ON: Local OS Authentication

SNMP OFF

Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora

Listener Log File /u01/app/11.2.0/grid/log/diag/tnslsnr/rac02/listener/alert/log.xml

Listening Endpoints Summary...

(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.209)(PORT=1521)))

Services Summary...

Service "+ASM" has 1 instance(s).

Instance "+ASM2", status READY, has 1 handler(s) for this service...

Service "rac" has 1 instance(s).

Instance "rac2", status READY, has 1 handler(s) for this service...

Service "racXDB" has 1 instance(s).

Instance "rac2", status READY, has 1 handler(s) for this service...

The command completed successfully

以節點1爲例

修改兩個節點的監聽文件：

[root@rac01 ~]# su - grid

[grid@rac01 ~]$ cd /u01/app/11.2.0/grid/network/admin

[grid@rac01 admin]$ ls

endpoints_listener.ora listener.ora shrept.lst

endpoints_listener.ora.bak.rac01 listener.ora.bak.rac01 sqlnet.ora

listener16112111AM0504.bak samples

[grid@rac01 admin]$ vim endpoints_listener.ora

#將HOST參數改成本機IP

LISTENER_RAC01=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.2.206)(PORT=1521)(IP=FIRST))))

修改TNS配置文件：

[oracle@rac01 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/network/admin

[oracle@rac01 admin]$ ls

samples shrept.lst tnsnames.ora

[oracle@rac01 admin]$ vim tnsnames.ora

RAC =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = max-cluster)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = rac)

)