asm學習之rebalance

時間 2019-11-19

標籤 asm 學習 rebalance 简体版

原文原文鏈接

rebalacne場景：

向asm磁盤組中添加，刪除，resize等操做都將會引發asm實例rebalacne。html

reblaacne過程:

第一階段planning:

　　計算出rebalance的計劃，會根據磁盤大小個數，磁盤吞吐，au大小等計算出大體計劃，該過程通常只需幾分鐘。ios

第二階段extent relocating：

　　真正進行重平衡的過程，將extent均勻的分配在各個磁盤中。該過程一般是整個rebalance最耗時的過程。數據庫

第三階段compacting:

　　asm11.1.0.7以上纔開始支持，將磁盤上存的數據儘量的移動到磁盤的外圈磁道上去（機械盤的外圈速度更快），花費時間較少。　　oracle

rebalance實際大小以及所需時間async

　　磁盤組的rebalance何時能完成？這多是不少人最關心的問題。一般狀況下能夠經過gv$asm_operation.est_minutes來知道還剩下幾分鐘能夠完成rebalance雖然這個值並不許確。ide

作個小實驗體驗下整個過程：

首先了解個公式函數

Formula: (TAU/N)*N-1
Where:
TAU= Total Allocation Unit to move.
N=number of disks3d

查看磁盤組多少個extentrest

13:59:11 SQL> Select count(pxn_kffxp) Extents, disk_kffxp disk_number, group_kffxp group_number from x$kffxp where group_kffxp=6 group by disk_kffxp, group_kffxp order by group_kffxp,disk_kffxp;日誌

EXTENTS DISK_NUMBER GROUP_NUMBER
---------- ----------- ------------
194 0 6

Elapsed: 00:00:00.00

13:59:16 SQL> select (total_mb-free_mb) REQUIRED_MIRROR_FREE_MB,USABLE_FILE_MB, type,ALLOCATION_UNIT_SIZE/1024 from v$asm_diskgroup where name='TEST';

REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB TYPE ALLOCATION_UNIT_SIZE/1024
----------------------- -------------- ------------ -------------------------
200 457144 EXTERN 1024

Elapsed: 00:00:00.03

200m使用大小跟查出來194個extent大小基本相同

23:27:36 SQL> show parameter asm ---------------------- 查看asm rebalance並行度

NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
asm_power_limit integer 1

23:06:28 SQL> alter diskgroup archdg drop disk ARCHDG_0005;

Diskgroup altered.

14:01:09 SQL> select * from v$asm_operation; ---------------------能夠看到實際須要rebalance的extent是129=194/3*2

GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----------------------------------------------------------------------------------------
6 REBAL RUN 1 1 2 129 5098 0

在看下 EST_MINUTES字段

23:06:45 SQL> select INST_ID, OPERATION, STATE, POWER, SOFAR, EST_WORK, EST_RATE, EST_MINUTES from GV$ASM_OPERATION where GROUP_NUMBER=1;

INST_ID OPERATION STATE POWER SOFAR EST_WORK EST_RATE EST_MINUTES
---------- ---------- -------- ---------- ---------- ---------- ---------- -----------
2 REBAL WAIT 1
1 REBAL RUN 1 6166 53989 10157 4

這裏顯示4分鐘完成，能夠看到後臺日誌真正完成時間從23:06-23:13是7分鐘，這裏的數據較少因此結果還不是特別明顯。

SQL> alter diskgroup archdg drop disk ARCHDG_0005
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
Wed Jul 04 23:06:42 2018
GMON updating for reconfiguration, group 1 at 61 for pid 35, osid 30893
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: group 1 PST updated.
Wed Jul 04 23:06:42 2018
NOTE: membership refresh pending for group 1/0x8ab98994 (ARCHDG)
GMON querying group 1 at 62 for pid 18, osid 20082
SUCCESS: refreshed membership for 1/0x8ab98994 (ARCHDG)
SUCCESS: alter diskgroup archdg drop disk ARCHDG_0005
NOTE: Attempting voting file refresh on diskgroup ARCHDG
NOTE: Refresh completed on diskgroup ARCHDG. No voting file found.
NOTE: starting rebalance of group 1/0x8ab98994 (ARCHDG) at power 1
Starting background process ARB0
Wed Jul 04 23:06:48 2018
ARB0 started with pid=36, OS id=41870
NOTE: assigning ARB0 to group 1/0x8ab98994 (ARCHDG) with 1 parallel I/O
cellip.ora not found.
Wed Jul 04 23:11:33 2018
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
Wed Jul 04 23:11:36 2018
GMON updating for reconfiguration, group 1 at 63 for pid 37, osid 52733
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: group 1 PST updated.
SUCCESS: grp 1 disk ARCHDG_0005 emptied
NOTE: erasing header on grp 1 disk ARCHDG_0005
NOTE: process _x000_+asm1 (52733) initiating offline of disk 5.3916003717 (ARCHDG_0005) with mask 0x7e in group 1
NOTE: initiating PST update: grp = 1, dsk = 5/0xe9697985, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 64 for pid 37, osid 52733
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: PST update grp = 1 completed successfully
NOTE: initiating PST update: grp = 1, dsk = 5/0xe9697985, mask = 0x7e, op = clear
GMON updating disk modes for group 1 at 65 for pid 37, osid 52733
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: cache closing disk 5 of grp 1: ARCHDG_0005
NOTE: PST update grp = 1 completed successfully
GMON updating for reconfiguration, group 1 at 66 for pid 37, osid 52733
NOTE: cache closing disk 5 of grp 1: (not open) ARCHDG_0005
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: group 1 PST updated.
Wed Jul 04 23:11:36 2018
NOTE: membership refresh pending for group 1/0x8ab98994 (ARCHDG)
GMON querying group 1 at 67 for pid 18, osid 20082
GMON querying group 1 at 68 for pid 18, osid 20082
NOTE: Disk ARCHDG_0005 in mode 0x0 marked for de-assignment
SUCCESS: refreshed membership for 1/0x8ab98994 (ARCHDG)
NOTE: Attempting voting file refresh on diskgroup ARCHDG
NOTE: Refresh completed on diskgroup ARCHDG. No voting file found.
Wed Jul 04 23:13:57 2018
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 1/0x8ab98994 (ARCHDG)

此時可使用ls -lrth +ASM_arb0*找到最新的arb trace文件，該文件會帶有文件號相似+ASM1_arb0_70552.trc，其中能夠看到arb0進程正在遷移數據

*** 2018-07-04 23:24:07.940
ARB0 relocating file +ARCHDG.2115.980618919 (120 entries)

*** 2018-07-04 23:24:09.072
ARB0 relocating file +ARCHDG.2115.980618919 (120 entries)

*** 2018-07-04 23:24:11.826
ARB0 relocating file +ARCHDG.2115.980618919 (120 entries)

*** 2018-07-04 23:24:14.527
ARB0 relocating file +ARCHDG.2115.980618919 (120 entries)

經過pstack也能夠看到這個進程使用了

kfgbRebalExecute - kfdaExecute - kffRelocate函數

#pstack 70552
#0 0x00007fe8467bc644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x0000000002baea09 in skgfrliopo ()
#2 0x0000000002bae801 in skgfospo ()
#3 0x00000000086ba843 in skgfrwat ()
#4 0x0000000008599e5d in ksfdwtio ()
#5 0x000000000220019b in ksfdwat_internal ()
#6 0x0000000003edce17 in kfk_reap_ufs_async_io ()
#7 0x0000000003edcd5f in kfk_reap_ios_from_subsys ()
#8 0x0000000000adefa4 in kfk_reap_ios ()
#9 0x0000000003edc382 in kfk_io1 ()
#10 0x0000000003edbf28 in kfkRequest ()
#11 0x0000000003ee272e in kfk_transitIO ()
#12 0x0000000003e35d5e in kffRelocateWait ()
#13 0x0000000003e5cbf6 in kffRelocate ()
#14 0x0000000003dd22df in kfdaExecute ()
#15 0x0000000003eb563f in kfgbRebalExecute ()
#16 0x0000000003ea2414 in kfgbDriver ()
#17 0x00000000021a86d7 in ksbabs ()
#18 0x0000000003eb964c in kfgbRun ()
#19 0x00000000021ad44b in ksbrdp ()
#20 0x00000000023d3eef in opirip ()
#21 0x000000000167e7b5 in opidrv ()
#22 0x0000000001c58477 in sou2o ()
#23 0x00000000008472c2 in opimai_real ()
#24 0x0000000001c5e795 in ssthrdmain ()
#25 0x00000000008471b9 in main ()

23:09:13 SQL> /

INST_ID OPERATION STATE POWER SOFAR EST_WORK EST_RATE EST_MINUTES
---------- ---------- -------- ---------- ---------- ---------- ---------- -----------
2 REBAL WAIT 1
1 REBAL RUN 1 53989 53989 0 0

當剩餘0min中的時候實際上是進入了第三階段compact
一樣可使用pstack看到使用了kfdCompact 函數

#pstack 85306
#0 0x00007fef99cdb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x0000000002baea09 in skgfrliopo ()
#2 0x0000000002bae801 in skgfospo ()
#3 0x00000000086ba843 in skgfrwat ()
#4 0x0000000008599e5d in ksfdwtio ()
#5 0x000000000220019b in ksfdwat_internal ()
#6 0x0000000003edce17 in kfk_reap_ufs_async_io ()
#7 0x0000000003edcd5f in kfk_reap_ios_from_subsys ()
#8 0x0000000000adefa4 in kfk_reap_ios ()
#9 0x0000000003edc382 in kfk_io1 ()
#10 0x0000000003edbf28 in kfkRequest ()
#11 0x0000000003ee272e in kfk_transitIO ()
#12 0x0000000003e35d5e in kffRelocateWait ()
#13 0x0000000003e5cbf6 in kffRelocate ()
#14 0x0000000003dd22df in kfdaExecute ()
#15 0x0000000003da1281 in kfdCompact ()
#16 0x0000000003da221a in kfdExecute ()
#17 0x0000000003eb5342 in kfgbRebalExecute ()
#18 0x0000000003ea2414 in kfgbDriver ()
#19 0x00000000021a86d7 in ksbabs ()
#20 0x0000000003eb964c in kfgbRun ()
#21 0x00000000021ad44b in ksbrdp ()
#22 0x00000000023d3eef in opirip ()
#23 0x000000000167e7b5 in opidrv ()
#24 0x0000000001c58477 in sou2o ()
#25 0x00000000008472c2 in opimai_real ()
#26 0x0000000001c5e795 in ssthrdmain ()
#27 0x00000000008471b9 in main ()

那麼如何提高rebalance速度

影響rebalance速度因素:：

Asm_power_limit
Diskgroup redundancy
Disks throughput
Disks RAID
Disk provider
AU size
How many disks are being added or dropped
Failure groups

這裏列出幾點不一一細說了：

一、不要過分rebalance

當使用alter diskgroup rebalance的時候，將會使asm承受很高的IO負載，因此咱們應該量避免過分使用rebalance，能用1次操做的不要使用兩次。

15:42:29 SQL> alter diskgroup testdd add disk '/dev/qdata/mpath-s03.3264.01.P0B00S05' rebalance power 20 wait;

Diskgroup altered.

Elapsed: 00:00:26.82

15:42:50 SQL> alter diskgroup testdd add disk '/dev/qdata/mpath-s02.3264.01.P0B00S05' rebalance power 20 wait;

Diskgroup altered.

Elapsed: 00:00:25.97

15:41:32 SQL> alter diskgroup testdd add disk '/dev/qdata/mpath-s03.3264.01.P0B00S05', '/dev/qdata/mpath-s02.3264.01.P0B00S05' rebalance power 20 wait;

Diskgroup altered.

Elapsed: 00:00:16.77

二、power參數越大，rebalance越快

15:48:00 SQL> alter diskgroup testdd add disk '/dev/qdata/mpath-s03.3264.01.P0B00S05', '/dev/qdata/mpath-s02.3264.01.P0B00S05' rebalance power 1 wait;

Diskgroup altered.

Elapsed: 00:00:20.11

三、跳過compact階段：

設置_DISABLE_REBALANCE_COMPACT=TRUE，ALTER DISKGROUP SET ATTRIBUTE "_rebalance_compact」="FALSE";在設置該參數前仍是建議諮詢SR，或者去問asm support guy Bane Radulovic。

rebalance新特性：

asm fast reblacne:

關閉數據庫，將asm實例restrict模式重啓，此時數據庫沒法鏈接，進行rebalance會快速一點，等待rebalacne完畢之後重啓asm實例，彷佛沒什麼用。

asm fast mirror resync:

當一個磁盤損壞修復後還原磁盤組冗餘度時，有時候會很耗時間。這種場景下，oracle asm fast mirror resync可以有效的減小重同步時間。必須將兼容性設置11.1.0以上才能生效。fast mirror resync在這樣的場景下，會跟蹤磁盤offline期間extents的變化，等待online之後再進行重平衡。

這個參數默認爲3.6h，經過設置disk_repaire_time來設置。在drop以前要肯定沒有offline disk

12c版本中，能夠經過V$ASM_OPERATION.pass字段觀察到resync階段。

SQL> SELECT GROUP_NUMBER, PASS, STATE FROM V$ASM_OPERATION;

GROUP_NUMBER PASS STAT

------------ --------- ----

1 RESYNC RUN

1 REBALANCE WAIT

1 COMPACT WAIT

二、power參數越大，rebalance越快