GreenPlum 大數據平臺--segment 失效問題恢復

1，問題檢查

[gpadmin@greenplum01 conf]$ psql -c "select * from gp_segment_configuration where status='d'" dbid | content | role | preferred_role | mode | status | port  |  hostname   |   address   | replication_por t ------+---------+------+----------------+------+--------+-------+-------------+-------------+---------------- --    12 |       2 | m    | m              | s    | d      | 43002 | greenplum03 | greenplum03 |            4400
2
    7 |       5 | m    | p              | s    | d      |  6001 | greenplum03 | greenplum03 |            3400
1 (2 rows)
發現狀態的

[gpadmin@greenplum01 conf]$ gpstate -m 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-Starting gpstate with args: -m 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jan 16 2019 02:32:15'
20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-Obtaining Segment details from master... 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:--------------------------------------------------------------
20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:--Current GPDB mirror list and status
20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:--Type = Group
20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:--------------------------------------------------------------
20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum03   /greenplum/data/mirror/gpseg0  43000 Passive Synchronized 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum03   /greenplum/data/mirror/gpseg1  43001 Passive Synchronized 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[WARNING]:-greenplum03   /greenplum/data2/mirror/gpseg2  43002   Failed                                <<<<<<<< 這個出現問題了
20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum03   /greenplum/data2/mirror/gpseg3  43003 Passive Synchronized 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum02   /greenplum/data/mirror/gpseg4  43000 Passive Synchronized 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum02   /greenplum/data/mirror/gpseg5  43001   Acting as Primary Change Tracking 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum02   /greenplum/data2/mirror/gpseg6  43002 Passive Synchronized 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum02   /greenplum/data2/mirror/gpseg7  43003 Passive Synchronized 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[INFO]:--------------------------------------------------------------
20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[WARNING]:-1 segment(s) configured as mirror(s) are acting as primaries 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[WARNING]:-1 segment(s) configured as mirror(s) have failed ------------看這裏 20190711:17:06:51:025238 gpstate:greenplum01:gpadmin-[WARNING]:-1 mirror segment(s) acting as primaries are in change tracking

01，鏈接問題

首先解決鏈接是否成功，ping 相應的主機看返回是不是成功狀態

ping greenplum03

02，激活失效的segment

gprecoverseg 恢復過程會啓動失效的Segment而且肯定須要同步的已更改文件 在gprecoverseg完成後，系統會進入到Resynchronizing模式而且開始複製更改過的文件。這個過程在後臺運行，而系統處於在線狀態而且可以接受數據庫請求。 當從新同步過程完成時，系統狀態是Synchronized


須要恢復兩個

日誌：

 1 [gpadmin@greenplum01 conf]$ gprecoverseg  2 20190711:17:10:44:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Starting gprecoverseg with args:  3 20190711:17:10:44:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
 4 20190711:17:10:44:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jan 16 2019 02:32:15'
 5 20190711:17:10:44:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Checking if segments are ready to connect  6 20190711:17:10:44:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Obtaining Segment details from master...  7 20190711:17:10:44:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Obtaining Segment details from master...  8 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Heap checksum setting is consistent between master and the segments that are candidates for recoverseg  9 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Greenplum instance recovery parameters 10 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:----------------------------------------------------------
11 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Recovery type              = Standard 12 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:----------------------------------------------------------
13 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Recovery 1 of 2
14 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:----------------------------------------------------------
15 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Synchronization mode                        = Incremental 16 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance host                        = greenplum03 17 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance address                     = greenplum03 18 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance directory                   = /greenplum/data2/mirror/gpseg2 19 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance port                        = 43002
20 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance replication port            = 44002
21 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance host               = greenplum02 22 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance address            = greenplum02 23 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance directory          = /greenplum/data2/primary/gpseg2 24 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance port               = 6002
25 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance replication port   = 34002
26 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Target                             = in-place 27 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:----------------------------------------------------------
28 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Recovery 2 of 2
29 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:----------------------------------------------------------
30 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Synchronization mode                        = Incremental 31 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance host                        = greenplum03 32 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance address                     = greenplum03 33 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance directory                   = /greenplum/data/primary/gpseg5 34 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance port                        = 6001
35 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Failed instance replication port            = 34001
36 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance host               = greenplum02 37 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance address            = greenplum02 38 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance directory          = /greenplum/data/mirror/gpseg5 39 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance port               = 43001
40 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Source instance replication port   = 44001
41 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-   Recovery Target                             = in-place 42 20190711:17:10:45:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:----------------------------------------------------------
43 
44 Continue with segment recovery procedure Yy|Nn (default=N): 45 > Y 46 20190711:17:11:31:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-2 segment(s) to recover 47 20190711:17:11:31:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Ensuring 2 failed segment(s) are stopped 48 
49 20190711:17:11:32:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments 50 updating flat files 51 20190711:17:11:32:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Updating configuration with new mirrors 52 20190711:17:11:33:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Updating mirrors 53 . 54 20190711:17:11:34:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Starting mirrors 55 20190711:17:11:34:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-era is 24a58010f9c5a05a_190711113124 56 20190711:17:11:34:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... 57 .. 58 20190711:17:11:36:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Process results... 59 20190711:17:11:36:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Updating configuration to mark mirrors up 60 20190711:17:11:36:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Updating primaries 61 20190711:17:11:36:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Commencing parallel primary conversion of 2 segments, please wait... 62 . 63 20190711:17:11:37:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Process results... 64 20190711:17:11:37:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Done updating primaries 65 20190711:17:11:37:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-******************************************************************
66 20190711:17:11:37:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Updating segments for resynchronization is completed. 67 20190711:17:11:37:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background. 68 20190711:17:11:37:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-
69 20190711:17:11:37:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-Use  gpstate -s  to check the resynchronization progress. 70 20190711:17:11:37:025375 gprecoverseg:greenplum01:gpadmin-[INFO]:-******************************************************************

View Code

03, 檢測同步

gpstate -m

[gpadmin@greenplum01 conf]$ gpstate -m 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-Starting gpstate with args: -m 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jan 16 2019 02:32:15'
20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-Obtaining Segment details from master... 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:--------------------------------------------------------------
20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:--Current GPDB mirror list and status
20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:--Type = Group
20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:--------------------------------------------------------------
20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum03   /greenplum/data/mirror/gpseg0    43000 Passive Synchronized 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum03   /greenplum/data/mirror/gpseg1    43001 Passive Synchronized 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum03   /greenplum/data2/mirror/gpseg2   43002 Passive Synchronized 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum03   /greenplum/data2/mirror/gpseg3   43003 Passive Synchronized 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum02   /greenplum/data/mirror/gpseg4    43000 Passive Synchronized 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum02   /greenplum/data/mirror/gpseg5    43001   Acting as Primary Synchronized 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum02   /greenplum/data2/mirror/gpseg6   43002 Passive Synchronized 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:-   greenplum02   /greenplum/data2/mirror/gpseg7   43003 Passive Synchronized 20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[INFO]:--------------------------------------------------------------
20190711:17:12:10:025484 gpstate:greenplum01:gpadmin-[WARNING]:-1 segment(s) configured as mirror(s) are acting as primaries

發現恢復出來了

04，恢復初始化狀態

　　由於宕機一個主segment，鏡像會激活另外一個，而且成爲主segment。運行gprecoverseg以後，主segment依舊沒變化，失效的segment沒有正式加進來，因此須要讓他變成初始化的時候的segment狀態，讓全部segment從新恢復平衡系統

檢查這個segment的狀態
gpstate -e

　　

運行gpstate -m來確保全部鏡像都是Synchronized。 gpstate -m

 

一直在運行了

假若有Resynchronizing模式 ，須要耐心等待

用-r選項運行gprecoverseg，讓Segment回到它們的首選角色。
 gprecoverseg -r

在從新平衡以後，運行gpstate -e來確認全部的Segment都處於它們的首選角色。

gpstate -e

這個就沒問題了