Redhat cluster 的心跳機制

Redhat cluster軟件的架構分爲兩層：

• 底層的cluster messaging layer（集羣信息層）負責在全部節點之間傳遞集羣信息包括心跳信息
  RHEL6及以前是CMAN，從RHEL7開始是Corosync。
• 上層的cluster resource manager（集羣資源管理器）管理劃歸集羣所屬的資源，包括IP地址、邏輯卷、文件系統等
  RHEL6及以前是RGManager，從RHEL7開始是Pacemaker。

心跳機制使用的是Totem協議，Totem是一個巨複雜的協議，心跳只是其中一個很小的部分，能夠這麼理解：Totem協議負責處理集羣內部全部節點之間的通訊問題，包含四個主要組件：Total Ordering Protocol, Membership Protocol, Recovery Protocol, Flow Control Mechanism。這裏只講與心跳有關的部分。

Totem的通訊是兩種方式互相配合進行的：multicast（多播）和token（令牌）。[注意multicast與broadcast(廣播)不一樣，它至關於限定範圍的廣播，只有分組內的成員能收到。] 一個節點只有在持有token的時候才能發言，經過multicast給集羣內全部節點發送信息，每一條multicast信息都附有順序號，發送完成以後就把順序號記錄到token中，而後把token傳給下一個節點，token的傳遞方式是一對一的，就像首尾相連的環形接力，從IP地址最小的節點依次傳給下一個IP地址更大的節點，每一個節點收到token以後，根據token裏記錄的順序號檢查本身是否漏收了multicast信息，若是漏收了，就在token裏添加劇傳申請，而後把token傳給下一個節點，下一個節點除了檢查本身有沒有漏收multicast以外，還會根據token裏記錄的重傳申請把相應的multicast信息從新發送一次（固然前提是該節點已經收到了這條multicast信息）。

集羣經過兩種機制檢測節點的健康狀態：

• Token超時

若是token中斷的時間超過了指定的期限，節點就會觸發membership protocol，重組cluster，這個期限經過如下參數設置：<totem token=」XXX」/>
注：XXX以毫秒爲單位。

在上述期限內，token會嘗試重傳，重傳的次數是如下參數指定的，缺省值是4：
<totem token_retransmits_before_loss_const=」X」/>

• Multicast retransmit(重傳)超過閾值

若是某個節點能收到token，但收不到multicast，那麼通過若干次token循環以後，也會觸發membership protocol，重組cluster，這個次數能夠經過參數 fail_recv_const 設置，缺省值是2500次。

Totem協議有4種狀態，表明集羣運行的不一樣階段，不一樣的狀態下運行的子協議也不一樣：

• GATHER
• COMMIT
  Gather和Commit狀態出如今集羣組建的階段，能夠比喻爲集合、報數的過程。運行的是membership protocol。
• RECOVERY
  Recovery狀態出如今集羣發生問題、進行恢復的階段。運行的是recovery protocol。
• OPERATIONAL
  Operational狀態表示集羣正常工做的階段。運行的是Totem Ordering Protocol。

有了這些基本概念，咱們就能夠大體看懂集羣的日誌了：

// 原本正常的集羣忽然發生token丟失（心跳信號丟失）： Apr 26 01:47:26 node1 openais[4252]: [TOTEM] The token was lost in the OPERATIONAL state. Apr 26 01:47:26 node1 openais[4252]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes). Apr 26 01:47:26 node1 openais[4252]: [TOTEM] Transmit multicast socket send buffer size (320000 bytes). // 集羣開始重組，進入Gather狀態，召集各節點： Apr 26 01:47:26 node1 openais[4252]: [TOTEM] entering GATHER state from 2. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering GATHER state from 0. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Creating commit token because I am the rep. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Storing new sequence id for ring 798 // 進入Commit狀態： Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering COMMIT state. // 進入Recovery狀態： Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering RECOVERY state. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] position [0] member 10.23.201.30: Apr 26 01:47:28 node1 openais[4252]: [TOTEM] previous ring seq 1940 rep 10.23.201.30 Apr 26 01:47:28 node1 openais[4252]: [TOTEM] aru 2b high delivered 2b received flag 1 Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Did not need to originate any messages in recovery. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Sending initial ORF token Apr 26 01:47:28 node1 openais[4252]: [CLM ] CLM CONFIGURATION CHANGE Apr 26 01:47:28 node1 openais[4252]: [CLM ] New Configuration: Apr 26 01:47:28 node1 openais[4252]: [CLM ] r(0) ip(10.23.201.30) Apr 26 01:47:28 node1 kernel: dlm: closing connection to node 2 Apr 26 01:47:28 node1 openais[4252]: [CLM ] Members Left: Apr 26 01:47:28 node1 openais[4252]: [CLM ] r(0) ip(10.23.201.31) Apr 26 01:47:28 node1 openais[4252]: [CLM ] Members Joined: Apr 26 01:47:28 node1 openais[4252]: [CLM ] CLM CONFIGURATION CHANGE Apr 26 01:47:28 node1 openais[4252]: [CLM ] New Configuration: Apr 26 01:47:28 node1 openais[4252]: [CLM ] r(0) ip(10.23.201.30) Apr 26 01:47:28 node1 openais[4252]: [CLM ] Members Left: Apr 26 01:47:29 node1 openais[4252]: [CLM ] Members Joined: Apr 26 01:47:29 node1 openais[4252]: [SYNC ] This node is within the primary component and will provide service. // 集羣重組完成，進入Operational狀態，只剩下一個節點： Apr 26 01:47:29 node1 openais[4252]: [TOTEM] entering OPERATIONAL state. Apr 26 01:47:29 node1 openais[4252]: [CLM ] got nodejoin message 10.23.201.30 Apr 26 01:47:29 node1 openais[4252]: [CPG ] got joinlist message from node 1 // 把不正常的節點fence掉： Apr 26 01:47:58 node1 fenced[4302]: node2 not a cluster member after 30 sec post_fail_delay Apr 26 01:47:58 node1 fenced[4302]: fencing node "node2" Apr 26 01:48:10 node1 fenced[4302]: fence "node2" success

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

// 原本正常的集羣忽然發生token丟失（心跳信號丟失）： Apr 26 01 : 47 : 26 node1 openais [ 4252 ] : [ TOTEM ] The token was lost in the OPERATIONAL state . Apr 26 01 : 47 : 26 node1 openais [ 4252 ] : [ TOTEM ] Receive multicast socket recv buffer size ( 320000 bytes ) . Apr 26 01 : 47 : 26 node1 openais [ 4252 ] : [ TOTEM ] Transmit multicast socket send buffer size ( 320000 bytes ) . // 集羣開始重組，進入Gather狀態，召集各節點： Apr 26 01 : 47 : 26 node1 openais [ 4252 ] : [ TOTEM ] entering GATHER state from 2. Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] entering GATHER state from 0. Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] Creating commit token because I am the rep . Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] Storing new sequence id for ring 798 // 進入Commit狀態： Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] entering COMMIT state . // 進入Recovery狀態： Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] entering RECOVERY state . Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] position [ 0 ] member 10.23.201.30 : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] previous ring seq 1940 rep 10.23.201.30 Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] aru 2b high delivered 2b received flag 1 Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] Did not need to originate any messages in recovery . Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] Sending initial ORF token Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] CLM CONFIGURATION CHANGE Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] New Configuration : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ]    r ( 0 ) ip ( 10.23.201.30 )    Apr 26 01 : 47 : 28 node1 kernel : dlm : closing connection to node 2 Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] Members Left : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ]    r ( 0 ) ip ( 10.23.201.31 )    Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] Members Joined : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] CLM CONFIGURATION CHANGE Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] New Configuration : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ]    r ( 0 ) ip ( 10.23.201.30 )    Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] Members Left : Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ CLM    ] Members Joined : Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ SYNC ] This node is within the primary component and will provide service . // 集羣重組完成，進入Operational狀態，只剩下一個節點： Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ TOTEM ] entering OPERATIONAL state . Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ CLM    ] got nodejoin message 10.23.201.30 Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ CPG    ] got joinlist message from node 1 // 把不正常的節點fence掉： Apr 26 01 : 47 : 58 node1 fenced [ 4302 ] : node2 not a cluster member after 30 sec post_fail_delay Apr 26 01 : 47 : 58 node1 fenced [ 4302 ] : fencing node "node2" Apr 26 01 : 48 : 10 node1 fenced [ 4302 ] : fence "node2" success

來源 http://linuxperf.com/?p=120