Oracle集羣（RAC）時間同步（ntp和CTSS）

Oracle集羣（RAC）時間同步（ntp和CTSS）

 

 

http://blog.itpub.net/26736162/viewspace-2157130/

 

 

crsctl stat res -t -init

ps -ef|grep ctss

crsctl check ctss

cluvfy comp clocksync -n all -verbose

 

 crsctl start res ora.ctssd -init

 crsctl stop res ora.ctssd -init

 

 

Network Time Protocol Setting

l  You have two options for time synchronization: an operating system configured network time protocol (NTP), or Oracle Cluster Time Synchronization Service.

l  Oracle Cluster Time Synchronization Service is designed for organizations whose cluster servers are unable to access NTP services.

l  If you use NTP, then the Oracle Cluster Time Synchronization daemon (ctssd) starts up in observer mode. If you do not have NTP daemons, then ctssd starts up in active mode and synchronizes time among cluster members without contacting an external time server..

 

能夠採用操做系統的NTP服務，也可使用Oracle自帶的服務ctss，若是ntp沒有啓用，那麼Oracle會自動啓用本身的ctssd進程。

從oracle 11gR2 RAC開始使用Cluster Time Synchronization Service(CTSS)同步各節點的時間,當安裝程序發現NTP協議處於非活動狀態時，安裝集羣時間同步服務將以活動模式（active）自動進行安裝並同步全部節點的時間。若是發現配置了 NTP，則以觀察者模式（observer mode）啓動集羣時間同步服務，Oracle Clusterware不會在集羣中進行活動的時間同步。

在RAC中，集羣的時間應該是保持同步的，不然可能致使不少問題，例如：依賴於時間的應用會形成數據的錯誤，各類日誌打印的順序紊亂，這將會影響問題的診斷，嚴重的可能會致使集羣宕機或者從新啓動集羣時節點沒法加入集羣。

在Oracle 11gR2前，集羣的時間是由NTP同步的，而在11gR2後，Oracle引入了CTSS組件，若是系統沒有配置NTP，則由CTSS來同步集羣時間。

NTP和CTSS是能夠共存的，且NTP的優先級要高於CTSS，也就是說，若是系統中同時有NTP和CTSS，那麼集羣的時間是由NTP同步的，CTSS會處於觀望（Observer）模式，只有當集羣關閉全部的NTP服務，CTSS纔會處於激活（Active）模式。在一個集羣中，只要有一個節點的ntp處於活動狀態，那麼集羣的全部節點的CTSS都會處於激活（Active）模式。

須要注意的是，要讓CTSS處於激活（Active）模式，則不只要關閉ntp服務（/sbin/service ntpd stop），還要刪除/etc/ntp.conf文件（mv /etc/ntp.conf /etc/ntp.conf.bak），不然不能啓用CTSS。

 

1.1.1      CTSS同步模式

關閉NTP：

/sbin/service ntpd stop

mv /etc/ntp.conf /etc/ntp.conf.bak

service ntpd status

chkconfig ntpd off

 

[root@raclhr-11gR2-N2 ~]# ps -ef|grep ctss

root     19678     1  0 19:22 ?        00:00:02 /u01/app/11.2.0/grid/bin/octssd.bin reboot

root     20970 20623  0 19:35 pts/4    00:00:00 grep ctss

[root@raclhr-11gR2-N2 ~]#

[root@raclhr-11gR2-N2 ~]# crsctl stat res -t -init

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS      

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

      1        ONLINE  ONLINE       raclhr-11gr2-n2          Started            

ora.cluster_interconnect.haip

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.crf

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.crsd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.cssd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.cssdmonitor

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.ctssd

      1        ONLINE  ONLINE       raclhr-11gr2-n2          ACTIVE:0           

ora.diskmon

      1        OFFLINE OFFLINE                                                  

ora.evmd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.gipcd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.gpnpd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.mdnsd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

[root@raclhr-11gR2-N2 ~]#

 

節點1的ctss狀態：

[root@raclhr-11gR2-N1 ~]# crsctl check ctss

CRS-4701: The Cluster Time Synchronization Service is in Active mode.

CRS-4702: Offset (in msec): 0

[root@raclhr-11gR2-N1 ~]#

節點1的octssd的日誌：

/u01/app/11.2.0/grid/log/raclhr-11gr2-n1/ctssd/octssd.log

2018-06-30 19:25:56.369: [    CTSS][899475200]sclsctss_gvss2: NTP default pid file not found

2018-06-30 19:25:56.369: [    CTSS][899475200]sclsctss_gvss8: Return [0] and NTP status [1].

2018-06-30 19:25:56.369: [    CTSS][899475200]ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].

2018-06-30 19:25:57.002: [    CTSS][916338432]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].

2018-06-30 19:26:01.263: [    CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].

2018-06-30 19:26:01.264: [    CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg

2018-06-30 19:26:01.264: [    CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc4] nodenum [2] hostname [raclhr-11gr2-n2] )

2018-06-30 19:26:09.267: [    CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].

節點1的octssd.log中記錄沒有發現ntp服務，ctss服務爲激活模式。

 

節點2的ctss狀態：

[root@raclhr-11gR2-N2 ~]# crsctl check ctss

CRS-4701: The Cluster Time Synchronization Service is in Active mode.

CRS-4702: Offset (in msec): 0

[root@raclhr-11gR2-N2 ~]#

節點2的octssd的日誌：

/u01/app/11.2.0/grid/log/raclhr-11gr2-n2/ctssd/octssd.log

2018-06-30 19:28:49.539: [    CTSS][839321344]sclsctss_gvss2: NTP default pid file not found

2018-06-30 19:28:49.539: [    CTSS][839321344]sclsctss_gvss8: Return [0] and NTP status [1].

2018-06-30 19:28:49.539: [    CTSS][839321344]ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].

2018-06-30 19:29:05.544: [    CTSS][839321344]ctsselect_msm: CTSS mode is [0xc4]

2018-06-30 19:29:05.544: [    CTSS][839321344]ctssslave_swm1_2: Ready to initiate new time sync process.

2018-06-30 19:29:05.545: [    CTSS][839321344]ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2].

2018-06-30 19:29:05.546: [    CTSS][845625088]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].

2018-06-30 19:29:05.546: [    CTSS][845625088]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].

2018-06-30 19:29:05.547: [    CTSS][839321344]ctssslave_swm2_3: Received time sync message from master.

2018-06-30 19:29:05.547: [    CTSS][839321344]ctssslave_swm: The system time difference is too small [243] usec. Not adjusting time.

2018-06-30 19:29:05.547: [    CTSS][839321344]ctssslave_swm17: LT [1530358145sec 546888usec], MT [1530358145sec 140655884523349usec], Delta [2314usec]

2018-06-30 19:29:05.547: [    CTSS][839321344]ctssslave_swm19: The offset is [243 usec] and sync interval set to [1]

2018-06-30 19:29:05.547: [    CTSS][839321344]ctssslave_swm: Received from master (mode [0xcc] nodenum [1] hostname [raclhr-11gr2-n1] )

2018-06-30 19:29:05.547: [    CTSS][839321344]ctsselect_msm: Sync interval returned in [1]

2018-06-30 19:29:05.547: [    CTSS][845625088]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler

2018-06-30 19:29:07.910: [    CTSS][860387072]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xc4], offset[0 ms]}, length=[8].

節點2的octssd.log中記錄沒有發現ntp服務，ctss服務爲激活模式，同步時間的主節點是節點1，而且會告訴集羣的時間有差別，可是由於差別太小，無需調整。

 

校驗集羣的時間：

 cluvfy comp clocksync -n all -verbose

 

雖然集羣時間不一致，可是這種狀況下校驗結果是經過的，並且略微的差別範圍內集羣也會自動同步回來。

[grid@raclhr-11gR2-N1 ~]$  cluvfy comp clocksync -n all -verbose

 

Verifying Clock Synchronization across the cluster nodes

 

Checking if Clusterware is installed on all nodes...

Check of Clusterware install passed

 

Checking if CTSS Resource is running on all nodes...

Check: CTSS Resource running on all nodes

  Node Name                             Status                 

  ------------------------------------  ------------------------

  raclhr-11gr2-n2                       passed                 

  raclhr-11gr2-n1                       passed                  

Result: CTSS resource check passed

 

 

Querying CTSS for time offset on all nodes...

Result: Query of CTSS for time offset passed

 

Check CTSS state started...

Check: CTSS state

  Node Name                             State                  

  ------------------------------------  ------------------------

  raclhr-11gr2-n2                       Active                 

  raclhr-11gr2-n1                       Active                 

CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...

Reference Time Offset Limit: 1000.0 msecs

Check: Reference Time Offset

  Node Name     Time Offset               Status                 

  ------------  ------------------------  ------------------------

  raclhr-11gr2-n2  0.0                       passed                 

  raclhr-11gr2-n1  0.0                       passed                 

 

Time offset is within the specified limits on the following set of nodes:

"[raclhr-11gr2-n2, raclhr-11gr2-n1]"

Result: Check of clock time offsets passed

 

 

Oracle Cluster Time Synchronization Services check passed

 

Verification of Clock Synchronization across the cluster nodes was successful.

 

1.1.2      NTP同步模式

開啓NTP:

mv /etc/ntp.conf.bak /etc/ntp.conf

service ntpd status

/sbin/service ntpd start

# chkconfig ntpd off

ps -ef|grep ntp

 

節點1 ：

[root@raclhr-11gR2-N1 ~]# crsctl check ctss

CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

 

[root@raclhr-11gR2-N1 ~]#  crsctl stat res -t -init

ora.ctssd

      1        ONLINE  ONLINE       raclhr-11gr2-n1          OBSERVER           

 

節點1的ctss日誌：

/u01/app/11.2.0/grid/log/raclhr-11gr2-n1/ctssd/octssd.log

2018-06-30 20:51:29.388: [    CTSS][899475200]sclsctss_gvss1: NTP default config file found

2018-06-30 20:51:29.389: [    CTSS][899475200]sclsctss_gvss8: Return [0] and NTP status [2].

2018-06-30 20:51:29.389: [    CTSS][899475200]ctss_check_vendor_sw: Vendor time sync software is detected. status [2].

2018-06-30 20:51:29.389: [    CTSS][899475200]ctss_check_vendor_sw: Ctssd is switching to observer role

2018-06-30 20:51:29.389: [    CTSS][899475200]clsctsselect_update_mbrdata: Updating pridata: { version[1] node[1] swversion[186647296] mode[0xee] }.

2018-06-30 20:51:29.639: [  CRSCCL][671086336]clsCclGetPriMemberData: Detected pridata change for node[1]. Retrieving it to the cache.

2018-06-30 20:51:31.434: [    CTSS][916338432]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xee], offset[0 ms]}, length=[8].

2018-06-30 20:51:35.258: [    CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].

2018-06-30 20:51:35.258: [    CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg

2018-06-30 20:51:35.259: [    CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc4] nodenum [2] hostname [raclhr-11gr2-n2] )

2018-06-30 20:51:35.656: [  CRSCCL][671086336]clsCclGetPriMemberData: Detected pridata change for node[2]. Retrieving it to the cache.

2018-06-30 20:51:43.240: [    CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].

2018-06-30 20:51:43.240: [    CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg

2018-06-30 20:51:43.240: [    CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc6] nodenum [2] hostname [raclhr-11gr2-n2] )

2018-06-30 20:51:51.217: [    CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].

2018-06-30 20:51:51.217: [    CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg

2018-06-30 20:51:51.218: [    CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc6] nodenum [2] hostname [raclhr-11gr2-n2] )

2018-06-30 20:51:59.194: [    CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].

2018-06-30 20:51:59.194: [    CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg

2018-06-30 20:51:59.195: [    CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc6] nodenum [2] hostname [raclhr-11gr2-n2] )

節點1的octssd.log中記錄發現ntp服務，ctss服務會自動切換到觀望模式。

 

2018-06-30 20:57:27.608: [    CTSS][839321344]ctsselect_msm: CTSS mode is [0xc6]

2018-06-30 20:57:27.608: [    CTSS][839321344]ctssslave_swm1_2: Ready to initiate new time sync process.

2018-06-30 20:57:27.609: [    CTSS][839321344]ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2].

2018-06-30 20:57:27.612: [    CTSS][845625088]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].

2018-06-30 20:57:27.613: [    CTSS][845625088]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].

2018-06-30 20:57:27.613: [    CTSS][839321344]ctssslave_swm2_3: Received time sync message from master.

2018-06-30 20:57:27.613: [    CTSS][839321344]ctssslave_swm17: LT [1530363447sec 613028usec], MT [1530363447sec 140655884569984usec], Delta [4410usec]

2018-06-30 20:57:27.613: [    CTSS][839321344]ctssslave_swm19: The offset is [19748 usec] and sync interval set to [1]

2018-06-30 20:57:27.613: [    CTSS][839321344]ctssslave_swm: Received from master (mode [0xee] nodenum [1] hostname [raclhr-11gr2-n1] )

2018-06-30 20:57:27.613: [    CTSS][839321344]ctsselect_msm: Sync interval returned in [1]

2018-06-30 20:57:27.613: [    CTSS][845625088]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler

節點2的octssd.log中也會記錄發現ntp服務，ctss服務爲觀望模式，而且同步時間的主節點是節點1。

 

 

 

1.1.3      模擬集羣時間不一致

若是在咱們生產系統中碰到集羣時間不一致會致使什麼結果，咱們的排查思路是怎麼樣的，如下是模擬集羣時間不一致的場景。

更改節點2的時間，向後推移2天：

將系統時間設定成2018年07月02日的命令以下：

#date -s 07/02/2018

將系統時間設定成下午23點23分06秒的命令以下。

#date -s 23:23:06

 

[root@raclhr-11gR2-N2 ctssd]# crsctl stat res -t -init

ora.ctssd

      1        ONLINE  ONLINE       raclhr-11gr2-n2          ACTIVE:172768000   

[root@raclhr-11gR2-N2 ctssd]# crsctl check ctss

CRS-4701: The Cluster Time Synchronization Service is in Active mode.

CRS-4702: Offset (in msec): 172768000

172768000微妙大約爲2天:

SYS@lhrrac11> select 172768000/1000/24/60/60 from dual;

 

172768000/1000/24/60/60

-----------------------

             1.99962963

 

更改節點2的時間後，在ASM和DB的alert日誌中產生了如下的告警信息：

Time drift detected. Please check VKTM trace file for more details.

 

drift表示漂移。

 

[grid@raclhr-11gR2-N2 trace]$ pwd

/u01/app/grid/diag/asm/+asm/+ASM2/trace

[grid@raclhr-11gR2-N2 trace]$ ll -lrt *vktm*

-rw-r----- 1 grid oinstall  136 May 17 14:09 +ASM2_vktm_29999.trm

-rw-r----- 1 grid oinstall 1847 May 17 14:09 +ASM2_vktm_29999.trc

-rw-r----- 1 grid oinstall  529 Jun  4 14:52 +ASM2_vktm_32504.trm

-rw-r----- 1 grid oinstall 7238 Jun  4 14:52 +ASM2_vktm_32504.trc

-rw-r----- 1 grid oinstall   78 Jun  4 14:59 +ASM2_vktm_14800.trm

-rw-r----- 1 grid oinstall 1079 Jun  4 14:59 +ASM2_vktm_14800.trc

-rw-r----- 1 grid oinstall   90 Jun  4 17:26 +ASM2_vktm_14991.trm

-rw-r----- 1 grid oinstall 1200 Jun  4 17:26 +ASM2_vktm_14991.trc

-rw-r----- 1 grid oinstall   89 Jun 29 10:05 +ASM2_vktm_17961.trm

-rw-r----- 1 grid oinstall 1200 Jun 29 10:05 +ASM2_vktm_17961.trc

-rw-r----- 1 grid oinstall  191 Jul  2 21:35 +ASM2_vktm_19774.trm

-rw-r----- 1 grid oinstall 3171 Jul  2 21:35 +ASM2_vktm_19774.trc

[grid@raclhr-11gR2-N2 trace]$ cat +ASM2_vktm_19774.trc

*** 2018-06-30 19:22:12.650

VKTM running at (1)millisec precision with DBRM quantum (100)ms

[Start] HighResTick = 1530357732650537

kstmrmtickcnt = 0 : ksudbrmseccnt[0] = 1530357732

kstmchkdrift (kstmhighrestimecntkeeper:highres): Time stalled at 1530363888044519

 

*** 2018-06-10 20:04:00.000

kstmchkdrift (kstmhighrestimecntkeeper:highres): Time jumped forward by

(172844812599)usec at (1528632240000738) whereas (1000000) is allowed

 

usec表明微秒，ms表示毫秒，1s=1000ms=1000000us

VKTM進程發現系統時間變了，alert日誌會產生相應的告警信息，從產生的trace文件中可知，系統向前推動了172844812599微秒，也即爲2天，也就是咱們模擬更改的時間，而容許的差別範圍爲1秒。

SYS@lhrrac11> select 172844812599/1000/1000/24/60/60 from dual;

 

172844812599/1000/1000/24/60/60

-------------------------------

                     2.00051866

 

節點2的octssd.log中和ctss狀態都記錄了偏移的時間：

2018-07-02 21:54:39.330: [    CTSS][1400497920]ctsselect_msm: CTSS mode is [0x84]

2018-07-02 21:54:39.330: [    CTSS][1400497920]ctssslave_swm1_2: Ready to initiate new time sync process.

2018-07-02 21:54:39.330: [    CTSS][1400497920]ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2].

2018-07-02 21:54:39.331: [    CTSS][1404700416]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].

2018-07-02 21:54:39.331: [    CTSS][1404700416]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].

2018-07-02 21:54:39.331: [    CTSS][1400497920]ctssslave_swm2_3: Received time sync message from master.

2018-07-02 21:54:39.331: [    CTSS][1400497920]ctssslave_swm: The magnitude [172757997797] of the offset [172757997797 usec] is larger than [86400000000 usec] sec which is the CTSS limit.

2018-07-02 21:54:39.331: [    CTSS][1400497920]ctssslave_swm: The magnitude of the systime diff is larger than max adjtime limit. Offset [172757997797] usec will be changed to max adjtime limit [+/- 131071].

2018-07-02 21:54:39.331: [    CTSS][1400497920]ctssslave_swm15: The CTSS master is behind this node. The local time offset [-131071 usec] is being adjusted. Sync method [2]

2018-07-02 21:54:39.331: [    CTSS][1400497920]ctssslave_swm17: LT [1530539679sec 331583usec], MT [1530366921sec 139882790197210usec], Delta [1267usec]

2018-07-02 21:54:39.331: [    CTSS][1400497920]ctssslave_swm19: The offset is [131071 usec] and sync interval set to [4]

2018-07-02 21:54:39.331: [    CTSS][1400497920]ctssslave_swm: Received from master (mode [0x8c] nodenum [1] hostname [raclhr-11gr2-n1] )

2018-07-02 21:54:39.331: [    CTSS][1400497920]ctsselect_msm: Sync interval returned in [4]

2018-07-02 21:54:39.331: [    CTSS][1404700416]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler

 

集羣的時間同步校驗也是失敗的，校驗結果是須要同步節點2的時間，此時由於集羣時間差別較大，同步服務每每是沒法作到的，只有手工同步才能修復。

校驗集羣的時間同步：

[grid@raclhr-11gR2-N2 ~]$ cluvfy comp clocksync -n all -verbose

 

Verifying Clock Synchronization across the cluster nodes

 

Checking if Clusterware is installed on all nodes...

Check of Clusterware install passed

 

Checking if CTSS Resource is running on all nodes...

Check: CTSS Resource running on all nodes

  Node Name                             Status                 

  ------------------------------------  ------------------------

  raclhr-11gr2-n2                       passed                 

  raclhr-11gr2-n1                       passed                 

Result: CTSS resource check passed

 

 

Querying CTSS for time offset on all nodes...

Result: Query of CTSS for time offset passed

 

Check CTSS state started...

Check: CTSS state

  Node Name                             State                  

  ------------------------------------  ------------------------

  raclhr-11gr2-n2                       Active                 

  raclhr-11gr2-n1                       Active                 

CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...

Reference Time Offset Limit: 1000.0 msecs

Check: Reference Time Offset

  Node Name     Time Offset               Status                 

  ------------  ------------------------  ------------------------

  raclhr-11gr2-n2  1.727568E8                failed                 

  raclhr-11gr2-n1  0.0                       passed                 

Result: PRVF-9661 :  Time offset is greater than acceptable limit on node "raclhr-11gr2-n2" [actual = "1.727568E8", acceptable = "1000.0" ]

 

PRVF-9652 : Cluster Time Synchronization Services check failed

 

Verification of Clock Synchronization across the cluster nodes was unsuccessful.

Checks did not pass for the following node(s):

        raclhr-11gr2-n2

1.727568E8表示科學計數法，爲1.7*10的8次方，即172756800ms，即2天。

在沒有同步時間以前，重啓節點2是沒法正常啓動的，從如下命令可知是在ctss這一步有問題，經過從新更改正確時間後，集羣才能正常啓動。

[root@raclhr-11gR2-N2 ~]# crsctl stat res -t -init

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS      

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

      1        ONLINE  OFFLINE                               Instance Shutdown  

ora.cluster_interconnect.haip

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.crf

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.crsd

      1        ONLINE  OFFLINE                                                  

ora.cssd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.cssdmonitor

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.ctssd

      1        ONLINE  OFFLINE                                                  

ora.diskmon

      1        OFFLINE OFFLINE                                                  

ora.evmd

      1        ONLINE  OFFLINE                                                  

ora.gipcd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

ora.gpnpd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                              

ora.mdnsd

      1        ONLINE  ONLINE       raclhr-11gr2-n2                             

查看集羣的告警日誌：

/u01/app/11.2.0/grid/log/raclhr-11gr2-n2/alertraclhr-11gr2-n2.log

 

2018-07-02 22:05:36.344

[ctssd(30350)]CRS-2405:The Cluster Time Synchronization Service on host raclhr-11gr2-n2 is shutdown by user

2018-07-02 22:05:40.689

[ctssd(30358)]CRS-2407:The new Cluster Time Synchronization Service reference node is host raclhr-11gr2-n1.

2018-07-02 22:05:40.689

[ctssd(30358)]CRS-2401:The Cluster Time Synchronization Service started on host raclhr-11gr2-n2.

2018-07-02 22:05:42.704

[ctssd(30358)]CRS-2404:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/raclhr-11gr2-n2/ctssd/octssd.log.

2018-07-02 22:05:43.395

[ctssd(30358)]CRS-2402:The Cluster Time Synchronization Service aborted on host raclhr-11gr2-n2. Details at  in /u01/app/11.2.0/grid/log/raclhr-11gr2-n2/ctssd/octssd.log.

2018-07-02 22:05:44.404

[ohasd(29989)]CRS-2807:Resource 'ora.asm' failed to start automatically.

2018-07-02 22:05:44.405

[ohasd(29989)]CRS-2807:Resource 'ora.crsd' failed to start automatically.

2018-07-02 22:05:44.405

[ohasd(29989)]CRS-2807:Resource 'ora.ctssd' failed to start automatically.

2018-07-02 22:05:44.405

[ohasd(29989)]CRS-2807:Resource 'ora.evmd' failed to start automatically.

查看octssd.log

2018-07-02 22:05:42.702: [    CTSS][1805252352]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [3].

2018-07-02 22:05:42.702: [    CTSS][1805252352]ctsscomm_recv_cb4_2: Receive active version change msg. Old active version [186647296] New active version [186647296].

2018-07-02 22:05:42.702: [    CTSS][1805252352]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].

2018-07-02 22:05:42.702: [    CTSS][1805252352]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].

2018-07-02 22:05:42.703: [    CTSS][1798948608]ctssslave_swm2_3: Received time sync message from master.

2018-07-02 22:05:42.703: [    CTSS][1798948608]ctssslave_swm: sendtime{sec[1530540340], usec[690191]}, receivetime{sec[1530540342], usec[702977]}.

2018-07-02 22:05:42.703: [    CTSS][1798948608]ctssslave_swm: The RTT of sync msg [2012786] is too large for time sync to be accurate. Recommends retry. Returns [17].

2018-07-02 22:05:42.703: [    CTSS][1798948608]ctssslave_swm: Received from master (mode [0x8c] nodenum [1] hostname [raclhr-11gr2-n1] )

2018-07-02 22:05:42.703: [    CTSS][1798948608]ctsselect_monitor_steysync_mode: Failed in clsctssslave_sync_with_master [17]. Retries [0/3].

2018-07-02 22:05:42.703: [    CTSS][1798948608]ctssslave_swm1_1: Waiting for last time sync process to finish. sync_state[6].

2018-07-02 22:05:42.703: [    CTSS][1805252352]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler

2018-07-02 22:05:42.703: [    CTSS][1798948608]ctssslave_swm1_2: Ready to initiate new time sync process.

2018-07-02 22:05:42.703: [    CTSS][1798948608]ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2].

2018-07-02 22:05:42.704: [    CTSS][1805252352]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].

2018-07-02 22:05:42.704: [    CTSS][1805252352]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].

2018-07-02 22:05:42.704: [    CTSS][1798948608]ctssslave_swm2_3: Received time sync message from master.

2018-07-02 22:05:42.704: [    CTSS][1798948608]ctssslave_swm: The magnitude [172752141259 usec] of the offset [172752141259 usec] is larger than [86400000000 usec] sec which is the CTSS limit.

2018-07-02 22:05:42.704: [    CTSS][1798948608]ctsselect_monitor_steysync_mode: Failed in clsctssslave_sync_with_master [12]: Time offset is too much to be corrected

2018-07-02 22:05:42.704: [    CTSS][1805252352]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler

2018-07-02 22:05:43.395: [    CTSS][2023593728]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xd0], offset[172752141 ms]}, length=[8].

2018-07-02 22:05:43.395: [    CTSS][1798948608]ctsselect_monitor_steysync_mode: CTSS daemon exiting [12].

2018-07-02 22:05:43.395: [    CTSS][1798948608]CTSS daemon aborting

2018-07-02 22:05:44.398: [    CTSS][2023593728]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xd0], offset[172752141 ms]}, length=[8].

 

下面開始修復系統：

 

將系統時間設定成2018年06月30日的命令以下：

#date -s 06/30/2018

將系統時間設定成下午23點23分06秒的命令以下。

#date -s 22:14:06

 

而後重啓CRS服務：

crsctl stop crs -f

crsctl start crs

 

而後ctss自動同步時間：

[root@raclhr-11gR2-N2 ctssd]# crsctl stat res -t -init

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS      

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.ctssd

      1        ONLINE  ONLINE       raclhr-11gr2-n2          ACTIVE:100         

[root@raclhr-11gR2-N2 ctssd]# crsctl stat res -t -init

ora.ctssd

      1        ONLINE  ONLINE       raclhr-11gr2-n2          ACTIVE:0           

 

 

 

小麥苗課程

小麥苗課堂開課啦，以下是現有的課程，歡迎諮詢小麥苗：

 

課程名稱	課時	上課時間	價格
OCP（從入門到專家）	每一年1期，35課時左右/期	20：00-22：00	1588(可優惠)
OCM認證	每一年N期，9課時/期	20：00-22：00	22888
高可用課程（rac+dg+ogg）	每一年1期，20課時左右/期	20：00-22：00	1888(可優惠)
Oracle初級入門	每一年1期，15課時左右/期	20：00-22：00	800
Oracle健康檢查腳本	可微信或微店購買。	88
Oracle數據庫技能直通車	包含以下3個課程： ①《11g OCP網絡課程培訓》（面向零基礎） 價值1600元 ②《11g OCM網絡班課程培訓》（Oracle技能合集）價值10000+元 ③《RAC + DG + OGG 高可用網絡班課程》 價值2000元 以上3個課程所有打包只要5888，只要5888全部課程帶回家，終身指導！全部課程都是在線講課，不是播放視頻，課件所有贈送！ 注意：以上OCP和OCM課程只包括培訓課程，不包括考試費用。OCM提供培訓+視頻，可是不提供練習環境和資料。報名一次，OCP和高可用的課程能夠免費終身循環聽課。	5888
OCP+高可用（rac+dg+ogg）	報名OCP+高可用課程，能夠優惠300元，優惠後的價格爲3188.	3188(可優惠)

 

注意：

一、每次上課前30分鐘答疑。

二、OCM實時答疑，提供和考試同樣的練習模擬環境，只要按照老師講的方式來練習，能夠保證100%經過。

三、授課方式：YY語音網絡直播講課（非視頻） + QQ互動答疑 + 視頻複習。

四、OCP課時能夠根據你們學習狀況進行增長或縮減。

五、以上全部課程都可循環聽課。

六、12c OCM課程私聊。

七、Oracle初級入門課程，只教你們最實用+最經常使用的Oracle操做維護知識。

八、以上全部課程，能夠加小麥苗微信（lhrbestxh）或QQ（646634621）詳聊，優惠多多。

 

培訓項目	鏈接地址
DB筆試面試歷史鏈接	http://mp.weixin.qq.com/s/Vm5PqNcDcITkOr9cQg6T7w
OCP培訓說明鏈接	https://mp.weixin.qq.com/s/2cymJ4xiBPtTaHu16HkiuA
OCM培訓說明鏈接	https://mp.weixin.qq.com/s/7-R6Cz8RcJKduVv6YlAxJA
高可用（RAC+DG+OGG）培訓說明鏈接	https://mp.weixin.qq.com/s/4vf042CnOdAD8zDyjUueiw
OCP最新題庫解析歷史鏈接（052）	http://mp.weixin.qq.com/s/bUgn4-uciSndji_pUbLZfA
微店地址	https://weidian.com/s/793741433?wfr=c&ifr=shopdetail

 

 

About Me

............................................................................................................................................. ● 本文做者：小麥苗，部份內容整理自網絡，如有侵權請聯繫小麥苗刪除 ● 本文在itpub（http://blog.itpub.net/26736162/abstract/1/）、博客園（http://www.cnblogs.com/lhrbest）和我的微信公衆號（xiaomaimiaolhr）上有同步更新 ● 本文itpub地址：http://blog.itpub.net/26736162/abstract/1/ ● 本文博客園地址：http://www.cnblogs.com/lhrbest ● 本文pdf版、我的簡介及小麥苗雲盤地址：http://blog.itpub.net/26736162/viewspace-1624453/ ● 數據庫筆試面試題庫及解答：http://blog.itpub.net/26736162/viewspace-2134706/ ● DBA寶典今日頭條號地址：http://www.toutiao.com/c/user/6401772890/#mid=1564638659405826 ............................................................................................................................................. ● QQ羣號：230161599（滿）、618766405 ● 微信羣：可加我微信，我拉你們進羣，非誠勿擾 ● 聯繫我請加QQ好友（646634621），註明添加原因 ● 於 2018-06-01 06:00 ~ 2018-06-31 24:00 在魔都完成 ● 最新修改時間：2018-06-01 06:00 ~ 2018-06-31 24:00 ● 文章內容來源於小麥苗的學習筆記，部分整理自網絡，如有侵權或不當之處還請諒解 ● 版權全部，歡迎分享本文，轉載請保留出處 ............................................................................................................................................. ● 小麥苗的微店：https://weidian.com/s/793741433?wfr=c&ifr=shopdetail ● 小麥苗出版的數據庫類叢書：http://blog.itpub.net/26736162/viewspace-2142121/ ● 小麥苗OCP、OCM、高可用網絡班：http://blog.itpub.net/26736162/viewspace-2148098/ ............................................................................................................................................. 使用微信客戶端掃描下面的二維碼來關注小麥苗的微信公衆號（xiaomaimiaolhr）及QQ羣（DBA寶典）、添加小麥苗微信，學習最實用的數據庫技術。    小麥苗的微信公衆號      小麥苗的DBA寶典QQ羣2       小麥苗的微信二維碼          小麥苗的微店 .............................................................................................................................................