drbd+nfs+keepalived

時間 2019-11-21

標籤 drbd+nfs+keepalived drbd nfs keepalived 欄目 Linux 简体版

原文原文鏈接

寫的很詳細理論知識：html

https://www.cnblogs.com/kevingrace/p/5740940.html前端

寫的很詳細負載：node

https://www.cnblogs.com/kevingrace/p/5740953.htmllinux

實際操做以下：nginx

一、修改主機名

11.11.11.2  主服務器     主機名：Primary
11.11.11.3  備服務器     主機名：Secondary
11.11.11.4  VIP

******主要操做上邊兩臺設備，下邊兩臺用於輔助測試******

11.11.11.8  web服務器   IIS  (用於掛載測試)
11.11.11.9  nginx代理    設置緩存   主機名LB

二、兩臺機器的防火牆要相互容許訪問。最好是關閉selinux和iptables防火牆（兩臺機器一樣操做）

[root@Primary ~]# setenforce 0        //臨時性關閉；永久關閉的話，須要修改/etc/sysconfig/selinux的SELINUX爲disabled
[root@Primary ~]# /etc/init.d/iptables stop

三、設置hosts文件（兩臺機器一樣操做）

[root@primary ~]# cat /etc/hosts
**********
11.11.11.2 primary
11.11.11.3 secondary

四、兩臺機器同步時間

[root@Primary ~]# yum install -y netpdate
[root@Primary ~]# ntpdate -u ntp1.aliyun.com

五、DRBD的安裝（兩臺機器上一樣操做）

rpm -ivh https://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum list drbd*
yum install -y drbd84-utils kmod-drbd84

加載模塊：
[root@Primary ~]# modprobe drbd
查看模塊是否已加上
[root@Primary ~]# lsmod |grep drbd
drbd                  332493  0

　六、DRBD配置（兩臺機器上一樣操做）web

[root@primary ~]# cat /etc/drbd.conf
# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";      //這是主要的兩個配置文件，這個定義軟件策略
include "drbd.d/*.res";                   //定義使用磁盤

[root@primary ~]# cp /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.bak

[root@primary ~]# cat  /etc/drbd.d/global_common.conf
global {
    usage-count no;
    udev-always-use-vnr;
}

common {
    protocol C;
    handlers {

    }

    startup {
             wfc-timeout          120;
             degr-wfc-timeout     120;
             outdated-wfc-timeout 120;
    }

    disk {
        on-io-error detach;
    }

    net {
       cram-hmac-alg md5;
       sndbuf-size 512k;
       shared-secret "testdrbd";
    }
        syncer {
              rate 300M;
        }
}
[root@primary ~]#

[root@primary ~]# cat /etc/drbd.d/r0.res 
resource r0 {
 on primary {
  device   /dev/drbd0;
  disk /dev/sdb1;     //這個磁盤須要分過區不須要格式化
  address 11.11.11.2:7789;
  meta-disk  internal;
 }
 on secondary {
  device /dev/drbd0;
  disk /dev/sdb1;
  address 11.11.11.3:7789;
  meta-disk internal;
 }
}
[root@primary ~]#

七、在兩臺機器上添加DRBD磁盤

在Primary機器上添加一塊30G的硬盤做爲DRBD，分區爲/dev/sdb1，不作格式化，並在本地系統建立/data目錄，不作掛載操做。
[root@Primary ~]# fdisk -l
......
[root@Primary ~]# fdisk /dev/sdb      
依次輸入"n->p->1->1->回車->w"                                    //分區建立後，再次使用"fdisk /dev/vdd"，輸入p，便可查看到建立的分區，好比/dev/vdd1
   
在Secondary機器上添加一塊30G的硬盤做爲DRBD，分區爲/dev/sdb1，不作格式化，並在本地系統建立/data目錄，不作掛載操做。
[root@Secondary ~]# fdisk -l
......
[root@Secondary ~]# fdisk /dev/sdb      
依次輸入"n->p->1->1->回車->w"

八、在兩臺機器上分別建立DRBD設備並激活r0資源（下面操做在兩臺機器上都要執行）

[root@Primary ~]# mknod /dev/drbd0 b 147 0
mknod: `/dev/drbd0': File exists

[root@Primary ~]# drbdadm create-md r0
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.

[root@Primary ~]# drbdadm create-md r0
You want me to create a v08 style flexible-size internal meta data block.
There appears to be a v08 flexible-size internal meta data block
already in place on /dev/vdd1 at byte offset 10737340416
Do you really want to overwrite the existing v08 meta-data?
[need to type 'yes' to confirm] yes                                  //這裏輸入"yes"
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
   
啓動drbd服務（注意：須要主從共同啓動方能生效）
service drbd start

 查看狀態（兩臺機器上都執行查看）

[root@primary ~]# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
0: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----- //顯示都爲備用；數據還在同步，同步完成後把第一個提高爲主
ns:3472496 nr:0 dw:0 dr:3472496 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:1594256
[============>.......] sync'ed: 68.6% (1556/4948)M
finish: 0:00:55 speed: 28,840 (27,776) K/sec
[root@primary ~]#vim

由上面兩臺主機的DRBD狀態查看結果裏的ro:Secondary/Secondary表示兩臺主機的狀態都是備機狀態，ds是磁盤狀態，顯示的狀態內容爲「不一致」，這是由於DRBD沒法判斷哪一方爲主機，centos

應以哪一方的磁盤數據做爲標準。

九、接着將Primary主機配置爲DRBD的主節點

[root@Primary ~]# drbdsetup /dev/drbd0 primary --force

分別查看主從DRBD狀態：
[root@primary ~]# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:5066752 nr:0 dw:0 dr:5068840 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

[root@secondary ~]# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:5066752 dw:5066752 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@secondary ~]#
 
ro在主從服務器上分別顯示 Primary/Secondary和Secondary/Primary
ds顯示UpToDate/UpToDate 表示主從配置成功

十、掛載DRBD (Primary主節點機器上操做)

從上面Primary主節點的DRBD狀態上看到mounted和fstype參數爲空，因此這步開始掛載DRBD到系統目錄api

先格式化/dev/drbd0
[root@Primary ~]# mkfs.ext4 /dev/drbd0

建立掛載目錄，而後執行DRBD掛載
[root@Primary ~]# mkdir /data
[root@Primary ~]# mount /dev/drbd0 /data
[root@primary ~]# df -h
文件系統                 容量  已用  可用 已用% 掛載點
/dev/mapper/centos-root   17G  1.4G   16G    8% /
devtmpfs                 899M     0  899M    0% /dev
/dev/sda1               1014M  145M  870M   15% /boot
/dev/drbd0                30G  1.7G   27G    6% /data

特別注意：
Secondary節點上不容許對DRBD設備進行任何操做，包括只讀，全部的讀寫操做只能在Primary節點上進行。
只有當Primary節點掛掉時，Secondary節點才能提高爲Primary節點

十一、DRBD主備故障切換測試

模擬Primary節點發生故障，Secondary接管並提高爲Primary緩存

下面是在Primary主節點上操做記錄：
[root@Primary ~]# cd /data
[root@Primary data]# touch wangshibo wangshibo1 wangshibo2 wangshibo3
[root@Primary data]# cd ../
[root@Primary /]# umount /data
   
[root@Primary /]# drbdsetup /dev/drbd0 secondary              //將Primary主機設置爲DRBD的備節點。在實際生產環境中，直接在Secondary主機上提權（即設置爲主節點）便可。
[root@Primary /]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res  cs         ro                   ds                 p  mounted  fstype
0:r0   Connected  Secondary/Secondary  UpToDate/UpToDate  C
   
注意：這裏實際生產環境若Primary主節點宕機，在Secondary狀態信息中ro的值會顯示爲Secondary/Unknown,只須要進行DRBD提權操做便可。
   
下面是在Secondary 備份節點上操做記錄：
先進行提權操做，即將Secondary手動升級爲DRBD的主節點
[root@Secondary ~]# drbdsetup /dev/drbd0 primary
[root@Secondary ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res  cs         ro                 ds                 p  mounted  fstype
0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C
   
而後掛載DRBD
[root@Secondary ~]# mkdir /data
[root@Secondary ~]# mount /dev/drbd0 /data
[root@Secondary ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      156G   13G  135G   9% /
tmpfs                 2.9G     0  2.9G   0% /dev/shm
/dev/vda1             190M   89M   92M  50% /boot
/dev/vdd              9.8G   23M  9.2G   1% /data2
/dev/drbd0            9.8G   23M  9.2G   1% /data
   
發現DRBD掛載目錄下已經有了以前在遠程Primary主機上寫入的內容
[root@Secondary ~]# cd /data
[root@Secondary data]# ls
wangshibo  wangshibo1  wangshibo2  wangshibo3

在Secondary節點上繼續寫入數據
[root@Secondary data]# touch huanqiu huanqiu1 huanqiu2 huanqiu3
  
而後模擬Secondary節點故障，Primary節點再提權升級爲DRBD主節點（操做同上，此處省略.......）

View Code

到此，DRBD的主從環境的部署工做已經完成。不過上面是記錄的是主備手動切換，至於保證DRBD主從結構的智能切換，實現高可用，還需裏用到Keepalived或Heartbeat來實現了（會在DRBD主端掛掉的狀況下，自動切換從端爲主端並自動掛載/data分區）

十二、安裝nfs

思路：
1）在兩臺機器上安裝keepalived，VIP爲192.168.1.200
2）將DRBD的掛載目錄/data做爲NFS的掛載目錄。遠程客戶機使用vip地址掛載NFS
3）當Primary主機發生宕機或NFS掛了的故障時，Secondary主機提權升級爲DRBD的主節點，而且VIP資源也會轉移過來。
   當Primary主機的故障恢復時，會再次變爲DRBD的主節點，並從新奪回VIP資源。從而實現故障轉移

Primary主機（11.11.11.2）默認做爲DRBD的主節點，DRBD掛載目錄是/data

Secondary主機（11.11.11.3）是DRBD的備份節點

在Primary和Secondary兩臺主機上安裝NFS
[root@Primary ~]# yum install rpcbind nfs-utils
[root@Primary ~]# vim /etc/exports
/data 11.11.11.0/24(rw,sync,no_root_squash)

service rpcbind start
service nfs start

關閉兩臺主機的iptables防火牆
防火牆最好關閉，不然可能致使客戶機掛載nfs時會失敗！
若開啓防火牆，須要在iptables中開放nfs相關端口機以及VRRP組播地址
[root@Primary ~]# /etc/init.d/iptables stop
 
兩臺機器上的selinux必定要關閉！！！！！！！！！！
不然下面在keepalived.conf裏配置的notify_master.sh等腳本執行失敗！這是曾經踩過的坑！
[root@Primary ~]# setenforce 0     //臨時關閉。永久關閉的話，還須要在/etc/sysconfig/selinux 文件裏將SELINUX改成disabled
[root@Primary ~]# getenforce
Permissive

1三、在兩臺主機上安裝Keepalived，配合keepalived實現自動fail-over

13.一、Primary端配置

安裝Keepalived
[root@Primary ~]# yum install -y openssl-devel popt-devel
[root@Primary ~]# cd /usr/local/src/
[root@Primary src]# wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz
[root@Primary src]# tar -zvxf keepalived-1.3.5.tar.gz
[root@Primary src]# cd keepalived-1.3.5
[root@Primary keepalived-1.3.5]# ./configure --prefix=/usr/local/keepalived
[root@Primary keepalived-1.3.5]# make && make install
        
[root@Primary keepalived-1.3.5]# cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/rc.d/init.d/
[root@Primary keepalived-1.3.5]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@Primary keepalived-1.3.5]# mkdir /etc/keepalived/
[root@Primary keepalived-1.3.5]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[root@Primary keepalived-1.3.5]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
[root@Primary keepalived-1.3.5]# echo "/etc/init.d/keepalived start" >> /etc/rc.local
         
[root@Primary keepalived-1.3.5]# chmod +x /etc/rc.d/init.d/keepalived      #添加執行權限
[root@Primary keepalived-1.3.5]# chkconfig keepalived on                   #設置開機啓動
[root@Primary keepalived-1.3.5]# service keepalived start                   #啓動
[root@Primary keepalived-1.3.5]# service keepalived stop                    #關閉
[root@Primary keepalived-1.3.5]# service keepalived restart                 #重啓

編譯安裝keepalived

yum install -y openssl-devel popt-devel
yum install keepalived

-----------Primary主機的keepalived.conf配置
[root@Primary ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-bak
[root@primary ~]# cat /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
    root@localhost
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id DRBD_HA_MASTER
}

vrrp_script chk_nfs {
        script "/etc/keepalived/check_nfs.sh"
        interval 5
    }

vrrp_instance VI_1 {
    state MASTER
    interface ens33   //注意修改爲本身的網卡名稱
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
       chk_nfs
    }
    virtual_ipaddress {
       11.11.11.4
    }
    nopreempt
    notify_stop "/etc/keepalived/notify_stop.sh"
    notify_master "/etc/keepalived/notify_master.sh"
}

啓動keepalived服務

[root@primary ~]# service keepalived start
Redirecting to /bin/systemctl start keepalived.service
[root@primary ~]# ps -ef|grep keepalived
root 7314 1 0 14:22 ? 00:00:00 /usr/sbin/keepalived -D
root 7315 7314 0 14:22 ? 00:00:00 /usr/sbin/keepalived -D
root 7316 7314 0 14:22 ? 00:00:00 /usr/sbin/keepalived -D
root 7431 7119 0 14:22 pts/1 00:00:00 grep --color=auto keepalived
[root@primary ~]#

查看VIP

一、primary的腳本內容

1）此腳本只在Primary機器上配置
[root@primary ~]# cat /etc/keepalived/check_nfs.sh 
/sbin/service nfs status &>/dev/null
if [ $? -ne 0 ];then
    ###若是服務狀態不正常，先嚐試重啓服務
    /sbin/service nfs restart
    /sbin/service nfs status &>/dev/null
    if [ $? -ne 0 ];then
        ###若重啓nfs服務後，仍不正常
        ###卸載drbd設備
        umount /dev/drbd0
        ###將drbd主降級爲備
        drbdadm secondary r0
        #關閉keepalived
        /sbin/service keepalived stop
    fi
fi
 [root@Primary ~]# chmod 755 /etc/keepalived/check_nfs.sh

2）此腳本只在Primary機器上配置
[root@Primary ~]# mkdir /etc/keepalived/logs
[root@primary ~]# cat /etc/keepalived/notify_stop.sh
#!/bin/bash
  
time=`date "+%F  %H:%M:%S"`
echo -e "$time  ------notify_stop------\n" >> /etc/keepalived/logs/notify_stop.log
/sbin/service nfs stop &>> /etc/keepalived/logs/notify_stop.log
/bin/umount /dev/drbd0 &>> /etc/keepalived/logs/notify_stop.log
/sbin/drbdadm secondary r0 &>> /etc/keepalived/logs/notify_stop.log
echo -e "\n" >> /etc/keepalived/logs/notify_stop.log
[root@primary ~]# 
[root@Primary ~]# chmod 755 /etc/keepalived/notify_stop.sh

二、此腳本在兩臺機器上都要配置

[root@primary ~]# cat /etc/keepalived/notify_master.sh
#!/bin/bash
  
time=`date "+%F  %H:%M:%S"`
echo -e "$time    ------notify_master------\n" >> /etc/keepalived/logs/notify_master.log
/sbin/drbdadm primary data1 &>> /etc/keepalived/logs/notify_master.log
/bin/mount /dev/drbd0 /data &>> /etc/keepalived/logs/notify_master.log
/sbin/service nfs restart &>> /etc/keepalived/logs/notify_master.log
echo -e "\n" >> /etc/keepalived/logs/notify_master.log
[root@primary ~]#

/etc/keepalived/notify_master.sh[root@Primary ~]# chmod 755

13.二、Secondary端配置

-----------Secondary主機的keepalived.conf配置
[root@Secondary ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-bak
[root@secondary ~]# cat /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id DRBD_HA_BACKUP
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33  //注意修改爲本身的網卡名稱
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }

   nopreempt
   notify_master "/etc/keepalived/notify_master.sh"    
   notify_backup "/etc/keepalived/notify_backup.sh"    
    virtual_ipaddress {
        11.11.11.4
    }
}//當此機器爲keepalived的master角色時執行這個腳本 //當此機器爲keepalived的backup角色時執行這個腳本

啓動keepalived服務

/etc/keepalived/notify_master.sh
/etc/keepalived/notify_backup.sh[root@Primary ~]# chmod 755[root@Primary ~]# chmod 755

[root@primary ~]# service keepalived start
Redirecting to /bin/systemctl start keepalived.service

一、此腳本只在Secondary機器上配置

[root@Secondary ~]# mkdir /etc/keepalived/logs
[root@Secondary ~]# vim /etc/keepalived/notify_backup.sh
#!/bin/bash
  
time=`date "+%F  %H:%M:%S"`
echo -e "$time    ------notify_backup------\n" >> /etc/keepalived/logs/notify_backup.log
/sbin/service nfs stop &>> /etc/keepalived/logs/notify_backup.log
/bin/umount /dev/drbd0 &>> /etc/keepalived/logs/notify_backup.log
/sbin/drbdadm secondary data1 &>> /etc/keepalived/logs/notify_backup.log
echo -e "\n" >> /etc/keepalived/logs/notify_backup.log
[root@Secondary ~]# chmod 755 /etc/keepalived/notify_backup.sh

1四、遠程客戶掛載NFS

客戶端只須要安裝rpcbind程序，並確認服務正常
[root@huanqiu ~]# yum install rpcbind nfs-utils
[root@huanqiu ~]# /etc/init.d/rpcbind start

linux端：
umount /web/
mount -t nfs 11.11.11.4:/data /web

windos2012端：
mount \\11.11.11.4\data -o nolock,rsize=1024,wsize=1024,timeo=15 z:     優，切換中幾乎不斷

主從出現UpToDate/DUnknown 故障恢復
https://blog.csdn.net/kjsayn/article/details/52958282


windos修改註冊表（掛載上去以後沒有寫權限）
https://jingyan.baidu.com/article/c910274bfd6800cd361d2df3.html

1五、前端nginx緩存

建立緩存空間
mkdir /cache 
nginx -t 
nginx -s reload

[root@localhost conf.d]# pwd
/etc/nginx/conf.d
[root@localhost conf.d]# cat web.conf 
upstream node {
    server 11.11.11.9:80;
}
proxy_cache_path /cache levels=1:2 keys_zone=cache:10m max_size=10g inactive=60m use_temp_path=off;
server {
    listen 80;
    server_name www.test.com;
    index index.html;
    location / {
    proxy_pass http://node;
    proxy_cache cache;
    proxy_cache_valid   200 304 12h;
    proxy_cache_valid   any 10m;
    add_header  Nginx-Cache "$upstream_cache_status";
    proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
        }
    }
[root@localhost conf.d]#

關於nginx緩存問題，詳情請查看另外一個博文

15.二、清除緩存

rm刪除已緩存的數據
rm -rf /cache/*

1六、測試

1）
先關閉Primary主機上的keepalived服務。就會發現VIP資源已經轉移到Secondary主機上了。
同時，Primary主機的nfs也會主動關閉，同時Secondary會升級爲DRBD的主節點
[root@Primary ~]# /etc/init.d/keepalived stop
Stopping keepalived:                                       [  OK  ]
[root@Primary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:35:d1:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.151/24 brd 192.168.1.255 scope global eth0
    inet6 fe80::f816:3eff:fe35:d1d6/64 scope link
       valid_lft forever preferred_lft forever
  
查看系統日誌，也能看到VIP資源轉移信息
[root@Primary ~]# tail -1000 /var/log/messages
........
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:51 localhost Keepalived[30937]: Stopping
May 25 11:58:51 localhost Keepalived_vrrp[30940]: VRRP_Instance(VI_1) sent 0 priority
May 25 11:58:51 localhost Keepalived_vrrp[30940]: VRRP_Instance(VI_1) removing protocol VIPs.
  
[root@Primary ~]# ps -ef|grep nfs
root       588 10364  0 12:13 pts/1    00:00:00 grep --color nfs
[root@Primary ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      156G   36G  112G  25% /
tmpfs                 2.9G     0  2.9G   0% /dev/shm
/dev/vda1             190M   98M   83M  55% /boot
[root@Primary ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res  cs         ro                   ds                 p  mounted  fstype
0:r0   Connected  Secondary/Secondary  UpToDate/UpToDate  C
  
登陸到Secondary備份機器上，發現VIP資源已經轉移過來
[root@Secondary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:4c:7e:88 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.152/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.200/32 scope global eth0
    inet6 fe80::f816:3eff:fe4c:7e88/64 scope link
       valid_lft forever preferred_lft forever
  
[root@Secondary ~]# tail -1000 /var/log/messages
........
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:58 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:58 localhost Keepalived_vrrp[17131]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.1.200
  
[root@Secondary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:4c:7e:88 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.152/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.200/32 scope global eth0
    inet6 fe80::f816:3eff:fe4c:7e88/64 scope link
       valid_lft forever preferred_lft forever
[root@Secondary ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      156G   13G  135G   9% /
tmpfs                 2.9G     0  2.9G   0% /dev/shm
/dev/vda1             190M   89M   92M  50% /boot
/dev/drbd0            9.8G   23M  9.2G   1% /data
  
當Primary機器的keepalived服務恢復啓動後，VIP資源又會強制奪回來（能夠查看/var/log/message系統日誌）
而且Primary還會再次變爲DRBD的主節點
  
2）
關閉Primary主機的nfs服務。根據監控腳本，會主動去啓動nfs，只要當啓動失敗時，纔會強制由DRBD的主節點降爲備份節點，並關閉keepalived。
從而跟上面流程同樣實現故障轉移
  
結論：
在上面的主從故障切換過程當中，對於客戶端來講，掛載NFS不影響使用，只是會有一點的延遲。
這也驗證了drbd提供的數據一致性功能（包括文件的打開和修改狀態等），在客戶端看來，真個切換過程就是"一次nfs重啓"（主nfs停，備nfs啓）。

Top：

一、`說一下"Split-Brain"（腦裂）的狀況：`

假設把Primary主機的的eth0設備宕掉，而後直接在Secondary主機上進行提權升級爲DRBD的主節點，而且mount掛載DRBD，這時會發現以前在Primary主機上寫入的數據文件確實同步過來了。
接着再把Primary主機的eth0設備恢復，看看有沒有自動恢復 主從關係。通過查看，發現DRBD檢測出了Split-Brain的情況，也就是兩個節點都處於standalone狀態，
故障描述以下：Split-Brain detected,dropping connection! 這就是傳說中的「腦裂」。
  
DRBD官方推薦的手動恢復方案：
  
1）Secondary主機上的操做
# drbdadm secondary r0
# drbdadm disconnect all
# drbdadm --discard-my-data connect r0            //或者"drbdadm  --  --discard-my-data  connect r0"
  
2）Primary主機上的操做
# drbdadm disconnect all
# drbdadm connect r0
# drbdsetup /dev/drbd0 primary

更多相關文章...

相關標籤/搜索

負載均衡

Linux

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。