最少三臺Centos7系統虛擬機用於部署Ceph集羣。硬件配置:2C4G,另外每臺機器最少掛載三塊硬盤(每塊盤5G).如下爲本次環境規劃:node
HOST | IP | Service | Mark |
---|---|---|---|
cephmanager01 | public network:192.168.10.51--ens224 <br>cluster network:172.168.10.51--ens192 | mon, mgr,mds | 8c8g<br>系統盤:200G |
cephmanager02 | public network:192.168.10.72--ens224 <br/>cluster network:172.168.10.72--ens192 | mon, mgr,mds | 8c8g<br/>系統盤:200G |
cephmanager03 | public network:192.168.10.75--ens224 <br/>cluster network:172.168.10.75--ens192 | mon, mgr,mds | 8c8g<br/>系統盤:200G |
cephnode001 | public network:192.168.10.76--ens224 <br/>cluster network:172.168.10.76--ens192 | osd | 4c8g<br>系統盤:200G<br>數據盤:50G/塊x4 |
cephnode002 | public network:192.168.10.82--ens224 <br/>cluster network:172.168.10.82--ens192 | osd | 4c8g<br/>系統盤:200G<br/>數據盤:50G/塊x4 |
cephnode003 | public network:192.168.10.101--ens224 <br/>cluster network:172.168.10.101--ens192 | osd | 4c8g<br/>系統盤:200G<br/>數據盤:50G/塊x4 |
組件 | 版本 | 建議配置 | 備註 |
CentOS | CentOS Linux release 7.4.1708 | 管理節點CPU要好 | |
Ceph | 13.2.10 mimic | 數據節點磁盤空間大 |
[在manager節點和node節點執行]python
關閉防火牆並設置開機關閉linux
# systemctl stop firewalld && systemctl disable firewalld && systemctl status firewalld
或者配置防火牆端口(根據狀況添加對應端口)shell
# firewall-cmd --zone=public --add-port=6789/tcp --permanent # firewall-cmd --zone=public --add-port=6800-7100/tcp --permanent # firewall-cmd --reload # firewall-cmd --zone=public --list-all
永久關閉(須要重啓)bootstrap
# sed -i 's/enforcing/disabled/' /etc/selinux/config
臨時關閉vim
# setenforce 0
確認關閉後端
# getenforce
# systemctl disable NetworkManager && systemctl stop NetworkManager && systemctl status NetworkManager
設置主機名[在對應節點上執行]服務器
# hostnamectl --static set-hostname cephmanager01 # hostnamectl --static set-hostname cephmanager02 # hostnamectl --static set-hostname cephmanager03 # hostnamectl --static set-hostname cephnode001 # hostnamectl --static set-hostname cephnode002 # hostnamectl --static set-hostname cephnode003
修改hosts網絡
# cat >> /etc/hosts << EOF 192.168.10.51 cephmanager01 192.168.10.72 cephmanager02 192.168.10.75 cephmanager03 192.168.10.76 cephnode001 192.168.10.82 cephnode002 192.168.10.101 cephnode003 EOF
1)設置時區並同步互聯網時間
# yum install -y ntpdate chrony # timedatectl set-timezone Asia/Shanghai # ntpdate time1.aliyun.com
2)配置全部服務器的時間和時間服務器同步或者同某一臺服務器同步[本例同步cephmanager01節點時間]
編輯 /etc/chrony.conf 文件,添加(互聯網上的時間服務器能夠刪除):
server 192.168.10.51 iburst
重啓服務並配置開機啓動
# systemctl restart chronyd.service && systemctl enable chronyd.service && systemctl status chronyd.service
驗證:是否與cephmanager01同步
# chronyc sources
# echo "ulimit -SHn 102400" >> /etc/rc.local # cat >> /etc/security/limits.conf << EOF * soft nofile 65535 * hard nofile 65535 EOF
# cat >> /etc/sysctl.conf << EOF kernel.pid_max = 4194303 vm.swappiness = 0 EOF # sysctl -p
經過數據預讀而且記載到隨機訪問內存方式提升磁盤讀操做
# echo "8192" > /sys/block/sda/queue/read_ahead_kb
SSD要用noop,SATA/SAS使用deadline
# echo "deadline" >/sys/block/sd[x]/queue/scheduler # echo "noop" >/sys/block/sd[x]/queue/scheduler
# yum install wget vim lrzsz -y
# useradd -d /home/cephuser -m cephuser # passwd cephuser Changing password for user cephuser. New password: ceph_pass Retype new password: ceph_pass
修改visudo文件,使cephuser用戶在sudo列表中;
# 在92行」 root ALL=(ALL) ALL」下新增一行:」 cephuser ALL=(ALL) ALL」 # visudo cephuser ALL=(ALL) ALL
設置cephuser用戶具有無密碼sudo(root)權限;切換到cephuser用戶下操做
# su - cephuser [cephuser@cephmanager01 ~]$ echo "cephuser ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephuser [sudo] password for cephuser: ceph_pass cephuser ALL = (root) NOPASSWD:AL [cephuser@cephmanager01 ~]$ sudo chmod 0440 /etc/sudoers.d/cephuser
1)生成祕鑰
ceph-deploy不支持密碼輸入,須要在管理控制節點生成ssh祕鑰,並將公鑰分發到各ceph節點;
在用戶cephuser下生成祕鑰,不能使用sudo或root用戶;默認在用戶目錄下生成~/.ssh目錄,含生成的祕鑰對;「Enter passphrase」時,回車,口令爲空;
備註:3個manager節點均設置爲ceph管理控制節點,應該使3個manager節點均能ssh免密登錄到其餘全部manager節點與node節點
# 分別在cephmanager01,cephmanager02,cephmanager03執行 [root@cephmanager01 ~]# su - cephuser Last login: Sun Jun 28 14:49:09 CST 2020 on pts/4 [cephuser@cephmanager01 ~]$ ssh-keygen -t rsa
2)分發密鑰
確認各控制manger與node節點已生成相關用戶cephuser;分發成功後,在~/.ssh/下生成known_hosts文件,記錄相關登錄信息;
# 須要分別在cephmanager0一、cephmanager0二、cephmanager03上分發到全部的manager和node節點 [cephuser@cephmanager01 ~]$ ssh-copy-id cephuser@cephmanager02
配置內網yum源服務器,或者配置對應的國內源,這裏配置ceph源和epel源
# cat > /etc/yum.repos.d/ceph.repo << EOF [ceph] name=ceph baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/x86_64/ gpgcheck=0 priority=1 [ceph-noarch] name=cephnoarch baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/noarch/ gpgcheck=0 priority=1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/SRPMS gpgcheck=0 priority=1 EOF # wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo # yum clean all # yum makecache # 查看yum源 # yum repolist
[在規劃的所有manager節點安裝ceph-deploy工具,本例以cephmanager01爲例]
推配置文件、變動維護,添加硬盤等操做都須要用到ceph-deploy,並確認ceph-deploy版本是否爲2.0.1
[root@cephmanager01 ~]# yum install ceph-deploy python-setuptools -y # 確認版本 [root@cephmanager01 ~]# ceph-deploy --version
[在規劃的所有manager節點執行,特殊說明處例外,本例以cephmanager01爲例]
在cephuser帳戶下操做,切忌使用sudo操做;在manager節點上建立一個目錄用於存放集羣相關配置文件;
[root@cephmanager01 ~]# su - cephuser [cephuser@cephmanager01 ~]$ mkdir cephcluster # 後續ceph-deploy相關操做所有在所建立的目錄執行 [cephuser@cephmanager01 ~]$ cd cephcluster/ # 將規劃中的MON(monitor)節點歸入集羣,即建立集羣, # [該操做任意一個cephmanager執行,以cephmanage01爲例] [cephuser@cephmanager01 cephcluster]$ ceph-deploy new cephmanager01 cephmanager02 cephmanager03 [cephuser@cephmanager01 cephcluster]$ ls ceph.conf ceph-deploy-ceph.log ceph.mon.keyrin
生成集羣后在集羣目錄下生成3個文件,其中ceph.conf便是配置文件;
默承認不修改,爲使服務按規劃啓動,可作適當修改;
如下紅色字體部分是在默認生成的conf文件上新增的配置
[該操做任意一個cephmanager執行,以cephmanage01爲例]
[cephde@controller01 cephcluster]$ cat ceph.conf [global] fsid = 6a71324c-6949-42bc-92ca-325c9d45ae06 mon_initial_members = cephmanager01, cephmanager02, cephmanager03 mon_host = 192.168.10.51,192.168.10.72,192.168.10.75 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # public network:前端mon網絡,client訪問網絡;確保public network與mon_host在相同網段,不然初始化時可能會有錯誤; # cluster network:後端osd心跳,數據/流複製恢復等網絡 public network = 192.168.10.0/24 cluster network = 172.168.10.0/24 # 默認的副本數爲3,根據實際狀況變動 osd pool default size = 3 # 默認保護機制不容許刪除pool,根據狀況設置 mon_allow_pool_delete = true
[在所有manager和node節點執行]
# 建議使用該命令安裝在每一個節點安裝 [root@cephmanager01 ~]# yum -y install epel-release [root@cephmanager01 ~]# yum -y install ceph # 不建議使用,該命令會在對應節點安裝,可是源會自動更換爲國外官方源,網速會很慢 [root@cephmanager01 ~]# ceph-deploy install --release mimic cephnode01 cephnode02 cephnode03 # 查看版本 [root@controller01 ~]# ceph -v
[該操做在任意一個cephmanager執行,以cephmanage01爲例]
[root@cephmanager01 ~]# su - cephuser Last login: Mon Jun 29 02:17:20 CST 2020 on pts/0 [cephuser@cephmanager01 ~]$ cd cephcluster/ [cephuser@cephmanager01 cephcluster]$ ceph-deploy mon create-initial # 初始化完成後,在集羣目錄下新增多個祕鑰文件 [cephuser@cephmanager01 cephcluster]$ ls -l
# 查看狀態 [cephuser@cephmanager01 cephcluster]$ sudo systemctl status ceph-mon@cephmanager01
# 分發ceph配置文件與祕鑰到其餘控制manager節點與node節點; # 注意分發節點自己也須要包含在內,默認沒有祕鑰文件,須要分發; # 若是被分發節點已經配置文件(統一變動配置文件場景),可使用以下命令:ceph-deploy --overwrite-conf admin xxx # 分發的配置文件與祕鑰到各節點/etc/ceph/目錄 [cephuser@cephmanager01 cephcluster]$ ceph-deploy admin cephmanager01 cephmanager02 cephmanager03 cephnode001 cephnode002 cephnode003
完後可使用ceph -s命令,tail -f ceph-deploy-ceph.log查看安裝日誌
將cephcluster目錄下面的文件拷貝到其他的manager節點:
[cephuser@cephmanager01 cephcluster]$ pwd /home/cephuser/cephcluster [cephuser@cephmanager01 cephcluster]$ scp ceph.conf ceph-deploy-ceph.log ceph.mon.keyring cephuser@cephmanager03:~/cephcluster/ [cephuser@cephmanager01 cephcluster]$ scp ceph.bootstrap-* cephuser@cephmanager02:~/cephcluster/
注意:L版以後才須要部署
[在任意manager節點執行,以cephmanager01爲例]
[cephuser@cephmanager01 cephcluster]$ ceph-deploy mgr create cephmanager01:cephmanager01_mgr cephmanager02:cephmanager02_mgr cephmanager03:cephmanager03_mgr # 查看狀態 [cephuser@cephmanager01 cephcluster]$ systemctl status ceph-mgr@cephmanager01_mgr [cephuser@cephmanager01 cephcluster]$ sudo ss -tunlp |grep mgr
[manager和node節點都可執行]
# 查看monitor狀態 [cephuser@cephmanager01 ~]$ sudo ceph mon stat # 查看ceph狀態 [cephuser@cephmanager01 ~]$ sudo ceph -s
osd位於node節點,可查看node節點磁盤情況,以cephnode001節點爲例;
或在manager節點採用命令:ceph-deploy disk list cephnode001 cephnode001 … cephnode00N;
[root@cephnode001 ~]# lsblk
實際建立osd時,可經過manager節點使用ceph-deploy建立[任意一manager節點執行,以cephmanager01爲例];
# 參數"--data"指定數據盤,參數"--journal"指定journal日誌盤,日誌盤能夠是logical volume(vg/lv)或GPT patition,是option操做; # 另有命令(L版本):ceph-deploy osd prepare {NODE}:{DISK}[:{/PATH/TO/JOURNAL}]; ceph-deploy osd activate {NODE}:{DISK}[:{/PATH/TO/JOURNAL}],其中[:{/PATH/TO/JOURNAL}]是option操做,可建立具備journal日誌的osd守護進程(節點) [cephuser@cephmanager01 cephcluster]$ ceph-deploy osd create cephnode001 --data /dev/sdb [cephuser@cephmanager01 cephcluster]$ ceph-deploy osd create cephnode001 --data /dev/sdc [cephuser@cephmanager01 cephcluster]$ ceph-deploy osd create cephnode001 --data /dev/sdd [cephuser@cephmanager01 cephcluster]$ ceph-deploy osd create cephnode001 --data /dev/sde
本例中有3個node節點(osd),每一個node節點運行4個osd進程(在6800~7300端口範圍內,每進程監聽1個本地端口)[以cephnode001爲例];
[root@cephnode001 ~]# lsblk [root@cephnode001 ~]# ps -aux |grep osd # osd進程端口號; [root@cephnode001 ~]# netstat -tunlp | grep osd # ceph-osd進程,根據啓動順序,每一個osd進程有特定的序號 [root@cephnode001 ~]# systemctl status ceph-osd@0
若是節點磁盤有數據,可經過命令初始化(謹慎):ceph-deploy disk zap {NODE} {DISK},這裏{DISK}能夠是物理盤符,也但是分區
在manager節點查看,以cephmanager01爲例
[cephuser@cephmanager01 cephcluster]$ ceph-deploy osd list cephnode001 [cephuser@cephmanager01 cephcluster]$ sudo ceph osd stat [cephuser@cephmanager01 cephcluster]$ sudo ceph osd tree [cephuser@cephmanager01 cephcluster]$ sudo ceph df
在node節點查看,以cephnode001爲例
[root@cephnode001 ~]# lsblk [root@cephnode001 ~]# ps -aux |grep osd [root@cephnode001 ~]# netstat -tunlp | grep osd [root@cephnode001 ~]# systemctl status ceph-osd@0