基於cephfs搭建高可用分佈式存儲並mount到本地

時間  2019-11-06
標籤 基於 cephfs 搭建 可用 分佈式 存儲 mount 本地 欄目 系統架構 简体版
原文   原文鏈接

 原文：https://www.fullstackmemo.com/2018/10/11/cephfs-ha-mount-storage/html

服務器硬件配置及環境

項目 說明
CPU 1核
內存 1GB
硬盤 40GB
系統 CentOS 7.5
時間同步服務 chrony
ceph 13.2.2-0

節點部署圖

節點部署圖

節點功能簡介

項目 說明
yum repo 若是部署環境每一個節點均可以訪問外網, 則無需作任何操做, 部署腳本會自動添加外網環境的yum源. 若是部署沒法訪問外網, 須要自行部署centos, epel, ceph三個yum源. 每一個節點都必須能夠訪問所提到的全部yum源
時間同步服務器 每一個節點都必須能夠訪問, 若是部署環境沒法訪問外網須要自行搭建時間同步服務器
client-x 須要掛載存儲的設備, 須要同時能夠訪問每一個storage-ha-x和yum源, 時間服務器
storage-deploy-1 用於統一部署ceph集羣的工做機, 系統爲CentOS 7.5
storage-ha-x 承載ceph各項服務的服務器節點, 系統爲CentOS 7.5
mon Monitors, 節點映射管理, 身份驗證管理, 須要達到冗餘和高可用至少須要3個節點
osd object storage daemon, 對象存儲服務, 須要達到冗餘和高可用至少須要3個節點
mgr Manager, 用於跟蹤運行指標和集羣狀態, 性能.
mds Metadata Serve, 提供cephfs的元數據存儲

參考:node

默認端口

項目 說明
ssh tcp: 22
mon tcp: 6789
mds/mgr/osd tcp: 6800~7300

參考:linux

默認路徑

項目 說明
主配置文件 /etc/ceph/ceph.conf
配置文件夾 /etc/ceph
日誌文件夾 /var/log/ceph
各服務認證key文件 /var/lib/ceph/{server name}/{hostname}/keyring
admin認證key文件 ceph.client.admin.keyring

部署腳本說明

  • node-init.sh: storage-ha-x節點初期運行的初始化腳本
  • admin-init.sh: storage-deploy-1節點初期運行的初始化腳本, 必需要在每一個storage-ha-x節點都運行完node-init.sh以後才能運行.
  • ceph-deploy.sh: ceph部署腳本, 僅在storage-deploy-1節點上運行便可, 須要在node-init.sh和admin-init.sh運行完成且成功後運行.

PS: 腳本中涉及到的ip和其它不一樣信息請先自行修改後再運行.git

腳本運行命令

請將 附錄: 腳本內容章節或腳本Git庫章節中的各個腳本放到各個對應服務器任意位置並使用如下命令按照順序運行.github

PS: 需嚴格按照部署腳本說明章節中的持續順序執行腳本.json

PS: 腳本中涉及到不一樣於當前環境的信息(如: ip, yum源, 密碼, 主機名等)請先自行修改後再運行.centos

  • 初始化ceph節點
1
 
/bin/bash node-init.sh

執行命令結果api

初始化ceph節點

  • 初始化部署節點
1
 
/bin/bash admin-init.sh

執行命令結果bash

初始化部署節點

初始化部署節點

  • 開始部署
1
 
/bin/bash ceph-deploy.sh

執行命令結果服務器

部署結果-1

能夠看到上方的pgs下方有個creating+peering, 這表示OSDs在建立和準備同步中.須要等待

這時能夠在任意有admin角色的storage-ha-x節點上執行如下命令看查是否完成準備

1
 
ceph -s

當pgs顯示爲下圖的active+clean表明各個節點同步完成.

部署結果-2

若是一直沒法達到active+clean狀態, 請參考如下操做文章:
TROUBLESHOOTING PGS

掛載存儲

建立測試用戶

如下命令在任意一個storage-ha-x服務器上運行

1
2
3
4
5
 
# 此命令含義是建立一個名爲client.fs-test-1的用戶, 對於掛載的根目錄'/'只有可讀權限, 對於掛載的'/test_1'目錄有讀寫權限.
ceph fs authorize cephfs client.fs-test-1 / r /test_1 rw
# 命令輸入完成後會返回類以下信息:
# [client.fs-test-1]
# key = AQA0Cr9b9afRDBAACJ0M8HxsP41XmLhbSxWkqA==

獲取用戶受權信息

如下命令在任意一個添加過admin角色的storage-ha-x服務器上運行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
 
# 獲取client.admin用戶的受權信息
ceph auth get client.admin
# 命令輸入後會返回相似以下信息
# [client.admin]
# key = AQAm4L5b60alLhAARxAgr9jQDLopr9fbXfm87w==
# caps mds = "allow *"
# caps mgr = "allow *"
# caps mon = "allow *"
# caps osd = "allow *"

# 獲取client.fs-test-1用戶的受權信息
ceph auth get client.fs-test-1
# 命令輸入後會返回相似以下信息
# [client.fs-test-1]
# key = AQA0Cr9b9afRDBAACJ0M8HxsP41XmLhbSxWkqA==
# caps mds = "allow r, allow rw path=/test-1"
# caps mon = "allow r"
# caps osd = "allow rw tag cephfs data=cephfs"

掛載方式

掛載方式分爲兩種, 分別是cephfs和fuse. 選擇其中一種方式進行掛載便可.

兩種掛載方式的區別和優點請參考如下文章:
WHICH CLIENT?

cephfs方式

如下命令在任意須要掛載存儲的client下執行

PS: 此掛載方式依賴於ceph, 須要先添加ceph和epel的yum源.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
 
# cephfs方式掛載
yum install ceph -y
mkdir -p /etc/ceph
mkdir -p /mnt/mycephfs
# 如下寫入的secret請根據'獲取用戶受權信息'章節中獲取到的'key'進行修改
cat > /etc/ceph/admin_secret.key << EOF
AQAm4L5b60alLhAARxAgr9jQDLopr9fbXfm87w==
EOF

# 如下寫入的secret請根據'獲取用戶受權信息'章節中獲取到的'key'進行修改
cat > /etc/ceph/test_cephfs_1_secret.key << EOF
AQA0Cr9b9afRDBAACJ0M8HxsP41XmLhbSxWkqA==
EOF

# 使用'admin'用戶掛載cephfs的根目錄
# ip或主機名請根據實際狀況修改
# 這裏填寫的'name=admin'是'client.admin'點後面的'admin'.
mount.ceph 192.168.60.111:6789,192.168.60.112:6789,192.168.60.113:6789:/ /mnt/mycephfs -o name=admin,secretfile=/etc/ceph/admin_secret.key

# 使用只讀的用戶掛載
mkdir -p /mnt/mycephfs/test_1
mkdir -p /mnt/test_cephfs_1
# 使用'fs-test-1'用戶掛載cephfs的根目錄
# ip或主機名請根據實際狀況修改
# 這裏填寫的'name=fs-test-1'是'client.fs-test-1'點後面的'fs-test-1'.
mount.ceph 192.168.60.111:6789,192.168.60.112:6789,192.168.60.113:6789:/ /mnt/test_cephfs_1 -o name=fs-test-1,secretfile=/etc/ceph/test_cephfs_1_secret.key

# 開機自動掛載
cat >> /etc/fstab << EOF
192.168.60.111:6789,192.168.60.112:6789,192.168.60.113:6789:/ /mnt/mycephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev 0 2
EOF

fuse方式

如下命令在任意須要掛載存儲的client下執行

PS: 此掛載方式依賴於ceph-fuse, 須要先添加ceph和epel的yum源.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
 
yum install ceph-fuse -y
mkdir -p /etc/ceph
mkdir -p /mnt/mycephfs

# 獲取storage-ha-x任意一個節點上的ceph配置文件
scp storage@storage-ha-1:/etc/ceph/ceph.conf /etc/ceph/ceph.conf

# 如下寫入的secret請根據'獲取用戶受權信息'章節中獲取到的'key'進行修改
cat > /etc/ceph/ceph.keyring << EOF
[client.admin]
 key = AQAm4L5b60alLhAARxAgr9jQDLopr9fbXfm87w==
 caps mds = "allow *"
 caps mgr = "allow *"
 caps mon = "allow *"
 caps osd = "allow *"
[client.fs-test-1]
 key = AQA0Cr9b9afRDBAACJ0M8HxsP41XmLhbSxWkqA==
 caps mds = "allow r, allow rw path=/test-1"
 caps mon = "allow r"
 caps osd = "allow rw tag cephfs data=cephfs"
EOF

# 使用'admin'用戶掛載cephfs的根目錄
# ip或主機名請根據實際狀況修改
ceph-fuse -m 192.168.60.111:6789,192.168.60.112:6789,192.168.60.113:6789 /mnt/mycephfs
# 開機自動掛載
cat >> /etc/fstab << EOF
none /mnt/ceph fuse.ceph ceph.id=admin,ceph.conf=/etc/ceph/ceph.conf,_netdev,defaults 0 0
EOF

# 使用只讀的用戶掛載
mkdir -p /mnt/mycephfs/test_1
mkdir -p /mnt/test_cephfs_1
# 使用'fs-test-1'用戶掛載cephfs的根目錄
# ip或主機名請根據實際狀況修改
# 這裏填寫的'-n client.fs-test-1'是完整的'client.fs-test-1'.
ceph-fuse -m 192.168.60.111:6789,192.168.60.112:6789,192.168.60.113:6789 -n client.fs-test-1 /mnt/test_cephfs_1

掛載結果

掛載結果可使用如下命令查看

1
 
df -h

掛載結果

運維命令

  • 狀態查看
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
 
# 查看集羣總體狀態
ceph -s

# 查看集羣健康狀態
ceph health

# 查看集羣健康狀態詳情
ceph health detail

# 查看cephfs列表
ceph fs ls

# 查看mds狀態
ceph mds stat

# 查看 osd節點狀態
ceph osd tree

# 查看監視器狀況
ceph quorum_status --format json-pretty
  • 簡單寫性能測試
1
2
 
# 在掛載了存儲的client下簡單測試寫性能
time dd if=/dev/zero of=/mnt/mycephfs/test.dbf bs=8k count=3000 oflag=direct

測試結果

  • 刪除cephfs和pool
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 
# 刪除cephfs前須要的操做
# 中止每一個mds節點的mds服務
# 每一個mds節點上都要執行
systemctl stop ceph-mds.target

# 僅在任意一臺'storage-ha-x'上執行
# 刪除cephfs
ceph fs rm cephfs --yes-i-really-mean-it

# 刪除pool
# 須要刪除pool的時候須要寫兩次pool名外帶'--yes-i-really-really-mean-it'參數
ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it
ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it

# 中止每一個mds節點的mds服務
# 每一個mds節點上都要執行
systemctl start ceph-mds.target
  • 同步ceph配置文件
1
2
3
4
5
6
7
8
9
10
11
 
# 同步的配置文件
# 若是節點上有配置文件且和當前要同步的配置文件不通, 須要帶'--overwrite-conf'參數
# 此命令會把執行此命令目錄下的'ceph.conf'文件同步到各個指定節點上
ceph-deploy --overwrite-conf config push storage-ha-1 storage-ha-2 storage-ha-3

# 重啓每一個節點的cepf相關服務
# 須要在有對應功能節點的節點上分別運行如下命令
systemctl restart ceph-osd.target
systemctl restart ceph-mds.target
systemctl restart ceph-mon.target
systemctl restart ceph-mgr.target

FAQ

  • Q: health_warn：clock skew detected on mon
    A: 使用chrony同步每臺服務器節點的時間

  • Q: Error ERANGE: pg_num 「*「 size 「*「 would mean 「*「 total pgs, which exceeds max 「*「 (mon_max_pg_per_osd 250 num_in_osds 「\「)
    A: ceph.conf配置文件中加入mon_max_pg_per_osd = 1000(參數中的數值本身根據實際狀況修改)並用同步ceph配置文件方式上傳到各個節點, 並重啓ceph-mon.target

  • Q: too many PGs per OSD
    A: ceph.conf配置文件中加入mon_max_pg_per_osd = 1000(參數中的數值本身根據實際狀況修改)並用同步ceph配置文件方式上傳到各個節點, 並重啓ceph-mon.target

參考

ceph cephx認證參考
設置cephfs訪問權限
ceph用戶管理
ceph-fuse方式掛載
Ceph 運維手冊
Red Hat Ceph存儲—《深刻理解Ceph架構》
Ceph常規操做及常見問題梳理

腳本Git庫

https://github.com/x22x22/cephfs-verify-script

附錄: 腳本內容

  • node-init.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
 
#!/bin/bash

# 禁用ipv6, 加大pid限制
cat >>/etc/sysctl.conf <<EOF
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
kernel.pid_max = 4194303
EOF

sysctl -p
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1

# 簡單代替dns服務器寫入當前環境中的主機名和ip的對應關係
cat >>/etc/hosts <<EOF

192.168.60.110 storage-deploy-1
192.168.60.111 storage-ha-1
192.168.60.112 storage-ha-2
192.168.60.113 storage-ha-3
EOF

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

# 添加一個storage用戶, 用於ceph-deploy工具進行節點的安裝和操做
useradd -d /home/storage -m storage
echo 'fullstackmemo***' | passwd --stdin storage
echo "storage ALL = (root) NOPASSWD:ALL" | tee /etc/sudoers.d/storage
chmod 0440 /etc/sudoers.d/storage

# 添加ceph的yum源, 若是沒法訪問外網請自行搭建並修改
cat >/etc/yum.repos.d/ceph.repo <<'EOF'
[Ceph]
name=Ceph packages for $basearch
baseurl=http://mirror.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirror.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirror.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirror.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1

[ceph-source]
name=Ceph source packages
baseurl=http://mirror.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirror.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1

EOF

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

# 修改CentOS的yum基礎源, 若是沒法訪問外網請自行搭建並修改
cat >/etc/yum.repos.d/CentOS-Base.repo <<'EOF'
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client. You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/os/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates]
name=CentOS-$releasever - Updates
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/updates/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/extras/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/centosplus/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF

yum makecache fast
# 安裝CentOS的yum epel源
yum install -y epel-release

# 修改CentOS的yum epel源, 若是沒法訪問外網請自行搭建並修改
cat >/etc/yum.repos.d/epel.repo <<'EOF'
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/$basearch
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/$basearch/debug
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/SRPMS
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
EOF

yum makecache
yum install yum-plugin-priorities chrony parted xfsprogs -y
mv /etc/chrony.conf /etc/chrony.conf.bk

# 添加時間同步服務器, 若是沒法訪問外網請自行搭建並修改
# 添加時間同步服務器, 若是沒法訪問外網請更換成yum.yfb.sunline.cn和nexus.yfb.sunline.cn
cat > /etc/chrony.conf << EOF
server ntp.api.bz iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony
EOF

systemctl enable chronyd
systemctl restart chronyd
sleep 10
chronyc activity
chronyc sources -v
hwclock -w

# 這裏將/dev/sdb做爲ceph的存儲池, 因此先格式化/dev/sdb, 請根據本身實際狀況修改
parted -s /dev/sdb mklabel gpt mkpart primary xfs 0% 100%
partprobe /dev/sdb
mkfs.xfs /dev/sdb -f
  • admin-init.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
 
#!/bin/bash

# 禁用ipv6
cat >>/etc/sysctl.conf <<EOF
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
EOF

sysctl -p
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1

# 簡單代替dns服務器寫入當前環境中的主機名和ip的對應關係
cat >>/etc/hosts <<EOF

192.168.60.110 storage-deploy-1
192.168.60.111 storage-ha-1
192.168.60.112 storage-ha-2
192.168.60.113 storage-ha-3
EOF

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

# 添加ceph的yum源, 若是沒法訪問外網請自行搭建並修改
cat >/etc/yum.repos.d/ceph.repo <<'EOF'
[Ceph]
name=Ceph packages for $basearch
baseurl=http://mirror.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirror.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirror.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirror.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1

[ceph-source]
name=Ceph source packages
baseurl=http://mirror.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirror.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1

EOF

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

# 修改CentOS的yum基礎源, 若是沒法訪問外網請自行搭建並修改
cat >/etc/yum.repos.d/CentOS-Base.repo <<'EOF'
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client. You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/os/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates]
name=CentOS-$releasever - Updates
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/updates/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/extras/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/$releasever/centosplus/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF

yum makecache fast
# 安裝CentOS的yum epel源
yum install -y epel-release

# 修改CentOS的yum epel源, 若是沒法訪問外網請自行搭建並修改
cat >/etc/yum.repos.d/epel.repo <<'EOF'
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/$basearch
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/$basearch/debug
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/SRPMS
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
EOF

yum makecache
yum install yum-plugin-priorities chrony sshpass ceph-deploy ceph -y
mv /etc/chrony.conf /etc/chrony.conf.bk

# 添加時間同步服務器, 若是沒法訪問外網請自行搭建並修改
cat > /etc/chrony.conf << EOF
server ntp.api.bz iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony
EOF

systemctl enable chronyd
systemctl restart chronyd
sleep 10
chronyc activity
chronyc sources -v
hwclock -w

rm -f "${HOME}"/.ssh/ceph_id_rsa
ssh-keygen -t rsa -b 4096 -f "${HOME}"/.ssh/ceph_id_rsa -N ''
cat >"${HOME}"/.ssh/config <<EOF
Host storage-ha-1
 Hostname storage-ha-1
 User storage
 IdentityFile ${HOME}/.ssh/ceph_id_rsa
 IdentitiesOnly yes
 StrictHostKeyChecking no
Host storage-ha-2
 Hostname storage-ha-2
 User storage
 IdentityFile ${HOME}/.ssh/ceph_id_rsa
 IdentitiesOnly yes
 StrictHostKeyChecking no
Host storage-ha-3
 Hostname storage-ha-3
 User storage
 IdentityFile ${HOME}/.ssh/ceph_id_rsa
 IdentitiesOnly yes
 StrictHostKeyChecking no
EOF
chmod 0400 "${HOME}"/.ssh/config
sshpass -p "fullstackmemo***" ssh-copy-id -i ~/.ssh/ceph_id_rsa storage@storage-ha-1
sshpass -p "fullstackmemo***" ssh-copy-id -i ~/.ssh/ceph_id_rsa storage@storage-ha-2
sshpass -p "fullstackmemo***" ssh-copy-id -i ~/.ssh/ceph_id_rsa storage@storage-ha-3

mkdir -p "${HOME}"/ceph-cluster
cd "${HOME}"/ceph-cluster || exit
  • ceph-deploy.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
 
#!/bin/bash

mkdir -p "${HOME}"/ceph-cluster
cd "${HOME}"/ceph-cluster || exit
ceph-deploy new storage-ha-1 storage-ha-2 storage-ha-3

cat >>ceph.conf <<EOF
# 'public network':
# 整個集羣所存在的網段
# 這裏須要根據實際狀況修改
public network = 192.168.60.0/24
osd pool default size = 3
osd pool default min size = 2
osd pool default pg num = 100
osd pool default pgp num = 100
# 'mon allow pool delete': 
# 此設置容許刪除pool的操做, poc環境爲方便操做加上此選項, 生產環境建議註釋
mon allow pool delete = true

[osd]
osd_max_backfills = 1
osd_recovery_max_active = 1
osd_recovery_op_priority = 1
EOF

# 在各個節點上安裝ceph, 並指定了外網的ceph yum源, 若是沒法訪問外網請自行搭建並修改
ceph-deploy install storage-ha-1 storage-ha-2 storage-ha-3 --repo-url http://mirrors.ustc.edu.cn/ceph/rpm-mimic/el7 --gpg-url 'http://mirrors.ustc.edu.cn/ceph/keys/release.asc'
# 初始化mon服務和key信息
ceph-deploy mon create-initial
ceph-deploy mon add storage-ha-2
ceph-deploy mon add storage-ha-3
ceph-deploy admin storage-ha-1 storage-ha-2 storage-ha-3
ceph-deploy mgr create storage-ha-1 storage-ha-2 storage-ha-3

# 添加存儲服務節點上的裸盤到存儲池中
ceph-deploy osd create --data /dev/sdb storage-ha-1
ceph-deploy osd create --data /dev/sdb storage-ha-2
ceph-deploy osd create --data /dev/sdb storage-ha-3

ceph-deploy mds create storage-ha-1 storage-ha-2 storage-ha-3

ssh storage@storage-ha-1 << EOF
# 建立兩個pool, 服務於cephfs, cephfs至少須要兩個pool, 分別作metadata和data
sudo ceph osd pool create cephfs_data 100
# 使用raid 5方式存儲數據即erasure類型, 當單個文件平均大小大於8k時erasure比replicated有優點.
# sudo ceph osd pool create cephfs_data 100 100 erasure
# sudo ceph osd pool set cephfs_data allow_ec_overwrites true
# sudo metadata pool必須使用replicated類型.
sudo ceph osd pool create cephfs_metadata 100
# 若是使用了erasure類型, 此步驟跳過
sudo ceph osd pool set cephfs_data size 3

sudo ceph osd pool set cephfs_metadata size 3
sudo ceph fs new cephfs cephfs_metadata cephfs_data

# 查看集羣各項信息
sudo ceph quorum_status --format json-pretty
sudo ceph fs ls
sudo ceph mds stat
sudo ceph health
sudo ceph -s
EOF
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程