SUSE Ceph 快速部署 - Storage6

時間 2019-11-24

標籤 suse ceph 快速部署 storage6 storage 简体版

原文原文鏈接

學習 SUSE Storage 系列文章

（1）SUSE Storage6 實驗環境搭建詳細步驟 - Win10 + VMware WorkStation
html

（2）SUSE Linux Enterprise 15 SP1 系統安裝node

（3）SUSE Ceph 快速部署 - Storage6python

（4）SUSE Ceph 增長節點、減小節點、刪除OSD磁盤等操做 - Storage6
git

（5）深刻理解 DeepSea 和 Salt 部署工具 - Storage6
github

1、安裝環境描述

整個環境採用VMware workstation搭建，一共5臺虛擬機，全部虛擬機安裝SLES15SP1系統，其中一臺安裝apache做爲倉庫使用，剩餘4臺用於搭建Storage6集羣。web

（1）硬件環境：apache

筆記本一臺，CPU、內存和磁盤空間足夠
筆記本上創建一塊環回接口用於分佈式存儲管理網絡

（2）軟件環境：vim

筆記本安裝 Win10 操做系統
虛擬環境：VMware Workstation 14 Pro

使用VMnet2和VMnet3 做爲分佈式存儲 public和cluster網絡windows

虛擬機操做系統：SLES15SP1 企業版操做系統

一、網絡bash

主機名 public網絡 管理網絡 集羣網絡 描述  
smt                       172.200.50.19                   SUSE倉庫
admin     192.168.2.39    172.200.50.39   192.168.3.39    管理主機
node001   192.168.2.41    172.200.50.41   192.168.3.41    MON
node002   192.168.2.42    172.200.50.42   192.168.3.42    MON
node003   192.168.2.43    172.200.50.43   192.168.3.43    MON

二、磁盤
每一個節點系統有2塊 OSD 盤和1塊 NVME 磁盤

 1 # lsblk 
 2 NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
 3 sda               8:0    0   20G  0 disk              # 操做系統盤
 4 ├─sda1            8:1    0    1G  0 part /boot
 5 └─sda2            8:2    0   19G  0 part 
 6   ├─vgoo-lvroot 254:0    0   17G  0 lvm  /
 7   └─vgoo-lvswap 254:1    0    2G  0 lvm  [SWAP]
 8 sdb               8:16   0   10G  0 disk              # osd 數據盤
 9 sdc               8:32   0   10G  0 disk              # osd 數據盤
10 nvme0n1         259:0    0   20G  0 disk              # wal db

2、操做系統初始化安裝

一、臨時IP地址設置

1 ip link set eth0 up
2 ip addr add 172.200.50.50/24 dev eth0

鏈接上了設置永久地址

yast lan list
yast lan edit id=0 ip=192.168.2.40 netmask=255.255.255.0

二、設置bash環境變量和別名

# vim /root/.bash_profile 
alias cd..='cd ..'
alias dir='ls -l'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias l='ls -alF'
alias la='ls -la'
alias ll='ls -l'
alias ls-l='ls -l'

三、配置after.local文件

touch /etc/init.d/after.local
chmod 744 /etc/init.d/after.local

複製該內容進去

#! /bin/sh
#
# Copyright (c) 2010 SuSE LINUX Products GmbH, Germany.  All rights reserved.
#
# Author: Werner Fink, 2010
#
# /etc/init.d/after.local
#
# script with local commands to be executed from init after all scripts
# of a runlevel have been executed.
#
# Here you should add things, that should happen directly after
# runlevel has been reached.
#

四、倉庫配置(全部節點和admin)

## Pool
zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Product-SLES/15-SP1/x86_64/product/    SLE-Product-SLES15-SP1-Pool
zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Basesystem/15-SP1/x86_64/product/ SLE-Module-Basesystem-SLES15-SP1-Pool
zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Server-Applications/15-SP1/x86_64/product/ SLE-Module-Server-Applications-SLES15-SP1-Pool
zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Legacy/15-SP1/x86_64/product/  SLE-Module-Legacy-SLES15-SP1-Pool
zypper ar  http://172.200.50.19/repo/SUSE/Products/Storage/6/x86_64/product/ SUSE-Enterprise-Storage-6-Pool

## Update
zypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Product-SLES/15-SP1/x86_64/update/ SLE-Product-SLES15-SP1-Updates
zypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Basesystem/15-SP1/x86_64/update/  SLE-Module-Basesystem-SLES15-SP1-Upadates
zypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Server-Applications/15-SP1/x86_64/update/ SLE-Module-Server-Applications-SLES15-SP1-Upadates
zypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Legacy/15-SP1/x86_64/update/ SLE-Module-Legacy-SLES15-SP1-Updates
zypper ar http://172.200.50.19/repo/SUSE/Updates/Storage/6/x86_64/update/ SUSE-Enterprise-Storage-6-Updates

# zypper lr
#  | Alias                                              | Name                                               
---+----------------------------------------------------+----------------------------------------------------
 1 | SLE-Module-Basesystem-SLES15-SP1-Pool              | SLE-Module-Basesystem-SLES15-SP1-Pool              
 2 | SLE-Module-Basesystem-SLES15-SP1-Upadates          | SLE-Module-Basesystem-SLES15-SP1-Upadates          
 3 | SLE-Module-Legacy-SLES15-SP1-Pool                  | SLE-Module-Legacy-SLES15-SP1-Pool                  
 4 | SLE-Module-Legacy-SLES15-SP1-Updates               | SLE-Module-Legacy-SLES15-SP1-Updates               
 5 | SLE-Module-Server-Applications-SLES15-SP1-Pool     | SLE-Module-Server-Applications-SLES15-SP1-Pool     
 6 | SLE-Module-Server-Applications-SLES15-SP1-Upadates | SLE-Module-Server-Applications-SLES15-SP1-Upadates 
 7 | SLE-Product-SLES15-SP1-Pool                        | SLE-Product-SLES15-SP1-Pool                        
 8 | SLE-Product-SLES15-SP1-Updates                     | SLE-Product-SLES15-SP1-Updates                     
 9 | SUSE-Enterprise-Storage-6-Pool                     | SUSE-Enterprise-Storage-6-Pool                     
10 | SUSE-Enterprise-Storage-6-Updates                  | SUSE-Enterprise-Storage-6-Updates

五、安裝基本軟件 (全部節點和admin)

zypper in -y -t pattern yast2_basis base
zypper in -y net-tools vim man sudo tuned irqbalance
zypper in -y ethtool rsyslog iputils less supportutils-plugin-ses
zypper in -y net-tools-deprecated tree wget

六、關閉IPV6 (全部節點和admin)

# vim /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 1                 # 關閉 IPV6
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
vm.min_free_kbytes = 2097152                       # 128GB的RAM，系統預留2GB
kernel.pid_max = 4194303                           # 線程數設置最大

執行生效

# sysctl -p

七、調整網絡優化參數 (全部節點和admin)

# tuned-adm profile throughput-performance
# tuned-adm active
# systemctl start tuned.service
# systemctl enable tuned.service

八、編輯hosts文件 (全部節點和admin)

# vim /etc/hosts
192.168.2.39    admin.example.com admin
192.168.2.40    node001.example.com node001
192.168.2.41    node002.example.com node002
192.168.2.42    node003.example.com node003

九、執行更新操做系統，並重啓 (全部節點和admin)

# zypper ref
# zypper -n update
# reboot

3、安裝Storage6集羣

一、安裝 salt（admin節點）

zypper -n in deepsea
systemctl restart salt-master.service
systemctl enable salt-master.service
systemctl status salt-master.service

osd節點和 admin節點

zypper -n in salt-minion
sed -i '17i\master: 192.168.2.39' /etc/salt/minion
systemctl restart salt-minion.service
systemctl enable salt-minion.service
systemctl status salt-minion.service

接受全部請求（admin節點）

salt-key
salt-key --accept-all
salt-key
salt '*' test.ping

二、Admin節點，配置NTP服務
若是沒有ntp服務器，默認以admin節點爲ntp server
Admin節點：

# vim /etc/chrony.conf
# Sync to local clock        # 添加本地時鐘源
server 127.0.0.1             
allow 127.0.0.0/8
allow 192.168.2.0/24
allow 172.200.50.0/24
local stratum 10

systemctl restart chronyd.service
systemctl enable chronyd.service
systemctl status chronyd.service

# chronyc sources      
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 127.127.1.0                  12   6    37    23  +1461ns[+3422ns] +/-  166us

# chronyc -n sources -v

三、修改組（admin節點）

cp -p /srv/pillar/ceph/deepsea_minions.sls /tmp/
sed -i "4c # deepsea_minions: 'G@deepsea:*'" /srv/pillar/ceph/deepsea_minions.sls
sed -i "6c deepsea_minions: '*'" /srv/pillar/ceph/deepsea_minions.sls

四、遠程鏈接終端監控（admin節點）

該程序監控提供一個詳細的，實時的可視化操做行爲，當在執行運行salt-run state.orch時，監控執行期間運行了什麼

# deepsea monitor

五、更新補丁，並收集硬件信息（admin節點）

salt-run state.orch ceph.stage.0
salt-run state.orch ceph.stage.1

報錯信息可忽略

No minions matched the target. No command was sent, no jid was assigned.
No minions matched the target. No command was sent, no jid was assigned.
[ERROR   ] Exception during resolving address: [Errno 2] Host name lookup failure
[ERROR   ] Exception during resolving address: [Errno 2] Host name lookup failure
[WARNING ] /usr/lib/python3.6/site-packages/salt/grains/core.py:2827: DeprecationWarning: This server_id is
computed nor by Adler32 neither by CRC32. Please use "server_id_use_crc" option and define algorithm youprefer (
default "Adler32"). The server_id will be computed withAdler32 by default.

GitHub🔗

https://github.com/SUSE/DeepSea/issues/1593

六、查看網絡配置文件（admin節點）

# vim /srv/pillar/ceph/proposals/config/stack/default/ceph/cluster.yml
 cluster_network: 192.168.3.0/24
 fsid: 10aca2da-ead5-438d-b104-da37870b50b8
 public_network: 192.168.2.0/24

七、配置集羣policy.cfg文件（admin節點）
（1）模板文件，複製policy.cfg-rolebased模板

# ll /usr/share/doc/packages/deepsea/examples/
total 12
-rw-r--r-- 1 root root 329 Jun 13 16:00 policy.cfg-generic
-rw-r--r-- 1 root root 489 Jun 13 16:00 policy.cfg-regex
-rw-r--r-- 1 root root 577 Jun 13 16:00 policy.cfg-rolebased

# cp /usr/share/doc/packages/deepsea/examples/policy.cfg-rolebased /srv/pillar/ceph/proposals/policy.cfg

（2）編輯模板文件（admin節點）

# vim /srv/pillar/ceph/proposals/policy.cfg
## Cluster Assignment
cluster-ceph/cluster/*.sls

## Roles
# ADMIN  
role-master/cluster/admin*.sls
role-admin/cluster/admin*.sls

# Monitoring
role-prometheus/cluster/admin*.sls
role-grafana/cluster/admin*.sls

# MON
role-mon/cluster/node00[1-3]*.sls

# MGR (mgrs are usually colocated with mons)
role-mgr/cluster/node00[1-3]*.sls

# COMMON
config/stack/default/global.yml
config/stack/default/ceph/cluster.yml

# Storage   # 定義爲 storage 角色
role-storage/cluster/node00*.sls

（3）執行stage2命令（admin節點）

# salt-run state.orch ceph.stage.2
# salt '*' pillar.items    # 查看設置是否正確

尤爲是NTP，role 角色定義，public network 網絡是否認義正確

    public_network:
        192.168.2.0/24
    roles:
        - mon
        - mgr
        - storage
    time_server:
        admin.example.com

（4）若是3個節點須要修改（admin節點）
因爲測試環境，只用到3臺OSD節點，官方建議生產環境必須是4臺節點或以上

# sed -i 's/if (not self.in_dev_env and len(storage) < 4/if (not self.in_dev_env and len(storage) < 2/g' /srv/modules/runners/validate.py

八、定義和建立 OSD 磁盤
（1）備份配置文件

# cp /srv/salt/ceph/configuration/files/drive_groups.yml /srv/salt/ceph/configuration/files/drive_groups.yml.bak

（2）查看OSD節點磁盤狀況（node001,node002,node003)

# ceph-volume inventory
 stderr: blkid: error: /dev/sr0: No medium found

Device Path               Size         rotates available Model name
/dev/nvme0n1              20.00 GB     False   True      VMware Virtual NVMe Disk
/dev/sdb                  10.00 GB     True    True      VMware Virtual S
/dev/sdc                  10.00 GB     True    True      VMware Virtual S
/dev/sda                  20.00 GB     True    False     VMware Virtual S
/dev/sr0                  1024.00 MB   True    False     VMware SATA CD01

（3）編輯配置文件

# vim /srv/salt/ceph/configuration/files/drive_groups.yml
drive_group_hdd_nvme:             # 目標爲 storage角色節點
  target: 'I@roles:storage'
  data_devices:
    size: '9GB:12GB'              # 數據設備按照磁盤大小來區分，9G到12G之間
  db_devices:
    rotational: 0                 # 非機械設備 SSD or NVME
  block_db_size: '2G'             # 指定 db大小爲2GB  （大小按實際狀況）

（4）顯示OSD配置報告

能夠清楚的看到一塊nvme0n1做爲BlueStore的DB設備，2G大小，分割成2個LV對應2塊OSD數據磁盤。

# salt-run disks.report
  node003.example.com:
      |_
        - 0
        -
          Total OSDs: 2
          
          Solid State VG:
            Targets:   block.db                  Total size: 19.00 GB                 
            Total LVs: 2                         Size per LV: 1.86 GB                  
            Devices:   /dev/nvme0n1
          
            Type            Path                LV Size         % of device
          ----------------------------------------------------------------
            [data]          /dev/sdb            9.00 GB         100.0%
            [block.db]      vg: vg/lv           1.86 GB         10%
          ----------------------------------------------------------------
            [data]          /dev/sdc            9.00 GB         100.0%
            [block.db]      vg: vg/lv           1.86 GB         10%

注意：若是磁盤沒法識別請使用以下命令格式化，不能有GPT分區

# ceph-volume lvm zap /dev/xx 
# ceph-volume lvm zap /dev/xx --destroy

（5）運行stage3 stage4

# salt-run state.orch ceph.stage.3
# salt-run state.orch ceph.stage.4

4、配置HAproxy和Dashboard

一、添加SUSE HA倉庫、安裝 Haproxy (admin節點)

# zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Product-HA/15-SP1/x86_64/product/ SLE-Products-HA-SLES15-SP1-Pool
# zypper -n in haproxy

二、配置

# vim /etc/haproxy/haproxy.cfg
    ……
frontend http_web
    option tcplog
    bind 0.0.0.0:8443                 # 定義bind綁定，監聽那個套接字，若是是node001做爲admin節點，改爲9443
    mode tcp
    default_backend dashboard

backend dashboard
    mode tcp
    option log-health-checks
    option httpchk GET /
    http-check expect status 200
    server mgr1 172.200.50.40:8443 check ssl verify none
    server mgr2 172.200.50.41:8443 check ssl verify none
    server mgr3 172.200.50.42:8443 check ssl verify none

3) 啓動haproxy服務

# systemctl start haproxy.service
# systemctl enable haproxy.service
# systemctl status haproxy.service

4) 查看dashboard管理員密碼：

# salt-call grains.get dashboard_creds
local:
    ----------
    admin:
        9KyIXZSrdW

5）windows主機添加域名解析

C:\Windows\System32\drivers\etc\host
127.0.0.1 localhost
172.200.50.39 admin.example.com

6）訪問SES6 Dashboard頁面

http://172.200.50.39:8443/#/dashboard

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。