Ceph 集羣監控之Calamari 安裝部署

部署環境說明

其中，Calamari Server的版本是從ceph社區百度網盤共享的鏡像安裝的。

網盤地址是：

http://pan.baidu.com/s/1o6BT8sa#path=%252FCalamari%252FCentos%252FCentOS7.1%252BCalamari1.3

注意：本文在部署calamari以前已經將ceph環境部署好

[root@bgw-os-node151 ~]# ceph -s

    cluster00000000-0000-0000-0000-000000000002

     health HEALTH_OK

     monmap e1: 3 mons at{bgw-os-node151=10.240.216.151:6789/0,bgw-os-node152=10.240.216.152:6789/0,bgw-os-node153=10.240.216.153:6789/0},election epoch 380540, quorum 0,1,2bgw-os-node151,bgw-os-node152,bgw-os-node153

     mdsmap e62: 1/1/1 up{0=bgw-os-node151=up:active}, 2 up:standby

     osdmap e213915: 12 osds: 12 up, 12 in

      pgmap v543085: 384 pgs, 6 pools, 123 MBdata, 488 kobjects

            56185 MB used, 3295 GB / 3350 GBavail

                 384 active+clean

Calamari Server端安裝配置

說明：紅色部分表示內容    藍色部分表示命令

• 系統安裝（省略）

• CalamariServer安裝

• 關閉selinux

# vim /etc/selinux/config

      SELINUX=disabled

    # setenforce 0

• 關閉iptables

# systemctl stop firewalld

# chkconfig firewalld off

• 設置hosts

# cat /etc/hosts        （須要在上述四個服務器上都執行）

1. 10.240.240.200 calamari

2. 10.240.216.151 bgw-os-node151

3. 10.240.216.152 bgw-os-node152

4. 10.240.216.153 bgw-os-node153

5. n   設置yum源

#上傳鏡像文件（省略）並建立下面的掛載點，掛載鏡像文件

# mkdir -p /iso/calamari

# mkdir -p /iso/centos7

# mount -o loop /opt/CentOS-7-x86_64-DVD-1503.iso/iso/centos7/

# mount -o loop /opt/Calamari-1.3-CentOS-7.1-x86-64-20150430.iso/iso/calamari/

# cat /etc/yum.repos.d/CentOS-Base.repo

[base]

name=CentOS-7

baseurl=file:///iso/centos7

gpgcheck=0

        # cat/etc/yum.repos.d/calamari.repo

            [calamari]

name=calamari

baseurl=file:///iso/calamari/

gpgcheck=0

n   安裝基本包

# yumupdate

#yum -y install vim bash-completion net-tools pciutils sysstat wget

n   安裝Calamari和diamond包

# yum-y install calamari-server diamond

n   初始化

#/usr/bin/calamari-ctl initialize

輸入用戶名：root（保持默認，也能夠更改）

輸入密碼：zdh1113（隨意設置，用於登陸，最好一次設置好，修改比較麻煩）

n  配置calamari-client

Calamari-client是用來配置calamari登陸、管理所須要的相關頁面

# cd ~

# cp/iso/calamari/packages/calamari-clients-20150430.tar.gz .

# tar xf calamari-clients-20150430.tar.gz

# cd calamari-clients/

# cp -r login/dist/opt/calamari/webapp/content/login

# cp -r manage/dist/opt/calamari/webapp/content/manage

# cp -r dashboard/dist/opt/calamari/webapp/content/dashboard

# cp -r admin/dist/opt/calamari/webapp/content/admin

至此，按照社區裏奇總分享的鏡像和文檔進行安裝部署都很順利就能完成。

n   登陸

登陸時使用上面初始化的用戶名和密碼便可。登陸後出現下面的界面

Ceph端安裝配置

這一步是一個難點，經過各類搜索，終於搞定了，過程以下

• 查閱資料

http://tracker.ceph.com/issues/7851

• 安裝epel源

#rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

• 上傳diamond-3.4.67-0.noarch.rpm

• 安裝diamond和salt-minion和 ipvsadm

# yum install diamond-3.4.67-0.noarch.rpm ipvsadmsalt-minion

• 配置cephdeploy.conf

# cat ~/.cephdeploy.conf

#

# ceph-deploy configuration file

#

[ceph-deploy-global]

# Overrides for some of ceph-deploy's globalflags, like verbosity or cluster

# name

[ceph-deploy-install]

# Overrides for some of ceph-deploy's installflags, like version of ceph to

# install

#

# Repositories section

# yum repos:

# [myrepo]

# baseurl = https://user:pass@example.org/rhel6

# gpgurl = https://example.org/keys/release.asc

# default = True

# extra-repos = cephrepo  # will install the cephrepo file too

#

# [cephrepo]

# name=ceph repo noarch packages

# baseurl=http://ceph.com/rpm-emperor/el6/noarch

# enabled=1

# gpgcheck=1

# type=rpm-md

# gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

# apt repos:

# [myrepo]

# baseurl = https://user:pass@example.org/

# gpgurl = https://example.org/keys/release.asc

# default = True

# extra-repos = cephrepo  # will install the cephrepo file too

#

# [cephrepo]

# baseurl=http://ceph.com/rpm-emperor/el6/noarch

#gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

# 下面是重點配置

[calamari-minion]

name=ceph repo noarch packages

baseurl=http://ceph.com/rpm-emperor/fc19/x86_64

enabled=1

gpgcheck=1

type=repo-md

gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

• 修改salt-minion配置

#cat /etc/salt/minion.d/calamari.conf

master: 10.240.240.200

• 將ceph節點加入calamari （三個ceph節點都要執行）

# ceph-deploy calamari --master 10.240.240.200connect bgw-os-node151

[ceph_deploy.conf][DEBUG] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.9): /usr/bin/ceph-deploycalamari --master 10.240.240.200 connect bgw-os-node151

[bgw-os-node151][DEBUG ]connected to host: bgw-os-node151

[bgw-os-node151][DEBUG ]detect platform information from remote host

[bgw-os-node151][DEBUG ]detect machine type

[ceph_deploy.calamari][INFO  ] Distro info: Red Hat Enterprise LinuxServer 6.5 Santiago

[bgw-os-node151][INFO  ] installing calamari-minion package onbgw-os-node151

[bgw-os-node151][INFO  ] adding custom repository file

[bgw-os-node151][INFO  ] Running command: yum clean all

[bgw-os-node151][DEBUG ]Loaded plugins: product-id, security, subscription-manager

[bgw-os-node151][DEBUG ]Cleaning repos: base calamari-minion epel extras os-base os-ceph.repo os-epel

[bgw-os-node151][DEBUG]               : os-i.repo puppetpuppet.repo update

[bgw-os-node151][DEBUG ]Cleaning up Everything

[bgw-os-node151][WARNIN]This system is not registered to Red Hat Subscription Management. You can usesubscription-manager to register.

[bgw-os-node151][INFO  ] Running command: rpm --importhttps://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

[bgw-os-node151][DEBUG ]set the contents of repo file to /etc/yum.repos.d/

[bgw-os-node151][DEBUG ]creating config dir: /etc/salt/minion.d

[bgw-os-node151][DEBUG ]creating the calamari salt config: /etc/salt/minion.d/calamari.conf

[bgw-os-node151][INFO  ] Running command: yum -y install salt-minion

[bgw-os-node151][DEBUG ]Loaded plugins: product-id, security, subscription-manager

[bgw-os-node151][WARNIN]No data was received after 300 seconds, disconnecting...

[bgw-os-node151][INFO  ] Running command: chkconfig salt-minion on

[bgw-os-node151][INFO  ] Running command: service salt-minion start

[bgw-os-node151][DEBUG] Starting salt-minion daemon: [  OK  ]

 

上面啓動成功後，就能夠在web界面上看到ceph節點的信息啦

錯誤彙總

Ceph端的配置參考了http://my.oschina.net/renguijiayi/blog/307313

1.     將diamond-<version>.noarch.rpm複製到全部的ceph服務器。
使用yum localinstall安裝，或者yuminstall python-configobj而後使用rpm -ivh安裝。

2.     在全部的ceph服務器上安裝salt-minion，建立/etc/salt/minion.d/calamari.conf，內容爲：

master: {fqdn}

{fqdn}對應calamari服務器的域名。
啓動salt-minion服務：

# service salt-minion restart

3.     在Calamari服務器上配置防火牆和saltstack認證（我這裏將防火牆關閉了，省略該步驟） 
防火牆（容許ceph服務器訪問salt-master和carbon）：

###salt-master

# iptables -A INPUT -m state --state NEW -mtcp -p tcp --dport 4505 -j ACCEPT

# iptables -A INPUT -m state --state NEW -mtcp -p tcp --dport 4506 -j ACCEPT

### carbon

# iptables -A INPUT -m state --state NEW -mtcp -p tcp --dport 2003 -j ACCEPT

# iptables -A INPUT -m state --state NEW -mtcp -p tcp --dport 2004 -j ACCEPT

# systemctl save firewalld

# systemctl restart firewalld

 

saltstack認證：
當ceph服務器上的salt-minion服務啓動以後，會自動向salt-master請求認證。
在Calamari服務器上能夠經過下面的命令查看salt-minion密鑰的列表：

# salt-key -L

剛剛啓動salt-minion服務的ceph服務器會出如今Unaccepted Keys列表以後，要使得Calamari可以經過saltstack管理ceph服務器，須要對這些密鑰進行認證：

# salt-key -A

1.     部署完成以後，能夠訪問calamari

可是出現了下面的錯誤，並無看到咱們想要的結果

根據上面的提示，我作了下面的操做：

無語了，出錯了，因而google了一下，查閱 http://tracker.ceph.com/issues/7851解決。注意根據這個解決的 時候，要記得把salt-minion先stop哦。

#cat /etc/ceph/ceph.conf

[global]

osdcrush location hook = /usr/bin/calamari-crush-location