ROOK

rook簡介

    Rook是一個開放源碼的雲本機存儲協調器，提供平臺、框架和對各類存儲解決方案的支持，以便與雲本機環境進行本機集成。

    Rook將存儲軟件轉變爲自我管理、自我擴展和自我修復的存儲服務。它經過自動化部署、引導、配置、供應、擴展、升級、遷移、災難恢復、監視和資源管理來實現這一點。Rook使用底層雲本地容器管理、調度和協調平臺提供的設施來執行其職責。

    Rook利用擴展點深刻集成到雲本機環境中，爲調度、生命週期管理、資源管理、安全、監控和用戶體驗提供無縫體驗。

 

 

下圖說明了Ceph Rook如何與Kubernetes集成：

 

 

 

ROOK  架構

 

 

部署環境準備

須要K8S環境一套，節點分配以下.每臺vm至少掛一塊50G的盤

[root@k8s-master ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
51.0.1.213 k8s-master
51.0.1.214 k8s-node1
51.0.1.215 k8s-node2

軟件版本:

        k8s 版本:   v1.14.1

        ROOK版本:  v1.0  （Kubernetes v1.10 最低K8S要求）

 

開始部署:

github拉去ROOK倉庫

[root@k8s-master ~]# git clone https://github.com/rook/rook.git

[root@k8s-master rook]# git checkout -b remotes/origin/release-1.0

 

進入ceph部署目錄

[root@k8s-master rook]# cd ./cluster/examples/kubernetes/ceph/

 1 [root@k8s-master ceph]# kubectl create -f common.yaml
 2 namespace/rook-ceph created
 3 customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
 4 customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
 5 customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created
 6 customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
 7 customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
 8 customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
 9 customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created
10 clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
11 clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt-rules created
12 role.rbac.authorization.k8s.io/rook-ceph-system created
13 clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
14 clusterrole.rbac.authorization.k8s.io/rook-ceph-global-rules created
15 clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
16 clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster-rules created
17 serviceaccount/rook-ceph-system created
18 rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
19 clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
20 serviceaccount/rook-ceph-osd created
21 serviceaccount/rook-ceph-mgr created
22 role.rbac.authorization.k8s.io/rook-ceph-osd created
23 clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created
24 clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system-rules created
25 role.rbac.authorization.k8s.io/rook-ceph-mgr created
26 rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
27 rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
28 rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
29 rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
30 clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created

 

 建立operator 和agent容器

1 [root@k8s-master ceph]# kubectl create -f operator.yaml
2 deployment.apps/rook-ceph-operator created

 

查看先關容器是否已經啓動了，部署rook-ceph-operator過程當中，會觸發以DaemonSet的方式在集羣部署Agent和Discoverpods。operator會在集羣內的每一個主機建立兩個pod:rook-discover,rook-ceph-agent：

[root@k8s-master ceph]# kubectl get pod -n rook-ceph -o wide

 

 建立ceph進羣

[root@k8s-master ceph]# kubectl create -f cluster.yaml
cephcluster.ceph.rook.io/rook-ceph created

 

查看容器狀態

 [root@k8s-master ceph]# kubectl get pod -n rook-ceph -o wide      

 

剛建立後須要等待一段時間才能夠所有建立完，osd容器仍是相對比較慢

 

 

查看下deployment 信息看看集羣

[root@k8s-master ~]# kubectl -n rook-ceph get deployment

 

 

 

配置dashboard

     在cluster.yaml文件中默認已經啓用了ceph dashboard，查看dashboard的service：

[root@k8s-master ~]# kubectl get service -n rook-ceph|grep dashboard

NAME                                            TYPE                 CLUSTER-IP            EXTERNAL-IP      PORT(S)             AGE
rook-ceph-mgr-dashboard        ClusterIP            10.105.208.128              <none>                8443/TCP              25h

 

 

rook-ceph-mgr-dashboard監聽的端口是8443，建立nodeport類型的service以便集羣外部訪問

[root@k8s-master ceph]# kubectl apply -f dashboard-external-https.yaml
service/rook-ceph-mgr-dashboard-external-https created

 

查看 dashboard 外網端口
[root@k8s-master ceph]# kubectl get service -n rook-ceph | grep dashboard
rook-ceph-mgr-dashboard                              ClusterIP     10.105.208.128        <none>           8443/TCP                15m
rook-ceph-mgr-dashboard-external-https      NodePort     10.100.241.136         <none>          8443:32299/TCP      6s

 

獲取Dashboard的登錄帳號和密碼

[root@k8s-master ~]# MGR_POD=`kubectl get pod -n rook-ceph | grep mgr | awk '{print $1}'` 
[root@k8s-master ceph]# kubectl -n rook-ceph logs $MGR_POD | grep password
debug 2019-05-16 06:36:41.934 7fc2c1822700 0 log_channel(audit) log [DBG] : from='client.14398 -' entity='client.admin' cmd=[{"username": "admin", "prefix": "dashboard set-login-credentials", "password": "RmXvxlnOf6", "target": ["mgr", ""], "format": "json"}]: dispatch

 

 

找到password字段，用戶 admin，密碼   RmXvxlnOf6
打開瀏覽器輸入任意一個Node的IP+nodeport端口，這裏使用master節點 ip訪問：

 

 

部署Ceph toolbox

默認啓動的Ceph集羣，是開啓Ceph認證的，這樣你登錄Ceph組件所在的Pod裏，是無法去獲取集羣狀態，

以及執行CLI命令，這時須要部署Ceph toolbox，命令以下：

[root@k8s-master ceph]# kubectl apply -f toolbox.yaml
deployment.apps/rook-ceph-tools created
[root@k8s-master ceph]# kubectl -n    rook-ceph get pods -o wide | grep ceph-tools
rook-ceph-tools-b8c679f95-mf5xn        1/1 Running            0          12s            51.0.1.215           k8s-node2     <none>  <none>

 

登陸容器運行命令

[root@k8s-master ceph]#  kubectl -n rook-ceph exec -it rook-ceph-tools-b8c679f95-mf5xn bash