ceph運維命令合集

1、集羣

一、啓動一個ceph進程

啓動mon進程

[root@ceph-adm ~]#203.109 service ceph start mon.ceph-mon1

啓動msd進程

[root@ceph-adm ~]#203.109 service ceph start mds.ceph-mds1

啓動osd進程

[root@ceph-adm ~]#203.109 service ceph start osd.0

二、查看機器的監控狀態

[root@ceph-adm ~]#203.109 ceph health

HEALTH_OK

三、查看ceph的實時運行狀態

[root@ceph-adm ~]#203.109 ceph -w

    cluster fdbb34ad-765a-420b-89e8-443aba4254dd

     health HEALTH_WARN

            clock skew detected on mon.ceph-mon3

            84 pgs stale

            84 pgs stuck stale

            5 requests are blocked > 32 sec

            mds cluster is degraded

            Monitor clock skew detected 

     monmap e1: 3 mons at {ceph-mon1=192.168.203.72:6789/0,ceph-mon2=192.168.203.73:6789/0,ceph-mon3=192.168.203.74:6789/0}

            election epoch 22, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3

     mdsmap e16: 1/1/1 up {0=ceph-mon2=up:replay}, 2 up:standby

     osdmap e56: 2 osds: 2 up, 2 in

      pgmap v137: 596 pgs, 7 pools, 10442 bytes data, 63 objects

            77032 kB used, 30622 MB / 30697 MB avail

                 512 active+clean

                  84 stale+active+clean

2016-08-28 20:42:51.043538 osd.0 [WRN] slow request 120.842103 seconds old, received at 2016-08-28 20:40:50.201254: osd_op(mds.0.3:4 400.00000000 [read 0~0] 2.64e96f8f ack+read+known_if_redirected e49) currently no flag points reached

四、檢查信息狀態信息

[root@ceph-adm ~]#203.109 ceph -s

    cluster fdbb34ad-765a-420b-89e8-443aba4254dd

     health HEALTH_WARN

            clock skew detected on mon.ceph-mon3

            84 pgs stale

            84 pgs stuck stale

            5 requests are blocked > 32 sec

            mds cluster is degraded

            Monitor clock skew detected 

     monmap e1: 3 mons at {ceph-mon1=192.168.203.72:6789/0,ceph-mon2=192.168.203.73:6789/0,ceph-mon3=192.168.203.74:6789/0}

            election epoch 22, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3

     mdsmap e16: 1/1/1 up {0=ceph-mon2=up:replay}, 2 up:standby

     osdmap e56: 2 osds: 2 up, 2 in

      pgmap v139: 596 pgs, 7 pools, 10442 bytes data, 63 objects

            77636 kB used, 30622 MB / 30697 MB avail

                 512 active+clean

                  84 stale+active+clean

五、查看ceph存儲空間

[root@ceph-adm ~]#203.109 ceph df

GLOBAL:

    SIZE       AVAIL      RAW USED     %RAW USED 

    30697M     30622M       77636k          0.25 

POOLS:

    NAME                ID     USED     %USED     MAX AVAIL     OBJECTS 

    rbd                 0         0         0        15310M           0 

    cephfs_data         1         0         0        15310M           0 

    cephfs_metadata     2      9594         0        15310M          20 

六、刪除一個節點的全部的ceph數據包

[root@ceph-adm ~]#203.109 ceph-deploy purge ceph-mon1

[root@ceph-adm ~]#203.109 ceph-deploy purgedata cdph-mon1

七、爲ceph建立一個admin用戶併爲admin用戶建立一個密鑰，把密鑰保存到/etc/ceph目錄下：

[root@ceph-adm ~]#203.109 ceph auth get-or-create client.admin mds 'allow' osd 'allow *' mon 'allow *' > /etc/ceph/ceph.client.admin.keyring

或

[root@ceph-adm ~]#203.109 ceph auth get-or-create client.admin mds 'allow' osd 'allow *' mon 'allow *' -o /etc/ceph/ceph.client.admin.keyring

八、爲osd.0建立一個用戶並建立一個key

[root@ceph-adm ~]#203.109 ceph auth get-or-create osd.0 mon 'allow rwx' osd 'allow *' -o /var/lib/ceph/osd/ceph-0/keyring

九、爲mds.ceph-mon1建立一個用戶並建立一個key

[root@ceph-adm ~]#203.109 ceph auth get-or-create mds.node1 mon 'allow rwx' osd 'allow *' mds 'allow *' -o /var/lib/ceph/mds/ceph-node1/keyring

十、查看ceph集羣中的認證用戶及相關的key

[root@ceph-adm ~]#203.109 ceph auth list

十一、刪除集羣中的一個認證用戶

[root@ceph-adm ~]#203.109 ceph auth del osd.0

十二、查看集羣健康狀態細節

[root@ceph-adm ~]#203.109 ceph health detail

HEALTH_WARN clock skew detected on mon.ceph-mon3; 84 pgs stale; 84 pgs stuck stale; 5 requests are blocked > 32 sec; 2 osds have slow requests; mds cluster is degraded; Monitor clock skew detected 

pg 0.22 is stuck stale for 13269.231220, current state stale+active+clean, last acting [1,0]

pg 0.21 is stuck stale for 13269.231219, current state stale+active+clean, last acting [1,0]

pg 0.20 is stuck stale for 13269.231222, current state stale+active+clean, last acting [1,0]

pg 0.1f is stuck stale for 13269.231224, current state stale+active+clean, last acting [0,1]

1三、查看ceph log日誌所在的目錄

[root@ceph-adm ~]#203.109 ceph-conf --name mon.ceph-adm --show-config-value log_file

/var/log/ceph/ceph-mon.ceph-adm.log

2、mon

一、查看mon的狀態信息

[root@ceph-mon1 ~]# ceph mon stat

e1: 3 mons at {ceph-mon1=192.168.203.72:6789/0,ceph-mon2=192.168.203.73:6789/0,ceph-mon3=192.168.203.74:6789/0}, election epoch 22, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3

二、查看mon的選舉狀態

[root@ceph-mon1 ~]# ceph quorum_status

{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["ceph-mon1","ceph-mon2","ceph-mon3"],"quorum_leader_name":"ceph-mon1","monmap":{"epoch":1,"fsid":"fdbb34ad-765a-420b-89e8-443aba4254dd","modified":"0.000000","created":"0.000000","mons":[{"rank":0,"name":"ceph-mon1","addr":"192.168.203.72:6789\/0"},{"rank":1,"name":"ceph-mon2","addr":"192.168.203.73:6789\/0"},{"rank":2,"name":"ceph-mon3","addr":"192.168.203.74:6789\/0"}]}}

三、查看mon的映射信息

[root@ceph-mon1 ~]# ceph mon dump

dumped monmap epoch 1

epoch 1

fsid fdbb34ad-765a-420b-89e8-443aba4254dd

last_changed 0.000000

created 0.000000

0: 192.168.203.72:6789/0 mon.ceph-mon1

1: 192.168.203.73:6789/0 mon.ceph-mon2

2: 192.168.203.74:6789/0 mon.ceph-mon3

四、刪除一個mon節點

[root@node1 ~]# ceph mon remove node1

removed mon.node1 at 10.39.101.1:6789/0, there are now 3 monitors

2014-07-07 18:11:04.974188 7f4d16bfd700  0 monclient: hunting for new mon

五、得到一個正在運行的mon map，並保存在1.txt文件中

[root@ceph-mon1 ~]# ceph mon getmap -o 1.txt

got monmap epoch 1

六、查看上面得到的map

[root@ceph-mon1 ~]# monmaptool --print 1.txt 

monmaptool: monmap file 1.txt

epoch 1

fsid fdbb34ad-765a-420b-89e8-443aba4254dd

last_changed 0.000000

created 0.000000

0: 192.168.203.72:6789/0 mon.ceph-mon1

1: 192.168.203.73:6789/0 mon.ceph-mon2

2: 192.168.203.74:6789/0 mon.ceph-mon3

七、把上面的mon map注入新加入的節點

[root@ceph-mon1 ~]# ceph-mon -i ceph-mon3 --inject-monmap 1.txt

八、查看mon的amin socket

[root@ceph-mon1 ~]# ceph-conf --name mon.ceph-mon1 --show-config-value admin_socket

/var/run/ceph/ceph-mon.ceph-mon1.asok

九、查看mon的詳細狀態

[root@ceph-mon1 ~]# ceph daemon mon.ceph-mon1  mon_status 

{

    "name": "ceph-mon1",

    "rank": 0,

    "state": "leader",

    "election_epoch": 22,

    "quorum": [

        0,

        1,

        2

    ],

    "outside_quorum": [],

    "extra_probe_peers": [],

    "sync_provider": [],

    "monmap": {

        "epoch": 1,

        "fsid": "fdbb34ad-765a-420b-89e8-443aba4254dd",

        "modified": "0.000000",

        "created": "0.000000",

        "mons": [

            {

                "rank": 0,

                "name": "ceph-mon1",

                "addr": "192.168.203.72:6789\/0"

            },

            {

                "rank": 1,

                "name": "ceph-mon2",

                "addr": "192.168.203.73:6789\/0"

            },

            {

                "rank": 2,

                "name": "ceph-mon3",

                "addr": "192.168.203.74:6789\/0"

            }

        ]

    }

}

十、刪除一個mon節點

[root@os-node1 ~]# ceph mon remove ceph-mon1

removed mon.ceph-mon1 at 192.168.203.72:6789/0, there are now 3 monitors

3、msd

一、查看msd狀態

[root@ceph-mon1 ~]# ceph mds dump

dumped mdsmap epoch 16

epoch 16

flags 0

created 2016-08-28 13:35:38.203824

modified 2016-08-28 20:40:50.048995

tableserver 0

root 0

session_timeout 60

session_autoclose 300

max_file_size 1099511627776

last_failure 0

last_failure_osd_epoch 49

compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}

max_mds 1

in 0

up {0=44104}

failed

stopped

data_pools 1

metadata_pool 2

inline_data disabled

44098: 192.168.203.72:6800/1163 'ceph-mon1' mds.-1.0 up:standby seq 1

44099: 192.168.203.74:6800/1140 'ceph-mon3' mds.-1.0 up:standby seq 1

44104: 192.168.203.73:6800/1140 'ceph-mon2' mds.0.3 up:replay seq 1

三、刪除一個mds節點

[root@ceph-mon1 ~]# ceph  mds rm 0 mds.ceph-mon1

 mds gid 0 dne

4、osd

一、查看ceph osd運行狀態

[root@ceph-osd1 ~]# ceph osd stat

     osdmap e56: 2 osds: 2 up, 2 in

二、查看osd映射信息

[root@ceph-osd1 ~]# ceph osd dump

epoch 56

fsid fdbb34ad-765a-420b-89e8-443aba4254dd

created 2016-08-28 12:27:23.497369

modified 2016-08-28 20:41:10.565810

flags 

pool 0 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0

pool 1 'cephfs_data' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 10 pgp_num 10 last_change 12 flags hashpspool crash_replay_interval 45 stripe_width 0

pool 2 'cephfs_metadata' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 10 pgp_num 10 last_change 11 flags hashpspool stripe_width 0

pool 4 '.rgw.root' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 42 flags hashpspool stripe_width 0

pool 5 '.rgw.control' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 50 flags hashpspool stripe_width 0

pool 6 '.rgw' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 52 flags hashpspool stripe_width 0

pool 7 '.rgw.gc' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 53 flags hashpspool stripe_width 0

max_osd 10

osd.0 up   in  weight 1 up_from 47 up_thru 54 down_at 19 last_clean_interval [0,0) 192.168.203.75:6800/1954 192.168.203.75:6801/1954 192.168.203.75:6802/1954 192.168.203.75:6803/1954 exists,up 56d8c516-f3b4-4b84-a4b5-6d6d051f125f

osd.1 up   in  weight 1 up_from 50 up_thru 54 down_at 19 last_clean_interval [0,0) 192.168.203.76:6800/1954 192.168.203.76:6801/1954 192.168.203.76:6802/1954 192.168.203.76:6803/1954 exists,up c4572946-2d7e-4144-8416-c62bbadb6d2b

blacklist 192.168.203.73:6800/1173 expires 2016-08-28 21:04:49.243225

三、查看osd的目錄樹

[root@ceph-osd1 ~]# ceph osd tree

ID WEIGHT  TYPE NAME          UP/DOWN REWEIGHT PRIMARY-AFFINITY 

-1 0.01999 root default                                         

-2 0.00999     host ceph-osd1                                   

 0 0.00999         osd.0           up  1.00000          1.00000 

-3 0.00999     host ceph-osd2                                   

 1 0.00999         osd.1           up  1.00000          1.00000 

四、down掉一個osd硬盤

[root@ceph-osd1 ~]# ceph osd down 0   #down掉osd.0節點

五、在集羣中刪除一個osd硬盤

[root@ceph-osd1 ~]# ceph osd rm 0

removed osd.0

六、在集羣中刪除一個osd 硬盤 crush map

[root@ceph-osd1 ~]# ceph osd crush rm osd.0

七、在集羣中刪除一個osd的host節點

[root@ceph-osd1 ~]# ceph osd crush rm ceph-osd1

removed item id -2 name 'ceph-osd1' from crush map

八、查看最大osd的個數 

[root@ceph-osd1 ~]# ceph osd getmaxosd

max_osd = 4 in epoch 514           #默認最大是4個osd節點

八、設置最大的osd的個數（當擴大osd節點的時候必須擴大這個值）

[root@ceph-osd1 ~]# ceph osd setmaxosd 10

九、設置osd crush的權重爲1.0

ceph osd crush set {id} {weight} [{loc1} [{loc2} ...]]

例如：

[root@ceph-osd1 ~]# ceph osd crush set 1 3.0 host=ceph-osd1

set item id 1 name 'osd.1' weight 3 at location {host=ceph-osd2} to crush map

[root@ceph-osd1 ~]# ceph osd tree

ID WEIGHT  TYPE NAME          UP/DOWN REWEIGHT PRIMARY-AFFINITY 

-1 3.00999 root default                                         

-2 0.00999     host ceph-osd1                                   

 0 0.00999         osd.0           up  1.00000          1.00000 

-3 3.00000     host ceph-osd2                                   

 1 3.00000         osd.1           up  1.00000          1.00000 

或者用下面的方式

[root@ceph-osd1 ~]# ceph osd crush reweight osd.0 1.0

reweighted item id 0 name 'osd.0' to 1 in crush map

[root@ceph-osd1 ~]# ceph osd tree

ID WEIGHT  TYPE NAME          UP/DOWN REWEIGHT PRIMARY-AFFINITY 

-1 2.00000 root default                                         

-2 1.00000     host ceph-osd1                                   

 0 1.00000         osd.0           up  1.00000          1.00000 

-3 1.00000     host ceph-osd2                                   

 1 1.00000         osd.1           up  1.00000          1.00000 

十、設置osd的權重

[root@ceph-osd1 ~]# ceph osd reweight 1 0.5

reweighted osd.1 to 0.5 (8000)

[root@ceph-osd1 ~]# ceph osd tree

ID WEIGHT  TYPE NAME          UP/DOWN REWEIGHT PRIMARY-AFFINITY 

-1 2.00000 root default                                         

-2 1.00000     host ceph-osd1                                   

 0 1.00000         osd.0           up  1.00000          1.00000 

-3 1.00000     host ceph-osd2                                   

 1 1.00000         osd.1           up  0.50000          1.00000 

十一、把一個osd節點逐出集羣

[root@ceph-osd1 ~]# ceph osd out osd.1

marked out osd.1. 

[root@ceph-osd1 ~]# ceph osd tree

ID WEIGHT  TYPE NAME          UP/DOWN REWEIGHT PRIMARY-AFFINITY 

-1 2.00000 root default                                         

-2 1.00000     host ceph-osd1                                   

 0 1.00000         osd.0           up  1.00000          1.00000 

-3 1.00000     host ceph-osd2                                   

 1 1.00000         osd.1           up        0          1.00000  # osd.1的reweight變爲0了就再也不分配數據，可是設備仍是存活的

 

十二、把逐出的osd加入集羣

[root@ceph-osd1 ~]# ceph osd in osd.1

marked in osd.1. 

[root@ceph-osd1 ~]# ceph osd tree

ID WEIGHT  TYPE NAME          UP/DOWN REWEIGHT PRIMARY-AFFINITY 

-1 2.00000 root default                                         

-2 1.00000     host ceph-osd1                                   

 0 1.00000         osd.0           up  1.00000          1.00000 

-3 1.00000     host ceph-osd2                                   

 1 1.00000         osd.1           up  1.00000          1.00000 

1三、暫停osd （暫停後整個集羣再也不接收數據）

[root@ceph-osd1 ~]# ceph osd pause

set pauserd,pausewr

      

1四、再次開啓osd （開啓後再次接收數據） 

[root@ceph-osd1 ~]# ceph osd unpause

unset pauserd,pausewr

5、PG組

一、查看pg組的映射信息

[root@ceph-osd1 ~]# ceph pg dump |more

dumped all in format plain

version 204

stamp 2016-08-28 21:11:57.715242

last_osdmap_epoch 90

last_pg_scan 53

full_ratio 0.95

nearfull_ratio 0.85

pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up up_pr

imary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp

7.78 0 0 0 0 0 0 0 0 active+clean 2016-08-28 21:11:03.832762 0'0 88:53

[1,0] 1 [1,0] 1 0'0 2016-08-28 20:41:06.761508 0'0 2016-08-28 20:41:06.761508

6.79 0 0 0 0 0 0 0 0 active+clean 2016-08-28 21:11:03.111729 0'0 87:53

[0,1] 0 [0,1] 0 0'0 2016-08-28 20:41:04.937732 0'0 2016-08-28 20:41:04.937732

4.7b 0 0 0 0 0 0 0 0 active+clean 2016-08-28 21:11:03.834298 0'0 88:50

[1,0] 1 [1,0] 1 0'0 2016-08-28 20:38:50.323737 0'0 2016-08-28 20:38:50.323737

5.7a 0 0 0 0 0 0 0 0 active+clean 2016-08-28 21:11:03.816663 0'0 88:38

[1,0] 1 [1,0] 1 0'0 2016-08-28 20:40:51.277112 0'0 2016-08-28 20:40:51.277112

下面部分省略

二、查看一個PG的map

[root@ceph-osd1 ~]# ceph pg map 0.3f

osdmap e90 pg 0.3f (0.3f) -> up [0,1] acting [0,1] 

三、查看PG狀態

[root@ceph-osd1 ~]#  ceph pg stat

v206: 596 pgs: 84 stale+active+clean, 512 active+clean; 10442 bytes data, 89652 kB used, 30610 MB / 30697 MB avail

四、查詢一個pg的詳細信息

[root@ceph-osd1 ~]# ceph pg  7.78 query

{

    "state": "active+clean",

    "snap_trimq": "[]",

    "epoch": 90,

    "up": [

        1,

        0

下面部分省略

五、查看pg中stuck的狀態

[root@ceph-osd1 ~]# ceph pg dump_stuck unclean

ok

[root@ceph-osd1 ~]# ceph pg dump_stuck inactive

ok

[root@ceph-osd1 ~]# ceph pg dump_stuck stale

ok

pg_stat state up up_primary acting acting_primary

0.22 stale+active+clean [1,0] 1 [1,0] 1

0.21 stale+active+clean [1,0] 1 [1,0] 1

0.20 stale+active+clean [1,0] 1 [1,0] 1

六、顯示一個集羣中的全部的pg統計

[root@ceph-osd1 ~]# ceph pg dump --format plain|more

dumped all in format plain

version 208

stamp 2016-08-28 21:15:57.802463

last_osdmap_epoch 90

last_pg_scan 53

full_ratio 0.95

nearfull_ratio 0.85

pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up up_pr

imary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 

7.78 0 0 0 0 0 0 0 0 active+clean 2016-08-28 21:11:03.832762 0'0 88:53

[1,0] 1 [1,0] 1 0'0 2016-08-28 20:41:06.761508 0'0 2016-08-28 20:41:06.761508

下面部分省略

七、恢復一個丟失的pg

ceph pg {pg-id} mark_unfound_lost revert

八、顯示非正常狀態的pg

ceph pg dump_stuck inactive|unclean|stale

6、pool

一、查看ceph集羣中的pool數量

[root@ceph-adm ~]#203.109 ceph osd lspools

0 rbd,1 cephfs_data,2 cephfs_metadata,4 .rgw.root,5 .rgw.control,6 .rgw,7 .rgw.gc,

二、在ceph集羣中建立一個pool

[root@ceph-adm ~]#203.109 ceph osd pool create xiao 100  #這裏的100指的是PG組

pool 'xiao' created

三、爲一個ceph pool配置配額

[root@ceph-adm ~]#203.109 ceph osd pool set-quota xiao max_objects 10000

set-quota max_objects = 10000 for pool xiao

三、顯示全部的pool

[root@ceph-adm ~]#203.109 ceph osd pool ls

rbd

cephfs_data

cephfs_metadata

xiao

四、在集羣中刪除一個pool

ceph osd pool delete xiao xiao --yes-i-really-really-mean-it  #集羣名字須要重複兩次

五、顯示集羣中pool的詳細信息

[root@ceph-adm ~]#203.109 rados df

pool name                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB

cephfs_data                0            0            0            0           0            0            0            0            0

cephfs_metadata           10           20            0            0           0            0            0           75           52

rbd                        0            0            0            0           0            0            0            0            0

xiao                       0            0            0            0           0            0            0            0            0

  total used           91444           20

  total avail       31343268

  total space       31434712

六、給一個pool建立一個快照

[root@ceph-adm ~]#203.109 ceph osd pool mksnap xiao xiao-snap

created pool xiao snap xiao-snap

七、刪除pool的快照

[root@ceph-adm ~]#203.109 ceph osd pool rmsnap xiao xiao-snap

removed pool xiao snap xiao-snap

八、查看data池的pg數量

[root@ceph-adm ~]#203.109 ceph osd pool get xiao pg_num

pg_num: 100

九、設置data池的最大存儲空間爲100T（默認是1T)

[root@ceph-adm ~]#203.109 ceph osd pool set xiao target_max_bytes 100000000000000

set pool 0 target_max_bytes to 100000000000000

十、設置data池的副本數是3

[root@ceph-adm ~]#203.109 ceph osd pool set xiao size 3

set pool 0 size to 3

十一、設置data池能接受寫操做的最小副本爲2

[root@ceph-adm ~]#203.109 ceph osd pool set xiao min_size 2

set pool 0 min_size to 2

十二、查看集羣中全部pool的副本尺寸

[root@ceph-adm ~]#203.109 ceph osd dump | grep 'replicated size'

pool 0 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0

pool 1 'cephfs_data' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 10 pgp_num 10 last_change 12 flags hashpspool crash_replay_interval 45 stripe_width 0

pool 2 'cephfs_metadata' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 10 pgp_num 10 last_change 11 flags hashpspool stripe_width 0

pool 8 'xiao' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 104 flags hashpspool max_objects 10000 stripe_width 0

1三、設置一個pool的pg數量

[root@ceph-adm ~]# ceph osd pool set xiao pg_num 100

set pool 0 pg_num to 100

1四、設置一個pool的pgp數量

[root@admin ~]# ceph osd pool set xiao pgp_num 100

set pool 0 pgp_num to 100

7、rados和rbd指令

一、rados命令使用方法

（1）、查看ceph集羣中有多少個pool （只是查看pool)

[root@ceph-adm ~]# rados lspools

rbd

cephfs_data

cephfs_metadata

xiao

（2）、查看ceph集羣中有多少個pool,而且每一個pool容量及利用狀況

[root@ceph-adm ~]# rados df 

pool name                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB

cephfs_data                0            0            0            0           0            0            0            0            0

cephfs_metadata           10           20            0            0           0            0            0           75           52

rbd                        0            0            0            0           0            0            0            0            0

xiao                       0            0            0            0           0            0            0            0            0

  total used           73960           20

  total avail       31360752

  total space       31434712

（3）、建立一個pool

[root@ceph-adm ~]# rados mkpool test

successfully created pool test

(4)、查看ceph pool中的ceph object （這裏的object是以塊形式存儲的）

[root@node-44 ~]# rados ls -p volumes | more

rbd_data.348f21ba7021.0000000000000866

rbd_data.32562ae8944a.0000000000000c79

rbd_data.589c2ae8944a.00000000000031ba

rbd_data.58c9151ff76b.00000000000029af

rbd_data.58c9151ff76b.0000000000002c19

rbd_data.58c9151ff76b.0000000000000a5a

rbd_data.58c9151ff76b.0000000000001c69

rbd_data.58c9151ff76b.000000000000281d

rbd_data.58c9151ff76b.0000000000002de1

rbd_data.58c9151ff76b.0000000000002dae

（5）、建立一個對象object 

[root@ceph-adm ~]# rados create test-object -p test

[root@ceph-adm ~]# rados -p test ls

test-object

（6）、刪除一個對象

[root@ceph-adm ~]# rados rm test-object -p test

[root@ceph-adm ~]# rados -p test ls

二、rbd命令的用法 

（1）、查看ceph中一個pool裏的全部鏡像

[root@ceph-osd2 ~]# rbd ls p_w_picpaths

2014-05-24 17:17:37.043659 7f14caa6e700  0 -- :/1025604 >> 10.49.101.9:6789/0 pipe(0x6c5400 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x6c5660).fault

2182d9ac-52f4-4f5d-99a1-ab3ceacbf0b9

34e1a475-5b11-410c-b4c4-69b5f780f03c

[root@ceph-osd2 ~]# rbd ls volumes

2014-05-24 17:22:18.649929 7f9e98733700  0 -- :/1010725 >> 10.49.101.9:6789/0 pipe(0x96a400 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x96a660).fault

volume-0788fc6c-0dd4-4339-bad4-e9d78bd5365c

volume-0898c5b4-4072-4cae-affc-ec59c2375c51

（2）、查看ceph pool中一個鏡像的信息

[root@ceph-osd2 ~]# rbd info -p p_w_picpaths --p_w_picpath 74cb427c-cee9-47d0-b467-af217a67e60a

rbd p_w_picpath '74cb427c-cee9-47d0-b467-af217a67e60a':

        size 1048 MB in 131 objects

        order 23 (8192 KB objects)

        block_name_prefix: rbd_data.95c7783fc0d0

        format: 2

        features: layering

        

（3）、在test池中建立一個命名爲zhanguo的10000M的鏡像

[root@ceph-osd2 ~]# rbd create -p test --size 10000 zhanguo

[root@ceph-osd2 ~]# rbd -p test info zhanguo    #查看新建的鏡像的信息

rbd p_w_picpath 'zhanguo':

        size 10000 MB in 2500 objects

        order 22 (4096 KB objects)

        block_name_prefix: rb.0.127d2.2ae8944a

        format: 1

（4）、刪除一個鏡像

[root@ceph-osd2 ~]# rbd rm  -p test  lizhanguo

Removing p_w_picpath: 100% complete...done.

（5）、調整一個鏡像的尺寸

[root@ceph-osd2 ~]# rbd resize -p test --size 20000 zhanguo

Resizing p_w_picpath: 100% complete...done.

[root@ceph-osd2 ~]# rbd -p test info zhanguo   #調整後的鏡像大小

rbd p_w_picpath 'zhanguo':

        size 20000 MB in 5000 objects

        order 22 (4096 KB objects)

        block_name_prefix: rb.0.127d2.2ae8944a

        format: 1

 

（6）、給一個鏡像建立一個快照

[root@ceph-osd2 ~]# rbd  snap create  test/zhanguo@zhanguo123  #池/鏡像@快照

[root@ceph-osd2 ~]# rbd   snap ls  -p test zhanguo

SNAPID NAME           SIZE 

     2 zhanguo123 20000 MB 

[root@ceph-osd2 ~]# rbd info test/zhanguo@zhanguo123

rbd p_w_picpath 'zhanguo':

        size 20000 MB in 5000 objects

        order 22 (4096 KB objects)

        block_name_prefix: rb.0.127d2.2ae8944a

        format: 1

        protected: False

（7）、查看一個鏡像文件的快照

[root@ceph-osd2 ~]# rbd snap ls  -p volumes volume-7687988d-16ef-4814-8a2c-3fbd85e928e4

SNAPID NAME                                               SIZE 

     5 snapshot-ee7862aa-825e-4004-9587-879d60430a12 102400 MB 

     

（8）、刪除一個鏡像文件的一個快照快照

                                 快照所在的池/        快照所在的鏡像文件           @ 快照

[root@ceph-osd2 ~]# rbd snap rm volumes/volume-7687988d-16ef-4814-8a2c-3fbd85e928e4@snapshot-ee7862aa-825e-4004-9587-879d60430a12

rbd: snapshot 'snapshot-60586eba-b0be-4885-81ab-010757e50efb' is protected from removal.

2014-08-18 19:23:42.099301 7fd0245ef760 -1 librbd: removing snapshot from header failed: (16) Device or resource busy

上面不能刪除顯示的報錯信息是此快照備寫保護了，下面命令是刪除寫保護後再進行刪除。

[root@ceph-osd2 ~]# rbd snap unprotect volumes/volume-7687988d-16ef-4814-8a2c-3fbd85e928e4@snapshot-ee7862aa-825e-4004-9587-879d60430a12

[root@ceph-osd2 ~]# rbd snap rm volumes/volume-7687988d-16ef-4814-8a2c-3fbd85e928e4@snapshot-ee7862aa-825e-4004-9587-879d60430a12

（9）刪除一個鏡像文件的全部快照

[root@ceph-osd2 ~]# rbd snap purge  -p volumes volume-7687988d-16ef-4814-8a2c-3fbd85e928e4

Removing all snapshots: 100% complete...done.

（10）、把ceph pool中的一個鏡像導出

導出鏡像

[root@ceph-osd2 ~]# rbd export -p p_w_picpaths --p_w_picpath 74cb427c-cee9-47d0-b467-af217a67e60a /root/aaa.img

2014-05-24 17:16:15.197695 7ffb47a9a700  0 -- :/1020493 >> 10.49.101.9:6789/0 pipe(0x1368400 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x1368660).fault

Exporting p_w_picpath: 100% complete...done.

導出雲硬盤

[root@ceph-osd2 ~]# rbd export -p volumes --p_w_picpath volume-470fee37-b950-4eef-a595-d7def334a5d6 /var/lib/glance/ceph-pool/volumes/Message-JiaoBenJi-10.40.212.24

2014-05-24 17:28:18.940402 7f14ad39f700  0 -- :/1032237 >> 10.49.101.9:6789/0 pipe(0x260a400 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x260a660).fault

Exporting p_w_picpath: 100% complete...done.

（11）、把一個鏡像導入ceph中 （可是直接導入是不能用的，由於沒有通過openstack,openstack是看不到的）

[root@ceph-osd2 ~]# rbd import /root/aaa.img -p p_w_picpaths --p_w_picpath 74cb427c-cee9-47d0-b467-af217a67e60a  

Importing p_w_picpath: 100% complete...done.