使用Glusterfs做爲kvm後端存儲

時間 2019-11-09

標籤使用 glusterfs 做爲 kvm 後端存儲简体版

原文原文鏈接

1.測試環境

centos6.4 x86-64git

gluster-3.4web

qemu-1.5.2centos

機器:api

192.168.1.100:glusterfs+kvmdom

192.168.1.101-103:glusterfs socket

2.Glusterfs存儲集羣部署

先部署好glusterfs集羣,部署教程參考這裏http://my.oschina.net/kisops/blog/151073tcp

gluster集羣部署完畢後,建立一個volume用於存放vm鏡像性能

   
 
    
 
   gluster volume create vm-images stripe  
 
   2 
 
    replica  
 
   2 
 
     
 
   192.168 
 
   . 
 
   1 
 
   . 
 
   { 
 
   100 
 
   , 
 
   101 
 
   , 
 
   102 
 
   , 
 
   103 
 
   }: 
 
   / 
 
   data 
 
   / 
 
   vm-images gluster volume  
 
   start 
 
    vm-images

3.虛擬化機器上安裝qemu

centos6自帶的qemu1.2版本並不支持glusterfs,這裏要編譯新版的qemu.測試

安裝qemu前先裝上glusterfs-develspa

   
 
    
 
   1 
 
    rpm –ivh http: 
 
   // 
 
   download 
 
   . 
 
   gluster 
 
   . 
 
   org 
 
   / 
 
   pub 
 
   / 
 
   gluster 
 
   / 
 
   glusterfs 
 
   / 
 
   3.4 
 
   / 
 
   3.4 
 
   . 
 
   0 
 
   / 
 
   CentOS 
 
   / 
 
   epel-6Server 
 
   / 
 
   x86_64 
 
   / 
 
   glusterfs-devel- 
 
   3.4 
 
   . 
 
   0 
 
   - 
 
   8 
 
   . 
 
   el6 
 
   . 
 
   x86_64 
 
   . 
 
   rpm

安裝依賴的包

   
 
    
 
   1 
 
   yum install zlib-devel glib2-devel -y

開始編譯qemu,加上參數--enable-glusterfs

   
 
    
 
   1 
 
   wget http: 
 
   // 
 
   wiki 
 
   . 
 
   qemu-project 
 
   . 
 
   org 
 
   / 
 
   download 
 
   / 
 
   qemu- 
 
   1.5 
 
   . 
 
   2 
 
   . 
 
   tar 
 
   . 
 
   bz2  
 
   2 
 
   tar jxvf qemu- 
 
   1.5 
 
   . 
 
   2 
 
   . 
 
   tar 
 
   . 
 
   bz2  
 
   3 
 
    
 
   cd 
 
    qemu- 
 
   1.5 
 
   . 
 
   2 
 
     
 
   4 
 
    
 
   ./ 
 
   configure --enable-glusterfs  
 
   # 
 
   這裏加上enable-glusterfs  
 
   5 
 
   make 
 
   ; 
 
   make install

4.使用

環境安裝完畢,使用qemu-img建立一個虛擬機磁盤

   
 
    
 
   1 
 
     
 
   / 
 
   usr 
 
   / 
 
   local 
 
   / 
 
   bin 
 
   / 
 
   qemu-img create -f qcow2 gluster: 
 
   // 
 
   192.168 
 
   . 
 
   1.100 
 
   / 
 
   vm-images 
 
   / 
 
   disk1 10G

建立一個虛擬機

   
 
    
 
   qemu-system-x86_64 --enable-kvm -m  
 
   1024 
 
    -drive file 
 
   = 
 
   gluster: 
 
   // 
 
   192.168 
 
   . 
 
   1.100 
 
   / 
 
   vm-images 
 
   / 
 
   disk1 -vnc : 
 
   15 
 
    -cdrom  
 
   / 
 
   data 
 
   / 
 
   CentOS- 
 
   6.4 
 
   -i386-minimal 
 
   . 
 
   iso

如今可使用VNC連上並安裝系統了.

5.後記

qemu鏈接glusterfs支持多種格式:

gluster://1.2.3.4/testvol/a.img
gluster+tcp://1.2.3.4/testvol/a.img
gluster+tcp://1.2.3.4:24007/testvol/dir/a.img
gluster+tcp://[1:2:3:4:5:6:7:8]/testvol/dir/a.img
gluster+tcp://[1:2:3:4:5:6:7:8]:24007/testvol/dir/a.img
gluster+tcp://server.domain.com:24007/testvol/dir/a.img
gluster+unix:///testvol/dir/a.img?socket=/tmp/glusterd.socket
gluster+rdma://1.2.3.4:24007/testvol/a.img

通過測試發現qemu鏈接glusterfs是支持高可用的,如gluster://1.2.3.4/testvol/a.img,1.2.3.4宕機將不會影響虛擬機運行.

另外測試過程發現gluster對xfs分區格式兼容性不是太好,會有使用空間大小不正確的現象,解決方法是:

   
 
    
 
    gluster volume  
 
   set 
 
     
 
   < 
 
   volname 
 
   > 
 
    cluster 
 
   . 
 
   stripe-coalesce enable

6.附上老外的性能測試

由測試結果能夠看到使用api鏈接glusterfs性能比fuse提升近一倍,而且接近本地磁盤的速度.

The following numbers from FIO benchmark are to show the performance advantage of using QEMU’s GlusterFS block driver instead of the usual FUSE mount while accessing the VM image.

Test setup

Host	Dual core x86_64 system running Fedora 17 kernel (3.5.6-1.fc17.x86_64)
Guest	Fedora 17 image, 4 way SMP, 2GB RAM, using virtio and cache=none QEMU options

QEMU options

FUSE mount	qemu-system-x86_64 –enable-kvm –nographic -smp 4 -m 2048 -drive file=/mnt/F17,if=virtio,cache=none => /mnt is GlusterFS FUSE mount point
GlusterFS block driver in QEMU (FUSE bypass)	qemu-system-x86_64 –enable-kvm –nographic -smp 4 -m 2048 -drive file=gluster://bharata/test/F17,if=virtio,cache=none
Base (VM image accessed directly from brick)	qemu-system-x86_64 –enable-kvm –nographic -smp 4 -m 2048 -drive file=/test/F17,if=virtio,cache=none => /test is brick directory

FIO load files

Sequential read direct IO	; Read 4 files with aio at different depths [global] ioengine=libaio direct=1 rw=read bs=128k size=512m directory=/data1 [file1] iodepth=4 [file2] iodepth=32 [file3] iodepth=8 [file4] iodepth=16
Sequential write direct IO	; Write 4 files with aio at different depths [global] ioengine=libaio direct=1 rw=write bs=128k size=512m directory=/data1 [file1] iodepth=4 [file2] iodepth=32 [file3] iodepth=8 [file4] iodepth=16

FIO READ numbers

	aggrb (KB/s)	minb (KB/s)	maxb (KB/s)
FUSE mount	15219	3804	5792
QEMU’s GlusterFS block driver (FUSE bypass)	39357	9839	12946
Base	43802	10950	12918

FIO WRITE numbers

	aggrb (KB/s)	minb (KB/s)	maxb (KB/s)
FUSE mount	24579	6144	8423
QEMU’s GlusterFS block driver (FUSE bypass)	42707	10676	17262
Base	42393	10598	15646

Updated numbers

Here are the recent FIO numbers averaged from 5 runs using latest QEMU (git commit: 03a36f17d77) and GlusterFS (git commit: cee1b62d01). The test environment remains same as above with the following two changes: