Linux分佈式mogilefs文件系統

時間 2020-08-04

原文原文鏈接

一、初識分佈式文件系統html

在生產環境中，後端存儲服務器服務器是一個很重要的環節。若是一旦前段服務器須要大量進行IO操做，會對後端存儲的文件系統形成很大的壓力。一旦後端的存儲文件系統down機，那會致使整個生產環境奔潰。由於在生產環境中，後端的存儲文件系統會成爲一個單點故障。而爲了不由於單點故障致使整個生產環境沒法正常使用，全部你們都會採用分佈式文件系統，以免由於單點故障致使生產環境奔潰。node

mogilefs組成部分mysql

MogileFS由3個部分組成：nginx

(1) server：主要包括mogilefsd和mogstored兩個應用程序。mogilefsd實現的是tracker，它經過數據庫來保存元數據信息，包括站點domain、class、host等；mogstored是存儲節點(store node)，它實際上是個WebDAV服務，默認監聽在7500端口，接受客戶端的文件存儲請求。在MogileFS安裝完後，要運行mogadm工具將全部的store node註冊到mogilefsd的數據庫裏，mogilefsd會對這些節點進行管理和監控。程序員

(2) utils（工具集）：主要是MogileFS的一些管理工具，例如mogadm等。sql

(3) 客戶端API：MogileFS的客戶端API不少，例如Perl、PHP、Java、Python等，用這個模塊能夠編寫客戶端程序，實現文件的備份管理功能等。數據庫

存儲主機(節點)vim

這個是 MogileFS 存儲文件存放在這些機器上,也是 mogstored 節點,也叫 Storage Server,一臺存儲主要都要啓動一個 mogstored 服務.擴容就是增長這些機器.後端

設備(device)centos

一個存儲節點,以就是上面的主機,能夠有多個 device, 就是用來存放文件的目錄(例如掛載的目錄),每一個設備都有一個設備id,須要在 mogstored 的配置文件中的 docroot 配置的項目指定的目錄下面建立相應的設備的目錄,目錄名爲 $docroot/dev$id,設備是不能刪除的.只能將其設備的狀態的值置爲dead,當一個設備 dead 以後,就真的 dead了,裏面的數據也沒法恢復了,且這個dead了的設備的 id 也不能再用.

3、分佈式文件系統須要的程序包

MogileFS-Server-2.46-2.el6.noarch.rpm

MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm

MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm

MogileFS-Utils-2.19-1.el6.noarch.rpm

Perlbal-1.78-1.el6.noarch.rpm

Perlbal-doc-1.78-1.el6.noarch.rpm

perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm

perl-IO-AIO-3.71-2.el6.x86_64

perl-MogileFS-Client-1.14-1.el6.noarch.rpm

perl-Net-Netmask-1.9015-8.el6.noarch.rpm

perl-Perlbal-1.78-1.el6.noarch.rpm

接着去安裝所提供的全部的軟件包

[root@mogilefs1 mogilefs]# yum install perl-IO-AIO perl-Net-Netmask perl-Sys-Syslog

[root@mogilefs1 mogilefs]# yum install MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm

perl-MogileFS-Client-1.14-1.el6.noarch.rpm MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm

或者在安裝包的目錄下把全部rpm一次性安裝

[root@mogilefs1 mogilefs]# yum install *.rpm

接着去mysql數據庫上進行受權

[root@mysql1 ~]# mysql

mysql> grant all on *.* to 'root'@'172.16.17.%' identified by 'xiaopeng';

Query OK, 0 rows affected (0.00 sec)

mysql> create database mogilefs;

Query OK, 1 row affected (0.03 sec)

mysql> grant all on mogilefs.* to 'moguser'@'172.16.17.%' identified by 'mogpass';

Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;

Query OK, 0 rows affected (0.00 sec)

[root@mysql1 ~]#

6、數據庫受權完成以後，接着去mogilefs文件系統的主控制檯上，對數據庫進行初始化。（建議此命令慎用，若是mogilefs文件系統在運行的時候，當心使用。）

[root@mogilefs1 mogilefs]# mogdbsetup --dbhost=172.16.17.144 --dbrootpass='xiaopeng' --dbuser='moguser' --dbpass='mogpass'

This will attempt to setup or upgrade your MogileFS database.

It won't destroy existing data.

Run with --help for more information. Run with --yes to shut up these prompts.

Continue? [N/y]: y

完成以後，能夠去數據庫確認初始化狀況

[root@mogilefs1 mogilefs]#

mysql> use mogilefs

Database changed

mysql> show tables;

+----------------------+

| Tables_in_mogilefs |

+----------------------+

數據庫準備完成以後，接着去修改控制檯文件

[root@mogilefs1 ~]# vim /etc/mogilefs/mogilefsd.conf

daemonize = 1

pidfile = /var/run/mogilefsd/mogilefsd.pid

db_dsn = DBI:mysql:mogilefs:host=172.16.100.68

db_user = moguser

db_pass = mogpass

listen = 0.0.0.0:7001

conf_port = 7001

query_jobs = 5

delete_jobs = 1

replicate_jobs = 2

reaper_jobs = 1

主配置文件修改完成以後，接着全部的存儲節點都須要進行一下的配置

[root@mogilefs1 ~]# mkdir -pv /data/mogilefs/dev1

[root@mogilefs1 ~]# chown -R mogilefs.mogilefs /data/mogilefs/

[root@mogilefs1 ~]# vim /etc/mogilefs/mogstored.conf

maxconns = 10000

httplisten = 0.0.0.0:7500

mgmtlisten = 0.0.0.0:7501

docroot = /data/mogilefs/

對於mogstord的啓動文件（全部的節點都的mogstored的啓動腳本文件須要加如下的幾項）

爲腳本添加：

pidfile=/var/run/mogilefsd/mogstored.pid

echo $(pidof mogstored) > ${pidfile}

rm -f ${pidfile}

全部的配置修改完成以後，接着去啓動服務器，進行配置資源

[root@mysql1 ~]# service mogstored start

Starting mogstored [ OK ]

[root@mysql1 ~]# ss -tanlp

[root@mogilefs1 init.d]# service mogilefsd start

Starting mogilefsd [ OK ]

[root@mogilefs1 ~]# mogadm check

Checking trackers...

127.0.0.1:7001 ... OK

Checking hosts...

No devices found on tracker(s).

[root@mogilefs1 ~]# clear

追蹤節點所能適應的管理命令

[root@mogilefs1 ~]# mogadm --tracker=172.16.17.141:7001

mogadm check Check the state of the MogileFS world.

mogadm stats Show MogileFS system statistics. mogadm host ...

host add ... Add a host to MogileFS.

host delete ... Delete a host.

host list List all hosts.

host mark ... Change the status of a host. (equivalent to 'modify --status')

host modify ... Modify a host's properties.

mogadm device ...

device add ... Add a device to a host.

device list ... List all devices, for each host.

device mark ... Mark a device as {alive,dead,down,drain,readonly}

device modify ... Modify a device's properties.

device summary ... List the summary of devices, for each host.

mogadm domain ...

domain add ... Add a domain (namespace)

domain delete ... Delete a domain.

domain list List all hosts.

mogadm class ...

class add ... Add a file class to a domain.

class delete ... Delete a file class from a domain.

class list List all classes, for each domain.

class modify ... Modify properties of a file class.

mogadm slave ...

slave add ... Add a slave node for store usage

slave delete ... Delete a slave node for store usage

slave list List current store slave nodes.

slave modify ... Modify a slave node for store usage

mogadm fsck ...

fsck clearlog Clear the fsck log

fsck printlog Display the fsck log

fsck reset ... Reset fsck position back to the beginning

fsck start Start (or resume) background fsck

fsck status Show fsck status

fsck stop Stop (pause) background fsck

fsck taillog Tail the fsck log

mogadm rebalance ...

rebalance policy ... Add or adjust the current policy

rebalance reset Reset an existing policy

rebalance settings Display rebalance settings

rebalance start Start a rebalance job

rebalance status Show status of current rebalance job

rebalance stop Stop a rebalance job

rebalance test Show what devices the current policy would match

mogadm settings ...

settings list List all server settings

settings set ... Set server setting 'key' to 'value'.

首先須要把節點設備添加至管理設備中。使其可以正常管理

[root@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.140 --ip=172.16.17.140 --status=alive

[root@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.141 --ip=172.16.17.141 --status=alive

[root@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.144 --ip=172.16.17.144 --status=alive

[root@mogilefs1 ~]#

添加完成以後，去確認設備添加狀況

[root@mogilefs1 ~]# mogadm host list

172.16.17.140 [1]: alive

IP: 172.16.17.140:7500

172.16.17.141 [2]: alive

IP: 172.16.17.141:7500

172.16.17.144 [3]: alive

IP: 172.16.17.144:7500

12、後端存儲的節點添加完成以後，接着去添加設備文件。

[root@mogilefs1 ~]# mogadm device add 172.16.17.140 1

[root@mogilefs1 ~]# mogadm device add 172.16.17.141 2

[root@mogilefs1 ~]# mogadm device add 172.16.17.144 3

[root@mogilefs1 ~]# mogadm device list

172.16.17.140 [1]: alive

used(G) free(G) total(G) weight(%)

dev1: alive 0.280 18.280 18.561 100

172.16.17.141 [2]: alive

used(G) free(G) total(G) weight(%)

dev2: alive 0.280 18.280 18.561 100

172.16.17.144 [3]: alive

used(G) free(G) total(G) weight(%)

dev3: alive 1.671 16.890 18.561 100

再次檢查mogadm的狀態信息

[root@mogilefs1 ~]# mogadm check

Checking trackers...

127.0.0.1:7001 ... OK

Checking hosts...

[ 1] 172.16.17.140 ... OK

[ 2] 172.16.17.141 ... OK

[ 3] 172.16.17.144 ... OK

Checking devices...

host device size(G) used(G) free(G) use% ob state I/O%

---- ------------ ---------- ---------- ---------- ------ ---------- -----

[ 1] dev1 18.561 0.281 18.280 1.51% writeable 0.0

[ 2] dev2 18.561 0.281 18.280 1.51% writeable 0.0

[ 3] dev3 18.561 1.671 16.890 9.00% writeable 1.7

---- ------------ ---------- ---------- ---------- ------

total: 55.683 2.233 53.451 4.01%

[root@mogilefs1 ~]#

[root@mogilefs1 ~]# mogadm domain add files

[root@mogilefs1 ~]# mogadm domain add p_w_picpaths

[root@mogilefs1 ~]# mogadm domain list

domain class mindevcount replpolicy hashtype

-------------------- -------------------- ------------- ------------ -------

files default 2 MultipleHosts() NONE

p_w_picpaths default 2 MultipleHosts() NONE

在建立domain的時候，根絕本身的須要進行建立。

建立好域名空間以後，若是須要有建立class類別。也能建立，這樣可以分類別存儲文件

基本使用方法

[root@mogilefs1 ~]# mogadm class help

mogadm class add <domain> <class> [opts] Add a file class to a domain.

<class> Name of class to add.

<domain> Domain to add class to.

--hashtype=s Hash algorithm string ('MD5', 'NONE').

--mindevcount=i Minimum number of replicas.

--replpolicy=s Replication policy string.

添加幾個基本的類試試

[root@mogilefs1 ~]# mogadm class add p_w_picpaths jpg

[root@mogilefs1 ~]# mogadm class add p_w_picpaths png

[root@mogilefs1 ~]# mogadm class add p_w_picpaths gif

[root@mogilefs1 ~]# mogadm class add files text --mindevcount=1

[root@mogilefs1 ~]# mogadm class add files html --replpolicy="MultipleHosts(3)"

[root@mogilefs1 ~]# mogadm class list

domain class mindevcount replpolicy hashtype

files default 2 MultipleHosts() NONE

files html 2 MultipleHosts(3) NONE

files plaintext 1 MultipleHosts() NONE

p_w_picpaths default 2 MultipleHosts() NONE

p_w_picpaths gif 2 MultipleHosts() NONE

p_w_picpaths jpeg 2 MultipleHosts() NONE

p_w_picpaths png 2 MultipleHosts() NONE

若是基本的屬性定義完成以後，接着能夠嘗試去上傳資源試試

[root@mogilefs1 ~]# mogupload

Usage: /usr/bin/mogupload --trackers=host --domain=foo --key='/hello.jpg' --file='./hello.jpg'

[root@mogilefs1 ~]# mogupload --trackers=172.16.17.140 --domain=files --key='/fstab.txt' --file='/etc/fstab' --class=plaintext

上傳完成以後，確認上傳狀況。可以，會生成http鏈接。可是鏈接會比較特殊。

[root@mogilefs1 ~]# mogfileinfo --trackers=172.16.17.140 --domain=files --key='/fstab.txt'

- file: /fstab.txt

class: plaintext

devcount: 1

domain: files

fid: 2

key: /fstab.txt

length: 921

- http://172.16.17.141:7500/dev2/0/000/000/0000000002.fid

[root@mogilefs1 ~]#

[root@mogilefs1 mogilefs]# mogupload --trackers=172.16.17.140 --domain=p_w_picpaths --key='/1.jpg' --file='1.jpg' --class=jpeg

[root@mogilefs1 mogilefs]# mogfileinfo --trackers=172.16.17.140 --domain=p_w_picpaths --key='/1.jpg'

- file: /1.jpg

class: jpeg

devcount: 2

domain: p_w_picpaths

fid: 3

key: /1.jpg

length: 170625

- http://172.16.17.140:7500/dev1/0/000/000/0000000003.fid

- http://172.16.17.141:7500/dev2/0/000/000/0000000003.fid

[root@mogilefs1 mogilefs]# mogstats --db_dsn="DBI:mysql:mogilefs:host=172.16.17.144" --db_user="moguser" --db_pass="mogpass" --verbose --stats="all"

Statistics for devices...

device host files status

---------- ---------------- ------------ ----------

dev1 172.16.17.140 1 alive

dev2 172.16.17.141 2 alive

---------- ---------------- ------------ ----------

Statistics for file ids...

Max file id: 3

Statistics for files...

domain class files size (m) fullsize (m)

-------------------- ----------- ---------- ----------- -------------

files plaintext 1 0 0

p_w_picpaths jpeg 1 0 0

-------------------- ----------- ---------- ----------- -------------

Statistics for replication...

domain class devcount files

-------------------- ----------- ---------- ----------

files plaintext 1 1

p_w_picpaths jpeg 2 1

-------------------- ----------- ---------- ----------

若是想要經過更加便捷的方式進行上傳數據須要，程序員本身寫API接口

前段基於nginx進行反代mogilefs文件系統

須要的安裝包組

[root@backproxy ~]# ll

-rw-r--r-- 1 root root 832104 Dec 4 15:32 nginx-1.8.0.tar.gz

-rw-r--r-- 1 root root 11208 Dec 4 18:54 nginx_mogilefs_module-1.0.4.tar.gz

須要注意的一點是，對於nginx而言，若是想實現對後端的mogilefs服務器的代理須要nginx_mogilefs_module模塊。不然不能實現此功能；同時，還須要注意的一點是模板的版本須要嚴格要求，模塊的版本信息要和nginx版本信息匹配不然會報內核溢出之類的錯誤

2015/12/04 18:36:42 [alert] 1880#0: worker process 1890 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1892 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1894 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1896 exited on signal 11 (core dumped)

在安裝以前須要解決須要的依賴環境

[root@backproxy ~]# yum groupinstall "Development Tools" "Server Platform Deveopment"

[root@backproxy ~]# yum install openssl-devel pcre-devel

還須要建立一個nginx用戶

[root@backproxy ~]# groupadd -r nginx

[root@backproxy ~]# useradd -r -g nginx nginx

接着解壓安裝包進行安裝

[root@backproxy ~]# ./configure \

--prefix=/usr \

--sbin-path=/usr/sbin/nginx \

--conf-path=/etc/nginx/nginx.conf \

--error-log-path=/var/log/nginx/error.log \

--http-log-path=/var/log/nginx/access.log \

--pid-path=/var/run/nginx/nginx.pid \

--lock-path=/var/lock/nginx.lock \

--user=nginx \

--group=nginx \

--with-http_ssl_module \

--with-http_flv_module \

--with-http_stub_status_module \

--with-http_gzip_static_module \

--http-client-body-temp-path=/var/tmp/nginx/client/ \

--http-proxy-temp-path=/var/tmp/nginx/proxy/ \

--http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \

--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \

--http-scgi-temp-path=/var/tmp/nginx/scgi \

--with-pcre \

--with-debug \

--add-module=

這裏須要特別注意的一項是，在編譯安裝nginx的時候，須要爲期指定mogilefs的模塊相對於的文件--add-module=../nginx_mogilefs_module-1.0.4

編譯完成以後進行

Make && make install

爲nginx提供一個啓動腳本以下

#!/bin/sh

# nginx - this script starts and stops the nginx daemon

# chkconfig: - 85 15

# description: Nginx is an HTTP(S) server, HTTP(S) reverse \

# proxy and IMAP/POP3 proxy server

# processname: nginx

# config: /etc/nginx/nginx.conf

# config: /etc/sysconfig/nginx

# pidfile: /var/run/nginx.pid

# Source function library.

. /etc/rc.d/init.d/functions

# Source networking configuration.

. /etc/sysconfig/network

# Check that networking is up.

[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/sbin/nginx"

prog=$(basename $nginx)

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/lock/subsys/nginx

make_dirs() {

# make required directories

user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=$[^ ]*$.*/\1/g' -`

options=`$nginx -V 2>&1 | grep 'configure arguments:'`

for opt in $options; do

if [ `echo $opt | grep '.*-temp-path'` ]; then

value=`echo $opt | cut -d "=" -f 2`

if [ ! -d "$value" ]; then

# echo "creating" $value

mkdir -p $value && chown -R $user $value

done

}

start() {

[ -x $nginx ] || exit 5

[ -f $NGINX_CONF_FILE ] || exit 6

make_dirs

echo -n $"Starting $prog: "

daemon $nginx -c $NGINX_CONF_FILE

retval=$?

echo

[ $retval -eq 0 ] && touch $lockfile

return $retval

}

stop() {

echo -n $"Stopping $prog: "

killproc $prog -QUIT

retval=$?

echo

[ $retval -eq 0 ] && rm -f $lockfile

return $retval

}

restart() {

configtest || return $?

stop

sleep 1

start

}

reload() {

configtest || return $?

echo -n $"Reloading $prog: "

killproc $nginx -HUP

RETVAL=$?

echo

}

force_reload() {

restart

}

configtest() {

$nginx -t -c $NGINX_CONF_FILE

}

rh_status() {

status $prog

}

rh_status_q() {

rh_status >/dev/null 2>&1

}

case "$1" in

start)

rh_status_q && exit 0

;;

stop)

rh_status_q || exit 0

;;

restart|configtest)

;;

reload)

rh_status_q || exit 7

;;

force-reload)

force_reload

;;

status)

rh_status

;;

condrestart|try-restart)

rh_status_q || exit 0

;;

echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"

exit 2

esac

然後爲此腳本賦予執行權限：

[root@backproxy ~]# chmod +x /etc/rc.d/init.d/nginx

添加至服務管理列表，並讓其開機自動啓動：

[root@backproxy ~]# chkconfig --add nginx

[root@backproxy ~]# chkconfig nginx on

然後就能夠啓動服務並測試了：

[root@backproxy ~]# service nginx start

接着去修改nginx的主配置文件

upstream trackers {

server 172.16.17.140:7001 weight=1;

server 172.16.17.141:7001 weight=1;

server 172.16.17.144:7001 weight=1;

}

Server {

location /p_w_picpaths {

mogilefs_tracker trackers;

mogilefs_domain p_w_picpaths;

mogilefs_methods GET;

mogilefs_noverify on;

mogilefs_pass {

proxy_pass $mogilefs_path;

proxy_hide_header Content-Type;

proxy_buffering off;

}

location /files {

mogilefs_tracker trackers;

mogilefs_domain files;

mogilefs_methods GET;

mogilefs_noverify on;

mogilefs_pass {

proxy_pass $mogilefs_path;

proxy_hide_header Content-Type;

proxy_buffering off;

}

配置結束以後，重啓服務器能夠去測試反代效果如何？

查看反代日誌

[root@backproxy ~]# tail -2 /var/log/nginx/access.log

172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /p_w_picpaths/love.jpg HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /p_w_picpaths/love.jpg HTTP/1.1" 200 190166 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

[root@backproxy ~]#