Minio 集羣擴容存儲空間，配合nginx 負載反向代理後端minio 集羣服務器，提高高可用性

時間  2020-12-12
標籤 html linux nginx web bash 服務器 dom tcp 分佈式 ide 欄目 負載均衡 简体版
原文   原文鏈接

環境：Centos  7

軟件：minio,Etcd

需求：經過聯盟兩個集羣實例，實現水平擴容存儲空間問題；

服務器使用阿里雲，一共4臺服務器(官方說明最好4臺服務器作分佈式，測試節省服務器因此咱們使用兩臺服務器每臺服務器兩個目錄，作一個假服務器代替，真實環境條件容許最好4臺服務器操做)：

集羣

Minio-1

Minio-2

IP 地址

172.31.78.244

172.31.78.243

172.31.78.241

172.31.78.242

計算名稱地址

topsysfile.server.1.itusye.co

topsysfile.server.2.itusye.co

topsysfile.server.3.itusye.co

topsysfile.server.4.itusye.co

目錄地址

/Data/minio/{data1,data2}

/Data/minio/{data1,data2}

 

 

 

 

 

 

 

 

官方說明：
可擴容：不一樣MinIO集羣能夠組成聯邦，並造成一個全局的命名空間，並跨越多個數據中心，但不支持動態擴容；

https://docs.min.io/docs/minio-federation-quickstart-guide.html  官方參考 文檔

1. 因爲多個集羣使用Etcd 聯邦，因此咱們先須要安裝配置ETCD 服務，etcd 與zookeeper 服務器都是奇數性3，5，7 模式增長，etcd 安裝配置我就簡單跳過了，列出安裝與配置；html

[root@Minio-Storage-Services ~]# yum -y install etcd [root@Minio-Storage-Services ~]# cat /etc/etcd/etcd.conf | grep -v '#' ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://172.31.78.244:2380" #內網通迅 ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" #外網通迅 ETCD_NAME="top-etcd-01" #這裏是集羣鏈接地址，請注意與cluster 對應，與zookeeper myid 是同樣的原理 ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.78.244:2380" ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379" ETCD_INITIAL_CLUSTER="top-etcd-01=http://172.31.78.244:2380,top-etcd-02=http://172.31.78.243:2380,top-etcd-03=http://172.31.78.241:2380" #etcd 集羣鏈接分配地址，注意三臺服務器同樣 ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_STRICT_RECONFIG_CHECK="true"

以上服務器配置是172.31.78.244 這臺linux

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:16:3e:08:72:67 brd ff:ff:ff:ff:ff:ff inet 172.31.78.243/20 brd 172.31.79.255 scope global dynamic eth0 valid_lft 315039419sec preferred_lft 315039419sec [root@Minio-Storage-Services ~]# cat /etc/etcd/etcd.conf  | grep -v '#' ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://172.31.78.243:2380" ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" ETCD_NAME="top-etcd-02" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.78.243:2380" ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379" ETCD_INITIAL_CLUSTER="top-etcd-01=http://172.31.78.244:2380,top-etcd-02=http://172.31.78.243:2380,top-etcd-03=http://172.31.78.241:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_STRICT_RECONFIG_CHECK="true" [root@Minio-Storage-Services ~]#
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:16:3e:08:74:04 brd ff:ff:ff:ff:ff:ff inet 172.31.78.241/20 brd 172.31.79.255 scope global dynamic eth0 valid_lft 315040516sec preferred_lft 315040516sec [root@Minio-Storage-Services ~]# cat /etc/etcd/etcd.conf | grep -v '#' ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://172.31.78.241:2380" ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" ETCD_NAME="top-etcd-03" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.78.241:2380" ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379" ETCD_INITIAL_CLUSTER="top-etcd-01=http://172.31.78.244:2380,top-etcd-02=http://172.31.78.243:2380,top-etcd-03=http://172.31.78.241:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_STRICT_RECONFIG_CHECK="true" [root@Minio-Storage-Services ~]#

 

etcd 已安裝完成；nginx

 

2.  下載minio , 4臺配置服務器一樣的配置以下；web

[root@Minio-Storage-Services ~]#  wget https://dl.min.io/server/minio/release/linux-amd64/minio
[root@Minio-Storage-Services ~]# chmod +x minio [root@Minio-Storage-Services ~]# mv minio /usr/local/bin/ [root@Minio-Storage-Services ~]# mkdir /usr/local/minio{bin,config} -p [root@Minio-Storage-Services ~]# mkdir /Data/minio/{data1,data2} -p [root@Minio-Storage-Services ~]# firewall-cmd --zone=public --add-port=2380/tcp --add-port=2379/tcp --permanent && firewall-cmd --reload [root@Minio-Storage-Services ~]# firewall-cmd --zone=public --add-port=9000/tcp --permanent [root@Minio-Storage-Services ~]# systemctl restart firewalld

 

3. 編輯啓動腳本，兩個集羣，兩份配置不同，因爲我配置了hosts，因此etcd 鏈接使用了域名bash

[root@Minio-Storage-Services ~]# more /etc/hosts ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 172.31.78.241   Minio-Storage-Services  Minio-Storage-Services 172.31.78.244   topsysfile.server.1.itusye.co 172.31.78.243   topsysfile.server.2.itusye.co 172.31.78.241   topsysfile.server.3.itusye.co 172.31.78.242   topsysfile.server.4.itusye.co [root@Minio-Storage-Services ~]#

Minio-1集羣配置，兩臺服務器一樣的配置；服務器

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:16:3e:08:77:6b brd ff:ff:ff:ff:ff:ff inet 172.31.78.244/20 brd 172.31.79.255 scope global dynamic eth0 valid_lft 315039437sec preferred_lft 315039437sec [root@Minio-Storage-Services ~]# more /usr/local/minio/bin/run.sh #!/bin/bash export MINIO_ACCESS_KEY=topsysfile export MINIO_SECRET_KEY=Hao123baidu! export MINIO_ETCD_ENDPOINTS="http://topsysfile.server.1.海itusye.co:2380,http://topsysfile.server.2.itusye.co:2380,http://topsysfile.server.3.itusye.co:2380" export MINIO_DOMAIN=topfile.storage.minio.itusye.co export MINIO_PUBLIC_IPS=172.31.78.244,172.31.78.243 minio server --config-dir /usr/local/minio/config/ \ http://172.31.78.244/Data/minio/data1 \
http://172.31.78.244/Data/minio/data2 \
http://172.31.78.243/Data/minio/data1 \
http://172.31.78.243/Data/minio/data2

 

Minio-2 集羣配置，兩臺服務器一樣的配置；dom

 valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:16:3e:08:74:04 brd ff:ff:ff:ff:ff:ff inet 172.31.78.241/20 brd 172.31.79.255 scope global dynamic eth0 valid_lft 315034589sec preferred_lft 315034589sec [root@Minio-Storage-Services ~]# more /usr/local/minio/bin/run.sh #!/bin/bash export MINIO_ACCESS_KEY=topsysfile # 登陸帳號 export MINIO_SECRET_KEY=Hao123baidu! # 登陸密碼 export MINIO_ETCD_ENDPOINTS="http://topsysfile.server.1.itusye.co:2380,http://topsysfile.server.2.itusye.co:2380,http://topsysfile.server.3.itusye.co:2380" # 配置etcd 鏈接 export MINIO_DOMAIN=topfile.storage.minio.itusye.co #此處需特別注意：此域名須要配置與nginx 配置反向代理時使用的域名一致，否則好比分享會直接訪問到xml, 開發鏈接也會有問題， export MINIO_PUBLIC_IPS=172.31.78.241,172.31.78.242 # 此處須要配置上Minio-1 集羣的全部ip ,可配置內網外網，由於我測試使用內網通迅，此處配置內網地址了 minio server --config-dir /usr/local/minio/config/ \   # 官方提示：--config-dir 選項要中止使用了，此選擇主要建立一個certs目錄存放ca 信息 http://172.31.78.241/Data/minio/data1 \ # 如下信息是minio 分佈式集羣minio-1 啓動狀況
http://172.31.78.241/Data/minio/data2 \
http://172.31.78.242/Data/minio/data1 \
http://172.31.78.242/Data/minio/data2
[root@Minio-Storage-Services ~]#

 

4 配置啓動腳本，全部服務器一樣的配置；tcp

[root@Minio-Storage-Services ~]# more /usr/lib/systemd/system/minio.service [Unit] Description=Minio service Documentation=https://docs.minio.io/
 [Service] WorkingDirectory=/usr/local/minio/ ExecStart=/usr/local/minio/bin/run.sh Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target [root@Minio-Storage-Services ~]#

5.配置nginx , https://docs.min.io/docs/setup-nginx-proxy-with-minio.html   官方參考文檔分佈式

主要須要注意：客戶端上傳限制的問題ide

client_max_body_size 2048M;
client_body_buffer_size 500M;

如下爲nginx Server 模塊配置

upstream topfilestorage-minio{ ip_hash; server topsysfile.server.1.toptech.ph:9000 weight=3 max_fails=3 fail_timeout=10s; server topsysfile.server.2.toptech.ph:9000 weight=3 max_fails=3 fail_timeout=10s; server topsysfile.server.3.toptech.ph:9000 weight=3 max_fails=3 fail_timeout=10s; server topsysfile.server.4.toptech.ph:9000 weight=3 max_fails=3 fail_timeout=10s; # check interval=3000 rise=2 fall=3 timeout=1000 default_down=true type=http; } server { listen 80; server_name localhost; if ($scheme = http ) { return 301 https://$host$request_uri;
 } } server { server_name localhost; listen 443; ssl on; access_log /home/logs/nginx/www_logs/web_access.log main; error_log /home/logs/nginx/error.log; ssl_certificate /usr/local/nginx/ssl/topfile.crt; ssl_certificate_key /usr/local/nginx/ssl/topfile.key; ssl_prefer_server_ciphers on; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:ECDHE-RSA-AES128-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA128:DHE-RSA-A
ES128-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA128:ECDHE-RSA-AES128-SHA384:ECDHE-RSA-AES128-SHA128:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES 128-SHA128:DHE-RSA-AES128-SHA128:DHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA384:AES128-GCM-SHA128:AES128-SHA128:AES12 8-SHA128:AES128-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
 ssl_ecdh_curve secp384r1; error_page 403 404 /403.html; location = /403.html { root /var/www/; } ignore_invalid_headers off; proxy_buffering off; location / { proxy_pass http://topfilestorage-minio;
                        include conf.d/proxy.conf; } }

如下proxy.conf 文檔

[root@Minio-Storage-Services ~]# more /usr/local/nginx/config/conf.d/proxy.conf | grep -v '#' proxy_http_version 1.1; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $http_host; proxy_set_header Connection ""; chunked_transfer_encoding off; [root@Minio-Storage-Services ~]#

 

 

以上集羣已配置完成，測試新建文件夾會自動選擇集羣存儲，而且測試若是固定用一個集羣建立一個根目錄文件夾，再用另一個集羣或者反向代理去上傳都是能夠把數據上傳存儲到文件夾所在的集羣空間裏，兩個集羣內的數據並不會同步，可是在存儲時minio 會自動計算選擇集羣存儲；

 

MinIO是一個開源的對象存儲解決方案，優勢以下：

  1. 高性能：做爲高性能對象存儲，在標準硬件條件下它能達到55GB/s的讀、35GG/s的寫速率
  2. 可擴容：不一樣MinIO集羣能夠組成聯邦，並造成一個全局的命名空間，並跨越多個數據中心
  3. 雲原生：容器化、基於K8S的編排、多租戶支持
  4. 兼容性：兼容S3 API這一事實上的對象存儲標準，最早支持S3 Select
  5. 簡單：這一設計原則讓MinIO不容易出錯、更快啓動
  6. 支持糾刪碼：MinIO使用糾刪碼、Checksum來防止硬件錯誤和靜默數據污染。在最高冗餘度配置下，即便丟失1/2的磁盤也能恢復數據

 

原文出處：https://www.cnblogs.com/itusye/p/12544516.html

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程