在開始以前,這篇文章並非初級入門的文章,在該文章發佈以前已經有兩篇文章講述了FastDFS和SpringBoot相關的知識,請參閱:前端
《Java高級架構之FastDFS分佈式文件集羣》:http://www.javashuo.com/article/p-awevzoil-x.htmljava
若是須要在FastDFS中實現HTTPS功能,請參閱:nginx
在下圖中,任何一個節點都支持水平橫向無限擴展。當在某一個Group中添加新的Storage服務器的時候,新加入該Group的服務器會自動同步全部數據。當新添加Group的時候,只須要修改處於前端的Nginx服務器便可。要實現主備自動切換還能夠加入KeepAlived中間件。c++
FastDFS分佈式文件集羣HA架構圖git
集羣中具體機器配置github
安裝軟件 | 機器主機名 | 應用IP地址 | 管理IP地址 |
---|---|---|---|
fastdfs+fastdfs-nginx-module | fastdfs-storage1-group1 | 192.168.80.11 | 192.168.10.11 |
fastdfs+fastdfs-nginx-module | fastdfs-storage2-group1 | 192.168.80.12 | 192.168.10.12 |
fastdfs+fastdfs-nginx-module | fastdfs-storage3-group1 | 192.168.80.13 | 192.168.10.13 |
fastdfs+fastdfs-nginx-module | fastdfs-storage4-group2 | 192.168.80.14 | 192.168.10.14 |
fastdfs+fastdfs-nginx-module | fastdfs-storage5-group2 | 192.168.80.15 | 192.168.10.15 |
fastdfs+fastdfs-nginx-module | fastdfs-storage6-group2 | 192.168.80.16 | 192.168.10.16 |
fastdfs+fastdfs-nginx-module | fastdfs-storage7-group3 | 192.168.80.17 | 192.168.10.17 |
fastdfs+fastdfs-nginx-module | fastdfs-storage8-group3 | 192.168.80.18 | 192.168.10.18 |
fastdfs+fastdfs-nginx-module | fastdfs-storage9-group3 | 192.168.80.19 | 192.168.10.19 |
fastdfs+fastdfs-nginx-module | fastdfs-storage10-group1-new | 192.168.80.20 | 192.168.10.20 |
fastdfs | fastdfs-tracker1 | 192.168.80.21 | 192.168.10.21 |
fastdfs | fastdfs-tracker2 | 192.168.80.22 | 192.168.10.22 |
fastdfs | fastdfs-tracker3 | 192.168.80.23 | 192.168.10.23 |
nginx | fastdfs-nginx0 | 192.168.80.50 | 192.168.10.50 |
nginx | fastdfs-nginx1 | 192.168.80.51 | 192.168.10.51 |
yum update mkdir /source cd /source yum install -y gcc gcc-c++ make cmake wget libevent wget https://github.com/happyfish100/libfastcommon/archive/V1.0.35.tar.gz wget https://github.com/happyfish100/fastdfs/archive/V5.10.tar.gz tar -zxvf V1.0.35.tar.gz tar -zxvf V5.10.tar.gz cd libfastcommon-1.0.35 ./make.sh ./make.sh install ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so cd ../ cd fastdfs-5.10/ ./make.sh ./make.sh install cd ../ cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf mkdir -p /data/fdfs/tracker mkdir -p /data/fdfs/storage mkdir -p /data/fdfs/client mkdir -p /data/fdfs/disks/volume0 mkdir -p /data/fdfs/disks/volume1 mkdir -p /data/fdfs/disks/volume2 mkdir -p /data/fdfs/disks/volume3 mkdir -p /data/fdfs/disks/volume4 mkdir -p /data/fdfs/disks/volume5
部署步驟正則表達式
# 添加用戶 useradd nginx -s /sbin/nologin -M # 安裝Nginx須要的pcre(Perl兼容正則表達式)庫,容許Nginx使用rewrite模塊提供URL重寫功能。 yum install pcre pcre-devel perl-ExtUtils-Embed -y # 安裝openssl-devel,容許Nginx提供HTTPS服務。 yum install openssl-devel -y # 下載軟件包 cd /source wget http://nginx.org/download/nginx-1.16.1.tar.gz wget http://nchc.dl.sourceforge.net/project/fastdfs/FastDFS%20Nginx%20Module%20Source%20Code/fastdfs-nginx-module_v1.16.tar.gz # 解壓軟件包 tar xvf fastdfs-nginx-module_v1.16.tar.gz tar -xvf nginx-1.16.1.tar.gz # 建立必要的軟鏈接 ln -s /usr/include/fastdfs/ /usr/local/include/fastdfs ln -s /usr/include/fastcommon/ /usr/local/include/fastcommon cp fastdfs-5.10/conf/http.conf /etc/fdfs/ cp fastdfs-5.10/conf/mime.types /etc/fdfs/ cp fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/ mkdir -p /data/fdfs/fastdfs-nginx-module # 編譯安裝 cd nginx-1.16.1 ./configure --user=nginx --group=nginx --prefix=/application/nginx-1.16.1 --with-http_ssl_module --with-http_gzip_static_module --with-poll_module --with-file-aio --with-http_realip_module --with-http_addition_module --with-http_random_index_module --with-pcre --with-http_stub_status_module --with-stream --add-module=/source/fastdfs-nginx-module/src/ make make install ln -s /application/nginx-1.16.1/ /application/nginx # 修改權限並建立systemd控制單元 chown -R nginx:nginx /application/nginx* touch /usr/lib/systemd/system/nginx.service
Nginx配置文件spring
user nginx; worker_processes 8; error_log logs/error.log; pid logs/nginx.pid; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; sendfile on; keepalive_timeout 65; gzip on; server { listen 80; # 配置爲主機域名 server_name fastdfs-storage1-group1; access_log logs/host.access.log main; location ~/group[0-9] { ngx_fastdfs_module; } } }
配置mod_fastdfs.conf文件緩存
base_path=/data/fdfs/fastdfs-nginx-module tracker_server=192.168.80.21:22122 tracker_server=192.168.80.22:22122 tracker_server=192.168.80.23:22122 # 配置所屬組的名稱 group_name=group1 url_have_group_name = true store_path_count=6 store_path0=/data/fdfs/disks/volume0 store_path1=/data/fdfs/disks/volume1 store_path2=/data/fdfs/disks/volume2 store_path3=/data/fdfs/disks/volume3 store_path4=/data/fdfs/disks/volume4 store_path5=/data/fdfs/disks/volume5
Nginx SystemD單元腳本bash
[Unit] Description=nginx Documentation=http://nginx.org/en/docs/ After=network.target [Service] Type=forking PIDFile=/application/nginx/logs/nginx.pid ExecStartPre=/application/nginx/sbin/nginx -t -c /application/nginx/conf/nginx.conf ExecStart=/application/nginx/sbin/nginx -c /application/nginx/conf/nginx.conf ExecReload=/application/nginx/sbin/nginx -s reload ExecStop=/application/nginx/sbin/nginx -s stop Restart=on-abort PrivateTmp=true [Install] WantedBy=multi-user.target
# 配置tracker的IP地址或者主機域名,前提要進行配置靜態hosts解析或者配置DNS bind_addr=192.168.80.21 # tracker的數據和日誌存儲目錄 base_path=/data/fdfs/tracker
# 配置該Storage服務屬於哪個Group group_name=group1 # 配置綁定的IP地址或者主機域名 bind_addr=192.168.80.12 # 配置日誌存儲目錄 base_path=/data/fdfs/storage # 配置數據存儲目錄的個數 store_path_count=6 # 配置數據存儲目錄 store_path0=/data/fdfs/disks/volume0 store_path1=/data/fdfs/disks/volume1 store_path2=/data/fdfs/disks/volume2 store_path3=/data/fdfs/disks/volume3 store_path4=/data/fdfs/disks/volume4 store_path5=/data/fdfs/disks/volume5 # 配置跟蹤服務器的IP地址或者主機域名 tracker_server=192.168.80.21:22122 tracker_server=192.168.80.22:22122 tracker_server=192.168.80.23:22122
在所有主機上執行啓動服務並設置爲開機自啓動
NGINX配置文件
user nginx; worker_processes 8; error_log logs/error.log; pid logs/nginx.pid; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; sendfile on; keepalive_timeout 65; gzip on; # Group1的Storage節點 upstream g1_pool { server 192.168.80.11 weight=1; server 192.168.80.12 weight=1; server 192.168.80.13 weight=1; } # Group2的Storage節點 upstream g2_pool { server 192.168.80.14 weight=1; server 192.168.80.15 weight=1; server 192.168.80.16 weight=1; } # Group3的Storage節點 upstream g3_pool { server 192.168.80.17 weight=1; server 192.168.80.18 weight=1; server 192.168.80.19 weight=1; } server { listen 80; server_name localhost; access_log logs/host.access.log main; location ~/group1 { proxy_pass http://g1_pool; } location ~/group2 { proxy_pass http://g2_pool; } location ~/group3 { proxy_pass http://g3_pool; } } }
啓動全部服務
systemctl daemon-reload systemctl start nginx systemctl enable nginx # 若是是跟蹤服務器執行這兩行 systemctl restart fdfs_trackerd systemctl enable fdfs_trackerd # 若是是存儲服務器執行這兩行 systemctl restart fdfs_storaged systemctl enable fdfs_storaged
在任意一臺服務器配置/etc/fdfs/client.conf
base_path=/data/fdfs/client tracker_server=192.168.80.21:22122 tracker_server=192.168.80.22:22122 tracker_server=192.168.80.23:22122
在這臺服務器進行上傳文件測試
能夠看到通過屢次上傳測試,多Group+多Tracker+多Storager生效了。
group2/M00/00/00/wKhQDl2y9aOAcngTACdpr-L7emo254.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group2/M00/00/00/wKhQD12y9aiARzwQACdpr-L7emo948.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group2/M00/00/00/wKhQEF2y9ayAZBcBACdpr-L7emo474.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQEV2y9a6Af_mrACdpr-L7emo390.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQEl2y9a-Ae1cWACdpr-L7emo570.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQE12y9bGAbWemACdpr-L7emo654.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQEV2y9bKADzEIACdpr-L7emo251.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQEl2y9bOAHT16ACdpr-L7emo211.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQE12y9bSAEwBUACdpr-L7emo190.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQEV2y9bWAc4CSACdpr-L7emo991.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQEl2y9baAJmqNACdpr-L7emo880.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQE12y9biAbE8IACdpr-L7emo426.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M03/00/00/wKhQEV2y9bmAGh5pACdpr-L7emo466.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M03/00/00/wKhQEl2y9bmATKQjACdpr-L7emo349.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M03/00/00/wKhQE12y9bmAXVOfACdpr-L7emo921.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M04/00/00/wKhQEV2y9bmAV9sqACdpr-L7emo420.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M04/00/00/wKhQEl2y9bqAOg2NACdpr-L7emo216.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M04/00/00/wKhQE12y9bqAcZKEACdpr-L7emo382.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M05/00/00/wKhQEV2y9bqAJVS9ACdpr-L7emo910.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M05/00/00/wKhQEl2y9buATzq2ACdpr-L7emo523.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M05/00/00/wKhQE12y9buALTHRACdpr-L7emo483.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQEV2y9buAJO-PACdpr-L7emo079.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQEl2y9buAP1NpACdpr-L7emo443.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png ^[[Agroup3/M00/00/00/wKhQE12y9byAdfYxACdpr-L7emo025.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQEV2y9byAJB2OACdpr-L7emo747.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQEl2y9byAIye6ACdpr-L7emo456.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQE12y9byADddiACdpr-L7emo585.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png ^[[Agroup3/M02/00/00/wKhQEV2y9b2AFuifACdpr-L7emo318.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQEl2y9b2AQ3prACdpr-L7emo501.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQE12y9b2AbG-UACdpr-L7emo027.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M00/00/00/wKhQC12y9b6AZ866ACdpr-L7emo376.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M00/00/00/wKhQDF2y9b6ABGW6ACdpr-L7emo312.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M00/00/00/wKhQDV2y9b6AFEd_ACdpr-L7emo639.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M01/00/00/wKhQC12y9b-AU07YACdpr-L7emo467.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M01/00/00/wKhQDF2y9b-ALDNrACdpr-L7emo209.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M01/00/00/wKhQDV2y9b-AAsKGACdpr-L7emo483.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M02/00/00/wKhQC12y9b-AfzkvACdpr-L7emo440.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M02/00/00/wKhQDF2y9cCATq8TACdpr-L7emo504.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M02/00/00/wKhQDV2y9cCAUPEnACdpr-L7emo555.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M03/00/00/wKhQC12y9cCABjj9ACdpr-L7emo576.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M03/00/00/wKhQDF2y9cGAHNLqACdpr-L7emo735.png [root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png
在SpringBoot中使用FastDFS的時候須要集成fastdfs-spring-boot-starter:
項目主頁GitHub地址:https://github.com/bluemiaomiao/fastdfs-spring-boot-starter
克隆項目到本地
git clone https://github.com/bluemiaomiao/fastdfs-spring-boot-starter.git cd fastdfs-spring-boot-starter
使用Maven編譯打包並安裝到本地
mvn clean install mvn source:jar install mvn javadoc:jar install
在POM文件中添加依賴
<dependency> <groupId>com.bluemiaomiao</groupId> <artifactId>fastdfs-spring-boot-starter</artifactId> <version>1.0-SNAPSHOT</version> </dependency>
在主配置類上添加註解 (
@EnableFastdfsClient
)
@EnableFastdfsClient @SpringBootApplication public class DemoApplication { @Autowired private FastdfsClientService fastdfsClientService; public static void main(String[] args) { SpringApplication.run(DemoApplication.class, args); } }
添加配置條目(application.properties)
fastdfs.nginx-servers=192.168.80.50:80,192.168.80.51:80 fastdfs.tracker-servers=192.168.80.21:22122,192.168.80.22:22122,192.168.80.23:22122 fastdfs.http-secret-key=2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrn # 在本項目中沒有使用Token功能 fastdfs.http-anti-steal-token=false fastdfs.http-tracker-http-port=8080 fastdfs.network-timeout=30 fastdfs.connect-timeout=5 fastdfs.connection-pool-max-idle=18 fastdfs.connection-pool-min-idle=2 fastdfs.connection-pool-max-total=18 fastdfs.charset=UTF-8
或者添加配置條目(application.yml)
fastdfs: charset: UTF-8 connect-timeout: 5 http-secret-key: 2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrn network-timeout: 30 http-anti-steal-token: false http-tracker-http-port: 8080 connection-pool-max-idle: 20 connection-pool-max-total: 20 connection-pool-min-idle: 2 nginx-servers: 192.168.80.50:80,192.168.80.51:80 tracker-servers: 192.168.80.21:22122,192.168.80.22:22122,192.168.80.23:22122
測試主要代碼
@RestController @RequestMapping("/file") public class FileController { @Autowired private FastdfsClientService fastdfsClientService; @PostMapping("/upload") public String[] upload(@RequestParam("file") MultipartFile file) { String[] remoteInfo = null; try { remoteInfo = fastdfsClientService.upload("group1", file.getBytes(), "png", null); } catch (Exception e) { e.printStackTrace(); } return remoteInfo; } @GetMapping("/download") public String download(@RequestParam("group") String group, @RequestParam("file_id") String fileId) { String url = ""; try { url = fastdfsClientService.autoDownloadWithoutToken(group, fileId, UUID.randomUUID().toString()); } catch (Exception e) { e.printStackTrace(); } return url; } }