下面的命令默認都使用root用戶進行操做,操做系統爲Centos7,mongodb3.6.x以上版本node
修改/etc/security/limits.conf持久化設置容許用戶/進程打開文件句柄數,這一步須要重啓系統,否則不起做用linux
* soft nofile 1048576git
* hard nofile 1048576github
* soft nproc 524288mongodb
* hard nproc 524288shell
修改/etc/profile,在最後添加ulimit -s 1024,而後保存並source /etc/profile數據庫
Centos7如下命令:數組
chkconfig iptables off && service iptables stopbash
使用命令查看chkconfig --list 是否設置自動啓動爲關閉服務器
Centos7以上命令:
systemctl stop firewalld
systemctl is-enabled firewalld
在/etc/hosts添加集羣全部的ip及對應的hostname
注意:配置的ip數與選擇的模板相關聯
每臺機器分別執行下面命令,這裏以5臺機器爲準:
ssh-keygen -t rsa -P "" -f /root/.ssh/id_rsa
ssh-copy-id node1
ssh-copy-id node2
ssh-copy-id node3
ssh-copy-id node4
ssh-copy-id node5
ifconfig、ssh須要可用
須要安裝numactl,使用rpm -qa | grep numactl查看numactl是否已安裝
注意:完成上面5步操做後重啓每臺機器。重啓完畢,使用命令ulimit -a 查看
要保證 open files 對應的值爲 1048576
Stack size 對應的值爲1024
Max user processes對應的值爲 524288
進入到shell目錄,執行chmod +x auto*加執行權限
進入mongodb的bin目錄,執行chmod +x ./*加執行權限
shell腳本文件目前放置在mongodb的安裝目錄下:
⑪config.properties,配置文件,包括mongodb安裝目錄、mongoCluster集羣安裝目錄、所需主機、數據庫等等。如圖:
上圖幾個參數可能須要隨環境不一樣而修改。
說明:
ips=192.168.187.201, 192.168.187.202, 192.168.187.203, 192.168.187.204, 192.168.187.205
這裏默認192.168.187.201爲mongdb集羣的節點node1,192.168.187.202爲mongdb集羣的節點node2,192.168.187.203爲mongdb集羣的節點node3,192.168.187.204爲mongdb集羣的節點node4,,192.168.187.205爲mongdb集羣的節點node5。
template與ips的ip數量對應,ips的ip數爲5,則意味着採用5臺機器的模板,這裏默認爲5。
⑬template 目錄包括3.conf、4.conf和5.conf,爲mongodb3臺機器、4臺機器和5臺機器安裝模板,有須要者能夠自行修改模板。
3.conf 3臺機器分配的mongodb節點模板爲
#全部mongo節點
nodes=config,shard1,shard2,shard3,mongos
#機器分配到的mongo節點
node1=mongos,config,shard1,shard2,shard3
node2=mongos,config,shard1,shard2,shard3
node3=mongos,config,shard1,shard2,shard3
#mongo節點分配到的機器,注意shard部分排在第一的爲master,第二爲secondary,第三位arbitery
config=node1,node2,node3
mongos=node1,node2,node3
shard1=node1,node2,node3
shard2=node2,node3,node1
shard3=node3,node2,node1
#mongo節點端口
config_port=29040
mongos_port=29050
shard1_port=29010
shard2_port=29020
shard3_port=29030
注意:這裏的node一、node二、node3不是機器hostname(固然也能夠以這種node一、node2方式去命名hostname),只是機器ip的代名稱,由於不一樣開發環境中的ip老是不一樣的,因此使用node一、node2...去代替變化的ip。這裏的node一、node二、node3默認與config.properties中的ips屬性對應,即ips中的第一個ip默認爲node1,第二個ip默認爲node2,第三個ip默認爲node3,4.conf和5.conf類同。查看下圖,能夠清晰看到mongodb分佈的節點:
4.conf 4臺機器分配mongodb節點的模版爲
#全部mongo節點
nodes=config,shard1,shard2,shard3,shard4,mongos
#機器分配到的mongo節點
node1=mongos,shard1,shard3,shard4
node2=mongos,config,shard1,shard2,shard4
node3=mongos,config,shard1,shard2,shard3
node4=config,shard2,shard3,shard4
#mongo節點分配到的機器,注意shard部分排在第一的爲master,第二爲secondary,第三位arbitery
config=node2,node3,node4
mongos=node1,node2,node3
shard1=node1,node2,node3
shard2=node2,node3,node4
shard3=node3,node4,node1
shard4=node4,node1,node2
#mongo節點端口
config_port=29040
mongos_port=29050
shard1_port=29000
shard2_port=29010
shard3_port=29020
shard4_port=29030
5.conf 5臺機器分配mongodb節點的模版爲
#全部mongo節點
nodes=config,shard1,shard2,shard3,shard4,shard5,mongos
#機器分配到的mongo節點
node1=mongos,shard1,shard4,shard5
node2=mongos,shard1,shard2,shard5
node3=config,shard1,shard2,shard3
node4=config,shard2,shard3,shard4
node5=config,shard3,shard4,shard5
#mongo節點分配到的機器, 注意shard部分排在第一的爲master,第二爲secondary,第三位arbitery
config=node3,node4,node5
mongos=node1,node2
shard1=node1,node2,node3
shard2=node2,node3,node4
shard3=node3,node4,node5
shard4=node4,node5,node1
shard5=node5,node1,node2
#mongo節點端口
config_port=29040
mongos_port=29050
shard1_port=29060
shard2_port=29070
shard3_port=29080
shard4_port=29090
shard5_port=29100
模板須要配置的幾項內容:
若有須要,可自行配置六、7或者更多的機器分配mongo模板
①autoCheckLive.sh腳本爲建立所在服務器對應的mongodb的進程存活監控,不存在則自動拉起。
#! /bin/bash #shell目錄的絕對路徑 shellPath=$1 configPath=$shellPath/config.properties #從config.properties文件讀取數據出來 clusterPath=`awk -F= -v k=clusterPath '{ if ( $1 == k ) print $2; }' $configPath` template=`awk -F= -v k=template '{ if ( $1 == k ) print $2; }' $configPath` templatePath=$shellPath/template/$template.conf ips=`awk -F= -v k=ips '{ if ( $1 == k ) print $2; }' $configPath` #ip數組 eval $(echo $ips | awk '{split($0, arr, ","); for(i in arr) print "ipArray["i"]="arr[i]}') #本地ip localIp=`/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6 | awk '{print $2}' | tr -d "addr:"` suffix="" #判斷linux系統cpu是否爲numa架構,若是是則傳回_numa,禁用numactl numaCount=`grep -i numa /var/log/dmesg | wc -l` nodeCount=`grep -i numa /var/log/dmesg | grep -wi "node" | wc -l` offCount=`grep -i numa /var/log/dmesg | grep -wi "numa=off" | wc -l` if [[ ($numaCount -gt 1) && ($nodeCount -gt 1) && ($offCount -eq 0) ]] then suffix="_numa" fi #遍歷ip數組 for((i=1; i<=${#ipArray[@]}; i++)) do #獲取本地ip在模板中的代稱號,根據本地ip所在ip數組中的下標數字獲取 if [[ $localIp = ${ipArray[i]} ]] then #獲取模板中本地ip對應的代稱號所擁有的mongo節點 mongodbNodes=`awk -F= -v k=node$i '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $mongodbNodes | awk '{split($0, mongodbArr, ","); for(y in mongodbArr) print "mongodbArray["y"]="mongodbArr[y]}') for n in ${mongodbArray[*]} do pid=`cat $clusterPath/$n/pid/$n.pid` #判斷是否存活 count=`ps -ef | grep $pid | grep -v grep | wc -l` #程序掛掉啦,啓動 if [ $count -eq 0 ];then cd $shellPath && $shellPath/autoStartUp.sh $n$suffix fi done fi done
②autoClusterClose.sh腳本爲mongodb集羣關閉腳本,用法:./autoClusterClose.sh shutdown或者./autoClusterClose.sh kill,這裏建議使用shutdown方式關閉。
#! /bin/bash configPath=config.properties #mongodb的安裝目錄 mongodb_home=`awk -F= -v k=mongodbHome '{ if ( $1 == k ) print $2; }' $configPath` #mongodb集羣安裝目錄 clusterPath=`awk -F= -v k=clusterPath '{ if ( $1 == k ) print $2; }' $configPath` #所採用的安裝模板 template=`awk -F= -v k=template '{ if ( $1 == k ) print $2; }' $configPath` #模板所在的位置 templatePath=template/$template.conf user=`awk -F= -v k=user '{ if ( $1 == k ) print $2; }' $configPath` localIp=`/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6 | awk '{print $2}' | tr -d "addr:"` ips=`awk -F= -v k=ips '{ if ( $1 == k ) print $2; }' $configPath` eval $(echo $ips | awk '{split($0, arr, ","); for(i in arr) print "ipArray["i"]="arr[i]}') #獲取全部的配置、路由、分片 nodes=`awk -F= -v k=nodes '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $nodes | awk '{split($0, nodeArr, ","); for(i in nodeArr) print "nodeArray["i"]="nodeArr[i]}') #查看/etc/crontab是否開啓定時任務,有則關閉 for((i=1; i<=${#ipArray[@]}; i++)) do if [[ $localIp = ${ipArray[i]} ]] then #kill本地的mongod、mongos echo "*****close ${ipArray[i]} 定時任務:日誌切割和mongo進程存活監控*****" $mongodb_home/shell/autoConfig.sh removeCronTask else echo "*****close ${ipArray[i]} 定時任務:日誌切割和mongo進程存活監控*****" ssh $user@${ipArray[i]} "cd $mongodb_home/shell && $mongodb_home/shell/autoConfig.sh removeCronTask" fi done if [[ $1 = "kill" ]] then for((i=1; i<=${#ipArray[@]}; i++)) do if [[ $localIp = ${ipArray[i]} ]] then #kill本地的mongod、mongos echo "*****close ${ipArray[i]} mongodb*****" ps -ef | grep $clusterPath/conf | grep -v grep | cut -c 9-15 | xargs kill -2 else echo "*****close ${ipArray[i]} mongodb*****" ssh $user@${ipArray[i]} "ps -ef | grep $clusterPath/conf | grep -v grep | cut -c 9-15 | xargs kill -2" fi done fi if [[ $1 = "shutdown" ]] then #1.先關閉mongos #找到mongos對應的機器節點 mongoss=`awk -F= -v k=mongos '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $mongoss | awk '{split($0, mongosArr, ","); for(i in mongosArr) print "mongosArray["i"]="mongosArr[i]}') for mongosNode in ${mongosArray[*]} do #刪除node,保留右邊字符。下面兩句是根據機器節點的代名稱數字找到其對應的ip mongosIpNum=${mongosNode#*node} mongosIp=${ipArray[$mongosIpNum]} if [[ $localIp = $mongosIp ]] then echo "****************close $mongosIp mongos****************" ps -ef | grep $clusterPath/conf/mongos.conf | grep -v grep | cut -c 9-15 | xargs kill -2 else echo "****************close $mongosIp mongos****************" ssh $user@$mongosIp "ps -ef | grep $clusterPath/conf/mongos.conf | grep -v grep | cut -c 9-15 | xargs kill -2" fi done #2.再關閉configs configs=`awk -F= -v k=config '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $configs | awk '{split($0, configArr, ","); for(i in configArr) print "configArray["i"]="configArr[i]}') #副本集,先關閉仲裁節點、從節點,最後關閉主節點 for((i=${#configArray[@]}; i>=1; i--)) do configNode=${configArray[i]} #刪除node,保留右邊字符 configIpNum=${configNode#*node} configIp=${ipArray[$configIpNum]} if [[ $localIp = $configIp ]] then echo "****************close $configIp config****************" $mongodb_home/bin/mongod -f $clusterPath/conf/config.conf --shutdown else echo "****************close $configIp config****************" ssh $user@$configIp "$mongodb_home/bin/mongod -f $clusterPath/conf/config.conf --shutdown" fi done #3.最後關閉shards for node in ${nodeArray[*]} do if [[ $node =~ "shard" ]] then shards=`awk -F= -v k=$node '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $shards | awk '{split($0, shardArr, ","); for(i in shardArr) print "shardArray["i"]="shardArr[i]}') #副本集,先關閉仲裁節點、從節點,最後關閉主節點 for((i=${#shardArray[@]}; i>=1; i--)) do shardNode=${shardArray[i]} #刪除node,保留右邊字符 shardIpNum=${shardNode#*node} shardIp=${ipArray[$shardIpNum]} if [[ $localIp = $shardIp ]] then echo "****************close $shardIp $node****************" $mongodb_home/bin/mongod -f $clusterPath/conf/$node.conf --shutdown else echo "****************close $shardIp $node****************" ssh $user@$shardIp "$mongodb_home/bin/mongod -f $clusterPath/conf/$node.conf --shutdown" fi done fi done fi
③autoClusterIndex.sh腳本爲mongodb集羣初始化配置文件和修改系統配置、config和shard副本集以及mongos路由啓動並初始化、mongodb集羣數據庫表分片和基礎數據初始化的腳本、添加定時任務。用法:./ autoClusterIndex.sh。使用該腳本,能夠全自動化安裝mongodb。
#! /bin/bash configPath=config.properties #從config.properties文件讀取數據出來 clusterPath=`awk -F= -v k=clusterPath '{ if ( $1 == k ) print $2; }' $configPath` mongodb_home=`awk -F= -v k=mongodbHome '{ if ( $1 == k ) print $2; }' $configPath` ips=`awk -F= -v k=ips '{ if ( $1 == k ) print $2; }' $configPath` localIp=`/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6 | awk '{print $2}' | tr -d "addr:"` user=`awk -F= -v k=user '{ if ( $1 == k ) print $2; }' $configPath` template=`awk -F= -v k=template '{ if ( $1 == k ) print $2; }' $configPath` templatePath=template/$template.conf #獲取全部的配置、路由、分片 nodes=`awk -F= -v k=nodes '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $nodes | awk '{split($0, nodeArr, ","); for(i in nodeArr) print "nodeArray["i"]="nodeArr[i]}') eval $(echo $ips | awk '{split($0, arr, ","); for(i in arr) print "ipArray["i"]="arr[i]}') #遍歷ip數組 for((i=1; i<=${#ipArray[@]}; i++)) do if [[ $localIp = ${ipArray[i]} ]] then #1.建立mongodbCluster並修改系統配置 echo "********************************start ${ipArray[i]} autoConfig && autoSystemProperties********************************" $mongodb_home/shell/autoConfig.sh && $mongodb_home/shell/autoSystemProperties.sh else #把遠程服務器舊的mongodb安裝包刪除 ssh $user@${ipArray[i]} "rm -rf $mongodb_home" #把mongodb安裝包拷貝到遠程服務器 scp -r $mongodb_home $user@${ipArray[i]}:$mongodb_home echo "********************************start ${ipArray[i]} autoConfig && autoSystemProperties********************************" ssh $user@${ipArray[i]} "cd $mongodb_home/shell && $mongodb_home/shell/autoConfig.sh && $mongodb_home/shell/autoSystemProperties.sh" fi done #2.mongodb集羣啓動config、shard一、shard二、shard3 echo "********************************start mongodb集羣啓動config、shard一、shard二、shard3********************************" #不啓動mongos,只啓動config,shards $mongodb_home/shell/autoClusterStartUp.sh notmongos #3.mongodb集羣配置、分片的副本集初始化 echo "********************************start mongodb集羣分片和副本集初始化********************************" $mongodb_home/shell/autoClusterInitSvr.sh cs #4.mongodb集羣啓動mongos echo "********************************start mongodb集羣mongos********************************" mongoss=`awk -F= -v k=mongos '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $mongoss | awk '{split($0, mongosArr, ","); for(i in mongosArr) print "mongosArray["i"]="mongosArr[i]}') for mongosNode in ${mongosArray[*]} do #刪除node,保留右邊字符 mongosIpNum=${mongosNode#*node} mongosIp=${ipArray[$mongosIpNum]} if [[ $localIp = $mongosIp ]] then echo "****************start $mongosIp mongos****************" $mongodb_home/shell/autoStartUp.sh mongos else echo "****************start $mongosIp mongos****************" ssh $user@$mongosIp "cd $mongodb_home/shell && ./autoStartUp.sh mongos" fi done #5.mongodb集羣mongos初始化 echo "********************************start mongodb集羣mongos初始化********************************" $mongodb_home/shell/autoClusterInitSvr.sh mongos #6.mongodb集羣數據庫表分片和初始化 echo "********************************start mongodb集羣數據庫表分片和初始化********************************" $mongodb_home/shell/autoClusterShardedAndInitDB.sh #7.mongodb集羣配置定時任務:日誌天天切割,保留7天日誌/每隔10分鐘監控mongo進程是否存活,不存活則自動拉起 for((i=1; i<=${#ipArray[@]}; i++)) do if [[ $localIp = ${ipArray[i]} ]] then echo "******************************** ${ipArray[i]} 添加定時任務:日誌切割和mongo存活監控 ********************************" $mongodb_home/shell/autoConfig.sh addCronTask else echo "******************************** ${ipArray[i]} 添加定時任務:日誌切割和mongo存活監控 ********************************" ssh $user@${ipArray[i]} "cd $mongodb_home/shell && $mongodb_home/shell/autoConfig.sh addCronTask" fi done
④autoClusterInitSvr.sh腳本爲mongodb集羣config、shard和mongos初始化腳本。用法:./autoClusterInitSvr.sh cs,初始化config和shard副本集配置;./autoClusterInitSvr.sh mongos,初始化mongos路由配置。
#! /bin/bash configPath=config.properties mongodb_home=`awk -F= -v k=mongodbHome '{ if ( $1 == k ) print $2; }' $configPath` template=`awk -F= -v k=template '{ if ( $1 == k ) print $2; }' $configPath` templatePath=template/$template.conf ips=`awk -F= -v k=ips '{ if ( $1 == k ) print $2; }' $configPath` eval $(echo $ips | awk '{split($0, arr, ","); for(i in arr) print "ipArray["i"]="arr[i]}') #獲取全部的配置、路由、分片 nodes=`awk -F= -v k=nodes '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $nodes | awk '{split($0, nodeArr, ","); for(i in nodeArr) print "nodeArray["i"]="nodeArr[i]}') param=$1 if [[ $param = "cs" ]] then #自動化設置配置副本集 config_numbers="" ccount=0 config_port=`awk -F= -v k=config_port '{ if ( $1 == k ) print $2; }' $templatePath` configs=`awk -F= -v k=config '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $configs | awk '{split($0, configArr, ","); for(i in configArr) print "configArray["i"]="configArr[i]}') configMasterNode=${configArray[1]} echo "configMasterNode: $configMasterNode" #刪除node,保留右邊字符 configMasterIpNum=${configMasterNode#*node} #echo "configMasterIpNum: $configMasterIpNum" configMasterIp=${ipArray[$configMasterIpNum]} #echo "configMasterIp: $configMasterIp" for((i=1; i<=${#configArray[@]}; i++)) do configNode=${configArray[i]} #刪除node,保留右邊字符 configIpNum=${configNode#*node} config_numbers=$config_numbers"{_id : $ccount, host : '${ipArray[$configIpNum]}:$config_port'}," ccount=`expr $ccount + 1` done #刪除最後一個,保留左邊字符 echo "********************************設置config副本集********************************" config_numbers=${config_numbers%,*} echo $config_numbers $mongodb_home/bin/mongo $configMasterIp:$config_port/admin << EOF config = {_id : "configs", members : [ $config_numbers ] }; rs.initiate(config); EOF #自動化設置分片shard副本集 for node in ${nodeArray[*]} do if [[ $node =~ "shard" ]] then shard_numbers="" shardMasterIp="" scount=0 shard_port=`awk -F= -v k=$node"_port" '{ if ( $1 == k ) print $2; }' $templatePath` shards=`awk -F= -v k=$node '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $shards | awk '{split($0, shardArr, ","); for(i in shardArr) print "shardArray["i"]="shardArr[i]}') for((i=1; i<=${#shardArray[@]}; i++)) do shardNode=${shardArray[i]} #echo "shardNode: $shardNode" #刪除node,保留右邊字符 shardIpNum=${shardNode#*node} #echo "shardIpNum: $shardIpNum" if [[ $scount = 0 ]] then shardMasterIp=${ipArray[$shardIpNum]} shard_numbers=$shard_numbers"{_id : $scount, host : '${ipArray[$shardIpNum]}:$shard_port', priority : 2}," fi if [[ $scount = 1 ]] then shard_numbers=$shard_numbers"{_id : $scount, host : '${ipArray[$shardIpNum]}:$shard_port', priority : 1}," fi if [[ $scount = 2 ]] then shard_numbers=$shard_numbers"{_id : $scount, host : '${ipArray[$shardIpNum]}:$shard_port', arbiterOnly : true}" fi scount=`expr $scount + 1` done #echo "shard_numbers: $shard_numbers" echo "********************************設置$node副本集********************************" $mongodb_home/bin/mongo $shardMasterIp:$shard_port/admin << EOF config = {_id : "$node", members : [ $shard_numbers ] }; rs.initiate(config); EOF fi done fi if [[ $param = "mongos" ]] then #自動化設置路由分片 mongos_numbers="" mongos_port=`awk -F= -v k=mongos_port '{ if ( $1 == k ) print $2; }' $templatePath` mongoss=`awk -F= -v k=mongos '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $mongoss | awk '{split($0, mongosArr, ","); for(i in mongosArr) print "mongosArray["i"]="mongosArr[i]}') mongosNode=${mongosArray[1]} #刪除node,保留右邊字符 mongosIpNum=${mongosNode#*node} mongosIp=${ipArray[$mongosIpNum]} for node in ${nodeArray[*]} do if [[ $node =~ "shard" ]] then shard_numbers="" shard_port=`awk -F= -v k=$node"_port" '{ if ( $1 == k ) print $2; }' $templatePath` shards=`awk -F= -v k=$node '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $shards | awk '{split($0, shardArr, ","); for(i in shardArr) print "shardArray["i"]="shardArr[i]}') for((i=1; i<=${#shardArray[@]}; i++)) do shardNode=${shardArray[i]} #刪除node,保留右邊字符 shardIpNum=${shardNode#*node} shardIp=${ipArray[$shardIpNum]} shard_numbers=$shard_numbers$shardIp:$shard_port"," done shard_numbers=${shard_numbers%,*} echo "mongos add $node shard_numbers:"$shard_numbers echo "********************************添加mongos分片$node********************************" $mongodb_home/bin/mongo $mongosIp:$mongos_port/admin << EOF sh.addShard("$node/$shard_numbers"); EOF fi done $mongodb_home/bin/mongo $mongosIp:$mongos_port/admin << EOF sh.status(); EOF fi
上述代碼實現的config、shards以及mongos配置內容以下:
config副本集配置,參考示例:
> config = {
... _id : "configs",
... members : [
... {_id : 0, host : "192.168.187.203:29040"},
... {_id : 1, host : "192.168.187.204:29040"},
... {_id : 2, host : "192.168.187.205:29040"}
... ]
... }
> rs.initiate(config)
shards副本集配置,參考示例:
> config = {
... _id : "shard1",
... members : [
... {_id : 0, host : "192.168.187.201:29060",priority:2},
... {_id : 1, host : "192.168.187.202:29060",priority:1},
... {_id : 2, host : "192.168.187.203:29060", arbiterOnly : true}
... ]
... }
> rs.initiate(config)
這裏默認,第一個ip的優先級爲2,第二個ip的優先級爲1,第三個ip爲仲裁節點,因此第一個ip爲primary主節點,第二位secondary從節點,第三個只參與投票,shard1根據分片的數據變化爲shard二、shard三、shard四、shard5等等。
mongos配置內容爲添加shards,參考示例爲:
sh.addShard("shard1/192.168.187.201:29060,192.168.187.202:29060,192.168.187.203:29060")
sh.addShard("shard2/192.168.187.202:29070,192.168.187.203:29070,192.168.187.204:29070")
sh.addShard("shard3/192.168.187.203:29080,192.168.187.204:29080,192.168.187.205:29080")
sh.addShard("shard4/192.168.187.204:29090,192.168.187.205:29090,192.168.187.201:29090")
sh.addShard("shard5/192.168.187.205:29100,192.168.187.201:29100,192.168.187.202:29100")
sh.status() #參考分片的狀況是否添加上
⑤autoClusterShardedAndInitDB.sh 腳本爲指定數據庫testDB一、testDB二、testDB3及其數據庫表分片。
#! /bin/bash configPath=config.properties mongodb_home=`awk -F= -v k=mongodbHome '{ if ( $1 == k ) print $2; }' $configPath` ips=`awk -F= -v k=ips '{ if ( $1 == k ) print $2; }' $configPath` template=`awk -F= -v k=template '{ if ( $1 == k ) print $2; }' $configPath` templatePath=template/$template.conf mongoss=`awk -F= -v k=mongos '{ if ( $1 == k ) print $2; }' $templatePath` testDB1=`awk -F= -v k=testDB1 '{ if ( $1 == k ) print $2; }' $configPath` testDB2=`awk -F= -v k=testDB2 '{ if ( $1 == k ) print $2; }' $configPath` testDB3=`awk -F= -v k=testDB3 '{ if ( $1 == k ) print $2; }' $configPath` eval $(echo $ips | awk '{split($0, arr, ","); for(i in arr) print "ipArray["i"]="arr[i]}') eval $(echo $mongoss | awk '{split($0, mongosArr, ","); for(i in mongosArr) print "mongosArray["i"]="mongosArr[i]}') for mongosNode in ${mongosArray[*]} do #刪除node,保留右邊字符 mongosIpNum=${mongosNode#*node} mongosIp=${ipArray[$mongosIpNum]} #指定數據庫分片生效 $mongodb_home/bin/mongo $mongosIp:29050/admin << EOF db.runCommand({enablesharding : "$testDB1"}); db.runCommand({shardcollection : "$testDB1.account", key : {_id : "hashed"}}); db.runCommand({shardcollection : "$testDB1.alarm", key : {_id : "hashed"}}); db.runCommand({shardcollection : "$testDB1.blackClass", key : {_id : "hashed"}}); db.runCommand({enablesharding : "$testDB2"}); db.runCommand({shardcollection : "$testDB2.device", key : {_id : "hashed"}}); db.runCommand({shardcollection : "$testDB2.deviceParam", key : {_id : "hashed"}}); db.runCommand({enablesharding : "$testDB3"}); db.runCommand({shardcollection : "$testDB3.blackImsiFace", key : {_id : "hashed"}}); db.runCommand({shardcollection : "$testDB3.face", key : {_id : "hashed"}}); EOF done mongosInitNode=${mongosArray[1]} #刪除node,保留右邊字符 mongosInitIpNum=${mongosInitNode#*node} mongosInitIp=${ipArray[$mongosInitIpNum]} port=`awk -F= -v k=mongos_port '{ if ( $1 == k ) print $2; }' $templatePath` #初始化數據庫 initDBPath=$mongodb_home/shell #解壓initConfig.zip包 -o:不提示的狀況下覆蓋文件,-d /opt:指明將文件解壓縮到/opt目錄 unzip -o -d $initDBPath/initConfig $initDBPath/initConfig.zip $mongodb_home/bin/mongorestore --host $mongosInitIp:$port --authenticationDatabase admin -d $testDB1 $initDBPath/initConfig/testDB1 $mongodb_home/bin/mongorestore --host $mongosInitIp:$port --authenticationDatabase admin -d $testDB2 $initDBPath/initConfig/testDB2 $mongodb_home/bin/mongorestore --host $mongosInitIp:$port --authenticationDatabase admin -d $testDB3 $initDBPath/initConfig/testDB3
⑥autoClusterStartUp.sh 腳本爲config和shard副本集、mongos路由啓動腳本,用法:./ autoClusterStartUp.sh,所有啓動,包括config和shard副本集、mongos路由;./ autoClusterStartUp.sh notmongos,只啓動config和shard副本集。
#! /bin/bash configPath=config.properties #從config.properties文件讀取數據出來 clusterPath=`awk -F= -v k=clusterPath '{ if ( $1 == k ) print $2; }' $configPath` mongodb_home=`awk -F= -v k=mongodbHome '{ if ( $1 == k ) print $2; }' $configPath` template=`awk -F= -v k=template '{ if ( $1 == k ) print $2; }' $configPath` ips=`awk -F= -v k=ips '{ if ( $1 == k ) print $2; }' $configPath` user=`awk -F= -v k=user '{ if ( $1 == k ) print $2; }' $configPath` localIp=`/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6 | awk '{print $2}' | tr -d "addr:"` templatePath=template/$template.conf #獲取全部的配置、路由、分片 nodes=`awk -F= -v k=nodes '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $nodes | awk '{split($0, nodeArr, ","); for(i in nodeArr) print "nodeArray["i"]="nodeArr[i]}') #ips切割爲數組array eval $(echo $ips | awk '{split($0, arr, ","); for(i in arr) print "ipArray["i"]="arr[i]}') suffix="" #判斷linux系統cpu是否爲numa架構 numaCount=`grep -i numa /var/log/dmesg | wc -l` nodeCount=`grep -i numa /var/log/dmesg | grep -wi "node" | wc -l` offCount=`grep -i numa /var/log/dmesg | grep -wi "numa=off" | wc -l` if [[ ($numaCount -gt 1) && ($nodeCount -gt 1) && ($offCount -eq 0) ]] then suffix="_numa" fi #輪詢啓動mongodb的節點,先config,後shard,再mongos for n in ${nodeArray[*]} do if [[ $n = "mongos" ]] then #傳參爲空,則mongos也執行,不爲空則不執行 if [[ $1 = "" ]] then mongoss=`awk -F= -v k=mongos '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $mongoss | awk '{split($0, mongosArr, ","); for(i in mongosArr) print "mongosArray["i"]="mongosArr[i]}') for((i=1; i<=${#mongosArray[@]}; i++)) do mongosNode=${mongosArray[i]} #刪除node,保留右邊字符 mongosIpNum=${mongosNode#*node} mongosIp=${ipArray[$mongosIpNum]} echo "********************************start $mongosIp $n********************************" #啓動本地服務或ssh遠程調用 if [[ $localIp = $mongosIp ]] then $mongodb_home/shell/autoStartUp.sh $n else ssh $user@$mongosIp "cd $mongodb_home/shell && ./autoStartUp.sh $n" fi done #mongos也啓動,即mongodb集羣全啓動 #查看/etc/crontab是否開啓定時任務,沒有則開啓 for((i=1; i<=${#ipArray[@]}; i++)) do if [[ $localIp = ${ipArray[i]} ]] then echo "*****start ${ipArray[i]} 定時任務:日誌切割和mongo進程存活監控*****" $mongodb_home/shell/autoConfig.sh addCronTask else echo "*****start ${ipArray[i]} 定時任務:日誌切割和mongo進程存活監控*****" ssh $user@${ipArray[i]} "cd $mongodb_home/shell && $mongodb_home/shell/autoConfig.sh addCronTask" fi done fi else shards=`awk -F= -v k=$n '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $shards | awk '{split($0, shardArr, ","); for(i in shardArr) print "shardArray["i"]="shardArr[i]}') for((i=1; i<=${#shardArray[@]}; i++)) do shardNode=${shardArray[i]} #刪除node,保留右邊字符 shardIpNum=${shardNode#*node} shardIp=${ipArray[$shardIpNum]} echo "********************************start $shardIp $n********************************" #啓動本地服務或ssh遠程調用 if [[ $localIp = $shardIp ]] then $mongodb_home/shell/autoStartUp.sh $n$suffix else ssh $user@$shardIp "cd $mongodb_home/shell && ./autoStartUp.sh $n$suffix" fi done fi done
⑦autoConfig.sh腳本爲建立所在服務器對應的mongodb的配置文件如config.conf、shard1.conf、mongos.conf,以及添加日誌切割和mongo進程存活監控定時任務,用法:./autoConfig.sh addCronTask。
#! /bin/bash #適用於3臺或5臺機器搭建15個或20個節點的mongodb高可用集羣,3個或5個分片,每一個分片(1主+1副+1仲裁)、3個配置、3個或2個路由 configPath=config.properties #從config.properties文件讀取數據出來 template=`awk -F= -v k=template '{ if ( $1 == k ) print $2; }' $configPath` clusterDataPath=`awk -F= -v k=clusterDataPath '{ if ( $1 == k ) print $2; }' $configPath` clusterLogPath=`awk -F= -v k=clusterLogPath '{ if ( $1 == k ) print $2; }' $configPath` mongodb_home=`awk -F= -v k=mongodbHome '{ if ( $1 == k ) print $2; }' $configPath` function createConfig() { ips=`awk -F= -v k=ips '{ if ( $1 == k ) print $2; }' $configPath` hostname=`hostname` localIp=`cat /etc/hosts | grep $hostname | awk -F " " '{print $1}'` #localIp=`/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6 | awk '{print $2}' | tr -d "addr:"` #配置mongodb_home #echo "配置mongodb_home" #editMongodbHome #建立conf文件夾 echo "********************************建立conf文件夾********************************" createConfFolders eval $(echo $ips | awk '{split($0, arr, ","); for(i in arr) print "ipArray["i"]="arr[i]}') templatePath=template/$template.conf #遍歷ip數組 for((i=1; i<=${#ipArray[@]}; i++)) do if [[ $localIp = ${ipArray[i]} ]] then nodes=`awk -F= -v k=node$i '{ if ( $1 == k ) print $2; }' $templatePath` echo "********************************${ipArray[i]}:$nodes********************************" eval $(echo $nodes | awk '{split($0, nodeArr, ","); for(i in nodeArr) print "nodeArray["i"]="nodeArr[i]}') for node in ${nodeArray[*]} do port=`awk -F= -v k=$node"_port" '{ if ( $1 == k ) print $2; }' $templatePath` case "$node" in "config") #建立config配置文件 echo "***************建立config配置文件***************" createConfigFoldersConf $port ;; "mongos") #建立mongos路由配置文件 echo "***************建立mongos路由配置文件***************" configPort=`awk -F= -v k=config_port '{ if ( $1 == k ) print $2; }' $templatePath` getConfigsIpsPort $configPort createMongosFoldersConf $port $ipsAndPorts ;; esac if [[ $node =~ "shard" ]] then #建立shard分片配置文件 echo "***************建立$node分片配置文件***************" createShardFoldersConf $node $port fi done fi done } function getConfigsIpsPort() { ipsAndPorts="" for((i=1; i<=${#ipArray[@]}; i++)) do nodes=`awk -F= -v k=node$i '{ if ( $1 == k ) print $2; }' $templatePath` if [[ $nodes =~ "config" ]] then ipsAndPorts=$ipsAndPorts"${ipArray[i]}:$1," fi done ipsAndPorts=${ipsAndPorts%,*} } #爲每臺機器建立6個目錄,shard一、shard二、shard三、config、mongos、conf function createConfFolders() { if [[ -d $clusterDataPath/conf ]] then rm -rf $clusterDataPath/conf fi #建立配置文件夾 mkdir -p $clusterDataPath/conf } function createMongosFoldersConf() { #建立mongos路由服務器的日誌文件夾log、進程文件夾pid mkdir -p $clusterLogPath/mongos/log mkdir -p $clusterDataPath/mongos/pid #設置路由服務器 cat >> $clusterDataPath/conf/mongos.conf << EOF systemLog: destination: file path: $clusterLogPath/mongos/log/mongos.log logAppend: true processManagement: fork: true pidFilePath: $clusterDataPath/mongos/pid/mongos.pid net: bindIp: $localIp port: $1 maxIncomingConnections: 20000 sharding: configDB: configs/$2 EOF } function createConfigFoldersConf() { #建立config配置服務器的數據文件夾data、日誌文件夾log、進程文件夾pid mkdir -p $clusterDataPath/config/data mkdir -p $clusterLogPath/config/journal mkdir -p $clusterLogPath/config/log mkdir -p $clusterDataPath/config/pid #把數據目錄的journal日誌映射到日誌目錄裏面 ln -s $clusterLogPath/config/journal $clusterDataPath/config/data/journal #設置配置服務器副本集 cat >> $clusterDataPath/conf/config.conf << EOF systemLog: destination: file path: $clusterLogPath/config/log/config.log logAppend: true processManagement: fork: true pidFilePath: $clusterDataPath/config/pid/config.pid net: bindIp: $localIp port: $1 maxIncomingConnections: 20000 storage: dbPath: $clusterDataPath/config/data journal: enabled: true commitIntervalMs: 500 directoryPerDB: true syncPeriodSecs: 300 engine: wiredTiger replication: oplogSizeMB: 10000 replSetName: configs sharding: clusterRole: configsvr EOF } function createShardFoldersConf() { #建立shard1分片服務器的數據文件夾data、日誌文件夾log、進程文件夾pid mkdir -p $clusterDataPath/$1/data mkdir -p $clusterLogPath/$1/journal mkdir -p $clusterLogPath/$1/log mkdir -p $clusterDataPath/$1/pid #把數據目錄的journal日誌映射到日誌目錄裏面 ln -s $clusterLogPath/$1/journal $clusterDataPath/$1/data/journal #設置第一個分片副本集 cat >> $clusterDataPath/conf/$1.conf << EOF systemLog: destination: file path: $clusterLogPath/$1/log/$1.log logAppend: true processManagement: fork: true pidFilePath: $clusterDataPath/$1/pid/$1.pid net: bindIp: $localIp port: $2 maxIncomingConnections: 20000 storage: dbPath: $clusterDataPath/$1/data journal: enabled: true commitIntervalMs: 500 directoryPerDB: true syncPeriodSecs: 300 engine: wiredTiger wiredTiger: engineConfig: cacheSizeGB: 103 statisticsLogDelaySecs: 0 journalCompressor: snappy directoryForIndexes: false collectionConfig: blockCompressor: snappy indexConfig: prefixCompression: true replication: oplogSizeMB: 10000 replSetName: $1 sharding: clusterRole: shardsvr EOF } #配置mongodb環境變量 function editMongodbHome() { profile=/etc/profile #配置MONGODB_HOME sed -i "/^export MONGODB_HOME/d" $profile echo "export MONGODB_HOME=$mongodb_home" >> $profile #配置PATH sed -i "/^export PATH=\$PATH:\$MONGODB_HOME\/bin/d" $profile echo "export PATH=\$PATH:\$MONGODB_HOME/bin" >> $profile #使/etc/profile文件生效 source /etc/profile } if [[ $1 = "" ]] then createConfig fi if [[ $1 = "addCronTask" ]] then cronfile=/etc/crontab shellPath=${mongodb_home//\//\\/}\\/shell logRotateCronTaskNum=`sed -n -e "/\0 \0 \* \* \* root $shellPath\/autoLogRotate.sh $shellPath/=" $cronfile` if [[ $logRotateCronTaskNum = "" ]] then #沒有則追加 echo "0 0 * * * root $mongodb_home/shell/autoLogRotate.sh $mongodb_home/shell > /dev/null 2>&1 &" >> $cronfile fi checkLiveCronTaskNum=`sed -n -e "/\*\/10 \* \* \* \* root $shellPath\/autoCheckLive.sh $shellPath/=" $cronfile` if [[ $checkLiveCronTaskNum = "" ]] then #沒有則追加 echo "*/10 * * * * root $mongodb_home/shell/autoCheckLive.sh $mongodb_home/shell > /dev/null 2>&1 &" >> $cronfile fi fi if [[ $1 = "removeCronTask" ]] then cronfile=/etc/crontab shellPath=${mongodb_home//\//\\/}\\/shell sed -i "/^\*\/10 \* \* \* \* root $shellPath\/autoCheckLive.sh $shellPath/d" $cronfile sed -i "/^\0 \0 \* \* \* root $shellPath\/autoLogRotate.sh $shellPath/d" $cronfile fi
⑧autoLogRotate.sh腳本爲建立所在服務器對應的mongodb的日誌切割腳本,並添加定時任務,默認爲天天凌晨0點0分,對mongodb數據庫日誌進行切割並刪除7天前的日誌。
#! /bin/bash #Rotate the MongoDB logs to prevent a single logfile from consuming too much disk space. shellPath=$1 configPath=$shellPath/config.properties #從config.properties文件讀取數據出來 clusterPath=`awk -F= -v k=clusterPath '{ if ( $1 == k ) print $2; }' $configPath` template=`awk -F= -v k=template '{ if ( $1 == k ) print $2; }' $configPath` templatePath=$shellPath/template/$template.conf ips=`awk -F= -v k=ips '{ if ( $1 == k ) print $2; }' $configPath` eval $(echo $ips | awk '{split($0, arr, ","); for(i in arr) print "ipArray["i"]="arr[i]}') localIp=`/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6 | awk '{print $2}' | tr -d "addr:"` #表明刪除7天前的備份,即只保留最近7天的備份 days=7 #遍歷ip數組 for((i=1; i<=${#ipArray[@]}; i++)) do if [[ $localIp = ${ipArray[i]} ]] then mongodbNodes=`awk -F= -v k=node$i '{ if ( $1 == k ) print $2; }' $templatePath` eval $(echo $mongodbNodes | awk '{split($0, mongodbArr, ","); for(i in mongodbArr) print "mongodbArray["i"]="mongodbArr[i]}') for((n=1; n<=${#mongodbArray[@]}; n++)) do pid=`cat $clusterPath/${mongodbArray[n]}/pid/${mongodbArray[n]}.pid` logdir=$clusterPath/${mongodbArray[n]}/log #切割日誌 /bin/kill -SIGUSR1 $pid find $logdir/ -mtime +$days -delete done fi done
⑨autoStartUp.sh腳本爲啓動該腳本所在服務器的config、shard、mongos節點,啓動分正常啓動和禁止numactl啓動兩種。
#! /bin/bash configPath=config.properties mongodb_home=`awk -F= -v k=mongodbHome '{ if ( $1 == k ) print $2; }' $configPath` clusterPath=`awk -F= -v k=clusterPath '{ if ( $1 == k ) print $2; }' $configPath` param=$1 #config if [[ $param = "config" ]] then $mongodb_home/bin/mongod -f $clusterPath/conf/config.conf fi if [[ $param = "config_numa" ]] then numactl --interleave=all $mongodb_home/bin/mongod -f $clusterPath/conf/config.conf fi #shard1 paramLegth=`echo "$param" |wc -L` if [[ ($param =~ "shard") && ($paramLegth -lt 11) && ($paramLegth -gt 5)]] then $mongodb_home/bin/mongod -f $clusterPath/conf/$param.conf fi if [[ ($param =~ "shard") && $param =~ "numa" ]] then shardName=${param%_*} numactl --interleave=all $mongodb_home/bin/mongod -f $clusterPath/conf/$shardName.conf fi #mongos if [[ $param = "mongos" ]] then $mongodb_home/bin/mongos -f $clusterPath/conf/mongos.conf fi
⑩autoSystemProperties.sh 腳本爲修改服務器的系統配置,TCP配置等。
#! /bin/bash #系統全局容許分配的最大文件句柄數 sysctl -w fs.file-max=2097152 sysctl -w fs.nr_open=2097152 echo 2097152 > /proc/sys/fs/nr_open #容許當前會話/進程打開文件句柄數 ulimit -n 1048576 #修改fs.file-max設置到 /etc/sysctl.conf 文件: #fs.file-max = 1048576 #修改/etc/security/limits.conf 持久化設置容許用戶/進程打開文件句柄數,手動添加 #* soft nofile 1048576 #* hard nofile 1048576 #* soft nproc 524288 #* hard nproc 524288 #併發鏈接backlog設置 sysctl -w net.core.somaxconn=32768 sysctl -w net.ipv4.tcp_max_syn_backlog=16384 sysctl -w net.core.netdev_max_backlog=16384 #可用知名端口範圍: sysctl -w net.ipv4.ip_local_port_range='80 65535' #sysctl -w net.core.rmem_default=262144 sysctl -w net.core.wmem_default=262144 sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216 sysctl -w net.core.optmem_max=16777216 #TCP Socket 讀寫 Buffer 設置: sysctl -w net.core.rmem_default=262144 sysctl -w net.core.wmem_default=262144 sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216 sysctl -w net.core.optmem_max=16777216 sysctl -w net.ipv4.tcp_rmem='1024 4096 16777216' sysctl -w net.ipv4.tcp_wmem='1024 4096 16777216' #修改系統內核參數: echo "never" > /sys/kernel/mm/transparent_hugepage/enabled echo "never" > /sys/kernel/mm/transparent_hugepage/defrag #TCP 鏈接追蹤設置: sysctl -w net.nf_conntrack_max=1000000 sysctl -w net.netfilter.nf_conntrack_max=1000000 sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30 #修改stack size爲1024,不啓做用,只能手動source #if [ `grep -c "ulimit -s 1024" /etc/profile` -eq '0' ];then # echo "ulimit -s 1024" >> /etc/profile # source /etc/profile #fi
⑫initConfig.zip 爲數據庫test及其數據庫表的初始化文件,有須要的能夠自行配置。
1. 把mongodb安裝目錄放置在5臺機器中的任意一臺上面便可,選定好安裝的路徑。
2.進入mongodb/shell目錄。
根據實際環境須要修改config.properties文件的clusterPath、mongodbHome、ips、template、數據庫5個屬性
3.autoClusterIndex.sh腳本所在的目錄執行以下命令,不然腳本會不起做用,執行命令:
./autoClusterIndex.sh
執行上面命令,會一鍵自動化部署。
至此,完成部署。
5臺機器的驗證:
服務器node1(192.168.187.201):
服務器node2(192.168.187.202):
服務器node3(192.168.187.203):
服務器node4(192.168.187.204):
服務器node5(192.168.187.205):
4臺機器的驗證:
服務器node1(192.168.187.201):
服務器node2(192.168.187.202):
服務器node3(192.168.187.203):
服務器node4(192.168.187.204):
3臺機器的驗證:
服務器node1(192.168.187.201)、服務器node2(192.168.187.202)、服務器node3(192.168.187.203)都會啓動以下服務:
一鍵化部署的shell命令github地址爲:
https://github.com/SwordfallYeung/MongodbCluster_autoDeploy