搭建高可用mongodb集羣（二）—— 副本集

時間 2019-11-14

原文原文鏈接

在上一篇文章《搭建高可用MongoDB集羣（一）——配置MongoDB》提到了幾個問題尚未解決。html

主節點掛了可否自動切換鏈接？目前須要手工切換。
主節點的讀寫壓力過大如何解決？
從節點每一個上面的數據都是對數據庫全量拷貝，從節點壓力會不會過大？
數據壓力大到機器支撐不了的時候可否作到自動擴展？

這篇文章看完這些問題就能夠搞定了。NoSQL的產生就是爲了解決大數據量、高擴展性、高性能、靈活數據模型、高可用性。可是光經過主從模式的架構遠遠達不到上面幾點，由此MongoDB設計了副本集和分片的功能。這篇文章主要介紹副本集：java

mongoDB官方已經不建議使用主從模式了，替代方案是採用副本集的模式，點擊查看，如圖：

那什麼是副本集呢？打魔獸世界總說打副本，其實這兩個概念差很少一個意思。遊戲裏的副本是指玩家集中在高峯時間去一個場景打怪，會出現玩家暴多怪物少的狀況，遊戲開發商爲了保證玩家的體驗度，就爲每一批玩家單獨開放一個一樣的空間一樣的數量的怪物，這一個複製的場景就是一個副本，無論有多少個玩家各自在各自的副本里玩不會互相影響。 mongoDB的副本也是這個，主從模式其實就是一個單副本的應用，沒有很好的擴展性和容錯性。而副本集具備多個副本保證了容錯性，就算一個副本掛掉了還有不少副本存在，而且解決了上面第一個問題「主節點掛掉了，整個集羣內會自動切換」。難怪mongoDB官方推薦使用這種模式。咱們來看看mongoDB副本集的架構圖：linux

由圖能夠看到客戶端鏈接到整個副本集，不關心具體哪一臺機器是否掛掉。主服務器負責整個副本集的讀寫，副本集按期同步數據備份，一但主節點掛掉，副本節點就會選舉一個新的主服務器，這一切對於應用服務器不須要關心。咱們看一下主服務器掛掉後的架構：mongodb

副本集中的副本節點在主節點掛掉後經過心跳機制檢測到後，就會在集羣內發起主節點的選舉機制，自動選舉一位新的主服務器。看起來很牛X的樣子，咱們趕忙操做部署一下！
官方推薦的副本集機器數量爲至少3個，那咱們也按照這個數量配置測試。shell

一、準備兩臺機器 192.168.1.13六、192.168.1.13七、192.168.1.138。 192.168.1.136 看成副本集主節點，192.168.1.13七、192.168.1.138做爲副本集副本節點。數據庫

二、分別在每臺機器上創建mongodb副本集測試文件夾bash

1 2 3 4 5 6 7 8	#存放整個mongodb文件 mkdir -p /data /mongodbtest /replset #存放mongodb數據文件 mkdir -p /data /mongodbtest /replset /data #進入mongodb文件夾 cd /data /mongodbtest

三、下載mongodb的安裝程序包服務器

1	wget http: //fastdl.mongodb.org /linux /mongodb-linux-x86_64-2.4.8.tgz

注意linux生產環境不能安裝32位的mongodb，由於32位受限於操做系統最大2G的文件限制。網絡

1 2	#解壓下載的壓縮包 tar xvzf mongodb-linux-x86_64-2.4.8.tgz

四、分別在每臺機器上啓動mongodb架構

/data/mongodbtest/mongodb-linux-x86_64-2.4.8/bin/mongod –dbpath /data/mongodbtest/replset/data –replSet repset

能夠看到控制檯上顯示副本集尚未配置初始化信息。

Sun Dec 29 20:12:02.953 [rsStart] replSet can’t get local.system.replset config from self or any seed (EMPTYCONFIG)
Sun Dec 29 20:12:02.953 [rsStart] replSet info you may need to run replSetInitiate — rs.initiate() in the shell — if that is not already done

五、初始化副本集

在三臺機器上任意一臺機器登錄mongodb

1 2 3 4	/data /mongodbtest /mongodb-linux-x86_64-2.4.8 /bin /mongo #使用admin數據庫 use admin

#定義副本集配置變量，這裏的 _id:」repset」和上面命令參數「 –replSet repset」要保持同樣。
config = { _id:」repset「, members:[
... {_id:0,host:"192.168.1.136:27017"},
... {_id:1,host:"192.168.1.137:27017"},
... {_id:2,host:"192.168.1.138:27017"}]
… }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18	#輸出 { "_id" : "repset", "members" : [ { "_id" : 0, "host" : "192.168.1.136:27017" }, { "_id" : 1, "host" : "192.168.1.137:27017" }, { "_id" : 2, "host" : "192.168.1.138:27017" } ] }

1 2 3 4 5 6 7 8	#初始化副本集配置 rs.initiate (config ); #輸出成功 { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 }

#查看日誌，副本集啓動成功後，138爲主節點PRIMARY，13六、137爲副本節點SECONDARY。

Sun Dec 29 20:26:13.842 [conn3] replSet replSetInitiate admin command received from client

Sun Dec 29 20:26:13.842 [conn3] replSet replSetInitiate config object parses ok, 3 members specified
Sun Dec 29 20:26:13.847 [conn3] replSet replSetInitiate all members seem up
Sun Dec 29 20:26:13.848 [conn3] ******
Sun Dec 29 20:26:13.848 [conn3] creating replication oplog of size: 990MB...
Sun Dec 29 20:26:13.849 [FileAllocator] allocating new datafile /data/mongodbtest/replset/data/local.1, filling with zeroes...
Sun Dec 29 20:26:13.862 [FileAllocator] done allocating datafile /data/mongodbtest/replset/data/local.1, size: 1024MB, took 0.012 secs
Sun Dec 29 20:26:13.863 [conn3] ******
Sun Dec 29 20:26:13.863 [conn3] replSet info saving a newer config version to local.system.replset
Sun Dec 29 20:26:13.864 [conn3] replSet saveConfigLocally done
Sun Dec 29 20:26:13.864 [conn3] replSet replSetInitiate config now saved locally. Should come online in about a minute.
Sun Dec 29 20:26:23.047 [rsStart] replSet I am 192.168.1.138:27017
Sun Dec 29 20:26:23.048 [rsStart] replSet STARTUP2
Sun Dec 29 20:26:23.049 [rsHealthPoll] replSet member 192.168.1.137:27017 is up
Sun Dec 29 20:26:23.049 [rsHealthPoll] replSet member 192.168.1.136:27017 is up
Sun Dec 29 20:26:24.051 [rsSync] replSet SECONDARY
Sun Dec 29 20:26:25.053 [rsHealthPoll] replset info 192.168.1.136:27017 thinks that we are down
Sun Dec 29 20:26:25.053 [rsHealthPoll] replSet member 192.168.1.136:27017 is now in state STARTUP2
Sun Dec 29 20:26:25.056 [rsMgr] not electing self, 192.168.1.136:27017 would veto with 'I don't think 192.168.1.138:27017 is electable'
Sun Dec 29 20:26:31.059 [rsHealthPoll] replset info 192.168.1.137:27017 thinks that we are down
Sun Dec 29 20:26:31.059 [rsHealthPoll] replSet member 192.168.1.137:27017 is now in state STARTUP2
Sun Dec 29 20:26:31.062 [rsMgr] not electing self, 192.168.1.137:27017 would veto with 'I don't think 192.168.1.138:27017 is electable'
Sun Dec 29 20:26:37.074 [rsMgr] replSet info electSelf 2
Sun Dec 29 20:26:38.062 [rsMgr] replSet PRIMARY
Sun Dec 29 20:26:39.071 [rsHealthPoll] replSet member 192.168.1.137:27017 is now in state RECOVERING
Sun Dec 29 20:26:39.075 [rsHealthPoll] replSet member 192.168.1.136:27017 is now in state RECOVERING
Sun Dec 29 20:26:42.201 [slaveTracking] build index local.slaves { _id: 1 }
Sun Dec 29 20:26:42.207 [slaveTracking] build index done. scanned 0 total records. 0.005 secs
Sun Dec 29 20:26:43.079 [rsHealthPoll] replSet member 192.168.1.136:27017 is now in state SECONDARY
Sun Dec 29 20:26:49.080 [rsHealthPoll] replSet member 192.168.1.137:27017 is now in state SECONDARY

#查看集羣節點的狀態
rs.status();

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48	#輸出 { "set" : "repset", "date" : ISODate ( "2013-12-29T12:54:25Z" ), "myState" : 1, "members" : [ { "_id" : 0, "name" : "192.168.1.136:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1682, "optime" : Timestamp ( 1388319973, 1 ), "optimeDate" : ISODate ( "2013-12-29T12:26:13Z" ), "lastHeartbeat" : ISODate ( "2013-12-29T12:54:25Z" ), "lastHeartbeatRecv" : ISODate ( "2013-12-29T12:54:24Z" ), "pingMs" : 1, "syncingTo" : "192.168.1.138:27017" }, { "_id" : 1, "name" : "192.168.1.137:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1682, "optime" : Timestamp ( 1388319973, 1 ), "optimeDate" : ISODate ( "2013-12-29T12:26:13Z" ), "lastHeartbeat" : ISODate ( "2013-12-29T12:54:25Z" ), "lastHeartbeatRecv" : ISODate ( "2013-12-29T12:54:24Z" ), "pingMs" : 1, "syncingTo" : "192.168.1.138:27017" }, { "_id" : 2, "name" : "192.168.1.138:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2543, "optime" : Timestamp ( 1388319973, 1 ), "optimeDate" : ISODate ( "2013-12-29T12:26:13Z" ), "self" : true } ], "ok" : 1 }

整個副本集已經搭建成功了。

六、測試副本集數據複製功能

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24	在主節點192.168.1.138 上鍊接到終端： mongo 127.0.0.1 創建 test 數據庫。 use test; 往testdb表插入數據。 > db.testdb.insert ( { "test1": "testval1" } ) 在副本節點 192.168.1.13六、192.168.1.137 上鍊接到mongodb查看數據是否複製過來。 /data /mongodbtest /mongodb-linux-x86_64-2.4.8 /bin /mongo 192.168.1.136: 27017 #使用test 數據庫。 repset:SECONDARY > use test; repset:SECONDARY > show tables; Sun Dec 29 21: 50: 48.590 error: { "$err" : "not master and slaveOk=false", "code" : 13435 } at src /mongo /shell /query.js: 128 #mongodb默認是從主節點讀寫數據的，副本節點上不容許讀，須要設置副本節點能夠讀。 repset:SECONDARY > db.getMongo ( ).setSlaveOk ( ); #能夠看到數據已經複製到了副本集。 repset:SECONDARY > db.testdb.find ( ); { "_id" : ObjectId ( "52c028460c7505626a93944f" ), "test1" : "testval1" }

七、測試副本集故障轉移功能

先停掉主節點mongodb 138，查看13六、137的日誌能夠看到通過一系列的投票選擇操做，137 當選主節點，136從137同步數據過來。

1 2 3 4 5 6 7 8 9 10 11 12 13 14	Sun Dec 29 22:03: 05.351 [rsBackgroundSync ] replSet sync source problem: 10278 dbclient error communicating with server: 192.168.1.138: 27017 Sun Dec 29 22:03: 05.354 [rsBackgroundSync ] replSet syncing to: 192.168.1.138: 27017 Sun Dec 29 22:03: 05.356 [rsBackgroundSync ] repl: couldn 't connect to server 192.168.1.138:27017 Sun Dec 29 22:03:05.356 [rsBackgroundSync] replSet not trying to sync from 192.168.1.138:27017, it is vetoed for 10 more seconds Sun Dec 29 22:03:05.499 [rsHealthPoll] DBClientCursor::init call() failed Sun Dec 29 22:03:05.499 [rsHealthPoll] replset info 192.168.1.138:27017 heartbeat failed, retrying Sun Dec 29 22:03:05.501 [rsHealthPoll] replSet info 192.168.1.138:27017 is down (or slow to respond): Sun Dec 29 22:03:05.501 [rsHealthPoll] replSet member 192.168.1.138:27017 is now in state DOWN Sun Dec 29 22:03:05.511 [rsMgr] not electing self, 192.168.1.137:27017 would veto with '192.168.1.136: 27017 is trying to elect itself but 192.168.1.138: 27017 is already primary and more up-to-date ' Sun Dec 29 22:03:07.330 [conn393] replSet info voting yea for 192.168.1.137:27017 (1) Sun Dec 29 22:03:07.503 [rsHealthPoll] replset info 192.168.1.138:27017 heartbeat failed, retrying Sun Dec 29 22:03:08.462 [rsHealthPoll] replSet member 192.168.1.137:27017 is now in state PRIMARY Sun Dec 29 22:03:09.359 [rsBackgroundSync] replSet syncing to: 192.168.1.137:27017 Sun Dec 29 22:03:09.507 [rsHealthPoll] replset info 192.168.1.138:27017 heartbeat failed, retrying

查看整個集羣的狀態，能夠看到138爲狀態不可達。

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52	/data /mongodbtest /mongodb-linux-x86_64-2.4.8 /bin /mongo 192.168.1.136: 27017 repset:SECONDARY > rs.status ( ); #輸出 { "set" : "repset", "date" : ISODate ( "2013-12-29T14:28:35Z" ), "myState" : 2, "syncingTo" : "192.168.1.137:27017", "members" : [ { "_id" : 0, "name" : "192.168.1.136:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9072, "optime" : Timestamp ( 1388324934, 1 ), "optimeDate" : ISODate ( "2013-12-29T13:48:54Z" ), "self" : true }, { "_id" : 1, "name" : "192.168.1.137:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 7329, "optime" : Timestamp ( 1388324934, 1 ), "optimeDate" : ISODate ( "2013-12-29T13:48:54Z" ), "lastHeartbeat" : ISODate ( "2013-12-29T14:28:34Z" ), "lastHeartbeatRecv" : ISODate ( "2013-12-29T14:28:34Z" ), "pingMs" : 1, "syncingTo" : "192.168.1.138:27017" }, { "_id" : 2, "name" : "192.168.1.138:27017", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : Timestamp ( 1388324934, 1 ), "optimeDate" : ISODate ( "2013-12-29T13:48:54Z" ), "lastHeartbeat" : ISODate ( "2013-12-29T14:28:35Z" ), "lastHeartbeatRecv" : ISODate ( "2013-12-29T14:28:23Z" ), "pingMs" : 0, "syncingTo" : "192.168.1.137:27017" } ], "ok" : 1 }

再啓動原來的主節點 138，發現138 變爲 SECONDARY，仍是137 爲主節點 PRIMARY。

1 2 3 4 5 6 7 8	Sun Dec 29 22: 21: 06.619 [rsStart ] replSet I am 192.168.1.138: 27017 Sun Dec 29 22: 21: 06.619 [rsStart ] replSet STARTUP2 Sun Dec 29 22: 21: 06.627 [rsHealthPoll ] replset info 192.168.1.136: 27017 thinks that we are down Sun Dec 29 22: 21: 06.627 [rsHealthPoll ] replSet member 192.168.1.136: 27017 is up Sun Dec 29 22: 21: 06.627 [rsHealthPoll ] replSet member 192.168.1.136: 27017 is now in state SECONDARY Sun Dec 29 22: 21: 07.628 [rsSync ] replSet SECONDARY Sun Dec 29 22: 21: 08.623 [rsHealthPoll ] replSet member 192.168.1.137: 27017 is up Sun Dec 29 22: 21: 08.624 [rsHealthPoll ] replSet member 192.168.1.137: 27017 is now in state PRIMARY

八、java程序鏈接副本集測試。三個節點有一個節點掛掉也不會影響應用程序客戶端對整個副本集的讀寫！

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37	public class TestMongoDBReplSet { public static void main ( String [ ] args ) { try { List <ServerAddress > addresses = new ArrayList <ServerAddress > ( ) ; ServerAddress address1 = new ServerAddress ( "192.168.1.136" , 27017 ) ; ServerAddress address2 = new ServerAddress ( "192.168.1.137" , 27017 ) ; ServerAddress address3 = new ServerAddress ( "192.168.1.138" , 27017 ) ; addresses. add (address1 ) ; addresses. add (address2 ) ; addresses. add (address3 ) ; MongoClient client = new MongoClient (addresses ) ; DB db = client. getDB ( "test" ) ; DBCollection coll = db. getCollection ( "testdb" ) ; // 插入 BasicDBObject object = new BasicDBObject ( ) ; object. append ( "test2", "testval2" ) ; coll. insert (object ) ; DBCursor dbCursor = coll. find ( ) ; while (dbCursor. hasNext ( ) ) { DBObject dbObject = dbCursor. next ( ) ; System. out. println (dbObject. toString ( ) ) ; } } catch ( Exception e ) { e. printStackTrace ( ) ; } } }

目前看起來支持完美的故障轉移了，這個架構是否是比較完美了？其實還有不少地方能夠優化，好比開頭的第二個問題：主節點的讀寫壓力過大如何解決？常見的解決方案是讀寫分離，mongodb副本集的讀寫分離如何作呢？

看圖說話：

常規寫操做來講並無讀操做多，因此一臺主節點負責寫，兩臺副本節點負責讀。

一、設置讀寫分離須要先在副本節點SECONDARY 設置 setSlaveOk。
二、在程序中設置副本節點負責讀操做，以下代碼：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33	public class TestMongoDBReplSetReadSplit { public static void main ( String [ ] args ) { try { List <ServerAddress > addresses = new ArrayList <ServerAddress > ( ) ; ServerAddress address1 = new ServerAddress ( "192.168.1.136" , 27017 ) ; ServerAddress address2 = new ServerAddress ( "192.168.1.137" , 27017 ) ; ServerAddress address3 = new ServerAddress ( "192.168.1.138" , 27017 ) ; addresses. add (address1 ) ; addresses. add (address2 ) ; addresses. add (address3 ) ; MongoClient client = new MongoClient (addresses ) ; DB db = client. getDB ( "test" ) ; DBCollection coll = db. getCollection ( "testdb" ) ; BasicDBObject object = new BasicDBObject ( ) ; object. append ( "test2" , "testval2" ) ; //讀操做從副本節點讀取 ReadPreference preference = ReadPreference. secondary ( ) ; DBObject dbObject = coll. findOne (object, null , preference ) ; System. out . println (dbObject ) ; } catch ( Exception e ) { e. printStackTrace ( ) ; } } }

讀參數除了secondary一共還有五個參數：primary、primaryPreferred、secondary、secondaryPreferred、nearest。

primary:默認參數，只從主節點上進行讀取操做；
primaryPreferred:大部分從主節點上讀取數據,只有主節點不可用時從secondary節點讀取數據。
secondary:只從secondary節點上進行讀取操做，存在的問題是secondary節點的數據會比primary節點數據「舊」。
secondaryPreferred:優先從secondary節點進行讀取操做，secondary節點不可用時從主節點讀取數據；
nearest:不論是主節點、secondary節點，從網絡延遲最低的節點上讀取數據。

好，讀寫分離作好咱們能夠數據分流，減輕壓力解決了「主節點的讀寫壓力過大如何解決？」這個問題。不過當咱們的副本節點增多時，主節點的複製壓力會加大有什麼辦法解決嗎？mongodb早就有了相應的解決方案。

看圖：

其中的仲裁節點不存儲數據，只是負責故障轉移的羣體投票，這樣就少了數據複製的壓力。是否是想得很周到啊，一看mongodb的開發兄弟熟知大數據架構體系，其實不僅是主節點、副本節點、仲裁節點，還有Secondary-Only、Hidden、Delayed、Non-Voting。

Secondary-Only:不能成爲primary節點，只能做爲secondary副本節點，防止一些性能不高的節點成爲主節點。
Hidden:這類節點是不可以被客戶端制定IP引用，也不能被設置爲主節點，可是能夠投票，通常用於備份數據。
Delayed：能夠指定一個時間延遲從primary節點同步數據。主要用於備份數據，若是實時同步，誤刪除數據立刻同步到從節點，恢復又恢復不了。
Non-Voting：沒有選舉權的secondary節點，純粹的備份數據節點。

到此整個mongodb副本集搞定了兩個問題：