部署ActiveMQ的Share File System Master-Slave

時間 2019-11-24

標籤部署 activemq share file system master slave 欄目 ActiveMQ 简体版

原文原文鏈接

以前在項目裏用MQ是用單節點，由於業務量不大沒有主從。這樣風險很大，會有單點問題。新項目起來了，須要一個高可用的MQ，故研究了下AMQ的幾種master-slave方式：html

一、Pure Master-Slavejava

5.8版本以後已經不支持了，主要緣由是Master掛了以後再起來沒法自動融入集羣，須要停掉MQ從新啓動才能夠。這裏略過。web

http://activemq.apache.org/pure-master-slave.htmlspring

二、Share File System Master-Slave數據庫

經過共享文件系統實現主從節點，解決單點問題。能夠理解爲將數據放在第三方。apache

http://activemq.apache.org/shared-file-system-master-slave.htmlapi

三、JDBC Master-Slave安全

數據放在庫裏，嚴重依賴數據庫的性能，要是數據庫掛了更麻煩。服務器

http://activemq.apache.org/jdbc-master-slave.htmlsocket

三、zookeeper Master-Slave

用過Hadoop的都知道zookeeper是一個強大的調度工具。它能夠自動選擇可用節點來實現Master-Slave，但太費勁，感興趣能夠戳下面連接

http://activemq.apache.org/replicated-leveldb-store.html

咱們準備採用第二種Share File System Master-Slave，簡單粗暴，快捷有效。

在192.168.3.31上下載安裝AMQ就不用說了，注意機器名hostname裏面不能有下劃線，一開始我被坑死了，死活啓動不了。。。

在192.168.3.31上安裝NFS：

yum install nfs-utils nfs4-acl-tools portmap

其中portmap是NFS依賴的服務，必須確認先安裝：（記住名字rpcbind）

[root@bxcsweb data]# yum install portmap
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.btte.net
 * extras: mirrors.btte.net
 * updates: mirrors.btte.net
Setting up Install Process
Package rpcbind-0.2.0-11.el6.x86_64 already installed and latest version
Nothing to do

直接啓動NFS會報錯：

[root@bxcsweb data]# service nfs restart
關閉 NFS 守護進程：                                        [失敗]
關閉 NFS mountd：                                          [失敗]
Shutting down RPC idmapd:                                  [失敗]
啓動 NFS 服務：                                            [肯定]
啓動 NFS mountd：                                          [失敗]
啓動 NFS 守護進程：rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
rpc.nfsd: unable to set any sockets for nfsd              [失敗]

先啓動依賴，再啓動NFS:

service rpcbind restart   
service nfs restart

而後在被掛載的主節點機器上設置共享哪些目錄、以及共享策略(從節點不能設置,不然會有問題)：（編輯/etc/exports文件）

這個文件默認是空的，編輯的格式：

[共享的目錄] [主機名1或IP1(參數1,參數2)] [主機名2或IP2(參數3,參數4)]
 下面是一些NFS共享的經常使用參數： 
 
 
        ro                      只讀訪問 
        rw                      讀寫訪問 
        sync                    全部數據在請求時寫入共享 
        async                   NFS在寫入數據前能夠相應請求 
        secure                  NFS經過1024如下的安全TCP/IP端口發送 
        insecure                NFS經過1024以上的端口發送 
        wdelay                  若是多個用戶要寫入NFS目錄，則歸組寫入（默認） 
        no_wdelay               若是多個用戶要寫入NFS目錄，則當即寫入，當使用async時，無需此設置。 
        hide                    在NFS共享目錄中不共享其子目錄 
        no_hide                 共享NFS目錄的子目錄 
        subtree_check           若是共享/usr/bin之類的子目錄時，強制NFS檢查父目錄的權限（默認） 
        no_subtree_check        和上面相對，不檢查父目錄權限 
        all_squash              共享文件的UID和GID映射匿名用戶anonymous，適合公用目錄。 
        no_all_squash           保留共享文件的UID和GID（默認） 
        root_squash             root用戶的全部請求映射成如anonymous用戶同樣的權限（默認） 
        no_root_squas           root用戶具備根目錄的徹底管理訪問權限 （不安全）
        anonuid=xxx             指定NFS服務器/etc/passwd文件中匿名用戶的UID 
        anongid=xxx             指定NFS服務器/etc/passwd文件中匿名用戶的GID

後來編輯爲：

使得3一、32機器能夠有NFS的/home/shareMQ目錄的讀寫權限

/home/shareMQ 192.168.3.32(rw,sync)  192.168.3.31(rw,sync)

而後restart31機器的NFS服務 （切記要重啓一下）

到32機器上嘗試mount MQ的數據到31的NFS Server上：

結果報錯，錯誤很明顯，不識別-t後面的類型：

[root@bxcscache data]# mount -t nfs 192.168.3.31:/home/shareMQ /usr/local/apache-activemq-5.10.0/data/kahadb/
mount: wrong fs type, bad option, bad superblock on 192.168.3.31:/home/shareMQ,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

是由於須要安裝NFS工具：

 yum install nfs-utils nfs-utils-lib

而後主從節點都執行掛載命令：

 mount -t nfs 192.168.3.31:/home/shareMQ /usr/local/apache-activemq-5.10.0/data/kahadb/

31機器上也掛載：

[root@bxcsweb shareMQ]# mount -t nfs 192.168.3.31:/home/shareMQ/ /usr/local/apache-activemq-5.10.0/data/kahadb/

查看已掛載的狀況：

31(31是主節點,能夠發現後面的服務端客戶端兩個ip都是本身並且要在主節點新建/home/shareMQ目錄,從節點不用建目錄)

[root@bxcsweb kahadb]# mount
/dev/vda2 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/vda1 on /main type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
192.168.3.31:/home/shareMQ/ on /usr/local/apache-activemq-5.10.0/data/kahadb type nfs (rw,vers=4,addr=192.168.3.31,clientaddr=192.168.3.31)

[root@bxcscache kahadb]# mount
/dev/vda2 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/vda1 on /main type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
192.168.3.31:/home/shareMQ on /usr/local/apache-activemq-5.10.0/data/kahadb type nfs (rw,vers=4,addr=192.168.3.31,clientaddr=192.168.3.32)

啓動兩臺機器的activeMQ：

發現有一臺一直打出以下日誌，注意紅色部分。後啓動這這個節點一直被看成slave節點，khaha的文件鎖已經被先啓動的master拿到了，因此10秒嘗試獲取一次鎖：

./activemq start

INFO: Using java '/usr/local/jdk1.6.0_45/bin/java'
INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details
INFO: pidfile created : '/usr/local/apache-activemq-5.10.0/data/activemq-bxcsweb.pid' (pid '3439')
[root@bxcsweb bin]# tail -f ../data/activemq.log
2014-06-13 04:36:09,628 | INFO  | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2014-06-13 04:36:09,631 | WARN  | Store limit is 102400 mb (current store usage is 0 mb). The data directory: /usr/local/apache-activemq-5.10.0/data/kahadb only has 7555 mb of usable space - resetting to maximum available disk space: 7555 mb | org.apache.activemq.broker.BrokerService | main
2014-06-13 04:36:09,645 | ERROR | Temporary Store limit is 51200 mb, whilst the temporary data directory: /usr/local/apache-activemq-5.10.0/data/localhost/tmp_storage only has 7554 mb of usable space - resetting to maximum available 7554 mb. | org.apache.activemq.broker.BrokerService | main
2014-06-13 04:36:10,104 | INFO  | ActiveMQ WebConsole available at http://0.0.0.0:8161/ | org.apache.activemq.web.WebConsoleStarter | main
2014-06-13 04:36:10,167 | INFO  | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
2014-06-13 04:36:10,439 | INFO  | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed | /api | main
2014-06-13 04:45:26,441 | INFO  | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@68814013: startup date [Fri Jun 13 04:45:26 CST 2014]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2014-06-13 04:45:27,850 | INFO  | PListStore:[/usr/local/apache-activemq-5.10.0/data/localhost/tmp_storage] started | org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2014-06-13 04:45:27,895 | INFO  | Using Persistence Adapter: KahaDBPersistenceAdapter[/usr/local/apache-activemq-5.10.0/data/kahadb] | org.apache.activemq.broker.BrokerService | main
2014-06-13 04:45:27,906 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2014-06-13 04:45:37,911 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2014-06-13 04:45:47,920 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2014-06-13 04:45:57,928 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2014-06-13 04:46:07,936 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2014-06-13 04:46:17,940 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main

若是要是master掛了，這個節點會立馬獲取到鎖，而後啓動起來：

好比我關掉master，在slave上看到：

2014-06-13 06:25:30,904 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2014-06-13 06:25:40,908 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2014-06-13 06:25:50,913 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2014-06-13 06:26:00,917 | INFO  | Database /usr/local/apache-activemq-5.10.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.10.0/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2014-06-13 06:26:11,381 | INFO  | KahaDB is version 5 | org.apache.activemq.store.kahadb.MessageDatabase | main
2014-06-13 06:26:11,399 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main
2014-06-13 06:26:11,454 | INFO  | Recovery replayed 1218 operations from the journal in 0.068 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
2014-06-13 06:26:11,622 | INFO  | Apache ActiveMQ 5.10.0 (localhost, ID:bxcsweb-43642-1402611971485-0:1) is starting | org.apache.activemq.broker.BrokerService | main
2014-06-13 06:26:11,642 | INFO  | Listening for connections at: tcp://bxcsweb:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2014-06-13 06:26:11,644 | INFO  | Connector openwire started | org.apache.activemq.broker.TransportConnector | main
2014-06-13 06:26:11,650 | INFO  | Listening for connections at: amqp://bxcsweb:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2014-06-13 06:26:11,651 | INFO  | Connector amqp started | org.apache.activemq.broker.TransportConnector | main
2014-06-13 06:26:11,658 | INFO  | Listening for connections at: stomp://bxcsweb:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2014-06-13 06:26:11,659 | INFO  | Connector stomp started | org.apache.activemq.broker.TransportConnector | main
2014-06-13 06:26:11,669 | INFO  | Listening for connections at: mqtt://bxcsweb:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2014-06-13 06:26:11,670 | INFO  | Connector mqtt started | org.apache.activemq.broker.TransportConnector | main
2014-06-13 06:26:11,821 | INFO  | Listening for connections at ws://bxcsweb:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.ws.WSTransportServer | main
2014-06-13 06:26:11,823 | INFO  | Connector ws started | org.apache.activemq.broker.TransportConnector | main
2014-06-13 06:26:11,825 | INFO  | Apache ActiveMQ 5.10.0 (localhost, ID:bxcsweb-43642-1402611971485-0:1) started | org.apache.activemq.broker.BrokerService | main
2014-06-13 06:26:11,826 | INFO  | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2014-06-13 06:26:11,827 | WARN  | Store limit is 102400 mb (current store usage is 0 mb). The data directory: /usr/local/apache-activemq-5.10.0/data/kahadb only has 7554 mb of usable space - resetting to maximum available disk space: 7554 mb | org.apache.activemq.broker.BrokerService | main
2014-06-13 06:26:11,829 | ERROR | Temporary Store limit is 51200 mb, whilst the temporary data directory: /usr/local/apache-activemq-5.10.0/data/localhost/tmp_storage only has 7554 mb of usable space - resetting to maximum available 7554 mb. | org.apache.activemq.broker.BrokerService | main
2014-06-13 06:26:12,296 | INFO  | ActiveMQ WebConsole available at http://0.0.0.0:8161/ | org.apache.activemq.web.WebConsoleStarter | main
2014-06-13 06:26:12,355 | INFO  | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
2014-06-13 06:26:12,680 | INFO  | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed | /api | main

至此，ActiveMQ的Share File System Master-Slave部署就算成功結束了。

而後是安全性配置：

activemq.xml已經默認打開了安全性。只用在conf/jetty-realm.properties配控制檯登陸用戶名、密碼。

另外鏈接隊列的用戶名密碼在activemq.xml的<broker>節點中添加如下元素以提供對創建鏈接時的用戶名/密碼的支持：

參考了這篇博文：http://www.cnblogs.com/zhongkl/archive/2011/08/12/2136553.html

 <plugins>
    <simpleAuthenticationPlugin>
      <users>
        <authenticationUser username="system" password="manager"
            groups="users,admins"/>
        <authenticationUser username="user" password="password"
            groups="users"/>
        <authenticationUser username="guest" password="password" groups="guests"/>
      </users>
    </simpleAuthenticationPlugin>

      <!--  lets configure a destination based authorization mechanism -->
    <authorizationPlugin>
        <map>
          <authorizationMap>
            <authorizationEntries>
              <authorizationEntry queue=">" read="admins" write="admins" admin="admins" />
              <authorizationEntry queue="USERS.>" read="users" write="users" admin="users" />
              <authorizationEntry queue="GUEST.>" read="guests" write="guests,users" admin="guests,users" />
              
              <authorizationEntry topic=">" read="admins" write="admins" admin="admins" />
              <authorizationEntry topic="USERS.>" read="users" write="users" admin="users" />
              <authorizationEntry topic="GUEST.>" read="guests" write="guests,users" admin="guests,users" />
              
              <authorizationEntry topic="ActiveMQ.Advisory.>" read="guests,users" write="guests,users" admin="guests,users"/>
            </authorizationEntries>
            
            <!-- let's assign roles to temporary destinations. comment this entry if we don't want any roles assigned to temp destinations  -->
            <tempDestinationAuthorizationEntry>  
              <tempDestinationAuthorizationEntry read="tempDestinationAdmins" write="tempDestinationAdmins" admin="tempDestinationAdmins"/>
           </tempDestinationAuthorizationEntry>               
          </authorizationMap>
        </map>
     </authorizationPlugin>
   </plugins>

接下來咱們看看spring裏怎麼配：

     <!--鏈接工廠-->
    <bean id="connectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
        <property name="brokerURL">
            <!--3一、32兩臺機器鏈接，斷線重連，10毫秒一次，最多嘗試60次-->
             <value>failover:(tcp://192.168.3.31:61616,tcp://192.168.3.32:61616)?initialReconnectDelay=10&amp;maxReconnectAttempts=60</value>
        </property>
        <property name="userName" value="***" />
        <property name="password" value="***" />
    </bean>

OK 結束。

其中，各類搭建過程部分參考了：

http://harbey.iteye.com/blog/1756607

http://shensy.iteye.com/blog/1752529

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。