Saltstack之multi-master

1、實驗環境：

一、salt版本：

[root@master master]# salt --versions-report
           Salt: 2015.5.10
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
 msgpack-python: 0.4.8
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 3.2.5
           Mako: Not Installed
        Tornado: Not Installed
        timelib: Not Installed
       dateutil: Not Installed

二、系統版本：

[root@master master]# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 

2、Salt搭建主備master

一、安裝新的master server

二、copy master keys到新的master對應的目錄(master.pem和master.pub)

三、啓動新的master進程

四、配置minions配置文件

五、Restart minions

六、在新的master上accept keys

七、測試兩個salt-master對salt-minion的test.ping

[root@master master]# salt -L 192.168.163.13 test.ping
192.168.163.13:
    True
[root@standby minions]# salt -L 192.168.163.13 test.ping
192.168.163.13:
    True

備註：

配置multi-master主要的問題是：每個master使用相同的private key. Private key在master第一次啓動時自動生成的(注意：配置multi-master時，必定要在啓動新master前將老master的private key copy到對應目錄)。

 

修改minion配置文件：

master

　　-saltmaster1.example.com

　　-saltmaster2.example.com

 

3、遇到的問題：

一、在minion端進行salt-call測試的時候(前提條件將主master stop)，發現：

[root@standby salt]# salt-call test.ping        
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 1 of 4)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 2 of 4)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 3 of 4)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 4 of 4)
[WARNING ] Attempted to authenticate with master 192.168.199.39 and failed
[WARNING ] Master ip address changed from 192.168.199.39 to 192.168.163.13
local:
    True
----以上測試結果是將minion中的auth_tries修改成4，默認值爲7.
將值改爲3次，並關閉主備master的測試結果：
[root@standby minion]# salt-call test.ping        
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 1 of 3)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 2 of 3)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 3 of 3)
[WARNING ] Attempted to authenticate with master 192.168.199.39 and failed
[WARNING ] Master ip address changed from 192.168.199.39 to 192.168.163.13
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 1 of 3)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 2 of 3)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 3 of 3)
[WARNING ] Attempted to authenticate with master 192.168.163.13 and failed
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
AttributeError: 'SMinion' object has no attribute 'functions'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 227, in salt_call
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 71, in run
    caller.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 236, in run
    ret = self.call()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 107, in call
    if fun not in self.minion.functions:
AttributeError: 'SMinion' object has no attribute 'functions'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 227, in salt_call
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 71, in run
    caller.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 236, in run
    ret = self.call()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 107, in call
    if fun not in self.minion.functions:
AttributeError: 'SMinion' object has no attribute 'functions'

 

二、master數據共享問題：

masters之間並不會共享信息，public keys須要在每臺master上accept，文件共享須要手工完成，或者使用相似git工具確保file_roots目錄文件一致。

具體須要同步的目錄有：

Minion keys:

• /etc/salt/pki/master/minions
• /etc/salt/pki/master/minions_pre
• /etc/salt/pki/master/minions_rejected

備註：直接共享/etc/salt/master目錄是強烈反對的。容許外部訪問master.pem key將帶來嚴重的安全風險。

 

四、minion keys能夠參考使用：

　　方案一：

　　　　*/10 * * * * rsync -av --progress --delete --timeout=30 root@192.168.199.39:/etc/salt/pki/master/minions/ /etc/salt/pki/master/minions/

　　方案二：修改salt-key的源代碼：

　　　　當主master有accept的時候同步給備master，在配置文件中配置備機IP，只有兩邊同步成功了纔算成功；

　　　　刪除minion的時候只用經過salt-key -d的方式刪除，或者配合rsync的方式，防止經過rm的方式刪除minion。

 

五、file_roots和pillar_roots等文件能夠放在git上。

 

須要說起的是，本實驗的salt版本須要修改minion.py文件：

由於minion註冊的時候會先往IP小的機器註冊，而沒法按你指定的IP順序註冊。

修改代碼以下：

從minion.py代碼中查看獲得for master in set(self.opts['master']):中

class MultiMinion(MinionBase):
    '''
    Create a multi minion interface, this creates as many minions as are
    defined in the master option and binds each minion object to a respective
    master.
    '''
    # timeout for one of the minions to auth with a master
    MINION_CONNECT_TIMEOUT = 5

    def __init__(self, opts):
        super(MultiMinion, self).__init__(opts)

    def minions(self):
        '''
        Return a dict of minion generators bound to the tune_in method

        dict of master -> minion_mapping, the mapping contains:

            opts: options used to create the minion
            last: last auth attempt time
            auth_wait: time to wait for next auth attempt
            minion: minion object
            generator: generator function (non-blocking tune_in)
        '''
        if not isinstance(self.opts['master'], list):
            log.error(
                'Attempting to start a multimaster system with one master')
            sys.exit(salt.defaults.exitcodes.EX_GENERIC)
        ret = {}
        #在這裏對master進行了一個排序
        for master in self.opts['master']:
        # for master in set(self.opts['master']):
            s_opts = copy.deepcopy(self.opts)
            s_opts['master'] = master
            s_opts['multimaster'] = True
            ret[master] = {'opts': s_opts,
                           'last': time.time(),
                           'auth_wait': s_opts['acceptance_wait_time']}
            try:
                minion = Minion(
                        s_opts,
                        self.MINION_CONNECT_TIMEOUT,
                        False,
                        'salt.loader.{0}'.format(master))
                ret[master]['minion'] = minion
                ret[master]['generator'] = minion.tune_in_no_block()
            except SaltClientError as exc:
                log.error('Error while bringing up minion for multi-master. Is master at {0} responding?'.format(master))
        return ret