多puppetmaster，多ca，keepalived+haproxy（nginx）puppet集羣搭建

多puppetmaster，多ca，keepalived+haproxy（nginx）puppet集羣搭建

 

1、服務器詳情

192.168.122.111 pm01.jq.com pm01 #（puppetmaster服務器）

192.168.122.112 pm02.jq.com pm02 #（puppetmaster服務器）

192.168.122.121 ag01.jq.com ag01 #（puppet agent服務器）

192.168.122.122 ag02.jq.com ag02 #（puppet agent服務器）

192.168.122.131 ca01.jq.com ca01 #（puppet ca服務器）

192.168.122.132 ca02.jq.com ca02 #（puppet ca服務器）

192.168.122.141 lvs01.jq.com lvs01 #（puppet 負載服務器）

192.168.122.142 lvs02.jq.com lvs02 #（puppet 負載服務器）

#vip暫時用於測試，使用ip addr的方式綁定，後續用高可用軟件實現bind

192.168.122.130 pc.jq.com pc #（ca服務器的vip，前期綁定在ca01服務器上）

192.168.122.115 lvs.jq.com lvs #（負載服務器的vip，前期綁定在puppetmaster上，後期須要綁定在lvs服務器上）

 

2、CA服務器部署

CA服務器單獨用於簽署和撤銷證書，當puppetCA服務不可用時，新的客戶端將不能得到證書，從而會影響使用，而已簽發證書的客戶端缺不受影響。所以將CA進行獨立隊架構，這對容錯性而言是很是有必要的。

 

2.1 安裝軟件包

[root@ca01 ~]# groupadd -g 3000 puppet

[root@ca01 ~]# useradd -u 3000 -g 3000 puppet

[root@ca01 ~]# yum install puppet puppet-server –y

2.2 bind vip

綁定ca的vip 192.168.122.130到ca01服務器

[root@ca01 ~]#ip addr add 192.168.122.130/24 dev eth0

 

2.3 生成證書

使用puppet cert命令生成CA服務器與服務器域名證書。生成pc.jq.com和lvs.jq.com兩個域名的受權證書文件,前面是證書別名，後面是證書名稱。

[root@ca01 ssl]# puppet  cert --generate --dns_alt_names pc pc.jq.com

[root@ca01 ssl]# puppet  cert --generate --dns_alt_names lvs lvs.jq.com [root@ca01 ssl]# puppet cert --list --all

+ "lvs.jq.com" (SHA256) D6:5B:51:D6:6E:35:61:A4:45:D8:37:17:5B:85:A1:1B:34:BB:2F:D7:48:E8:44:57:B7:1D:42:8E:11:18:81:34 (alt names: "DNS:lvs", "DNS:lvs.jq.com")

+ "pc.jq.com"  (SHA256) A7:71:E1:46:1E:F0:F1:70:72:E3:B5:16:03:91:17:6D:68:5B:55:39:B6:79:6B:30:DD:41:ED:10:21:27:2A:33 (alt names: "DNS:pc", "DNS:pc.jq.com")

2.4 配置puppet.conf，添加標籤[master]

[root@ca01 ~]# cat /etc/puppet/puppet.conf  | grep -v "#"

[main]

    logdir = /var/log/puppet

    rundir = /var/run/puppet

    ssldir = $vardir/ssl

    pluginsync = false

[agent]

    classfile = $vardir/classes.txt

    localconfig = $vardir/localconfig

    server = lvs.jq.com

    ca_server = pc.jq.com

    environment = jqprd

[master]

    confdir = /etc/puppet

    certname = pc.jq.com

ca = true #開啓CA認證

 

2.5 啓動puppetmaster，CA部署完成

[root@ca01 ~]# /etc/init.d/puppetmaster start

[root@ca01 ~]# chkconfig puppetmaster on

 

2.6 ca02服務部署

ca02跟ca01的部署方式徹底一致，證書是從ca01 拷貝過來的。直接copy /var/lib/ssl/puppet目錄

 

3、PuppetMaster服務器部署

PuppetMaster服務器部署可採用默認的WebRick方式，也能夠採用apache+passenger或nginx+passenger方式。

3.1 WebRick方式：

3.1.1 安裝軟件包

[root@pm01 ~]# groupadd -g 3000 puppet

[root@pm01 ~]# useradd -u 3000 -g 3000 puppet

[root@pm01 ~]# yum install puppet puppet-server -y

3.1.2 設置hosts文件

[root@pm01 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

3.1.3 bind master vip

綁定LVS的vip 192.168.122.115到pm01服務器，測試用，在沒有負載以前，綁定在master上。

ip addr add 192.168.122.115/24 dev eth0

3.1.4 建立證書目錄

[root@pm01 ~]# mkdir /var/lib/puppet/ssl/{certs,ca,private_keys} –p

 

3.1.5 將puppetca上生成的puppetmaster公鑰、私鑰和根證書複製到pm01

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/signed/lvs.jq.com.pem /var/lib/puppet/ssl/certs/lvs.jq.com.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/private_keys/lvs.jq.com.pem /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crl.pem /var/lib/puppet/ssl/ca/ca_crl.pem

3.1.6 配置puppet.conf，添加標籤[master]，關閉ca

 

[root@pm01 ~]# grep -v "#" /etc/puppet/puppet.conf

[main]

    logdir = /var/log/puppet

    rundir = /var/run/puppet

    ssldir = $vardir/ssl

    privatekeydir = $ssldir/private_keys { group = service } 

    hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

 

[agent]

    classfile = $vardir/classes.txt

    localconfig = $vardir/localconfig

    server = lvs.jq.com #puppetmaster域名，應該與以前手動生成的證書匹配

    ca_server = pc.jq.com #ca證書服務器

 

[master]

    certname = lvs.jq.com #puppetmaster的域名，應該與以前手動生成的證書匹配

    ca = false #關閉ca驗證

 

3.1.7 啓動puppetmaster服務，puppetmaster部署完成

[root@pm01 ssl]# /etc/init.d/puppetmaster restart

 

3.1.8 運行puppet命令進行本地證書申請

[root@pm01 ssl]# puppet  agent -t

Info: Creating a new SSL key for pm01.jq.com

Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml

Info: Creating a new SSL certificate request for pm01.jq.com

Info: Certificate Request fingerprint (SHA256): 2C:09:32:E1:13:CA:0F:44:3B:93:4B:0F:0E:2D:46:19:3A:37:E1:47:C7:D3:E8:2C:A6:83:44:B3:D3:94:63:D6

Exiting; no certificate found and waitforcert is disabled

 

3.1.9 登陸puppetca進行證書籤發

[root@ca01 ~]# puppet cert --sign pm01.jq.com

Notice: Signed certificate request for pm01.jq.com

Notice: Removing file Puppet::SSL::CertificateRequest pm01.jq.com at '/var/lib/puppet/ssl/ca/requests/pm01.jq.com.pem'

3.1.10 再次運行puppet命令進行測試連通性

[root@pm01 ssl]# puppet  agent -t

Info: Caching certificate for pm01.jq.com

Info: Caching certificate_revocation_list for ca

Info: Caching certificate for pm01.jq.com

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for pm01.jq.com

Info: Applying configuration version '1425526708'

Notice: Finished catalog run in 0.17 seconds

3.1.11 在kspupt-ca上申請本地證書

[root@ca01 ~]# vim /etc/puppet/puppet.conf

[agent]

    server    = lvs.jq.com

    ca_server = pc.jq.com

[root@ca01 ~]# puppet agent -t

[root@ca01 ~]# puppet cert --sign ca01.jq.com

[root@ca01 ~]# puppet agent –t

 

3.2 Nginx+Passenger方式：

注：可參考 http://kisspuppet.com/2014/10/20/puppet_learning_ext4/

3.2.一、配置nginx

[root@pm01 ssl]# cat /usr/local/nginx/conf/vhosts/passenger.conf

    server {

        listen 8140                ssl;

    server_name                puppetmaster;

    passenger_enabled          on;

    passenger_set_cgi_param    HTTP_X_CLIENT_DN $ssl_client_s_dn;

    passenger_set_cgi_param    HTTP_X_CLIENT_VERIFY $ssl_client_verify;

    proxy_buffer_size 4000k;

    proxy_buffering on;

    proxy_buffers 32 1280k;

    proxy_busy_buffers_size 17680k;

    client_max_body_size 10m;

    client_body_buffer_size 4096k;

    access_log /var/log/nginx/puppet_access.log;

    error_log /var/log/nginx/puppet_error.log;

    root /etc/puppet/rack/public;

    ssl off;

    ssl_session_timeout 5m;

    ssl_certificate /var/lib/puppet/ssl/certs/lvs.jq.com.pem;

    ssl_certificate_key /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem;

    ssl_client_certificate /var/lib/puppet/ssl/certs/ca.pem;

    ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;

    ssl_verify_client optional;

    ssl_ciphers SSLv2:-LOW:-EXPORT:RC4+RSA;

    ssl_prefer_server_ciphers on;

    ssl_verify_depth 1;

    ssl_session_cache shared:SSL:128m;

    # File sections

    location /production/file_content/files/ {

      types { }

      default_type application/x-raw;

      alias /etc/puppet/files/;

  }

 }

 

3.2.二、配置puppet.conf

[root@pm01 ssl]# grep -v "#" /etc/puppet/puppet.conf

[main]

    logdir = /var/log/puppet

    rundir = /var/run/puppet

    ssldir = $vardir/ssl

    privatekeydir = $ssldir/private_keys { group = service } 

    hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

 

[agent]

    classfile = $vardir/classes.txt

    localconfig = $vardir/localconfig

    server = lvs.jq.com

    ca_server = pc.jq.com

 

[master]

    certname = lvs.jq.com

    ca = false

    ssl_client_verify_header = HTTP_X_CLIENT_VERIFY

ssl_client_header = HTTP_X_CLIENT_DN

 

3.3 master02服務器部署

master02的部署與master01的徹底一致，包括拷貝證書部分

 

4 Puppet LB負載均衡器部署

4.1 puppet認證創建

4.1.一、安裝軟件包

[root@lvs01 ~]# groupadd -g 3000 puppet

[root@lvs01 ~]# useradd -u 3000 -g 3000 puppet

[root@lvs01 ~]# yum install puppet

4.1.二、編輯hosts文件

[root@lvs01 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

 

4.1.三、建立證書目錄

[root@lvs01 ~]# mkdir /var/lib/puppet/ssl/{certs,ca,private_keys} –p

 

4.1.四、將ca上生成的puppetmaster公鑰、私鑰和根證書複製到lvs01

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/signed/lvs.jq.com.pem /var/lib/puppet/ssl/certs/lvs.jq.com.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/private_keys/lvs.jq.com.pem /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crl.pem /var/lib/puppet/ssl/ca/ca_crl.pem

 

4.1.五、配置puppet.conf，修改標籤[agent]，增長server和ca_server字段

[root@lvs01 ~]# vim /etc/puppet/puppet.conf

[agent]

    server      = lvs.jq.com

    ca_server   = pc.jq.com

4.1.六、運行puppet命令進行本地證書申請

[root@lvs01 ~]# puppet  agent -t

4.1.七、登陸ca進行證書籤發

[root@ca01 ~]# puppet  cert --sign lvs01.jq.com

4.1.八、再次運行puppet命令進行測試連通性

[root@lvs01 ~]# puppet agent -t

Info: Caching certificate for lvs01.jq.com

Info: Caching certificate_revocation_list for ca

Info: Caching certificate for lvs01.jq.com

Info: Loading facts

Info: Caching catalog for lvs01.jq.com

Info: Applying configuration version '1425527450'

Notice: Finished catalog run in 0.24 seconds

 

4.2 安裝並配置nginx負載均衡器

4.2.一、安裝nginx軟件

[root@lvs01 ~]# groupadd -g 3001 nginx

[root@lvs01 ~]# useradd -u 3001 -g 3001 nginx

[root@lvs01 ~]# yum install nginx

4.2.二、臨時設置VIP地址（後面經過高可用軟件代替）

[root@lvs01 ~]# ip addr add 192.168.122.115/24 dev eth0

此處請將以前bind到pm01的vip取消

4.2.三、配置nginx虛擬主機，添加upstrem

[root@lvs01 ~]# cat /etc/nginx/conf.d/puppetmaster.conf

upstream puppet-master {

   server 192.168.122.111:8140;

   server 192.168.122.112:8140;

}

server {

   listen         8140 ssl;

   server_name    puppetmaster;

   access_log     /var/log/nginx/puppet_access.log;

   error_log      /var/log/nginx/puppet_error.log;

   ssl_protocols SSLv3 TLSv1;

   ssl_ciphers ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;

   proxy_set_header             X-SSL-Subject  $ssl_client_s_dn;

   proxy_set_header             X-Client-DN  $ssl_client_s_dn;

   proxy_set_header             X-Client-Verify  $ssl_client_verify;

   client_max_body_size 100m;

   client_body_buffer_size 1024k;

   proxy_buffer_size 100m;

   proxy_buffers 8 100m;

   proxy_busy_buffers_size 100m;

   proxy_temp_file_write_size 100m;

   proxy_read_timeout 500;

 

   ssl                     on;

   ssl_session_timeout     5m;

   ssl_certificate         /var/lib/puppet/ssl/certs/lvs.jq.com.pem;

   ssl_certificate_key     /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem;

   ssl_client_certificate  /var/lib/puppet/ssl/certs/ca.pem;

   ssl_crl                 /var/lib/puppet/ssl/ca/ca_crl.pem;

   ssl_verify_client       optional;

   ssl_prefer_server_ciphers  on;

   ssl_verify_depth           1;

   ssl_session_cache          shared:SSL:128m;

   location / {

         proxy_redirect    off;

         proxy_pass        https://puppet-master;

  }

}

 

4.2.四、編輯hosts文件，puppetmaster解析指向VIP

[root@lvs01 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

4.2.五、修改ca01和pm01的hosts文件puppetmaster解析

[root@kspupt-ca1 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

 

[root@pm01 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

4.2.六、啓動nginx服務器

[root@lvs01 ~]# /etc/init.d/nginx start

4.2.七、再次運行puppet命令進行測試連通性

[root@kspupt-ca1 ~]# puppet  agent -t

[root@pm01 ~]# puppet  agent -t

[root@lvs01 ~]# puppet  agent -t

[root@pm01 ~]# tailf  /var/log/nginx/puppet_access.log

[root@lvs01 ~]# tailf /var/log/nginx/puppet_access.log

 

4.3 安裝配置Haproxy負載均衡

安裝haproxy和keepalived過程略去，網上不少教程

 

[root@lvs01 keepalived]# cat /etc/haproxy/haproxy.cfg

global

        maxconn         40000

        ulimit-n        500000

        log             127.0.0.1 local0

        uid             99

        gid             99

        chroot          /tmp

#       nbproc          4

        daemon

 

defaults

        log     global

        retries 2

        option redispatch

        option dontlognull

        balance roundrobin

        timeout connect 30000ms

        timeout client 30000ms

        timeout server 30000ms

        timeout check 2000

 

listen admin_stats

    bind 0.0.0.0:8080

    mode http

    stats refresh 5s

    stats enable

    stats hide-version

    stats realm Haproxy\ Statistics

    stats uri /haproxy

    stats auth admin:password

 

listen puppetmaster *:8140

    mode tcp

    option ssl-hello-chk

#    option tcplog

    #balance source

#    balance roundrobin

    balance source

    server pm01 pm01.jq.com:8140 check inter 2000 fall 3

server pm02 pm02.jq.com:8140 check inter 2000 fall 3

 

4.4 配置keepalived，取消vip 經過ip addr 綁定

Keepalived的備機配置略去，網上也能夠搜索到，須要修改的地方不多。

[root@lvs01 ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

 

global_defs {

   notification_email {

     test@gmail.com

   }

   notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL

}

 

vrrp_script chk_http_port {

   script "/etc/keepalived/check_haproxy.sh"

   interval 2

   weight 2

}

 

vrrp_instance VI_1 {

    state MASTER

    interface eth0

    virtual_router_id 51

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

track_script {

    chk_http_port

}

 

    virtual_ipaddress {

        192.168.122.115 #此處爲負載的VIP，配置keepalived以後，切記去掉以前ip addr的綁定

    }

}

4.4.1 Keepalived監控haproxy腳本

[root@lvs01 ~]# cat /etc/keepalived/check_haproxy.sh

#!/bin/bash

. /etc/profile

A=`ps -C haproxy --no-header |wc -l`

if [ $A -eq 0 ];then

        /etc/init.d/haproxy start

sleep 3

if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then

/etc/init.d/keepalived stop

fi

fi

 

4.5 Lvs02服務器部署

Lvs02的配置與lvs01的配置徹底一致，將此服務器做爲lvs01的備機，包括keepalived+haproxy。

 

幾乎照搬了http://kisspuppet.com/2014/10/21/puppet_learning_ext6/ 的文章，很是感謝kisspuppet！