centos7.5系統elasticsearch使用滾動和全新安裝升級到最新的elasticsearch7.4.2版本

背景：
    生產環境大量使用 elasticsearch 集羣，不一樣的業務使用不一樣版本的elasticsearch
    es常常曝出一些大的漏洞，須要進行版本升級，而且使用x-pack的基本驗證功能，避免用戶數據泄露


x-pack免費版本特徵：

基本的TLS 功能，可對通訊進行加密
文件和原生 Realm，可用於建立和管理用戶
基於角色的訪問控制，可用於控制用戶對集羣 API 和索引的訪問權限；
經過針對 Kibana Spaces 的安全功能，還可容許在 Kibana 中實現多租戶。



升級的兩種策略：
1.滾動性升級，即不中斷業務服務，一臺一臺進行升級
2.全新部署新版本，而後將數據遷移到新版本的es集羣中


這兩種方式都須要將數據恢復到新版本的es集羣中，能夠先進行快照備份


1.升級前先備份低版本的elasticserch的數據：快照方式

原理：即將老版本的es數據打個快照備份出來寫入到 /opt/esback 目錄中並進行nfs掛載，新、舊兩個es集羣的配置文件中都引用配置 path.repo: ["/opt/esback/"]，
這樣新的集羣也能對這個目錄進行操做了，等待新集羣搭建好後，直接把 /opt/esback 目錄中的文件恢復到新集羣的 索引 index 中便可

使用Mount nfs進行掛載共享(全部的es集羣節點均可以訪問)：

目標：將本地es備份出來的數據目錄/opt/esback 目錄掛載到nfs的共享目錄 /opt/es_snapshot，這樣恢復的時候就均可以訪問這個共享目錄進行恢復了


// 在10.10.18.92上建立共享目錄
建立共享目錄，即做爲nfs的共享目錄
mkdir /opt/es_snapshot

建立本地備份出來的目錄
/opt/esback

# 在集羣全部節點中建立 /opt/esback 目錄，即將es數據備份出來的目標目錄


# 將其中一臺es客戶端做爲nfs服務端
#nfs服務端的操做
# vim /etc/exports

# 注意此處的anonuid和gid要和運行es程序的用戶保持一致

# 添加指定 uid 和 gid 的用戶
groupadd -g 1000 elastic
useradd -u 1000 -g elastic elastic

# 修改 gid和 uid爲500 命令示例：
usermod -u 500 es
groupmod -g 500 es

/opt/es_snapshot *(insecure,rw,no_root_squash,sync,anonuid=1000,anongid=1000)

// 查看共享文件夾
yum install -y exportfs
exportfs -rv

// nfs服務端修改nfs配置
 vim /etc/sysconfig/nfs
修改以下：
RPCNFSDARGS="-N 2 -N 3"
        ----->啓用
# Turn off v4 protocol support
RPCNFSDARGS="-N 4"     ---->啓用
重啓生效
systemctl restart nfs

// 客戶端操做
yum install -y nfs-utils
  

// 重啓啓動新集羣機器的NFS服務  
systemctl restart nfs

//每一臺es節點服務器上進行Mount掛載
mount -t nfs 10.10.18.90:/opt/es_snapshot /opt/esback -o proto=tcp -o nolock


列出nfs服務端共享的目錄:
[root@sz_kp_wanghong_dev02_18_93:/home/wanxing]# showmount -e 10.10.18.90
Export list for 10.10.18.92:
/opt/es_snapshot *


// 在舊機器上將共享目錄的權限付給ES的運行用戶  
chown elastic:elastic -R /opt/esback

2.建立ES倉庫my_backup

修改配置文件：
vim elasticsearch.yml
# 添加以下配置(須要在舊集羣的每一個節點上添加)，從新啓動集羣
path.repo: ["/opt/esback"]


建立快照倉庫 my_backup 命令：
curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup -d '
{
    "type": "fs",
    "settings": {
        "location": "/opt/esback",
    "compress": true
    }
}
'
# 返回值
{"acknowledged":true}


# 報錯的處理
'RemoteTransportException[[ictr_node1][10.10.18.93:9300][internal:admin/repository/verify]]

# 權限不夠
chown -R es.es /opt/es_snapshot/
chown -R es.es /opt/esback_20191104/


# 建立全部索引的備份

# curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107
{"accepted":true}


查看備份

[elastic@szyyelk01t slave02]$ curl -XGET http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107?pretty
{
  "snapshots" : [
    {
      "snapshot" : "snapshot20191107",
      "uuid" : "0_4SOntVS1GH-7irHjKBMQ",
      "version_id" : 6030299,
      "version" : "6.3.2",
      "indices" : [
        "support_faq_categorys",
        "ticket_list",
        "templates_search",
        "site_page_search",
        "support",
        "templates_page_search",
        "support_new_articles",
        "article_version",
        "blocks_version",
        "search",
        "version",
        "article_search",
        "templates",
        "learn",
        "templates_version",
        "blocks_search",
        "templates_page_version"
      ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2019-11-07T01:35:00.811Z",
      "start_time_in_millis" : 1573090500811,
      "end_time" : "2019-11-07T01:35:03.702Z",
      "end_time_in_millis" : 1573090503702,
      "duration_in_millis" : 2891,
      "failures" : [ ],
      "shards" : {
        "total" : 71,
        "failed" : 0,
        "successful" : 71
      }
    }
  ]
}


升級方式1：滾動升級 elasticsearch5.6.16 --> elasticsearch6.8.4

1.備份數據，避免出現問題後回滾
2.先升級到新版本，而後安裝x-pack，此時再要求開發同事修改代碼適配
a.先下載新版本的6.8.4


①關閉自動分片
curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}'

[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{
>   "persistent": {
>     "cluster.routing.allocation.enable": "none"
>   }
> }'
* Hostname was NOT found in DNS cache
*   Trying 10.10.18.92...
* Connected to 10.10.18.92 (10.10.18.92) port 9200 (#0)
> PUT /_cluster/settings HTTP/1.1
> User-Agent: curl/7.36.0
> Host: 10.10.18.92:9200
> Accept: */*
> Content-Length: 73
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 73 out of 73 bytes
< HTTP/1.1 200 OK
< Warning: 299 Elasticsearch-5.6.15-fe7575a "Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header." "Tue, 05 Nov 2019 08:14:44 GMT"
< content-type: application/json; charset=UTF-8
< content-length: 106
< 
* Connection #0 to host 10.10.18.92 left intact
{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"none"}}}},"transient":{}}


②暫時禁用非必要的索引並執行同步刷新

curl -XPOST http://10.10.18.92:9200/_flush/synced

[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# curl -XPOST http://10.10.18.92:9200/_flush/synced
{"_shards":{"total":28,"successful":28,"failed":0},"channel_rel":{"total":4,"successful":4,"failed":0},".kibana":{"total":2,"successful":2,"failed":0},"channel":{"total":6,"successful":6,"failed":0},"video":{"total":4,"successful":4,"failed":0},"channel_list":{"total":6,"successful":6,"failed":0},"influecer":{"total":6,"successful":6,"failed":0}}

注意: 若是是從6.3以前的版本升級上來的，須要注意提早要移除X-Pack插件，而後再去升級版本。執行bin/elasticsearch-plugin remove x-pack


a. 備份原來的elasticsearch目錄，而後解壓新版的elasticsearch。
b. 若是使用外部的配置路徑，配置ES_PATH_CONF環境變量到那個位置。若是沒有的話，拷貝老的配置目錄過來新的elasticsearch目錄就能夠了。
c. 檢查path.data是否指向正確的數據目錄
d. 檢查path.log是否指向正確的日誌目錄

新集羣的配置文件
[es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ more config/elasticsearch.yml 
cluster.name: kp-dev-application
node.name: ictr_node2
node.master: true
node.attr.rack: r1
node.max_local_storage_nodes: 3
network.host: 10.10.18.92
http.port: 9200
transport.tcp.port: 9300
path.repo: ["/opt/esback_20191104"]
discovery.zen.minimum_master_nodes: 1
http.cors.enabled: true
http.cors.allow-origin: "*"
# 新集羣的數據仍是指向老版本 es5.6.15 的數據存儲目錄
path.data: /opt/es-node/elasticsearch-5.6.15/data
path.logs: /opt/es-node/elasticsearch-5.6.15/logs
# 啓用安全認證
xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12


③關閉節點
④從新啓動節點，注意要切換到es用戶，不能使用root用戶
chown -R es.es elasticsearch-6.8.4

[es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ bin/elasticsearch -d


在其餘節點重複以上過程


啓動升級後的節點，並經過查看日誌和使用下面命令來檢查節點是否正確加入到集羣
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cat/nodes
10.10.18.93 16 98 56 1.22 0.50 0.29 di  - ictr_node1
10.10.18.92 16 88  8 0.08 0.26 0.31 mdi * ictr_node2
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cat/indices
yellow open channel      vRFQoIhmT8WmSbDCfph0ag 3 1   53374      0  44.2mb  44.2mb
yellow open channel_rel  ZeeBbkogT5KtxzziUYtu_Q 2 1  459528      0 168.8mb 168.8mb
yellow open channel_list 1dk8uH8bTeikez0lFR2mJg 3 1 5509390  78630     7gb     7gb
yellow open video        HNhyt9ioSEayAotGVXRCVg 2 1  798369 228155   1.6gb   1.6gb
yellow open .kibana      lY82G_-XSniyd_bnMOLuQg 1 1      15      1 146.3kb 146.3kb
yellow open influecer    RQtQWXKIRE2UYyZlCvv7bA 3 1  148526  48641 272.8mb 272.8mb


節點加入集羣后，刪除cluster.routing.allocation.enable設置以啓用分片分配並開始使用節點：


curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}'


從新打開分片報錯：
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{
>   "persistent": {
>     "cluster.routing.allocation.enable": "true"
>   }
> }'

* Hostname was NOT found in DNS cache
*   Trying 10.10.18.92...
* Connected to 10.10.18.92 (10.10.18.92) port 9200 (#0)
> PUT /_cluster/settings HTTP/1.1
> User-Agent: curl/7.36.0
> Host: 10.10.18.92:9200
> Accept: */*
> Content-Length: 73
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 73 out of 73 bytes
< HTTP/1.1 406 Not Acceptable
< content-type: application/json; charset=UTF-8
< content-length: 97
< 
* Connection #0 to host 10.10.18.92 left intact
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}


[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cluster/health?pretty
{
  "cluster_name" : "kp-dev-application",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 14,
  "active_shards" : 28,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}


安裝新版本中文分詞插件

https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.8.4/elasticsearch-analysis-ik-6.8.4.zip

# 解壓在plugin目錄從新啓動elasticsearch便可
cd /opt/es-node/elasticsearch-6.8.4/plugins
unzip -d elasticsearch-analysis-ik elasticsearch-analysis-ik-6.8.4.zip


ot@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4/plugins]# curl http://10.10.18.92:9200/_xpack?pretty
{
  "build" : {
    "hash" : "bca0c8d",
    "date" : "2019-10-16T06:19:49.319352Z"
  },
  "license" : {
    "uid" : "4de9d1c1-59f6-4dfd-8d48-baefd0a583d0",
    "type" : "basic",
    "mode" : "basic",
    "status" : "active"
  },
  "features" : {
    "ccr" : {
      "description" : "Cross Cluster Replication",
      "available" : false,
      "enabled" : true
    },
    "graph" : {
      "description" : "Graph Data Exploration for the Elastic Stack",
      "available" : false,
      "enabled" : true
    },
    "ilm" : {
      "description" : "Index lifecycle management for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "logstash" : {
      "description" : "Logstash management component for X-Pack",
      "available" : false,
      "enabled" : true
    },
    "ml" : {
      "description" : "Machine Learning for the Elastic Stack",
      "available" : false,
      "enabled" : true,
      "native_code_info" : {
        "version" : "6.8.4",
        "build_hash" : "93ad89b02ff490"
      }
    },
    "monitoring" : {
      "description" : "Monitoring for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "rollup" : {
      "description" : "Time series pre-aggregation and rollup",
      "available" : true,
      "enabled" : true
    },
    "security" : {
      "description" : "Security for the Elastic Stack",
      "available" : true,
      "enabled" : false
    },
    "sql" : {
      "description" : "SQL access to Elasticsearch",
      "available" : true,
      "enabled" : true
    },
    "watcher" : {
      "description" : "Alerting, Notification and Automation for the Elastic Stack",
      "available" : false,
      "enabled" : true
    }
  },
  "tagline" : "You know, for X"
}

3.啓用x-pack的密碼驗證

# 生成證書

[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# bin/elasticsearch-certutil ca
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]: 
Enter password for elastic-stack-ca.p12 : 
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# ls
bin  config  elastic-stack-ca.p12  lib  LICENSE.txt  logs  modules  NOTICE.txt  plugins  README.textile
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X.509 certificate and private keys.
    * By default, this generates a single certificate and key for use
       on a single instance.
    * The '-multiple' option will prompt you to enter details for multiple
       instances and will generate a certificate and key for each one
    * The '-in' option allows for the certificate generation to be automated by describing
       the details of each instance in a YAML file

    * An instance is any piece of the Elastic Stack that requires an SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.

    * All certificates generated by this tool will be signed by a certificate authority (CA).
    * The tool can automatically generate a new CA for you, or you can provide your own with the
         -ca or -ca-cert command line options.

By default the 'cert' mode produces a single PKCS#12 output file which holds:
    * The instance certificate
    * The private key for the instance certificate
    * The CA certificate

If you specify any of the following options:
    * -pem (PEM formatted output)
    * -keep-ca-key (retain generated CA key)
    * -multiple (generate multiple certificates)
    * -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files

Enter password for CA (elastic-stack-ca.p12) : 
Please enter the desired output file [elastic-certificates.p12]: 
Enter password for elastic-certificates.p12 : 

Certificates written to /opt/es-node/elasticsearch-6.8.4/elastic-certificates.p12

This file should be properly secured as it contains the private key for 
your instance.

This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.


# 修改config/elasticsearch.yml配置

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/local/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/local/elasticsearch/config/elastic-certificates.p12

# 配置密碼
[es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana]: 
Reenter password for [kibana]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]


密碼：espass

[es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ curl --user elastic:espass -XGET 'http://10.10.18.92:9200/_cat/indices'
green open channel_rel  ZeeBbkogT5KtxzziUYtu_Q 2 1  459528      0 337.7mb 168.8mb
green open .security-6  iQHndFBqRe2Ss2o7KMxyFg 1 1       6      0  38.3kb  19.1kb
green open .kibana      lY82G_-XSniyd_bnMOLuQg 1 1      15      1 292.6kb 146.3kb
green open influecer    RQtQWXKIRE2UYyZlCvv7bA 3 1  148526  48641 545.6mb 272.8mb
green open channel      vRFQoIhmT8WmSbDCfph0ag 3 1   53374      0  88.4mb  44.2mb
green open channel_list 1dk8uH8bTeikez0lFR2mJg 3 1 5522172  78630    14gb     7gb
green open video        HNhyt9ioSEayAotGVXRCVg 2 1  798369 228155   3.3gb   1.6gb


升級方式2：徹底重啓集羣升級
即配置好全新的elasticsearch7.4.2集羣，而後把數據恢復到新集羣中

下載地址：wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.2-linux-x86_64.tar.gz

老版本的配置
# cms elasticsearch

[root@szyyelk01t opt]# egrep -v '^#|^$' elk-master/config/elasticsearch.yml 
cluster.name: cms-uat-elastic
node.name: master
path.data: /opt/elk-master/data/data01,/opt/elk-master/data/data02
path.logs: /opt/elk-master/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.90
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"

[root@szyyelk01t elk-slave]# egrep -v '^$|^#' slave01/config/elasticsearch.yml 
cluster.name: cms-uat-elastic
node.name: slave01
path.data: /opt/elk-slave/slave01/data/data01,/opt/elk-slave/slave01/data/data02
path.logs: /opt/elk-slave/slave01/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.90
http.port: 8200
discovery.zen.ping.unicast.hosts: ["10.10.18.90"]
http.cors.enabled: true
http.cors.allow-origin: "*"

[root@szyyelk01t elk-slave]# egrep -v '^$|^#' slave02/config/elasticsearch.yml 
cluster.name: cms-uat-elastic
node.name: slave02
path.data: /opt/elk-slave/slave02/data/data01,/opt/elk-slave/slave02/data/data02
path.logs: /opt/elk-slave/slave02/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.90
http.port: 8201
discovery.zen.ping.unicast.hosts: ["10.10.18.90"]
http.cors.enabled: true
http.cors.allow-origin: "*"


# 已經升級的線上配置參考：

[root@eus_filmora_db01:/usr/local/elasticsearch-7.4.1]# egrep -v '^$|^#' config/elasticsearch.yml
cluster.name: UOS_CLUSTER_ES
node.name: uos_node_1
path.data: /data/elasticsearch_data/data
path.logs: /data/elasticsearch_data/logs
bootstrap.memory_lock: true
network.host: 172.20.103.199
http.port: 9200
transport.tcp.port: 9300
node.master: true
node.data: true
discovery.seed_hosts: ["172.20.103.199:9300", "172.20.73.200:9300", "172.20.73.212:9300"]
cluster.initial_master_nodes: ["172.20.103.199", "172.20.73.200", "172.20.73.212"]
gateway.recover_after_nodes: 2
transport.tcp.compress: true
path.repo: ["/data/bak_es"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/local/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/local/elasticsearch/config/elastic-certificates.p12


############## cms 系統的 elasticsearch 6升級7.4.2
總體策略：
1.升級cms測試環境的es到7.4.2，而後作適配性的開發，再次升級內網的生產環境 --> cms海外環境(找海外業務不繁忙的時候操做，先和開發協商好)

測試環境其餘人依賴進行測試，因此升級須要兩套並存，新版本的es7.4.2使用自帶的openjdk13.0


1.配置新版本的elasticsearch使用指定的jdk環境
# vim bin/elasticsear

export JAVA_HOME=/opt/elk7_onenode/elasticsearch-7.4.2/jdk
export PATH=$JAVA_HOME/bin:$PATH

mkdir /opt/elk7_onenode/elasticsearch-7.4.2/data



# 主節點配置
[elastic@szyyelk01t elasticsearch-7.4.2]$ more config/elasticsearch.yml
cluster.name: cms-uat-elastic7
node.name: cms_node01
node.master: true
node.data: true
discovery.seed_hosts: ["10.10.18.90:19300", "10.10.18.117:19300"]
cluster.initial_master_nodes: ["10.10.18.90"]
path.data: /opt/cms_elk7/elasticsearch-7.4.2/data
path.logs: /opt/cms_elk7/elasticsearch-7.4.2/logs
discovery.zen.minimum_master_nodes: 1
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.90
http.cors.enabled: true
http.cors.allow-origin: "*"
transport.tcp.compress: true
path.repo: ["/opt/esback/"]
gateway.recover_after_nodes: 1

# 增長新的參數head插件能夠訪問es
http.port: 19200
transport.tcp.port: 19300

gateway.recover_after_time: 8m

# 如下配置能夠減小當es節點短期宕機或重啓時shards從新分佈帶來的磁盤io讀寫浪費
discovery.zen.fd.ping_timeout: 300s
discovery.zen.fd.ping_retries: 8
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 180s

# 啓用安全認證
xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12


# 第二個節點配置

[elastic@cms-test:/opt/cms_elk7/elasticsearch-7.4.2]$ more config/elasticsearch.yml
cluster.name: cms-uat-elastic7
node.name: cms_node02
node.master: false
node.data: true
discovery.seed_hosts: ["10.10.18.90:19300", "10.10.18.117:19300"]
cluster.initial_master_nodes: ["10.10.18.90"]
path.data: /opt/cms_elk7/elasticsearch-7.4.2/data
path.logs: /opt/cms_elk7/elasticsearch-7.4.2/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.117
http.cors.enabled: true
http.cors.allow-origin: "*"
transport.tcp.compress: true
path.repo: ["/opt/esback/"]
gateway.recover_after_nodes: 1

# 增長新的參數head插件能夠訪問es
http.port: 19200
transport.tcp.port: 19300

gateway.recover_after_time: 8m

# 如下配置能夠減小當es節點短期宕機或重啓時shards從新分佈帶來的磁盤io讀寫浪費
discovery.zen.fd.ping_timeout: 300s
discovery.zen.fd.ping_retries: 8
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 180s

# 啓用安全認證
xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12


# 設置密碼

# 啓用安全認證，只添加這個選項
xpack.security.enabled: true

#xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate 
#xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 
#xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

elastic密碼：
espass

在集羣上配置TLS:
若是你在操做單節點ES則能夠跳過本內容。


1.生成CA證書 :
bin/elasticsearch-certutil ca
將產生新文件 elastic-stack-ca.p12。該 elasticsearch-certutil 命令還會提示你輸入密碼以保護文件和密鑰，請保留該文件的副本並記住其密碼，此處咱們設置爲空

2.爲集羣中的每一個節點生成證書和私鑰 
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

將產生新文件 elastic-certificates.p12。系統還會提示你輸入密碼，你能夠輸入證書和密鑰的密碼，也能夠按Enter鍵將密碼留空。默認狀況下 elasticsearch-certutil 生成沒有主機名信息的證書，這意味着你能夠將證書用於集羣中的每一個節點，另外要關閉主機名驗證。
將 elastic-certificates.p12 文件複製到每一個節點上Elasticsearch配置目錄中
無需將 elastic-stack-ca.p12 文件複製到此目錄。
mkdir config/certs
mv elastic-certificates.p12 config/certs/

配置集羣中的每一個節點以使用其簽名證書標識自身並在傳輸層上啓用TLS
啓用TLS並指定訪問節點證書所需的信息，將如下信息添加到每一個節點的 elasticsearch.yml 文件中：

xpack.security.enabled: true


3.設置密碼

# 報錯
[elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive

Failed to determine the health of the cluster running at http://10.10.18.90:19200
Unexpected response code [503] from calling GET http://10.10.18.90:19200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

Do you want to continue with the password setup process [y/N]y

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y               


Enter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana]: 
Reenter password for [kibana]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 

Unexpected response code [503] from calling PUT http://10.10.18.90:19200/_security/user/apm_system/_password?pretty
Cause: Cluster state has not been recovered yet, cannot write to the [null] index

Possible next steps:
* Try running this tool again.
* Try running with the --verbose parameter for additional messages.
* Check the elasticsearch logs for additional error details.
* Use the change password API manually. 

ERROR: Failed to set password for user [apm_system].


[elastic@szyyelk01t elasticsearch-7.4.2]$  bin/elasticsearch-setup-passwords interactive

Connection failure to: http://10.10.18.90:19200/_security/_authenticate?pretty failed: Connection refused

ERROR: Failed to connect to elasticsearch at http://10.10.18.90:19200/_security/_authenticate?pretty. Is the URL correct and elasticsearch running?
[elastic@szyyelk01t elasticsearch-7.4.2]$  bin/elasticsearch-setup-passwords interactive

Failed to determine the health of the cluster running at http://10.10.18.90:19200
Unexpected response code [503] from calling GET http://10.10.18.90:19200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

Do you want to continue with the password setup process [y/N]^C[elastic@szyyelk01t elasticsearch-7.4.2]$  bin/elasticsearch-setup-passwords interactive

Failed to determine the health of the cluster running at http://10.10.18.90:19200
Unexpected response code [503] from calling GET http://10.10.18.90:19200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

Do you want to continue with the password setup process [y/N]


解決辦法：只配置一個主節點：cluster.initial_master_nodes: ["10.10.18.90"]


# 錯誤2處理

[2019-11-07T16:12:31,563][INFO ][o.e.c.c.JoinHelper       ] [cms_node02] failed to join {cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, xpack.installed=true, ml.max_open_jobs=20}, targetNode={cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [cms_node01][10.10.18.90:19300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalArgumentException: can't add node {cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, ml.max_open_jobs=20, xpack.installed=true}, found existing node {cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, xpack.installed=true, ml.max_open_jobs=20} with the same id but is a different node instance
    at org.elasticsearch.cluster.node.DiscoveryNodes$Builder.add(DiscoveryNodes.java:618) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.cluster.coordination.JoinTaskExecutor.execute(JoinTaskExecutor.java:147) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.cluster.coordination.JoinHelper$1.execute(JoinHelper.java:119) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.4.2.jar:7.4.2]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.4.2.jar:7.4.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]

# 拷貝連着目錄 都拷貝，刪除 data 下面的全部文件重啓便可


# 最終密碼配置成功

[elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana]: 
Reenter password for [kibana]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]


# 查看集羣狀態
[elastic@szyyelk01t elasticsearch-7.4.2]$ curl -H "Content-Type: application/json" -u elastic:espass http://10.10.18.90:19200/_cluster/health?pretty
{
  "cluster_name" : "cms-uat-elastic7",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}


# 查看剛纔建立全部索引的備份

# curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107
{"accepted":true}

# 恢復全索引快照

#保證elasticsearch用戶擁有快照目錄的權限
chown -R elastic.elastic /opt/esback

#建立倉庫
curl -H "Content-Type: application/json" -XPUT -u elastic:espass http://10.10.18.90:19200/_snapshot/backup -d '
{
"type":"fs",
"settings":{"location":"/opt/esback"}
}'

#查詢全索引快照備份
$ curl -XGET -u elastic:espass "http://10.10.18.90:19200/_snapshot/backup/_all" | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   700  100   700    0     0   160k      0 --:--:-- --:--:-- --:--:--  170k
{
    "snapshots": [
        {
            "duration_in_millis": 2891,
            "end_time": "2019-11-07T01:35:03.702Z",
            "end_time_in_millis": 1573090503702,
            "failures": [],
            "include_global_state": true,
            "indices": [
                "support_faq_categorys",
                "ticket_list",
                "templates_search",
                "site_page_search",
                "support",
                "templates_page_search",
                "support_new_articles",
                "article_version",
                "blocks_version",
                "search",
                "version",
                "article_search",
                "templates",
                "learn",
                "templates_version",
                "blocks_search",
                "templates_page_version"
            ],
            "shards": {
                "failed": 0,
                "successful": 71,
                "total": 71
            },
            "snapshot": "snapshot20191107",
            "start_time": "2019-11-07T01:35:00.811Z",
            "start_time_in_millis": 1573090500811,
            "state": "SUCCESS",
            "uuid": "0_4SOntVS1GH-7irHjKBMQ",
            "version": "6.3.2",
            "version_id": 6030299
        }
    ]
}


#恢復全索引快照
[elastic@szyyelk01t elasticsearch-7.4.2]$ curl -XPOST -u elastic:espass 'http://10.10.18.90:19200/_snapshot/backup/snapshot20191107/_restore'?wait_for_completion=true

{"snapshot":{"snapshot":"snapshot20191107","indices":["templates_page_search","article_search","blocks_version","learn","templates_page_version","templates","version","site_page_search","support_new_articles","support_faq_categorys","search","templates_search","blocks_search","ticket_list","article_version","support","templates_version"],"shards":{"total":71,"failed":0,"successful":71}}}

# 查看已經恢復成功
[elastic@szyyelk01t elasticsearch-7.4.2]$ curl -H "Content-Type: application/json" -u elastic:espass http://10.10.18.90:19200/_cat/indices
green open templates_page_search  tUKh1vaHRla6QamphIByLQ 5 1  104  10 965.3kb 482.6kb
green open article_search         _LE5n_-KRSGVH6Z3I1YLNQ 5 1   44   2   1.5mb 797.8kb
green open blocks_version         VRmv8fyESY6iclBYkhKJ_w 5 1    9   0 145.5kb  72.7kb
green open learn                  W4RyJnkrStaRJwQgS4MAug 3 1   89   1 841.6kb 420.8kb
green open templates_page_version _hHckKOfRuCPEojviySxVw 5 1  945   0   1.5mb   777kb
green open templates              7iJqDoBwTbOEHcyEzPLHbA 5 1  138   0     2mb     1mb
green open version                mLbfHoA7SAu4RWHSHM3vtw 3 1    1   0  39.9kb  19.9kb
green open support_new_articles   HvGe-CklRU-iua-_T1pLNA 3 1 1534 170    12mb     6mb
green open site_page_search       xxk8IetTSr2HF2tEe2Vc1w 5 1  516   2   1.5mb 817.2kb
green open .security-7            xdRnCeykQGGPcqM3-_WFCw 1 1    6   0  39.5kb  19.8kb
green open search                 fOteaZd0QfaU_2fKBaWPdA 3 1    0   0   1.5kb    783b
green open support_faq_categorys  h61nZp5bSQqV1UGVyHL7WA 3 1    0   0   1.5kb    783b
green open templates_search       ru8oFeQDTtKovOmkjP6A0w 5 1  111   3   1.5mb 802.8kb
green open blocks_search          8vMOY6ebTs-4iJIwM2VG0Q 5 1    0   0   2.5kb   1.2kb
green open article_version        qcF3Nft6QMezKqtPHyYLlA 5 1  344   0     5mb   2.5mb
green open ticket_list            xpvXuhlqRFq5Y_zugq0qKw 3 1  403   0   2.1mb     1mb
green open support                LypmJq0pRDy428-TKOy6Yg 3 1    0   0   1.5kb    783b
green open templates_version      gI28sYWJT3GVgfBeyJhSLg 5 1  220   0   4.2mb   2.1mb