Elasticsearch7.6開啓xpack以及java代碼配置客戶端

最近這兩天在研究es7.6的配置以及開啓xpack後用java代碼配置TransportClient、RestClient和JestClient，在這裏記錄一下。

1、安裝Elasticsearch7.6

1. 首先下載es

   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.1-linux-x86\_64.tar.gz

2. 下載後對文件進行解壓，而後修改配置文件
   咱們暫時修改一下幾個地方：

   cluster.name: es-7.6.1
   node.name: node-7.6.1
   network.host: 127.0.0.1
   http.port: 9200

   保存後，能夠去啓動es

   ./bin/elasticsearch

   若是遇到如下錯誤，須要配置一下他們三個裏邊任意一個屬性便可：

   at least one of [discovery.seed_hosts,discovery.seed_providers,cluster.initial_master_nodes] must be configured

   這裏咱們配置一下cluster.initial_master_nodes:

   cluster.initial_master_nodes: ["node-7.6.1"]

2、開啓xpack，配置transport安全驗證

1. 首先咱們先生成ca文件

   konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-certutil ca
    This tool assists you in the generation of X.509 certificates and certificate
    signing requests for use with SSL/TLS in the Elastic stack.
   
    The 'ca' mode generates a new 'certificate authority'
    This will create a new X.509 certificate and private key that can be used
    to sign certificate when running in 'cert' mode.
   
    Use the 'ca-dn' option if you wish to configure the 'distinguished name'
    of the certificate authority
   
    By default the 'ca' mode produces a single PKCS#12 output file which holds:
        *The CA certificate
        *The CA's private key
    If you elect to generate PEM format certificates (the -pem option), then the output will
    be a zip file containing individual files for the CA certificate and private key
    // 這裏會提示咱們輸入輸出文件的名稱，默認爲elastic-stack-ca.p12,不更改直接回車便可
    Please enter the desired output file [elastic-stack-ca.p12]:
    // 輸入之後使用elastic-stack-ca.p12文件的密碼，若是不須要密碼直接回車
    Enter password for elastic-stack-ca.p12 :

   通過此步驟後，會在當前目錄下生成elastic-stack-ca.p12文件。

2. 而後咱們根據ca文件生成certificates文件

   konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
    This tool assists you in the generation of X.509 certificates and certificate
    signing requests for use with SSL/TLS in the Elastic stack.
   
    The 'cert' mode generates X.509 certificate and private keys.
        *By default, this generates a single certificate and key for use
           on a single instance.
        *The '-multiple' option will prompt you to enter details for multiple
           instances and will generate a certificate and key for each one
        *The '-in' option allows for the certificate generation to be automated by describing
           the details of each instance in a YAML file
   
        *An instance is any piece of the Elastic Stack that requires an SSL certificate.
          Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
          may all require a certificate and private key.
        *The minimum required value for each instance is a name. This can simply be the
          hostname, which will be used as the Common Name of the certificate. A full
          distinguished name may also be used.
        *A filename value may be required for each instance. This is necessary when the
          name would result in an invalid file or directory name. The name provided here
          is used as the directory name (within the zip) and the prefix for the key and
          certificate files. The filename is required if you are prompted and the name
          is not displayed in the prompt.
        *IP addresses and DNS names are optional. Multiple values can be specified as a
          comma separated string. If no IP addresses or DNS names are provided, you may disable hostname verification in your SSL configuration.
        *All certificates generated by this tool will be signed by a certificate authority (CA).
        *The tool can automatically generate a new CA for you, or you can provide your own with the -ca or -ca-cert command line options.
    By default the 'cert' mode produces a single PKCS#12 output file which holds:
        *The instance certificate
        *The private key for the instance certificate
        *The CA certificate
    If you specify any of the following options:
        *-pem (PEM formatted output)
        *-keep-ca-key (retain generated CA key)
        *-multiple (generate multiple certificates)
        *-in (generate certificates from an input file)
    then the output will be be a zip file containing individual certificate/key files
    // 輸入使用ca文件的密碼，在上一步中設置，若是沒有設置直接回車
    Enter password for CA (elastic-stack-ca.p12) :
    // 輸入輸出certificates文件的名稱
    Please enter the desired output file [elastic-certificates.p12]:
    // 輸入之後使用certificates文件的密碼
    Enter password for elastic-certificates.p12 :
   
    Certificates written to /home/konghang/elasticsearch-7.6.1/elastic-certificates.p12
   
    This file should be properly secured as it contains the private key for
    your instance.
   
    This file is a self contained file and can be copied and used 'as is'
    For each Elastic product that you wish to configure, you should copy
    this '.p12' file to the relevant configuration directory
    and then follow the SSL configuration instructions in the product guide.
   
    For client applications, you may only need to copy the CA certificate and
    configure the client to trust this certificate.

   以後會在當前目錄下生成certificates文件。

3. 啓用xpack，配置transport的安全認證
   把第二節中生成的兩個文件都移動到es目錄的config配置目錄中，而後修改配置文件，在配置文件末尾添加以下配置：

   xpack.security.enabled: true
   xpack.security.transport.ssl.enabled: true
   xpack.security.transport.ssl.verification_mode: certificate
   xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
   xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

   而後重啓es便可。

4. 配置用戶密碼
   咱們開啓xpack之後，訪問es是須要用戶名和密碼的，因此咱們須要配置一下訪問的密碼，配置密碼有兩種方式，一種是自動生成的，自動生成各類用戶的密碼；另外一種是交互式的，咱們須要手動去輸入每一個用戶的密碼。

   • 自動生成

     konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-setup-passwords auto

   • 手動生成

     konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-setup-passwords interactive
     Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_moni
     toring_user.
     You will be prompted to enter passwords as the process progresses.
     // 咱們輸入y，而後輸入每一個用戶的密碼
     Please confirm that you would like to continue [y/N]y
     Enter password for [elastic]:
     Reenter password for [elastic]:
     Enter password for [apm_system]:
     Reenter password for [apm_system]:
     Enter password for [kibana]:
     Reenter password for [kibana]:
     Enter password for [logstash_system]:
     Reenter password for [logstash_system]:
     Enter password for [beats_system]:
     Reenter password for [beats_system]:
     Enter password for [remote_monitoring_user]:
     Reenter password for [remote_monitoring_user]:
     Changed password for user [apm_system]
     Changed password for user [kibana]
     Changed password for user [logstash_system]
     Changed password for user [beats_system]
     Changed password for user [remote_monitoring_user]
     Changed password for user [elastic]

5. java配置TransportClient
   通過上邊四步配置後，咱們就能夠去用java實現TransportClient來實現查詢了，詳細的java代碼以下：

   @Bean
    public TransportClient getTransportClient() throws UnknownHostException, URISyntaxException {
        logger.info("init TransportClient");
        String path = Paths.get(getClass().getClassLoader().getResource(elasticsearchProperties.getPkcsTransportFilePath()).toURI()).toString();
        TransportClient client = new PreBuiltXPackTransportClient(Settings.builder()
                .put("cluster.name", elasticsearchProperties.getClusterName())
                // missing authentication credentials for action
                .put("xpack.security.user", String.format("%s:%s", elasticsearchProperties.getUsername(), elasticsearchProperties.getPassword()))
                .put("xpack.security.transport.ssl.enabled", true)
                .put("xpack.security.transport.ssl.verification_mode", "certificate")
                .put("xpack.security.transport.ssl.keystore.path", path)
                .put("xpack.security.transport.ssl.truststore.path", path)
                .build()).addTransportAddress(new TransportAddress(InetAddress.getByName(elasticsearchProperties.getHost()), elasticsearchProperties.getTcpPort()));
        return client;
    }

   詳細參考官方文檔Java Client and security，不過因爲TransportClient已通過時，不建議使用，咱們這裏只是研究一下怎麼可以成功配置它。

配置加密客戶端HTTP通訊

該配置有兩種模式，一種是pem模式，一種是pkcs模式。參考官方文檔，經過命令bin/elasticsearch-certutil http

1. pem模式

   • 執行命令後會先詢問是否生成csr文件，此時選y

     konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-certutil http
      ##Elasticsearch HTTP Certificate Utility
     
      The 'http' command guides you through the process of generating certificates
      for use on the HTTP (Rest) interface for Elasticsearch.
     
      This tool will ask you a number of questions in order to generate the right
      set of files for your needs.
     
      ##Do you wish to generate a Certificate Signing Request (CSR)?
     
      A CSR is used when you want your certificate to be created by an existing
      Certificate Authority (CA) that you do not control (that is, you don't have
      access to the keys for that CA).
     
      If you are in a corporate environment with a central security team, then you
      may have an existing Corporate CA that can generate your certificate for you.
      Infrastructure within your organisation may already be configured to trust this
      CA, so it may be easier for clients to connect to Elasticsearch if you use a
      CSR and send that request to the team that controls your CA.
     
      If you choose not to generate a CSR, this tool will generate a new certificate
      for you. That certificate will be signed by a CA under your control. This is a
      quick and easy way to secure your cluster with TLS, but you will need to
      configure all your clients to trust that custom CA.
     
      Generate a CSR? [y/N]y

   • 是否爲每個節點生成certificates，選y

     ## Do you wish to generate one certificate per node?
     
      If you have multiple nodes in your cluster, then you may choose to generate a
      separate certificate for each of these nodes. Each certificate will have its
      own private key, and will be issued for a specific hostname or IP address.
     
      Alternatively, you may wish to generate a single certificate that is valid
      across all the hostnames or addresses in your cluster.
     
      If all of your nodes will be accessed through a single domain
      (e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
      simpler to generate one certificate with a wildcard hostname (*.es.example.com)
      and use that across all of your nodes.
     
      However, if you do not have a common domain name, and you expect to add
      additional nodes to your cluster in the future, then you should generate a
      certificate per node so that you can more easily generate new certificates when
      you provision new nodes.
     
      Generate a certificate per node? [y/N]y

   • 而後輸入節點名稱，下邊的hostnames和ip addresses默認爲空，enter進入下一步

     ## What is the name of node #1?
     You can use any descriptive name that you like, but we recommend using the name
     of the Elasticsearch node.
     node #1 name: node-7.6.1
     
     ## Which hostnames will be used to connect to node-7.6.1?
     Enter all the hostnames that you need, one per line.
     When you are done, press <ENTER> once more to move on to the next step.
     
     You did not enter any hostnames.
     Clients are likely to encounter TLS hostname verification errors if they
     connect to your cluster using a DNS name.
     Is this correct [Y/n]Y
     ## Which IP addresses will be used to connect to node-7.6.1?
     Enter all the IP addresses that you need, one per line.
     When you are done, press <ENTER> once more to move on to the next step.
     
     You did not enter any IP addresses.
     Is this correct [Y/n]Y

   • 而後是其餘配置，根據須要進行配置，這裏也要特別注意這個subject DN：CN=node-7，後續在java代碼初始化RestClient的時候用到的域名要和這裏相同，要否則會報錯。

     ## Other certificate options
     
     The generated certificate will have the following additional configuration
     values. These values have been selected based on a combination of the
     information you have provided above and secure defaults. You should not need to
     change these values unless you have specific requirements.
     
     Key Name: node-7.6.1
     Subject DN: CN=node-7, DC=6, DC=1
     Key Size: 2048
     # 是否要更改這些選項，根據須要進行更改，必定要注意上邊的CN值
     Do you wish to change any of these options? [y/N]N
     # 是否生成其餘證書，這裏根據須要選擇
     Generate additional certificates? [Y/n]n

     若是到時候RestClient配置的域名和這裏的CN值不一致，則會報如下錯誤IOException: Host name '127.0.0.1' does not match the certificate subject provided by the peer (CN=node7.example.com,DC-example, DC=com),若是在本地開發，記得改hosts。

   • 接下來根據須要設置密碼，以及生成的zip文件名稱

     ## What password do you want for your private key(s)?
     
     Your private key(s) will be stored as a PEM formatted file.
     We recommend that you protect your private keys with a password
     
     If you do not wish to use a password, simply press <enter> at the prompt below.
     # 若是須要密碼，則輸入密碼，不須要直接enter下一步
     Provide a password for the private key:  [<ENTER> for none]
     
     ## Where should we save the generated files?
     
     A number of files will be generated including your private key(s),
     certificate request(s), and sample configuration options for Elastic Stack products.
     
     These files will be included in a single zip archive.
     # 默認給了文件名稱，若是須要更改，則輸入你想要的名稱
     What filename should be used for the output zip file? [/home/konghang/elasticsearch-7.6.1/elasticsearch-ssl-http.zip]
     
     Zip file written to /home/konghang/elasticsearch-7.6.1/elasticsearch-ssl-http.zip

     壓縮包的elasticsearch目錄下有兩個重要文件，一個csr格式，另外一個key格式。

   • 利用key文件生成ca文件（ca.crt）

     konghang@Surface-Pro7:~/elasticsearch-7.6.1/config$ openssl req -new -x509 -key http-node-7.6.1.key -out ca.crt -days 3650
     Can't load /home/konghang/.rnd into RNG
     140055252832704:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/home/konghang/.rnd
     You are about to be asked to enter information that will be incorporated
     into your certificate request.
     What you are about to enter is what is called a Distinguished Name or a DN.
     There are quite a few fields but you can leave some blank
     For some fields there will be a default value,
     If you enter '.', the field will be left blank.
     # 如下信息根據本身需求填寫，暫時也沒有發現有什麼特殊需求，隨便填，沒有什麼影響。若是有人知道這裏有什麼特殊用途能夠告訴我
     -----
     Country Name (2 letter code) [AU]:zh
     State or Province Name (full name) [Some-State]:gd
     Locality Name (eg, city) []:gz
     Organization Name (eg, company) [Internet Widgits Pty Ltd]:snc
     Organizational Unit Name (eg, section) []:
     Common Name (e.g. server FQDN or YOUR name) []:node7.example.com
     Email Address []:

   • 而後再根據csr文件、ca文件、key文件生成須要的crt文件

     konghang@Surface-Pro7:~/elasticsearch-7.6.1/config$ openssl x509 -req -days 3650 -in http-node-7.6.1.csr -CA ca.crt -CAkey http-node-7.6.1.key -CAcreateserial -out http-node-7.6.1.crt
     Signature ok
     subject=DC = 1, DC = 6, CN = node7.example.com
     Getting CA Private Key

   • 而後移動http-node-7.6.1.crt和http-node-7.6.1.key到es的配置目錄config下，而後配置es配置文件，添加如下配置

     xpack.security.http.ssl.enabled: true
     xpack.security.http.ssl.certificate: http-node-7.6.1.crt
     xpack.security.http.ssl.key: http-node-7.6.1.key

     至此pem模式的es服務器端已經配置完成了，重啓便可生效。

2. pkcs模式

   • 一樣執行命令，只是在詢問是否生成csr文件的時候，選擇N

     ## Do you wish to generate a Certificate Signing Request (CSR)?
     
     A CSR is used when you want your certificate to be created by an existing
     Certificate Authority (CA) that you do not control (that is, you don't have
     access to the keys for that CA).
     
     If you are in a corporate environment with a central security team, then you
     may have an existing Corporate CA that can generate your certificate for you.
     Infrastructure within your organisation may already be configured to trust this
     CA, so it may be easier for clients to connect to Elasticsearch if you use a
     CSR and send that request to the team that controls your CA.
     
     If you choose not to generate a CSR, this tool will generate a new certificate
     for you. That certificate will be signed by a CA under your control. This is a
     quick and easy way to secure your cluster with TLS, but you will need to
     configure all your clients to trust that custom CA.
     
     Generate a CSR? [y/N]N

   • 而後就會讓你輸入一個現存在的ca文件路徑，ca文件用咱們在配置TransportClient的時候生成的，輸入後繼續，輸入ca文件的密碼

     ## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?
     
     If you have an existing CA certificate and key, then you can use that CA to
     sign your new http certificate. This allows you to use the same CA across
     multiple Elasticsearch clusters which can make it easier to configure clients,
     and may be easier for you to manage.
     
     If you do not have an existing CA, one will be generated for you.
     # 選擇使用現有的ca
     Use an existing CA? [y/N]y
     
     ## What is the path to your CA?
     
     Please enter the full pathname to the Certificate Authority that you wish to
     use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
     (.jks) or PEM (.crt, .key, .pem) format.
     # 輸入咱們的ca文件位置
     CA Path: /home/konghang/elasticsearch-7.6.1/config/elastic-stack-ca.p12
     Reading a PKCS12 keystore requires a password.
     It is possible for the keystore's password to be blank,
     in which case you can simply press <ENTER> at the prompt
     # 若是ca有密碼則輸入密碼，而後enter下一步
     Password for elastic-stack-ca.p12:

   • 輸入認證文件的有效時間

     ## How long should your certificates be valid?
     
     Every certificate has an expiry date. When the expiry date is reached clients
     will stop trusting your certificate and TLS connections will fail.
     
     Best practice suggests that you should either:
     (a) set this to a short duration (90 - 120 days) and have automatic processes
     to generate a new certificate before the old one expires, or
     (b) set it to a longer duration (3 - 5 years) and then perform a manual update
     a few months before it expires.
     
     You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)
     # 輸入須要的有效時間
     For how long should your certificate be valid? [5y]

   • 而後就和pem模式中從【是否爲每個節點生成certificates】開始往下邊3步，到【Other certificate options】配置CN。

   • 而後就是輸入private key的密碼，能夠爲空，而後就會生成一個名稱爲elasticsearch-ssl-http.zip的壓縮包，裏邊的elasticsearch文件下生成一個名稱爲http.p12的文件

     ## What password do you want for your private key(s)?
     
     Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
     This type of keystore is always password protected, but it is possible to use a
     blank password.
     
     If you wish to use a blank password, simply press <enter> at the prompt below.
     # 須要密碼的根據本身需求進行配置
     Provide a password for the "http.p12" file:  [<ENTER> for none]
     
     ## Where should we save the generated files?
     
     A number of files will be generated including your private key(s),
     public certificate(s), and sample configuration options for Elastic Stack products.
     
     These files will be included in a single zip archive.
     # 生成的文件名稱，自行配置
     What filename should be used for the output zip file? [/home/konghang/elasticsearch-7.6.1/elasticsearch-ssl-http.zip]
     
     Zip file written to /home/konghang/elasticsearch-7.6.1/elasticsearch-ssl-http.zip

     壓縮包裏的elasticsearch文件夾內在一個http.p12的文件,就是咱們須要的了。

   • 把http.p12移到es的配置目錄config下，而後在配置文件末尾添加如下配置:

     xpack.security.http.ssl.enabled: true
     xpack.security.http.ssl.keystore.path: http.p12

     至此pkcs模式的es服務器端已經配置完成了

java代碼實現RestCilent初始化

代碼編寫參考官方文檔Encrypted communication。代碼以下：

@Bean
// @ConditionalOnExpression("${es.properties.certificates_type}==1&&${es.properties.certificates_type:true}")
// @ConditionalOnExpression("elasticsearchProperties.getCertificatesType().equals('pkcs')")
@ConditionalOnExpression("'${es.properties.certificates_type}'.equals('pkcs')")
public RestClient getRestClient() throws CertificateException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
    logger.info("init RestClient, type:pkcs");
    RestClientBuilder builder = initRestClientBuilder("pkcs");
    RestClient restClient = builder.build();
    return restClient;
}

@Bean
@ConditionalOnExpression("'${es.properties.certificates_type}'.equals('pem')")
public RestClient getRestClientPem() throws CertificateException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
    logger.info("init RestClient, type:pem");
    RestClientBuilder builder = initRestClientBuilder(elasticsearchProperties.getCertificatesType());
    RestClient restClient = builder.build();
    return restClient;
}

private RestClientBuilder initRestClientBuilder(String type) throws KeyStoreException, CertificateException, NoSuchAlgorithmException, KeyManagementException {
    CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
    credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(elasticsearchProperties.getUsername(), elasticsearchProperties.getPassword()));
    KeyStore keyStore = getKeyStoreByType(type);
    SSLContextBuilder sslContextBuilder = SSLContexts.custom().loadTrustMaterial(keyStore, (x509Certificates, s) -> true);
    final SSLContext sslContext = sslContextBuilder.build();
    RestClientBuilder builder = RestClient.builder(new HttpHost(elasticsearchProperties.getHost(), elasticsearchProperties.getHttpPort(), elasticsearchProperties.getSchema()));
    builder.setHttpClientConfigCallback(httpClientBuilder -> {
        httpClientBuilder.setSSLContext(sslContext);
        httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
        return httpClientBuilder;
    });
    return builder;
}

private KeyStore getKeyStoreByType(String type) throws KeyStoreException, CertificateException, NoSuchAlgorithmException {
    KeyStore keyStore = null;
    if (elasticsearchProperties.getCertificatesType().equalsIgnoreCase(type)){
        keyStore = KeyStore.getInstance("pkcs12");
        try (InputStream is = this.getClass().getClassLoader().getResourceAsStream(elasticsearchProperties.getPkcsClientFilePath())) {
            keyStore.load(is, "".toCharArray());
        } catch (IOException e) {
            e.printStackTrace();
        } catch (CertificateException e) {
            e.printStackTrace();
        } catch (NoSuchAlgorithmException e) {
            e.printStackTrace();
        }
    } else {
        CertificateFactory factory = CertificateFactory.getInstance("X.509");
        Certificate trustedCa;
        try (InputStream is = this.getClass().getClassLoader().getResourceAsStream(elasticsearchProperties.getPemFilePath())) {
            trustedCa = factory.generateCertificate(is);
            keyStore = KeyStore.getInstance("pkcs12");
            keyStore.load(null, null);
            keyStore.setCertificateEntry("ca", trustedCa);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    return keyStore;
}

完整代碼github地址spring-boot-demo，eslasticsearch7.6模塊。裏邊還包括JestClient的pkcs模式配置。

結尾

整個過程如上所述，但願能夠幫助一些朋友。若是以上過程當中有任何不對的地方歡迎在評論區指正，很是感謝。