Spring boot整合ElasticSearch案例分享

時間 2019-11-08

標籤 spring boot 整合 elasticsearch 案例分享欄目 Spring 简体版

原文原文鏈接

The best elasticsearch highlevel java rest api-----bboss java

Spring boot整合ElasticSearch HighLevel Rest Client案例分享，本文涉及內容mysql

集成bboss es starter
單es集羣整合
多es集羣整合

本文內容適合於:git

spring boot 1.x,2.x
elasticsearch 1.x,2.x,5.x,6.x,+

1.集成bboss es starter

在spring boot項目中導入bboss es starter github

maven工程spring

<dependency>
            <groupId>com.bbossgroups.plugins</groupId>
            <artifactId>bboss-elasticsearch-spring-boot-starter</artifactId>
            <version>5.6.2</version>
        </dependency>

gradle工程sql

compile "com.bbossgroups.plugins:bboss-elasticsearch-spring-boot-starter:5.6.2"

2.建立spring boot啓動類

新建Application類：shell

package org.bboss.elasticsearchtest.springboot;


import org.frameworkset.elasticsearch.ElasticSearchHelper;
import org.frameworkset.elasticsearch.client.ClientInterface;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

/**
 *  @author yinbp [122054810@qq.com]
 *  
 */

@SpringBootApplication

public class Application {

    private Logger logger = LoggerFactory.getLogger(Application.class);

   
    public static void main(String[] args) {

        SpringApplication.run(Application.class, args);
    }
}

Application類中定義了一個main方法用來啓動spring boot測試應用。數據庫

Application類將被用於啓動下面兩個測試用例：express

單es集羣測試用例
多es集羣測試用例

3.單es集羣配置和使用

3.1 配置單es集羣

bboss es starter配置單集羣能夠採用properties文件也能夠採用yml進行配置，兩者任選其一。apache

3.1.1 application.properties

修改spring boot配置文件application.properties內容

##ES集羣配置，支持x-pack和searchguard
spring.elasticsearch.bboss.elasticUser=elastic
spring.elasticsearch.bboss.elasticPassword=changeme


spring.elasticsearch.bboss.elasticsearch.rest.hostNames=10.21.20.168:9200
#spring.elasticsearch.bboss.elasticsearch.rest.hostNames=10.180.211.27:9280,10.180.211.27:9281,10.180.211.27:9282
##https配置，添加https://協議頭
#spring.elasticsearch.bboss.default.elasticsearch.rest.hostNames=https://10.180.211.27:9280,https://10.180.211.27:9281,https://10.180.211.27:9282
spring.elasticsearch.bboss.elasticsearch.dateFormat=yyyy.MM.dd
spring.elasticsearch.bboss.elasticsearch.timeZone=Asia/Shanghai
spring.elasticsearch.bboss.elasticsearch.ttl=2d
#在控制檯輸出腳本調試開關showTemplate,false關閉，true打開，同時log4j至少是info級別
spring.elasticsearch.bboss.elasticsearch.showTemplate=true
spring.elasticsearch.bboss.elasticsearch.discoverHost=false
# dsl配置文件熱加載掃描時間間隔，毫秒爲單位，默認5秒掃描一次，<= 0時關閉掃描機制
spring.elasticsearch.bboss.dslfile.refreshInterval = -1

##es client http鏈接池配置
spring.elasticsearch.bboss.http.timeoutConnection = 5000
spring.elasticsearch.bboss.http.timeoutSocket = 5000
spring.elasticsearch.bboss.http.connectionRequestTimeout=5000
spring.elasticsearch.bboss.http.retryTime = 1
spring.elasticsearch.bboss.http.maxLineLength = -1
spring.elasticsearch.bboss.http.maxHeaderCount = 200
spring.elasticsearch.bboss.http.maxTotal = 400
spring.elasticsearch.bboss.http.defaultMaxPerRoute = 200
spring.elasticsearch.bboss.http.soReuseAddress = false
spring.elasticsearch.bboss.http.soKeepAlive = false
spring.elasticsearch.bboss.http.timeToLive = 3600000
spring.elasticsearch.bboss.http.keepAlive = 3600000
spring.elasticsearch.bboss.http.keystore =
spring.elasticsearch.bboss.http.keyPassword =
# ssl 主機名稱校驗，是否採用default配置，
# 若是指定爲default，就採用DefaultHostnameVerifier,不然採用 SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER
spring.elasticsearch.bboss.http.hostnameVerifier =

## 數據庫數據源配置，使用db-es數據導入功能時須要配置
#spring.elasticsearch.bboss.db.name = test
#spring.elasticsearch.bboss.db.user = root
#spring.elasticsearch.bboss.db.password = 123456
#spring.elasticsearch.bboss.db.driver = com.mysql.jdbc.Driver
#spring.elasticsearch.bboss.db.url = jdbc:mysql://localhost:3306/bboss
#spring.elasticsearch.bboss.db.usePool = false
#spring.elasticsearch.bboss.db.validateSQL = select 1

單ES集羣配置項都是以spring.elasticsearch.bboss開頭。

3.1.2 application.yml

若是採用application.yml配置，內容以下

spring:
  elasticsearch:
    bboss:
      elasticUser: elastic
      elasticPassword: changeme
      elasticsearch:
        rest:
          hostNames: 192.168.8.25:9200
          ##hostNames: 192.168.8.25:9200,192.168.8.26:9200,192.168.8.27:9200  ##集羣地址配置
        dateFormat: yyyy.MM.dd
        timeZone: Asia/Shanghai
        ttl: 2d
        showTemplate: true
        discoverHost: false
      dslfile:
        refreshInterval: -1
      http:
         timeoutConnection: 5000
         timeoutSocket: 5000
         connectionRequestTimeout: 5000
         retryTime: 1
         maxLineLength: -1
         maxHeaderCount: 200
         maxTotal: 400
         defaultMaxPerRoute: 200
         soReuseAddress: false
         soKeepAlive: false
         timeToLive: 3600000
         keepAlive: 3600000
         keystore:
         keyPassword:
         hostnameVerifier:

3.2 單集羣測試用例

編寫es單集羣測試用例BBossESStarterTestCase

/*
 * Copyright 1999-2018 Alibaba Group Holding Ltd.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.bboss.elasticsearchtest.springboot;


import org.frameworkset.elasticsearch.client.ClientInterface;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

/**
 * 單集羣演示功能測試用例，spring boot配置項以spring.elasticsearch.bboss開頭
 * 對應的配置文件爲application.properties文件
 * @author  yinbp [122054810@qq.com]
 */
@RunWith(SpringRunner.class)
@SpringBootTest(classes = Application.class)
public class BBossESStarterTestCase {
	@Autowired
	private BBossESStarter bbossESStarter;
	@Autowired
	DocumentCRUD documentCRUD;

    @Test
    public void testBbossESStarter() throws Exception {
//        System.out.println(bbossESStarter);

		//驗證環境,獲取es狀態
//		String response = serviceApiUtil.restClient().executeHttp("_cluster/state?pretty",ClientInterface.HTTP_GET);

//		System.out.println(response);
		//判斷索引類型是否存在，false表示不存在，正常返回true表示存在
		boolean exist = bbossESStarter.getRestClient().existIndiceType("twitter","tweet");

		//判讀索引是否存在，false表示不存在，正常返回true表示存在
		exist =  bbossESStarter.getRestClient().existIndice("twitter");

		exist =  bbossESStarter.getRestClient().existIndice("agentinfo");

    }

    @Test
	public void testCRUD() throws Exception {

		//刪除/建立文檔索引表
		documentCRUD.testCreateIndice();
		//添加/修改單個文檔

		documentCRUD.testAddAndUpdateDocument();
		//批量添加文檔
		documentCRUD.testBulkAddDocument();
		//檢索文檔
		documentCRUD.testSearch();
		//批量修改文檔
		documentCRUD.testBulkUpdateDocument();

		//檢索批量修改後的文檔
		documentCRUD.testSearch();
		//帶list複雜參數的文檔檢索操做
		documentCRUD.testSearchArray();
		//帶from/size分頁操做的文檔檢索操做
		documentCRUD.testPagineSearch();
		//帶sourcefilter的文檔檢索操做
		documentCRUD.testSearchSourceFilter();

		documentCRUD.updateDemoIndice();
		documentCRUD.testBulkAddDocuments();
	}

	@Test
	public void testPerformaceCRUD() throws Exception {

		//刪除/建立文檔索引表
		documentCRUD.testCreateIndice();

		documentCRUD.testBulkAddDocuments();
	}

}

直接經過junit運行上述測試用例便可。

其中

BBossESStarter：由bboss提供，直接在代碼中聲明引用，並使用便可

DocumentCRUD:各類增刪改查操做實例，在demo工程中提供

4.多ES集羣測試用例

4.1 配置多es集羣

bboss es starter配置多集羣能夠採用properties文件也能夠採用yml進行配置，兩者任選其一。

4.1.1properties配置

修改spring boot配置文件application-multi-datasource.properties，內容以下：

##多集羣配置樣例，若是須要作多集羣配置，請將參照本文內容修改application.properties文件內容
spring.elasticsearch.bboss.default.name = default
##default集羣配配置
spring.elasticsearch.bboss.default.elasticUser=elastic
spring.elasticsearch.bboss.default.elasticPassword=changeme


spring.elasticsearch.bboss.default.elasticsearch.rest.hostNames=10.21.20.168:9200
#spring.elasticsearch.bboss.default.elasticsearch.rest.hostNames=10.180.211.27:9280,10.180.211.27:9281,10.180.211.27:9282
spring.elasticsearch.bboss.default.elasticsearch.dateFormat=yyyy.MM.dd
spring.elasticsearch.bboss.default.elasticsearch.timeZone=Asia/Shanghai
spring.elasticsearch.bboss.default.elasticsearch.ttl=2d
#在控制檯輸出腳本調試開關showTemplate,false關閉，true打開，同時log4j至少是info級別
spring.elasticsearch.bboss.default.elasticsearch.showTemplate=true
spring.elasticsearch.bboss.default.elasticsearch.discoverHost=false

##default鏈接池配置
spring.elasticsearch.bboss.default.http.timeoutConnection = 5000
spring.elasticsearch.bboss.default.http.timeoutSocket = 5000
spring.elasticsearch.bboss.default.http.connectionRequestTimeout=5000
spring.elasticsearch.bboss.default.http.retryTime = 1
spring.elasticsearch.bboss.default.http.maxLineLength = -1
spring.elasticsearch.bboss.default.http.maxHeaderCount = 200
spring.elasticsearch.bboss.default.http.maxTotal = 400
spring.elasticsearch.bboss.default.http.defaultMaxPerRoute = 200
spring.elasticsearch.bboss.default.http.keystore =
spring.elasticsearch.bboss.default.http.keyPassword =
# ssl 主機名稱校驗，是否採用default配置，
# 若是指定爲default，就採用DefaultHostnameVerifier,不然採用 SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER
spring.elasticsearch.bboss.default.http.hostnameVerifier =

##logs集羣配置
spring.elasticsearch.bboss.logs.name = logs
spring.elasticsearch.bboss.logs.elasticUser=elastic
spring.elasticsearch.bboss.logs.elasticPassword=changeme


spring.elasticsearch.bboss.logs.elasticsearch.rest.hostNames=127.0.0.1:9200

#spring.elasticsearch.bboss.default.elasticsearch.rest.hostNames=10.180.211.27:9280,10.180.211.27:9281,10.180.211.27:9282
spring.elasticsearch.bboss.logs.elasticsearch.dateFormat=yyyy.MM.dd
spring.elasticsearch.bboss.logs.elasticsearch.timeZone=Asia/Shanghai
spring.elasticsearch.bboss.logs.elasticsearch.ttl=2d
#在控制檯輸出腳本調試開關showTemplate,false關閉，true打開，同時log4j至少是info級別
spring.elasticsearch.bboss.logs.elasticsearch.showTemplate=true
spring.elasticsearch.bboss.logs.elasticsearch.discoverHost=false

##logs集羣對應的鏈接池配置
spring.elasticsearch.bboss.logs.http.timeoutConnection = 5000
spring.elasticsearch.bboss.logs.http.timeoutSocket = 5000
spring.elasticsearch.bboss.logs.http.connectionRequestTimeout=5000
spring.elasticsearch.bboss.logs.http.retryTime = 1
spring.elasticsearch.bboss.logs.http.maxLineLength = -1
spring.elasticsearch.bboss.logs.http.maxHeaderCount = 200
spring.elasticsearch.bboss.logs.http.maxTotal = 400
spring.elasticsearch.bboss.logs.http.defaultMaxPerRoute = 200
# https證書配置
spring.elasticsearch.bboss.logs.http.keystore =
spring.elasticsearch.bboss.logs.http.keyPassword =
# ssl 主機名稱校驗，是否採用default配置，
# 若是指定爲default，就採用DefaultHostnameVerifier,不然採用 SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER
spring.elasticsearch.bboss.logs.http.hostnameVerifier =
# dsl配置文件熱加載掃描時間間隔，毫秒爲單位，默認5秒掃描一次，<= 0時關閉掃描機制
spring.elasticsearch.bboss.dslfile.refreshInterval = -1

配置說明：

上面配置了兩個集羣：default和logs

每一個集羣配置項的前綴爲：spring.elasticsearch.bboss.集羣名字，其中的集羣名字是一個自定義的邏輯名稱，用來在client api中引用集羣。

注意：這裏的集羣名字是根據集羣的做用命名的名字，和Elasticsearch自己的cluster.name沒有關係，可自行根據須要命名

default集羣的配置項前綴爲：

spring.elasticsearch.bboss.default

logs集羣的配置項前綴爲：

spring.elasticsearch.bboss.logs

同時每一個集羣的配置項目裏面必須包含name項目的配置

default集羣name配置：

spring.elasticsearch.bboss.default.name = default

logs集羣name配置：

##logs集羣配置
spring.elasticsearch.bboss.logs.name = logs

4.1.2 yml配置（默認）

spring:
  elasticsearch:
    bboss:
      default:
        name: default
        elasticUser: elastic
        elasticPassword: changeme
        elasticsearch:
          rest:
            hostNames: 127.0.0.1:9200
            ##hostNames: 127.0.0.1:9200,127.0.0.1:9201,127.0.0.1:9202 
          dateFormat: yyyy.MM.dd
          timeZone: Asia/Shanghai
          ttl: 2d
          showTemplate: true
          discoverHost: false
        dslfile:
          refreshInterval: -1
        http:
           timeoutConnection: 5000
           timeoutSocket: 5000
           connectionRequestTimeout: 5000
           retryTime: 1
           maxLineLength: -1
           maxHeaderCount: 200
           maxTotal: 400
           defaultMaxPerRoute: 200
           soReuseAddress: false
           soKeepAlive: false
           timeToLive: 3600000
           keepAlive: 3600000
           keystore:
           keyPassword:
           hostnameVerifier:
      logs:
          name: logs
          elasticUser: elastic
          elasticPassword: changeme
          elasticsearch:
            rest:
              hostNames: 127.0.0.1:9200
            dateFormat: yyyy.MM.dd
            timeZone: Asia/Shanghai
            ttl: 2d
            showTemplate: true
            discoverHost: false
          dslfile:
            refreshInterval: -1
          http:
             timeoutConnection: 5000
             timeoutSocket: 5000
             connectionRequestTimeout: 5000
             retryTime: 1
             maxLineLength: -1
             maxHeaderCount: 200
             maxTotal: 400
             defaultMaxPerRoute: 200
             soReuseAddress: false
             soKeepAlive: false
             timeToLive: 3600000
             keepAlive: 3600000
             keystore:
             keyPassword:
             hostnameVerifier:

4.2 定義加載多es集羣配置的spring boot Configuration類

新建類MultiESSTartConfigurer，聲明分別對應兩個不一樣集羣的bboss工廠組件

package org.bboss.elasticsearchtest.springboot;
/*
 *  Copyright 2008 biaoping.yin
 *
 *  Licensed under the Apache License, Version 2.0 (the "License");
 *  you may not use this file except in compliance with the License.
 *  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software
 *  distributed under the License is distributed on an "AS IS" BASIS,
 *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 *  See the License for the specific language governing permissions and
 *  limitations under the License.
 */

import org.frameworkset.elasticsearch.ElasticSearchHelper;
import org.frameworkset.elasticsearch.boot.BBossESStarter;
import org.frameworkset.elasticsearch.client.ClientInterface;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.context.annotation.Profile;

/**
 * 配置多個es集羣
 * 指定多es數據源profile：multi-datasource
 */
@Configuration
public class MultiESSTartConfigurer {
	@Primary
	@Bean(initMethod = "start")
	@ConfigurationProperties("spring.elasticsearch.bboss.default")
	public BBossESStarter bbossESStarterDefault(){
		return new BBossESStarter();

	}

	@Bean(initMethod = "start")
	@ConfigurationProperties("spring.elasticsearch.bboss.logs")
	public BBossESStarter bbossESStarterLogs(){
		return new BBossESStarter();

	}

}

說明：

MultiESSTartConfigurer經過如下兩個方法分別加載default和logs兩個es集羣的配置

default集羣配置加載

@Primary
@Bean(initMethod = "start")
@ConfigurationProperties("spring.elasticsearch.bboss.default")
public BBossESStarter bbossESStarterDefault()

logs集羣配置加載

@Bean(initMethod = "start")
@ConfigurationProperties("spring.elasticsearch.bboss.logs")
public BBossESStarter bbossESStarterLogs()

BBossESStarter bbossESStarterDefault 對應spring.elasticsearch.bboss.default配置的elasticsearch集羣

BBossESStarter bbossESStarterLogs 對應spring.elasticsearch.bboss.logs配置的elasticsearch集羣

兩個組件的聲明都是必須的，在程序中只要用其中任意一個均可以獲取到兩個對應集羣的ClientInterface組件，具體看後面的示例。

4.3 定義多es集羣測試用例

多es集羣測試用例MultiBBossESStartersTestCase

/*
 * Copyright 1999-2018 Alibaba Group Holding Ltd.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.bboss.elasticsearchtest.springboot;


import org.frameworkset.elasticsearch.client.ClientInterface;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.context.junit4.SpringRunner;

/**
 * 多集羣演示功能測試用例，spring boot配置項以spring.elasticsearch.bboss.集羣名稱開頭，例如：
 * spring.elasticsearch.bboss.default 默認es集羣
 * spring.elasticsearch.bboss.logs  logs es集羣
 * 兩個集羣經過 org.bboss.elasticsearchtest.springboot.MultiESSTartConfigurer加載
 * 對應的配置文件爲application-multi-datasource.properties文件
 * 經過ActiveProfiles指定並激活多es集羣配置：multi-datasource 
 * @author yinbp [122054810@qq.com]
 */
@RunWith(SpringRunner.class)
@SpringBootTest(classes = Application.class)
@ActiveProfiles("multi-datasource")
public class MultiBBossESStartersTestCase {
	@Autowired
	private BBossESStarter bbossESStarterDefault;
	@Autowired
	MultiESDocumentCRUD multiESDocumentCRUD;
    @Test
    public void testMultiBBossESStarters() throws Exception {

		//驗證環境,獲取es狀態
//		String response = bbossESStarterDefault.getRestClient().executeHttp("_cluster/state?pretty",ClientInterface.HTTP_GET);
//		System.out.println(response);


		//判斷索引類型是否存在，false表示不存在，正常返回true表示存在
		boolean exist = bbossESStarterDefault.getRestClient().existIndiceType("twitter","tweet");
		System.out.println("default twitter/tweet:"+exist);
		//獲取logs對應的Elasticsearch集羣客戶端，並進行existIndiceType操做
		exist = bbossESStarterDefault.getRestClient("logs").existIndiceType("twitter","tweet");
		System.out.println("logs twitter/tweet:"+exist);
		//獲取logs對應的Elasticsearch集羣客戶端，判讀索引是否存在，false表示不存在，正常返回true表示存在
		exist = bbossESStarterDefault.getRestClient("logs").existIndice("twitter");
		System.out.println("logs  twitter:"+exist);
		//獲取logs對應的Elasticsearch集羣客戶端，判斷索引是否認義
		exist = bbossESStarterDefault.getRestClient("logs").existIndice("agentinfo");
		System.out.println("logs agentinfo:"+exist);
    }

	@Test
	public void testCRUD() throws Exception {

		//刪除/建立文檔索引表
		multiESDocumentCRUD.testCreateIndice();
		//添加/修改單個文檔

		multiESDocumentCRUD.testAddAndUpdateDocument();
		//批量添加文檔
		multiESDocumentCRUD.testBulkAddDocument();
		//檢索文檔
		multiESDocumentCRUD.testSearch();
		//批量修改文檔
		multiESDocumentCRUD.testBulkUpdateDocument();

		//檢索批量修改後的文檔
		multiESDocumentCRUD.testSearch();
		//帶list複雜參數的文檔檢索操做
		multiESDocumentCRUD.testSearchArray();
		//帶from/size分頁操做的文檔檢索操做
		multiESDocumentCRUD.testPagineSearch();
		//帶sourcefilter的文檔檢索操做
		multiESDocumentCRUD.testSearchSourceFilter();

		multiESDocumentCRUD.updateDemoIndice();
		multiESDocumentCRUD.testBulkAddDocuments();
	}

	@Test
	public void testPerformaceCRUD() throws Exception {

		//刪除/建立文檔索引表
		multiESDocumentCRUD.testCreateIndice();

		multiESDocumentCRUD.testBulkAddDocuments();
	}
}

直接經過junit運行上述測試用例便可。

其中

BBossESStarter bbossESStarterDefault：實現由bboss提供，在直接在代碼中MultiESSTartConfigurer 定義

MultiESDocumentCRUD:各類增刪改查操做實例，在demo工程中提供

5.完整的demo工程

https://gitee.com/bboss/eshelloword-spring-boot-starter

https://github.com/bbossgroups/eshelloword-spring-boot-starter

6 開發交流

elasticsearch交流Q：166471282

elasticsearch：

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。