分佈式全文搜索引擎——Elasticsearch

時間 2019-11-09

標籤分佈式全文搜索引擎 elasticsearch 欄目系統架構简体版

原文原文鏈接

1.安裝Elasticsearchnode

　　a.下載：從官網下載 Elasticsearch，地址：https://www.elastic.co/cn/downloads/past-releases#elasticsearchgit

　　b.啓動：解壓後，在 bin 目錄打開 elasticsearch.bat 啓動服務github

　　c.訪問：在瀏覽器中訪問 http://127.0.0.1:9200/ 檢查是否安裝成功web

　　　　注意：端口9300：Java程序訪問的端口；端口9200：瀏覽器訪問的端口spring

　　　　　　　Elasticsearch 的版本須要跟後面的 Spring-Data-Elasticsearch 的版本相符，否則會不兼容（Spring 5.1.1 對應 Elasticsearch 6.8.0）
數據庫

2.安裝Head管理Elasticsearch插件npm

　　a.安裝nodejs：在nodejs官網下載msi，地址：https://nodejs.org/en/download/，安裝後在CMD命令行輸入 node -v 查看是否安裝成功json

　　b.安裝grunt：在CMD命令行輸入 npm install -g grunt-cli 安裝grunt，安裝後輸入 grunt -version 查看是否安裝成功瀏覽器

　　c.下載Head：在GitHub下載並解壓 elasticsearch-head-master，地址：https://github.com/mobz/elasticsearch-headspring-mvc

　　d.修改Head配置：修改Head根目錄下Gruntfile.js，添加：hostname:'*',

　　e.修改Elasticsearch配置：修改 Elasticsearch 的 config 目錄下的 elasticsearch.yml 文件，添加以下配置。並重啓

network.host: 127.0.0.1
# 解決elasticsearch-head 集羣健康值: 未鏈接問題
http.cors.enabled: true 
http.cors.allow-origin: "*"

　　f.安裝Head：在 elasticsearch-head-master 根目錄下執行命令 npm install 進行安裝

　　g.啓動Head：一樣在 elasticsearch-head-master 根目錄下執行命令 grunt server（或者npm run start）

　　h.訪問Head：在瀏覽器中訪問 http://localhost:9100/ 便可進入管理界面

3.安裝IK中文分詞器

　　a.下載：在GitHub上下載對應版本的IK分詞器zip包，地址：https://github.com/medcl/elasticsearch-analysis-ik/releases

　　b.注意：Elasticsearch和IK分詞器必須版本統一

　　c.解壓安裝：解壓到 elasticsearch 的 plugins 目錄下，並更名爲 ik。並重啓 elasticsearch

4.使用 Spring-Data-Elasticsearch 操做 Elasticsearch

　　a.導入pom依賴

 <properties>

    ......

    <!-- spring -->
    <spring.version>5.1.1.RELEASE</spring.version>

  </properties>

  <dependencies>
    <!-- elasticsearch -->
    <dependency>
      <groupId>org.springframework.data</groupId>
      <artifactId>spring-data-elasticsearch</artifactId>
      <version>3.1.1.RELEASE</version>
    </dependency>

    <!-- spring-5.X -->
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-core</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-webmvc</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-oxm</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-tx</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-aop</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-context-support</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-test</artifactId>
      <version>${spring.version}</version>
    </dependency>

    <!-- AOP-AspectJ spring-aop依賴 -->
    <dependency>
      <groupId>org.aspectj</groupId>
      <artifactId>aspectjrt</artifactId>
      <version>1.8.6</version>
    </dependency>
    <dependency>
      <groupId>org.aspectj</groupId>
      <artifactId>aspectjweaver</artifactId>
      <version>1.8.6</version>
    </dependency>

    <!-- jackson-json spring-mvc依賴 -->
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.9.4</version>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-core</artifactId>
      <version>2.9.4</version>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-annotations</artifactId>
      <version>2.9.4</version>
    </dependency>

    <!-- fastjson -->
    <dependency>
      <groupId>com.alibaba</groupId>
      <artifactId>fastjson</artifactId>
      <version>1.2.47</version>
    </dependency>

    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

　　b.建立配置文件

　　　　spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

    <!-- 自動掃描的包名 -->
    <context:component-scan base-package="com.wode" />

    <!-- 開啓AOP代理 -->
    <aop:aspectj-autoproxy proxy-target-class="true" />

    <!--開啓註解處理器 -->
    <context:annotation-config>
    </context:annotation-config>

    <context:property-placeholder location="classpath:elasticsearch.properties"/>
    <!-- Spring中引入其餘配置文件 -->
    <import resource="classpath*:/spring-elasticsearch.xml" />

</beans>

　　　　spring-elasticsearch.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/data/elasticsearch http://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch-1.0.xsd">

    <!-- 搜索DAO 掃描 -->
    <elasticsearch:repositories base-package="com.wode.dao" />

    <!-- 配置Client -->
    <elasticsearch:transport-client id="client" cluster-nodes="${elasticsearch.host}:${elasticsearch.port}"/>

    <!-- 配置搜索模板  -->
    <bean id="elasticsearchTemplate" class="org.springframework.data.elasticsearch.core.ElasticsearchTemplate">
        <constructor-arg name="client" ref="client" />
    </bean>

</beans>

　　　　elasticsearch.properties

elasticsearch.host=localhost
elasticsearch.port=9300

　　b.建立BaseEntity基礎實體類

public class BaseEntity {

    @Id
    private String id;

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

}

　　c.建立Article和Author實體類

//index至關於數據庫，type至關於表
@Document(indexName = "elasticsearch", type = "article")
public class Article extends BaseEntity {

    private String title;
    private String content;
    //內嵌對象
    @Field(type = FieldType.Nested)
    private Author author;

    public String getTitle() {
        return title;
    }

    public void setTitle(String title) {
        this.title = title;
    }

    public Author getAuthor() {
        return author;
    }

    public void setAuthor(Author author) {
        this.author = author;
    }

    public String getContent() {
        return content;
    }

    public void setContent(String content) {
        this.content = content;
    }

    @Override
    public String toString() {
        return "Article: id[" + this.getId() + "], title[" + title + "], content[" + content + "], author[" + author + "]";
    }
}

public class Author {

    private String name;

    private int year;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public int getYear() {
        return year;
    }

    public void setYear(int year) {
        this.year = year;
    }

    @Override
    public String toString() {
        return "Author: name[" + name + "], year[" + year + "]";
    }
}

　　d.建立BaseDao基礎DAO

public class BaseDao<T extends BaseEntity> {

    @Resource(name="elasticsearchTemplate")
    protected ElasticsearchTemplate esTemplate;
    protected Class<T> clazz;

    @PostConstruct
    private void construct(){
        clazz = (Class)((ParameterizedType)this.getClass().getGenericSuperclass()).getActualTypeArguments()[0];
    }

    /**
     * 保存單個對象
     * @param t 對象
     * @return
     */
    public boolean save(T t){
        String id = t.getId();
        if (id == null) {
            id = UUID.randomUUID().toString().replaceAll("-", "");
            t.setId(id);
        }
        IndexQuery indexQuery = new IndexQueryBuilder().withId(id).withObject(t).build();
        esTemplate.index(indexQuery);
        return true;
    }

    /**
     * 保存多個
     * @param list 對象集合
     * @return
     */
    public boolean save(List<T> list){
        List<IndexQuery> queries = new ArrayList<IndexQuery>();
        for (T t : list) {
            String id = t.getId();
            if (id == null) {
                id = UUID.randomUUID().toString().replaceAll("-", "");
                t.setId(id);
            }
            IndexQuery indexQuery = new IndexQueryBuilder().withId(id).withObject(t).build();
            queries.add(indexQuery);
        }
        esTemplate.bulkIndex(queries);
        return true;
    }

    /**
     * 根據ID刪除
     * @param id 對象ID
     * @return
     */
    public boolean deleteById(String id){
        esTemplate.delete(clazz, id);
        return true;
    }

    /**
     * 根據多個ID刪除
     * @param idList 對象ID集合
     * @return
     */
    public boolean deleteByIds(List<String> idList) {
        DeleteQuery deleteQuery = new DeleteQuery();
        Map<String, Object> filter = new HashMap<>();
        filter.put("ids", idList);
        BoolQueryBuilder queryBuilder = this.getQueryBuilder(filter, QueryBuilders.boolQuery());
        deleteQuery.setQuery(queryBuilder);;
        esTemplate.delete(deleteQuery, clazz);
        return true;
    }

    /**
     * 根據過濾條件刪除
     * @param filter 過濾條件Map
     * @return
     */
    public boolean delete(Map<String,Object> filter){
        DeleteQuery deleteQuery = new DeleteQuery();
        BoolQueryBuilder queryBuilder = this.getQueryBuilder(filter, QueryBuilders.boolQuery());
        deleteQuery.setQuery(queryBuilder);
        esTemplate.delete(deleteQuery, clazz);
        return true;
    }

    /**
     * 根據條件查詢集合
     * @param filter 過濾條件Map
     * @param highFields 高亮字段
     * @param sortField 排序字段
     * @param order 正序倒序
     * @return
     */
    public List<T> queryList(Map<String, Object> filter, final List<String> highFields, String sortField, SortOrder order) {
        NativeSearchQueryBuilder searchBuilder = new NativeSearchQueryBuilder();

        //-----------------------------高亮-----------------------------
        List<Field> fieldList = new ArrayList<>();
        if(highFields != null) {
            for (String highField : highFields) {
                fieldList.add(new HighlightBuilder.Field(highField).preTags("<em>").postTags("</em>").fragmentSize(250));
            }
        }
        searchBuilder.withHighlightFields(fieldList.toArray(new Field[fieldList.size()]));

        //-----------------------------排序-----------------------------
        if (sortField != null && order != null){
            searchBuilder.withSort(new FieldSortBuilder(sortField + ".keyword").order(order));
        }

        //-----------------------------過濾條件-----------------------------
        BoolQueryBuilder queryBuilder = this.getQueryBuilder(filter, QueryBuilders.boolQuery());
        searchBuilder.withQuery(queryBuilder);

        //-----------------------------查詢創建-----------------------------
        SearchQuery searchQuery = searchBuilder.build();
        Page<T> page = null;
        //若是設置高亮
        if (highFields != null && highFields.size() > 0) {
            page = esTemplate.queryForPage(searchQuery, clazz, new SearchResultMapper() {
                @Override
                public <T> AggregatedPage<T> mapResults(SearchResponse response,Class<T> clazz, Pageable pageable) {
                    List<T> list = new ArrayList<T>();
                    for (SearchHit searchHit : response.getHits()) {
                        if (response.getHits().getHits().length <= 0) {
                            return null;
                        }
                        Map<String, Object> entityMap = searchHit.getSourceAsMap();
                        for (String highName : highFields) {
                            String highValue = searchHit.getHighlightFields().get(highName).fragments()[0].toString();
                            entityMap.put(highName, highValue);
                        }
                        T t = JSONObject.parseObject(JSONObject.toJSONString(entityMap), clazz);
                        list.add(t);
                    }
                    if (list.size() > 0) {
                        return new AggregatedPageImpl<T>(list);
                    }
                    return null;
                }
            });
            //若是不設置高亮
        } else{
            page = esTemplate.queryForPage(searchQuery, clazz);
        }
        List<T> resultList = new ArrayList<>();
        if(page != null){
            resultList = page.getContent();
        }
        return resultList;
    }

    /**
     * 根據條件查詢分頁列表
     * @param filter 過濾條件Map
     * @param highFields 高亮字段
     * @param sortField 排序字段
     * @param order 正序倒序
     * @param pageIndex 當前頁
     * @param pageSize 分頁大小
     * @return
     */
    public Map<String, Object> queryPage(Map<String,Object> filter, final List<String> highFields, String sortField, SortOrder order, int pageIndex, int pageSize) {
        NativeSearchQueryBuilder searchBuilder = new NativeSearchQueryBuilder();

        //-----------------------------高亮-----------------------------
        List<Field> fieldList = new ArrayList<>();
        if(highFields != null) {
            for (String highField : highFields) {
                fieldList.add(new HighlightBuilder.Field(highField).preTags("<em>").postTags("</em>").fragmentSize(250));
            }
        }
        searchBuilder.withHighlightFields(fieldList.toArray(new Field[fieldList.size()]));

        //-----------------------------排序-----------------------------
        if (sortField != null && order != null){
            searchBuilder.withSort(new FieldSortBuilder(sortField + ".keyword").order(order));
        }

        //-----------------------------分頁-----------------------------
        searchBuilder.withPageable(PageRequest.of(pageIndex, pageSize));

        //-----------------------------過濾條件-----------------------------
        BoolQueryBuilder queryBuilder = this.getQueryBuilder(filter, QueryBuilders.boolQuery());
        searchBuilder.withQuery(queryBuilder);

        //-----------------------------查詢創建-----------------------------
        SearchQuery searchQuery = searchBuilder.build();
        Page<T> page = null;
        //若是設置高亮
        if (highFields != null && highFields.size() > 0) {
            page = esTemplate.queryForPage(searchQuery, clazz, new SearchResultMapper() {
                @Override
                public <T> AggregatedPage<T> mapResults(SearchResponse response,Class<T> clazz, Pageable pageable) {
                    List<T> list = new ArrayList<T>();
                    for (SearchHit searchHit : response.getHits()) {
                        if (response.getHits().getHits().length <= 0) {
                            return null;
                        }
                        Map<String, Object> entityMap = searchHit.getSourceAsMap();
                        for (String highName : highFields) {
                            String highValue = searchHit.getHighlightFields().get(highName).fragments()[0].toString();
                            entityMap.put(highName, highValue);
                        }
                        T t = JSONObject.parseObject(JSONObject.toJSONString(entityMap), clazz);
                        list.add(t);
                    }
                    if (list.size() > 0) {
                        return new AggregatedPageImpl<T>(list);
                    }
                    return null;
                }
            });
            //若是不設置高亮
        } else{
            page = esTemplate.queryForPage(searchQuery, clazz);
        }

        List<T> resultList = new ArrayList<>();
        if(page != null){
            resultList = page.getContent();
        }

        //-----------------------------查詢數據總條數-----------------------------
        searchQuery = new NativeSearchQueryBuilder().withQuery(queryBuilder).build();
        long totalCount = esTemplate.count(searchQuery, clazz);

        //組裝結果
        Map<String, Object> resultMap = new HashMap<>();
        resultMap.put("pageIndex", pageIndex);
        resultMap.put("pageSize", pageSize);
        resultMap.put("data", resultList);
        resultMap.put("totalCount", totalCount);
        return resultMap;
    }

    /**
     * 根據過濾條件獲取BoolQueryBuilder
     * @param filter 過濾條件
     * @param query BoolQueryBuilder
     * @return
     */
    protected BoolQueryBuilder getQueryBuilder(Map<String, Object> filter, BoolQueryBuilder query){
        if(query == null){
            query = QueryBuilders.boolQuery();
        }
        if(CollectionUtils.isEmpty(filter)){
            return query;
        }
        for(Map.Entry<String, Object> entry : filter.entrySet()){
            String key = entry.getKey();
            Object value = entry.getValue();
            switch (key){
                case "id":
                    query = query.must(QueryBuilders.matchQuery("id.keyword", value));
                    break;
                case "ids":
                    List<String> idList = (List<String>) value;
                    for (String id : idList) {
                        query.should(QueryBuilders.matchQuery("id.keyword", id));
                    }
                    break;
                case "idNot":
                    query = query.mustNot(QueryBuilders.matchQuery("id.keyword", value));
                default:
                    break;
            }
        }
        return query;
    }

}

　　e.建立 ArticleDao 繼承 BaseDao

@Repository
public class ArticleDao extends BaseDao<Article> {

    @Override
    protected BoolQueryBuilder getQueryBuilder(Map<String, Object> filter, BoolQueryBuilder query) {
        query = super.getQueryBuilder(filter, query);
        for(Map.Entry<String, Object> entry : filter.entrySet()){
            switch (entry.getKey()){
                case "contentMatch":
                    query = query.must(QueryBuilders.matchQuery("content", entry.getValue()));
                    break;
                case "title":
                    // ".keyword" 表示不進行分詞（精準查詢），解決 "title" 分詞後 term 查不到數據的問題
                    query = query.must(QueryBuilders.matchQuery("title.keyword", entry.getValue()));
                    break;
                case "authorYearBt":
                    query = query.must(QueryBuilders.rangeQuery("author.year").gt(entry.getValue()));
                default:
                    break;
            }
        }
        return super.getQueryBuilder(filter, query);
    }
}