elasticsearch搭建並經過go-mysql-elasticsearch同步db數據達到搜索引擎的目的

logstash-input-jdbc/elasticsearch-jdbc缺點：刪除記錄沒有辦法同步，只能兩邊執行本身的刪除命令，版本16年後未更新。
go-mysql-elasticsearch缺點：國內的不是很穩定
elasticsearch-river-MYSQL 版本太舊，12年以後未更新。

es安裝：

下載

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.6.zip

解壓

unzip elasticsearch-5.6.6.zip

移動到/usr/local目錄

mv elasticsearch-5.6.6 /usr/local/elsearch

按需求修改config/elasticsearch.yml文件

vim /usr/local/elsearch/config/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /usr/local/elsearch/data
#
# Path to log files:
#
path.logs: /usr/local/elsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 172.21.0.15
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
http.cors.enabled: true
http.cors.allow-origin: "*"

安裝中文分詞設置

cd /usr/local/elsearch/
./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.6.6/elasticsearch-analysis-ik-5.6.6.zip

測試

curl -X PUT 'localhost:9200/accounts' -d '
{
  "mappings": {
    "person": {
      "properties": {
        "user": {
          "type": "text",
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_max_word"
        },
        "title": {
          "type": "text",
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_max_word"
        },
        "desc": {
          "type": "text",
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_max_word"
        }
      }
    }
  }
}'

參考連接 http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html

es-head安裝

下載

wget https://github.com/mobz/elasticsearch-head/archive/master.zip

解壓

unzip master.zip

放到/usr/local/下

mv elasticsearch-head-master /usr/local/elsearch-head

安裝node
於head插件本質上仍是一個nodejs的工程，所以須要安裝node，使用npm來安裝依賴的包

yum -y install gcc make gcc-c++ openssl-devel wget
wget http://nodejs.org/dist/v0.10.26/node-v0.10.26.tar.gz
tar -zvxf node-v0.10.26.tar.gz
cd node-v0.10.26
make && make install
驗證是否安裝配置成功：
node -v

安裝grunt
grunt是一個很方便的構建工具，能夠進行打包壓縮、測試、執行等等的工做，5.6裏的head插件就是經過grunt啓動的。所以須要安裝一下grunt：

cd /usr/local/elsearch-head
npm install -g grunt-cli  //執行後會生成node_modules文件夾
npm install

修改vim Gruntfile.js文件：增長hostname屬性，設置爲*

connect: {
    server: {
        options: {
            port: 9100,
            hostname: '*',
            base: '.',
            keepalive: true
        }
    }
}

修改 vim _site/app.js 文件：修改head的鏈接地址:

this.base_uri = this.config.base_uri || this.prefs.get(「app-base_uri」) || 「http://192.168.0.153:9200「;

啓動

cd elsearch-head
grunt server &  #後臺啓動

安裝go-mysql-elasticsearch進行同步

yum install go
go get github.com/tools/godep
go get github.com/siddontang/go-mysql-elasticsearch
cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch 
make

修改配置文件

vim /root/go/src/github.com/siddontang/go-mysql-elasticsearch/etc/river.toml

# MySQL address, user and password
# user must have replication privilege in MySQL.
my_addr = "172.21.0.15:3308"#mysql地址
my_user = "root"#用戶名
my_pass = "123456"#密碼
# Elasticsearch address
es_addr = "172.21.0.15:9200"#es安裝地址
# Path to store data, like master.info, and dump MySQL data
data_dir = "./var"
# Inner Http status address
stat_addr = "172.21.0.15:12800"
# pseudo server id like a slave
server_id = 1
# mysql or mariadb
flavor = "mysql"#mysql數據庫
# mysqldump execution path
# if not set or empty, ignore mysqldump.
mysqldump = "mysqldump"
# MySQL data source
[[source]]
schema = "test"#數據庫名稱
# Only below tables will be synced into Elasticsearch.
# "test_river_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023
# I don't think it is necessary to sync all tables in a database.
tables = ["travelrecord10","travelrecord4","travelrecord5","travelrecord1"]#數據表名稱，能夠多個
# Below is for special rule mapping
#[[rule]]
#schema = "test"
#table = "test_river"
#index = "river"
#type = "river"
# only MySQL field in filter will be synced
#filter=["title","tags"]
# title is MySQL test_river field name, es_title is the customized name in Elasticsearch
#[rule.field]
# This will map column title to elastic search my_title
#title="es_title"
# This will map column tags to elastic search my_tags and use array type
#tags="my_tags,list"
# This will map column keywords to elastic search keywords and use array type
#keywords=",list"
# wildcard table rule, the wildcard table must be in source tables
[[rule]]
schema = "test"#數據庫名稱
table = "travelrecord10"#數據表名稱
index = ""#生成es數據索引名稱，對應schema
type = ""#生成es數據類型，對應table

#[[rule]]
#schema = "test"#數據庫名稱
#table = "travelrecord8"#數據表名稱
#index = ""#生成es數據索引名稱，對應schema
#type = ""#生成es數據類型，對應table
# title is MySQL test_river field name, es_title is the customized name in Elasticsearch
[[rule.fields]]
mysql = "title"
elastic = "es_title"

運行

cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch 
./bin/go-mysql-elasticsearch -config=./etc/river.toml  >  /home/heron/wwwlogs/go-mysql.log 2>&1 &

測試：

db增長數據，查看頁面是否顯示。




curl -XPOST -u elastic '10.168.2.10:9200/_xpack/security/user/demo' -d '{   "password" : "123456",   "full_name" : " elastic",   "email" : 「elastic@163.com",  "roles" : [ "clicks_admin" ] }'

不穩定現況，會本身掛掉，暫時沒找到解決辦法，只能掛掉當即重啓