elasticsearh6.3.2+kibana-6.3.2+logstash-6.3.2+ik
1.文件準備（elasticsearch的jdk須要1.8以上）html
去官網下載最新的elasticsearch-6.3.2和kibana-6.3.2java
https://www.elastic.co/downloadsnode
2.下載完成後，上傳至linux，解壓到指定目錄。mysql
tar -zxvf /opt/elasticserach-6.3.2 -C /var/opt/
3.修改elasticsearch的配置文件linux
cd elasticsearch-6.3.2/config vim elasticsearh.yml cluster.name: elasticsearch-6 node.name: node-1 network.host: 0.0.0.0 http.port: 9200
4.直接root用戶啓動會報錯，6.0後直接修改配置文件，用root用戶啓動已經不行了，因此須要建立一個其餘用戶來啓動elasticsearch。git
#建立elasticsearch用戶組 groupadd elasticsearch useradd es -g elasticsearch-p /var/opt/elasticsearch-6.3.2 #更改elasticsearch文件夾及內部文件的所屬用戶及組爲es:elasticsearch cd /var/opt chown -R es:elasticsearch elasticsearch-6.3.2 #切換到elsearch用戶再啓動 su es cd elasticsearch-6.3.2/bin ./elasticsearh
5.啓動發現報錯：github
1.max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
2.max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]sql
修改配置文件：json
#修改第一個錯誤 vim /etc/security/limits.conf # 在最後面追加下面內容 es hard nofile 65536 es soft nofile 65536 #es即爲啓動用戶的名稱 ---修改第二個錯誤 #切換到root用戶修改配置sysctl.conf vi /etc/sysctl.conf #添加下面配置： vm.max_map_count=655360 #並執行命令： sysctl -p
參考文檔：vim
https://blog.csdn.net/jiankunking/article/details/65448030
https://www.cnblogs.com/yidiandhappy/p/7714481.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/zip-targz.html
kibana-6.3.2安裝
#解壓至指定目錄 tar -zxvf kibana-6.3.2-linux-x86_64.tar.gz -C /var/opt/ #修改配置config/kibana.yml cd /var/opt/kibana-6.3.2-linux-x86_64/config vim kibana.yml #端口 server.port: 5601 #主機 server.host: "172.16.20.11" #es的地址 elasticsearch.url: "http://172.16.20.11:9200" #kibana在es中的索引 kibana.index: ".kibana" #啓動kibana #前臺啓動 ./kibana #後臺啓動 nohup ./kibana &
參考文檔：
https://my.oschina.net/shea1992/blog/1925414
安裝ik分詞器elasticsearch-analysis-ik-6.3.2
下載地址：https://github.com/medcl/elasticsearch-analysis-ik/releases
須要下載對應的的elasticseach的版本的分詞器，而後直接解壓zip到es的plugins目錄中
#建立ik分詞器的目錄
cd /var/opt/elaticsearch-6.3.2/plugins/
mkdir ik
#解壓至指定目錄
unzip /var/opt/elasticsearch-analysis-ik-6.3.2.zip -d /var/opt/elaticsearch-6.3.2/plugins/ik
#重啓es
#在kibana中測試分詞器是否加載成功
#點擊Dev_Tools
GET _analyze?pretty
{
"analyzer":"ik_smart",
"text":"我愛北京天安門"
}
#顯示以下
{
"tokens": [
{
"token": "我",
"start_offset": 0,
"end_offset": 1,
"type": "CN_CHAR",
"position": 0
},
{
"token": "愛",
"start_offset": 1,
"end_offset": 2,
"type": "CN_CHAR",
"position": 1
},
{
"token": "北京",
"start_offset": 2,
"end_offset": 4,
"type": "CN_WORD",
"position": 2
},
{
"token": "天安門",
"start_offset": 4,
"end_offset": 7,
"type": "CN_WORD",
"position": 3
}
]
}
logstash-6.3.2安裝
下載logstash-6.3.2
https://www.elastic.co/downloads/logstash
#解壓loastash至指定文件 tar -zxvf logstash-6.3.2.tar.gz -C /var/opt/ #進入到bin目錄下安裝jdbc插件：用於將mysql數據同步到elasticsearch ./logstash-plugin install logstash-input-jdbc
再下載mysql的連接工具：
https://dev.mysql.com/downloads/connector/j/
而後放入到指定目錄--/var/opt/
從mysql導入數據到elasticsearch經過logstash
1.首先建立索引---經過kibana的Div_Tools
PUT hisms_ns
{
"mappings": {
"t_kc21k1": {
"properties": {
"id": {
"type": "text",
"index": "false"
},
"aac003": {
"type": "text",
"index": "false"
},
"yka002": {
"type": "text",
"index": "false"
},
"yka003": {
"type": "text",
"index": "false"
},
"ykd018": {
"type": "text",
"index": "false"
},
"ykd018_name": {
"type": "text",
"index": "false"
},
"akc190": {
"type": "text",
"index": "false"
},
"aac001": {
"type": "text",
"index": "false"
},
"akc194": {
"type": "text",
"format": "dateOptionalTime"
},
"yka055": {
"type": "double"
},
"money_sum": {
"type": "double"
},
"id_drg": {
"type": "text",
"index": "false"
},
"name_drg": {
"type": "text",
"index": "false"
},
"id_mdc": {
"type": "text",
"index": "false"
},
"name_mdc": {
"type": "text",
"index": "false"
},
"if_ss": {
"type": "short"
},
"if_death": {
"type": "short"
},
"zysc": {
"type": "short"
},
"sq_zysc": {
"type": "short"
},
"if_ry_cy_fh": {
"type": "short"
},
"if_zdfx_death": {
"type": "short"
},
"if_zyhz": {
"type": "short"
},
"if_yngr": {
"type": "short"
},
"if_sscf": {
"type": "short"
},
"yp_fee": {
"type": "double"
},
"if_tskss": {
"type": "short"
},
"if_3tskss": {
"type": "short"
},
"ghc_fee": {
"type": "double"
},
"jc_fee": {
"type": "double"
},
"oneself_fee": {
"type": "double"
},
"if_34_ss": {
"type": "short"
},
"id_depa": {
"type": "text",
"index": "false"
},
"id_doctor_org": {
"type": "text",
"index": "false"
},
"zyys_org": {
"type": "text",
"index": "false"
},
"if_zzy": {
"type": "short"
},
"if_tzzy": {
"type": "short"
},
"if_op10_death": {
"type": "short"
},
"if_bl_lc_fh": {
"type": "short"
},
"if_yx_bl_fh": {
"type": "short"
},
"yjqk_yh_sum": {
"type": "short"
},
"yjqk_sum": {
"type": "short"
},
"bazl": {
"type": "short"
},
"rescue_true": {
"type": "short"
},
"rescue": {
"type": "short"
},
"zhylfwf": {
"type": "double"
},
"syszdf": {
"type": "double"
},
"yxxzdf": {
"type": "double"
},
"sszlf": {
"type": "double"
},
"fsszlf": {
"type": "double"
},
"xyf": {
"type": "double"
},
"kjywf": {
"type": "double"
},
"zcyjzyf": {
"type": "double"
},
"hcf": {
"type": "double"
},
"qtf": {
"type": "double"
},
"zdbm": {
"type": "text",
"index": "false"
},
"zdmc": {
"type": "text",
"index": "false"
},
"ssbm": {
"type": "text",
"index": "false"
},
"ssmc": {
"type": "text",
"index": "false"
},
"group_status": {
"type": "text",
"index": "false"
},
"drg_error_msg": {
"type": "text",
"index": "false"
},
"bz_name": {
"type": "text",
"index": "false"
},
"bz_id": {
"type": "text",
"index": "false"
},
"belong_ks": {
"type": "text",
"index": "false"
},
"id_doctor_add": {
"type": "text",
"index": "false"
},
"zyys_add": {
"type": "text",
"index": "false"
},
"id_doctor": {
"type": "text",
"index": "false"
},
"zyys": {
"type": "text",
"index": "false"
}
}
}
}
}'
#安裝完後，而後在進入到config的目錄下，
#建立配置文件jdbc.conf（本身取名便可
vim jdbc.conf
#輸入如下內容：
input {
jdbc {
jdbc_driver_library => "/var/opt/mysql-connector-java-8.0.11.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://10.111.121.236:3306/hisms_ns?serverTimezone=GMT%2B8"
jdbc_user => "root"
jdbc_password => "root"
schedule => "* * * * *"
statement => "SELECT * from t_kc21k1"
}
}
output {
elasticsearch {
hosts => [ "172.16.20.11:9200" ]
index => "hisms_ns"
document_type => "t_kc21k1"
document_id => "%{id}"
}
stdout {
codec => json_lines
}
}
##參數解析
index => "hisms_ns"#指定導入的索引
document_type => "t_kc21k1"#指定導入的該索引下的type
##若是要將不一樣數據導入不一樣索引，能夠參考以下：
input {
jdbc {
jdbc_driver_library => "/var/opt/mysql-connector-java-8.0.11.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://10.111.121.236:3306/hisms_ns?serverTimezone=GMT%2B8"
jdbc_user => "root"
jdbc_password => "root"
schedule => "* * * * *"
statement => "SELECT * from t_kc21k1"
type => "01"
}
jdbc {
jdbc_driver_library => "/var/opt/mysql-connector-java-8.0.11.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://10.111.121.236:3306/hisms_ns?serverTimezone=GMT%2B8"
jdbc_user => "root"
jdbc_password => "root"
schedule => "* * * * *"
statement => "SELECT * from t_kc21k1"
type => "02"
}
}
output {
if [type] == "01" {
elasticsearch {
hosts => [ "172.16.20.11:9200" ]
index => "hisms_ns"
document_type => "t_kc21k1"
document_id => "%{id}"
}
stdout {
codec => line {
format => "Crawl: %{id} %{title}"
}
}
} else {
elasticsearch {
hosts => [ "172.16.20.11:9200" ]
index => "hisms_ns"
document_type => "t_kc21k1_2"
document_id => "%{id}"
}
stdout {
codec => line {
format => "VMS: %{id} %{title}"
}
}
}
}
最後執行導入命令便可：
./bin/logstash -f ./config/jdbc.conf --config.reload.automatic
或者
./bin/logstash -f ./config/jdbc.conf
參考文檔：
https://blog.csdn.net/chenjianhuideyueding/article/details/80864345
https://blog.csdn.net/lilongsy/article/details/78283122
https://elasticsearch.cn/question/3753
補充：kibana中的索引問題
1.經過logstash導入數據後kibana不能顯示出字段問題
解決方法：
logstash導入數據後能夠在kibana看見數據，此時點擊management，再點擊index patterns，最後點擊右上角的刷新按鈕，刷新全部字段便可顯示。
2.針對string字段不能使用Aggregatabel問題。
case1：不手動建立索引映射的狀況
直接經過logstash自動導入，此時string字段是能夠被聚合的，默認自帶keyword標籤。
case2：須要手動建立索引映射的狀況
在建立索引的時候，添加字段描述，方式以下：
"yka003": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. 在windows下的虛擬機中,安裝華爲電腦的deepin操作系統
- 2. 強烈推薦款下載不限速解析神器
- 3. 【區塊鏈技術】孫宇晨：區塊鏈技術帶來金融服務的信任變革
- 4. 搜索引起的鏈接分析-計算網頁的重要性
- 5. TiDB x 微衆銀行 | 耗時降低 58%，分佈式架構助力實現普惠金融
- 6. 《數字孿生體技術白皮書》重磅發佈（附完整版下載）
- 7. 雙十一「避坑」指南：區塊鏈電子合同爲電商交易保駕護航！
- 8. 區塊鏈產業，怎樣「鏈」住未來？
- 9. OpenglRipper使用教程
- 10. springcloud請求一次好用一次不好用zuul Name or service not known
歡迎關注本站公眾號,獲取更多信息
