搭建 ELK 問題排查

時間 2019-12-12

標籤搭建 elk 問題排查简体版

原文原文鏈接

最近在電腦上開了三個虛擬機鼓搗了一下 ELK，配置成功以後，關閉虛擬機從新打開老是出現一些奇怪的問題，使得 kibana 處於不健康的狀態，真是讓人操碎了心。html

1、前言

在搭建過程當中，本人是依據如下兩篇文章進行的，步驟明確，效果清晰。java

一、搭建ELK日誌分析平臺（上）—— ELK介紹及搭建 Elasticsearch 分佈式集羣node

二、搭建ELK日誌分析平臺（下）—— 搭建kibana和logstash服務器linux

如下記錄本人在實現過程當中遇到的問題以及最終解決的思路。bash

2、elasticsearch 集羣狀態不健康

一、問題描述

elasticsearch （如下簡稱 es）集羣狀態處於 yellow 或者 red 狀態，2 個數據節點未成功接入主節點，number_of_nodes 數量仍爲 1，kibana 界面報錯 503。服務器

[root@server ~]# curl '192.168.100.15:9200/_cluster/health?pretty'
{
  "cluster_name" : "server-node",
  "status" : "red",             # 爲 green 則表明健康沒問題，若是是 yellow 或者 red 則是集羣有問題
  "timed_out" : false,          # 是否有超時
  "number_of_nodes" : 1,        # 集羣中的節點數量
  "number_of_data_nodes" : 0,   # 集羣中data節點的數量
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 12,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 0.0          # 可用性百分比，此處爲 0 不可用
}

二、排查思路

1）首先確保 es 主節點最早啓動，隨後啓動數據節點；curl

2）容許 selinux（非必要），關閉 iptables；elasticsearch

3）確保數據節點的配置文件正確。分佈式

三、排查後狀態

[root@server ~]# curl '192.168.100.15:9200/_cluster/health?pretty'        
{
  "cluster_name" : "server-node",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

3、索引狀態不健康

一、問題描述

es 啓動正常，kibana 報錯 503，management 頁面無顯示，點擊無反應，查看索引狀態爲 red。fetch

[root@server ~]# curl '192.168.100.15:9200/_cat/indices?v'
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
red    open   system-syslog-2018.09 JPDsnK_qSym-sjOiZS9zAw   5   1        548            0    719.6kb        345.3kb

二、排查思路

1）確認 logstash 是否正常啓動，端口（9600以及各日誌索引配置端口）是否存在；

2）刪除不正常索引，從新啓動 logstash；

[root@server ~]# curl -XDELETE http://localhost:9200/system-syslog-2018.09
{"acknowledged":true}

3）確認 kibana 狀態

[root@server ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2018-09-29 09:01:44 CST; 31min ago
 Main PID: 646 (node)
   CGroup: /system.slice/kibana.service
           └─646 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml...

Sep 29 09:01:44 server systemd[1]: Started Kibana.
Sep 29 09:01:44 server systemd[1]: Starting Kibana...

4）從新查看索引狀態

三、排查後狀態

[root@server ~]# curl '192.168.100.15:9200/_cat/indices?v'                
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   system-syslog-2018.09 TR_gdOb8RDSRtHj_g4a4_g   5   1          3            0     62.2kb         31.1kb

4、es 的 node 日誌報錯

一、問題描述

kibana 界面報錯，菜單點擊無反應，日誌信息部分以下。

[2018-09-28T21:23:20,487][DEBUG][o.e.a.s.TransportSearchAction] [server] All shards failed for phase: [query]
[2018-09-28T21:23:20,488][WARN ][r.suppressed             ] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:210) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:189) [elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.1.jar:6.4.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
[2018-09-28T21:23:20,487][WARN ][r.suppressed             ] path: /.kibana/doc/config%3A6.4.1, params: {index=.kibana, id=config:6.4.1, type=doc}
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][doc][config:6.4.1]: routing [null]]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:207) ~[elasticsearch-6.4.1.jar:6.4.1]

二、排查思路

1）刪除索引

[root@server ~]# curl -XDELETE http://localhost:9200/.kibana
{"acknowledged":true}

2）在 kibana 界面開啓或關閉

三、排查後狀態

[root@server ~]# curl '192.168.100.15:9200/_cat/indices?v'        
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   system-syslog-2018.09 TR_gdOb8RDSRtHj_g4a4_g   5   1         20            0    240.6kb        120.3kb
green  open   .kibana               GGWwf7gdTwCKMn3BqRaGcQ   1   1          2            0       22kb           11kb