快速搭建應用服務日誌收集系統(Filebeat + ElasticSearch + kibana)

時間 2019-11-11

標籤快速搭建應用服務日誌收集系統 filebeat elasticsearch kibana 欄目日誌分析简体版

原文原文鏈接

快速搭建應用服務日誌收集系統(Filebeat + ElasticSearch + kibana)

概要說明html

需求場景，系統環境是CentOS，多個應用部署在多臺服務器上，平時查看應用日誌及排查問題十分不變。索性搭建一個服務器日誌收集系統，因爲每日日誌規模僅在G級別，全部前期暫先不搭建集羣。 
技術方案是 Filebeat + ElasticSearch + kibana (日誌服務器上安裝ElasticSearch,Kibana,其餘應用服務器上安裝Filebeat); 沒有考慮加上Logstash,Flume，Kafka，Redis等，一是Filebeat比較輕量級，佔用資源少，且可直接將日誌輸出到elasticsearch,僅是方便查看線上服務日誌；不須要對日誌字段進行解析，因此不打算引入技術太多，增長複雜性；沒用Redis,緣由僅是當前已有Redis集羣，但主要是交易系統使用，不想增長其額外風險。
另外，當前的選用的日誌服務器內存等配置不高。不適合安裝過多軟件，僅此而已。


也可參考我以前寫的另外一篇博客：
Windows下ELK-5.4.3環境搭建 http://www.cnblogs.com/huligong1234/p/7108109.html

1、安裝ElasticSearch

1.安裝JDK8環境

[root@app-001 src]# cd /usr/local/src/
[root@app-001 src]# rpm -qa | grep jdk
java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el6_8.x86_64
[root@app-001 src]# rpm -e java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el6_8.x86_64
[root@app-001 src]# curl -L -O http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.rpm?AuthParam=1506173332_32b98eb52c8955419974ec3efcba2209
[root@app-001 src]# rpm -ivh jdk-8u144-linux-x64.rpm
[root@app-001 src]# java -version

2.安裝ElasticSearch

[root@app-001 src]# curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.1.rpm
[root@app-001 src]# rpm -ivh elasticsearch-5.6.1.rpm
[root@app-001 src]# chkconfig --add elasticsearch

安裝支持中文分詞插件
[root@app-001 src]# /usr/share/elasticsearch/bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.6.1/elasticsearch-analysis-ik-5.6.1.zip

3.配置ElasticSearch

[root@app-001 src]# vi /etc/elasticsearch/elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /data/elasticsearch/data
#
# Path to log files:
#
#path.logs: /data/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true

#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.1.106
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "*"
#http.cors.allow-origin: "esmgr.domain.com"

注意：若是CentOS版本低於7.0還須要進行以下配置調整java

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

若是非內網訪問，network.host需改爲以下配置：node

network.host: 0.0.0.0

4.啓動ElasticSearch

[root@app-001 src]# service elasticsearch start
[root@app-001 src]# curl "http://192.168.1.234:9200" #查看啓動狀況

5.防火牆放開iptables 9200端口，容許內網其餘機器訪問

[root@app-001 src]# vi /etc/sysconfig/iptables
增長以下內容：
-A INPUT -s 192.168.1.0/24 -p tcp -m state --state NEW -m tcp --dport 9200 -j ACCEPT

[root@app-001 src]# service iptables restart

2、安裝Filebeat

1.下載

[root@app-001 src]# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.1-x86_64.rpm

2.安裝

[root@app-001 src]# rpm -ivh filebeat-5.6.1-x86_64.rpm
[root@app-001 src]# chkconfig --add filebeat

3.配置

[root@app-001 src]# vi /etc/filebeat/filebeat.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /opt/tomcat-myapp/logs/myapp.log
    - /data/production/tomcat-myapp/logs/catalina.out
    #- c:\programdata\elasticsearch\logs\*
  fields_under_root: true
  fields:
    log_type: myapp
  tags: ["myapp","tomcat-log"]
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ["^ERR", "^WARN"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after
  encoding: utf-8
  
- input_type: log
  paths:
    - /opt/tomcat-apiserver/logs/apiserver.log
    - /data/production/tomcat-apiserver/logs/catalina.out
  fields_under_root: true
  fields:
    log_type: apiserver
  tags: ["tomcat-log"]
  encoding: utf-8
  exclude_lines: ["^DBG"]
  multiline.pattern: ^\[
  multiline.match: after
  
- input_type: log
  paths:
    - /usr/local/tengine/logs/error.log
  fields_under_root: true
  fields:
    log_type: nginx-error
  tags: ["nginx-log"]
  encoding: utf-8
  
- input_type: log
  paths:
    - /var/log/*.log
  fields_under_root: true
  fields:
    log_type: system
  tags: ["system-log"]
  encoding: utf-8
#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
fields:
    log_host: ip-106
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.1.106:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

4.啓動

[root@app-001 src]# /usr/bin/filebeat.sh -configtest
[root@app-001 src]# service filebeat start

[root@app-001 src]# service filebeat status #檢查狀態

[root@app-001 src]# tail -f /var/log/filebeat/filebeat

3、安裝kibana

1.下載

[root@app-001 src]# curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-5.6.1-x86_64.rpm

2.安裝

[root@app-001 src]# rpm -ivh kibana-5.6.1-x86_64.rpm
[root@app-001 src]# chkconfig --add kibana

3.配置

[root@app-001 src]# vi /etc/kibana/kibana.yml

server.port: 5601
server.host: "192.168.1.106"
elasticsearch.url: "http://192.168.1.106:9200"

4.啓動

[root@app-001 src]# service kibana start

瀏覽器訪問 http://192.168.1.106:5601/

4、安裝插件

1.elasticsearch-head

1.1.安裝NodeJS環境

[root@app-001 src]# curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -
[root@app-001 src]# yum install -y nodejs
[root@app-001 src]# node -v 
[root@app-001 src]# npm -v

1.2.下載 elasticsearch-head

[root@app-001 src]# wget https://codeload.github.com/mobz/elasticsearch-head/zip/master
[root@app-001 src]# unzip master

1.3.修改配置

1.3.1.修改elasticsearch.yml，增長跨域的配置(須要重啓es才能生效)
http.cors.enabled: true
http.cors.allow-origin: "*"

1.3.2.編輯elasticsearch-head/Gruntfile.js，修改服務器監聽地址，connect節點增長hostname屬性，將其值設置爲*

connect: {
    server: {
        options: {
            hostname:'*',
            port: 9100,
            base: '.',
            keepalive: true
        }
    }
}

1.3.3.編輯elasticsearch-head/_site/app.js,
修改默認es地址http://localhost:9200/爲http://192.168.1.106:9200/。

1.4.啓動

[root@app-001 src]# npm run start
瀏覽器訪問 http://192.168.1.106:9100/

2.bigdesk

http://bigdesk.org/

5、使用Basic Auth給ElasticSearch和Kibana配置訪問權限

1.建立密碼

[root@app-001 src]# htpasswd -c /usr/local/tengine/db/passwd.db loguser

2.修改Nginx配置文件nginx.conf

[root@app-001 src]# vi /usr/local/tengine/conf/nginx.conflinux

server {
            listen          80;
            server_name     esmgr.domain.com;
            auth_basic "basic auth esmgr";
            auth_basic_user_file /usr/local/nginx/db/passwd.db;
            location / { 
                proxy_pass        http://192.168.1.106:9200; 
            }
            location /head/ {
                proxy_pass    http://192.168.1.106:9100/; 
            }
        }

3.更新Nginx生效

[root@app-001 src]# /usr/local/tengine/sbin/nginx -t
nginx: the configuration file /usr/local/tengine/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/tengine/conf/nginx.conf test is successful
[root@app-001 src]# /usr/local/tengine/sbin/nginx -s reload

6、CURL查詢方式示例

curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {"message": "JD_08011137015349778"}
  }
}'

curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    
    "match": {"log_type": "myapp"}
  },"size": 5
}'

curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{"query":{"bool":{"must":[{"match":{"message":"JD_08011137015349778"}}],"filter":[{"range":{"@timestamp":{"from":"now-1d","to":"now"}}}]}}}
'

curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{"query":{"bool":{"must":[{"match":{"message":"JD_08011137015349778"}}],"filter":[{"range":{"@timestamp":{"gte":"1506441600000","lte":"1506527999000"}}}]}}}
'

7、Java HttpClient 方式調用簡單封裝示例

package org.jeedevframework.common.es;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;

import javax.servlet.http.HttpServletRequest;

import org.apache.commons.lang3.StringUtils;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.BasicHttpParams;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.util.EntityUtils;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

import org.jeedevframework.util.DateUtil;
/**
針對以下CURL請求方式的Java封裝示例
curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{"query":{"bool":{"must":[{"match":{"message":"JD_08011137015349778"}}],
"filter":[{"range":{"@timestamp":{"gte":"1506441600000","lte":"1506527999000"}}}]}}}
'
 * */
public class EsQueryService {
    
    public static DefaultHttpClient httpClient = null;
    public static DefaultHttpClient getHttpClientInstance() {
        if (null == httpClient) {
            BasicHttpParams httpParams = new BasicHttpParams();  
            HttpConnectionParams.setConnectionTimeout(httpParams, 120000);  
            HttpConnectionParams.setSoTimeout(httpParams, 120000);  
            DefaultHttpClient httpClient = new DefaultHttpClient(httpParams);
            httpClient.getParams().setParameter("http.protocol.content-charset", "UTF-8");
            
            return httpClient;
        }
        return httpClient;
    }

    public static String query(HttpServletRequest request) throws JSONException, ParseException {
         String keywords = request.getParameter("keywords");
         String startTime = request.getParameter("startTime");
         String endTime = request.getParameter("endTime");
         int pageSize = Integer.valueOf(StringUtils.defaultIfEmpty(request.getParameter("pageSize"), "10"));
         String log_type = StringUtils.defaultIfEmpty(request.getParameter("log_type"), "");

        if(StringUtils.isEmpty(startTime)){
            startTime = DateUtil.format(new Date(),DateUtil.C_DATE_PATTON_DEFAULT)+" 00:00:00";
        }
        if(StringUtils.isEmpty(endTime)){
            endTime = DateUtil.format(new Date(),DateUtil.C_DATE_PATTON_DEFAULT)+" 23:59:59";
        }

        long startTimeDt = new SimpleDateFormat(DateUtil.C_TIME_PATTON_DEFAULT).parse(startTime).getTime();
        long endTimeDt = new SimpleDateFormat(DateUtil.C_TIME_PATTON_DEFAULT).parse(endTime).getTime();

        DefaultHttpClient httpClient = getHttpClientInstance();

        HttpPost httpPost = new HttpPost("http://192.168.1.106:9200/_search");

        JSONObject esQueryJo = new JSONObject();
        JSONObject queryJo = new JSONObject();
        JSONObject boolJo = new JSONObject();


        //esQueryJo.put("min_score", 1.2);

        JSONArray mustJoArr = new JSONArray();
        if(StringUtils.isNotEmpty(keywords)){
            JSONObject matchJo = new JSONObject();
            matchJo.put("message", keywords);
            JSONObject matchWrapJo = new JSONObject();
            matchWrapJo.put("match", matchJo);
            mustJoArr.put(matchWrapJo);
        }

        if(StringUtils.isNotEmpty(log_type)){
            JSONObject matchJo = new JSONObject();
            matchJo.put("log_type", log_type);
            JSONObject matchWrapJo = new JSONObject();
            matchWrapJo.put("match", matchJo);
            mustJoArr.put(matchWrapJo);
        }


        JSONArray filterJoArr = new JSONArray();
        JSONObject rangeJo = new JSONObject();
        JSONObject timestampJo = new JSONObject();
        timestampJo.put("gte", startTimeDt);
        timestampJo.put("lte", endTimeDt);
        //timestampJo.put("from", "now-1d");
        //timestampJo.put("to", "now");
        rangeJo.put("@timestamp", timestampJo);
        //mustJoArr.put("match", matchJo);
        JSONObject rangeWrapJo = new JSONObject();
        rangeWrapJo.put("range", rangeJo);
        filterJoArr.put(rangeWrapJo);

        boolJo.put("must",mustJoArr);
        boolJo.put("filter",filterJoArr);

        queryJo.put("bool", boolJo);
        esQueryJo.put("query", queryJo);
        esQueryJo.put("size", pageSize);
        String esQueryString = esQueryJo.toString();
        String resultContent = "";
        if(mustJoArr.length()>0){
            StringEntity reqEntity = new StringEntity(esQueryString ,ContentType.APPLICATION_JSON);
            httpPost.setEntity(reqEntity);
            try{
                 HttpResponse resp = httpClient.execute(httpPost);
                 resultContent = EntityUtils.toString(resp.getEntity(), "UTF-8");
                 return resultContent;
            }catch(Exception e){
                e.printStackTrace();
            }finally {
            }   
        }

        return "";
    }
        

}

8、相關參考資料

ELK 性能(1) — Logstash 性能及其替代方案
http://www.cnblogs.com/richaaaard/p/6109595.html

CentOS下RPM安裝ElasticSearch
http://www.netpc.com.cn/2361.html

Elasticsearch在Centos 7上的安裝與配置
https://www.biaodianfu.com/centos-7-install-elasticsearch.html


ElasticSearch 5.0.0 安裝部署常見錯誤或問題
http://www.dajiangtai.com/community/18136.do?origin=csdn-geek&dt=1214

elasticsearch5.0啓動出現的錯誤 
http://blog.csdn.net/qq942477618/article/details/53414983

ElasticSearch 經常使用的查詢過濾語句
http://www.cnblogs.com/ghj1976/p/5293250.html


filebeat專題
http://www.cnblogs.com/louis2008/p/filebeat.html

filebeat.yml（中文配置詳解）
http://www.cnblogs.com/zlslch/p/6622079.html

28.Filebeat的高級配置-Filebeat部分
http://blog.csdn.net/a464057216/article/details/51233375


ELK+Filebeat+Nginx集中式日誌解決方案（一）
http://zhengmingjing.blog.51cto.com/1587142/1907456

ELK日誌服務使用-filebeat多文件發送
http://bbotte.com/logs-service/use-elk-processing-logs-multiple-log-file-send/

初探ELK-以收集 nginx 日誌爲例示範搭建一個 ELK 環境的基本步驟
http://nosmoking.blog.51cto.com/3263888/1855680


filebeat+kafka+ELK5.4安裝與部署
http://xiangcun168.blog.51cto.com/4788340/1933509


Filebeat5+Kafka+ELK Docker搭建日誌系統
http://www.jianshu.com/p/9dfac37885cb


經過HTTP RESTful API 操做elasticsearch搜索數據 
http://blog.csdn.net/stark_summer/article/details/48830493

Elasticsearch+Logstash+Kibana教程
http://www.cnblogs.com/xing901022/p/4704319.html