CentOS 6.x ELK(Elasticsearch+Logstash+Kibana)

前言

Elasticsearch + Logstash + Kibana（ELK）是一套開源的日誌管理方案，分析網站的訪問狀況時咱們通常會藉助Google/百度/CNZZ等方式嵌入JS作數據統計，可是當網站訪問異常或者被攻擊時咱們須要在後臺分析如Nginx的具體日誌，而Nginx日誌分割/GoAccess/Awstats都是相對簡單的單節點解決方案，針對分佈式集羣或者數據量級較大時會顯得愛莫能助，而ELK的出現能夠使咱們從容面對新的挑戰。

Logstash：負責日誌的收集，處理和儲存

Elasticsearch：負責日誌檢索和分析

Kibana：負責日誌的可視化

ELK(Elasticsearch + Logstash + Kibana)

更新記錄

2015年08月31日 - 初稿

閱讀原文 - http://wsgzao.github.io/post/elk/

擴展閱讀

CentOS 7.x安裝ELK(Elasticsearch+Logstash+Kibana) - http://www.chenshake.com/centos-install-7-x-elk-elasticsearchlogstashkibana/

Centos 6.5 安裝nginx日誌分析系統 elasticsearch + logstash + redis + kibana - http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=17291169&id=4898582

logstash-forwarder and grok examples - https://www.ulyaoth.net/threads/logstash-forwarder-and-grok-examples.32413/

三斗室 - http://chenlinux.com/

elastic - https://www.elastic.co/guide

LTMP索引 - http://wsgzao.github.io/index/#LTMP

組件預覽

JDK - http://www.oracle.com/technetwork/java/javase/downloads/index.html

Elasticsearch - https://www.elastic.co/downloads/elasticsearch

Logstash - https://www.elastic.co/downloads/logstash

Kibana - https://www.elastic.co/downloads/kibana

redis - http://redis.io/download

設置FQDN

建立SSL證書的時候須要配置FQDN

#修改hostname

cat /etc/hostname

elk

#修改hosts

cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1 10-10-87-19

10.10.87.19 elk.ooxx.com elk

#刷新環境

hostname -F /etc/hostname

#複查結果

hostname -f

elk.ooxx.com

hostname

elk

服務端

Java

cat /etc/redhat-release

CentOS release 6.5 (Final)

yum install java-1.7.0-openjdk

java -version

java version "1.7.0_85"

OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01)

OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)

Elasticsearch

#下載安裝

wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm

yum localinstall elasticsearch-1.7.1.noarch.rpm

#啓動相關服務

service elasticsearch start

service elasticsearch status

#查看Elasticsearch的配置文件

rpm -qc elasticsearch

/etc/elasticsearch/elasticsearch.yml

/etc/elasticsearch/logging.yml

/etc/init.d/elasticsearch

/etc/sysconfig/elasticsearch

/usr/lib/sysctl.d/elasticsearch.conf

/usr/lib/systemd/system/elasticsearch.service

/usr/lib/tmpfiles.d/elasticsearch.conf

#查看端口使用狀況

netstat -nltp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   

tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java           

tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java           

tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd           

tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd 

#測試訪問

curl -X GET http://localhost:9200/

Kibana

#下載tar包

wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz

#解壓

tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/

cd /usr/local/

mv kibana-4.1.1-linux-x64 kibana

#建立kibana服務

vi /etc/rc.d/init.d/kibana

#!/bin/bash

### BEGIN INIT INFO

# Provides:          kibana

# Default-Start:     2 3 4 5

# Default-Stop:      0 1 6

# Short-Description: Runs kibana daemon

# Description: Runs the kibana daemon as a non-root user

### END INIT INFO

# Process name

NAME=kibana

DESC="Kibana4"

PROG="/etc/init.d/kibana"

# Configure location of Kibana bin

KIBANA_BIN=/usr/local/kibana/bin

# PID Info

PID_FOLDER=/var/run/kibana/

PID_FILE=/var/run/kibana/$NAME.pid

LOCK_FILE=/var/lock/subsys/$NAME

PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN

DAEMON=$KIBANA_BIN/$NAME

# Configure User to run daemon process

DAEMON_USER=root

# Configure logging location

KIBANA_LOG=/var/log/kibana.log

# Begin Script

RETVAL=0

if [ `id -u` -ne 0 ]; then

        echo "You need root privileges to run this script"

        exit 1

fi

# Function library

. /etc/init.d/functions

start() {

        echo -n "Starting $DESC : "

pid=`pidofproc -p $PID_FILE kibana`

        if [ -n "$pid" ] ; then

                echo "Already running."

                exit 0

        else

        # Start Daemon

if [ ! -d "$PID_FOLDER" ] ; then

                        mkdir $PID_FOLDER

                fi

daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &

                sleep 2

                pidofproc node > $PID_FILE

                RETVAL=$?

                [[ $? -eq 0 ]] && success || failure

echo

                [ $RETVAL = 0 ] && touch $LOCK_FILE

                return $RETVAL

        fi

}

reload()

{

    echo "Reload command is not implemented for this service."

    return $RETVAL

}

stop() {

        echo -n "Stopping $DESC : "

        killproc -p $PID_FILE $DAEMON

        RETVAL=$?

echo

        [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE

}

case "$1" in

  start)

        start

;;

  stop)

        stop

        ;;

  status)

        status -p $PID_FILE $DAEMON

        RETVAL=$?

        ;;

  restart)

        stop

        start

        ;;

  reload)

reload

;;

  *)

# Invalid Arguments, print the following message.

        echo "Usage: $0 {start|stop|status|restart}" >&2

exit 2

        ;;

esac

#修改啓動權限

chmod +x /etc/rc.d/init.d/kibana

#啓動kibana服務

service kibana start

service kibana status

#查看端口

netstat -nltp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   

tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java           

tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java           

tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd           

tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      1876/node           

tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd

Logstash

#下載rpm包

wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm

#安裝

yum localinstall logstash-1.5.4-1.noarch.rpm

#設置ssl，以前設置的FQDN是elk.ooxx.com

cd /etc/pki/tls

#openssl req -x509  -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -subj /CN=logstash.example.com

openssl req -subj '/CN=elk.ooxx.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

#建立一個01-logstash-initial.conf文件

cat > /etc/logstash/conf.d/01-logstash-initial.conf << EOF

input {

  lumberjack {

    port => 5000

    type => "logs"

    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"

    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"

  }

}

filter {

  if [type] == "syslog" {

    grok {

      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

      add_field => [ "received_at", "%{@timestamp}" ]

      add_field => [ "received_from", "%{host}" ]

    }

    syslog_pri { }

    date {

      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

    }

  }

}

output {

  elasticsearch { host => localhost }

  stdout { codec => rubydebug }

}

EOF

#啓動logstash服務

service logstash start

service logstash status

#查看5000端口

netstat -nltp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   

tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java           

tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java           

tcp        0      0 0.0.0.0:9301                0.0.0.0:*                   LISTEN      2309/java           

tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd           

tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      1876/node           

tcp        0      0 0.0.0.0:5000                0.0.0.0:*                   LISTEN      2309/java           

tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd 

#啓動服務

service logstash-forwarder start

service logstash-forwarder status

#訪問Kibana，Time-field name 選擇 @timestamp

http://localhost:5601/

#增長節點和客戶端配置同樣，注意同步證書

/etc/pki/tls/certs/logstash-forwarder.crt

客戶端

Logstash Forwarder

#登錄到客戶端，安裝Logstash Forwarder

wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm

yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm

#查看logstash-forwarder的配置文件位置

rpm -qc logstash-forwarder

/etc/logstash-forwarder.conf

#備份配置文件

cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.save

#編輯 /etc/logstash-forwarder.conf，須要根據實際狀況進行修改

cat > /etc/logstash-forwarder.conf << EOF

{

  "network": {

    "servers": [ "elk.ooxx.com:5000" ],

    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

    "timeout": 15

  },

  "files": [

    {

      "paths": [

        "/var/log/messages",

        "/var/log/secure"

      ],

      "fields": { "type": "syslog" }

    }

  ]

}

EOF

配置Nginx日誌策略

#修改客戶端配置

vi /etc/logstash-forwarder.conf

{

  "network": {

    "servers": [ "elk.ooxx.com:5000" ],

    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

    "timeout": 15

  },

  "files": [

    {

      "paths": [

        "/var/log/messages",

        "/var/log/secure"

      ],

      "fields": { "type": "syslog" }

    }, {

      "paths": [

        "/app/local/nginx/logs/access.log"

      ],

      "fields": { "type": "nginx" }

    }

  ]

}

#服務端增長patterns

mkdir /opt/logstash/patterns

vi /opt/logstash/patterns/nginx

NGUSERNAME [a-zA-Z\.\@\-\+_%]+

NGUSER %{NGUSERNAME}

NGINXACCESS %{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

#官網pattern的debug在線工具

https://grokdebug.herokuapp.com/

#修改logstash權限

chown -R logstash:logstash /opt/logstash/patterns

#修改服務端配置

vi /etc/logstash/conf.d/01-logstash-initial.conf

input {

  lumberjack {

    port => 5000

    type => "logs"

    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"

    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"

  }

}

filter {

  if [type] == "syslog" {

    grok {

      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

      add_field => [ "received_at", "%{@timestamp}" ]

      add_field => [ "received_from", "%{host}" ]

    }

    syslog_pri { }

    date {

      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

    }

  }

  if [type] == "nginx" {

    grok {

       match => { "message" => "%{NGINXACCESS}" }

    }

  }

}

output {

  elasticsearch { host => localhost }

  stdout { codec => rubydebug }

}

其它注意事項

修改kibana端口

#編輯kibana.yaml

vi /usr/local/kibana/config/kibana.yml

# Kibana is served by a back end server. This controls which port to use.

#port: 5601

port: 80

# The host to bind the server to.

host: "0.0.0.0"

# The Elasticsearch instance to use for all your queries.

elasticsearch_url: "http://localhost:9200"

# preserve_elasticsearch_host true will send the hostname specified in `elasticsearch`. If you set it to false,

# then the host you use to connect to *this* Kibana instance will be sent.

elasticsearch_preserve_host: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations

# and dashboards. It will create a new index if it doesn't already exist.

kibana_index: ".kibana"

# If your Elasticsearch is protected with basic auth, this is the user credentials

# used by the Kibana server to perform maintence on the kibana_index at statup. Your Kibana

# users will still need to authenticate with Elasticsearch (which is proxied thorugh

# the Kibana server)

# kibana_elasticsearch_username: user

# kibana_elasticsearch_password: pass

# If your Elasticsearch requires client certificate and key

# kibana_elasticsearch_client_crt: /path/to/your/client.crt

# kibana_elasticsearch_client_key: /path/to/your/client.key

# If you need to provide a CA certificate for your Elasticsarech instance, put

# the path of the pem file here.

# ca: /path/to/your/CA.pem

# The default application to load.

default_app_id: "discover"

# Time in milliseconds to wait for elasticsearch to respond to pings, defaults to

# request_timeout setting

# ping_timeout: 1500

# Time in milliseconds to wait for responses from the back end or elasticsearch.

# This must be > 0

request_timeout: 300000

# Time in milliseconds for Elasticsearch to wait for responses from shards.

# Set to 0 to disable.

shard_timeout: 0

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

# startup_timeout: 5000

# Set to false to have a complete disregard for the validity of the SSL

# certificate.

verify_ssl: true

# SSL for outgoing requests from the Kibana Server (PEM formatted)

# ssl_key_file: /path/to/your/server.key

# ssl_cert_file: /path/to/your/server.crt

# Set the path to where you would like the process id file to be created.

# pid_file: /var/run/kibana.pid

# If you would like to send the log output to a file you can set the path below.

# This will also turn off the STDOUT log output.

# log_file: ./kibana.log

# Plugins that are included in the build, and no longer found in the plugins/ folder

bundled_plugin_ids:

 - plugins/dashboard/index

 - plugins/discover/index

 - plugins/doc/index

 - plugins/kibana/index

 - plugins/markdown_vis/index

 - plugins/metric_vis/index

 - plugins/settings/index

 - plugins/table_vis/index

 - plugins/vis_types/index

 - plugins/visualize/index

JVM調優

#修改elasticsearch.in.sh

vi /usr/share/elasticsearch/bin/elasticsearch.in.sh

if [ "x$ES_MIN_MEM" = "x" ]; then

    ES_MIN_MEM=1g

fi

if [ "x$ES_MAX_MEM" = "x" ]; then

    ES_MAX_MEM=1g

來自爲知筆記(Wiz)