ELK介紹
http://blog.51cto.com/zero01/2082794html
https://www.cnblogs.com/aresxin/p/8035137.htmljava
需求背景： node
業務發展愈來愈龐大，服務器愈來愈多，各類訪問日誌、應用日誌、錯誤日誌量愈來愈多，開發人員排查問題，須要到服務器上查日誌，不方便，運營人員須要一些數據，須要咱們運維到服務器上分析日誌。mysql
ELK安裝準備工做
1.準備三臺機器，三臺機器都安裝es，主節點131，數據節點132,133，es主節點上安裝kibana，132上安裝logstash數據收集linux
2.三臺機器都配置下hostnginx
[root@centos-01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.242.131 centos-01 192.168.242.132 centos-02 192.168.242.133 centos-03
[root@centos-01 ~]#
3.三臺機器都安裝jdksql
[root@centos-01 ~]# yum install -y java-1.8.0-openjdk
安裝es
http://www.mamicode.com/info-detail-2207957.htmlexpress
1.三臺機器都導入keybootstrap
[root@centos-01 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch [root@centos-01 ~]#
2.三臺機器都寫一個repo文件vim
[root@centos-01 ~]# cat !$ cat /etc/yum.repos.d/elastic.repo [elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md [root@centos-01 ~]#
3.檢查下yum源是否能夠找的elastic
[root@centos-01 ~]# yum list|grep elastic apm-server.i686 6.3.0-1 elasticsearch-6.x apm-server.x86_64 6.3.0-1 elasticsearch-6.x auditbeat.i686 6.3.0-1 elasticsearch-6.x auditbeat.x86_64 6.3.0-1 elasticsearch-6.x elasticsearch.noarch 6.3.0-1 elasticsearch-6.x filebeat.i686 6.3.0-1 elasticsearch-6.x filebeat.x86_64 6.3.0-1 elasticsearch-6.x heartbeat-elastic.i686 6.3.0-1 elasticsearch-6.x heartbeat-elastic.x86_64 6.3.0-1 elasticsearch-6.x kibana.x86_64 6.3.0-1 elasticsearch-6.x kibana-oss.x86_64 6.3.0-1 elasticsearch-6.x logstash.noarch 1:6.3.0-1 elasticsearch-6.x metricbeat.i686 6.3.0-1 elasticsearch-6.x metricbeat.x86_64 6.3.0-1 elasticsearch-6.x packetbeat.i686 6.3.0-1 elasticsearch-6.x packetbeat.x86_64 6.3.0-1 elasticsearch-6.x pcp-pmda-elasticsearch.x86_64 3.12.2-5.el7 base rsyslog-elasticsearch.x86_64 8.24.0-16.el7_5.4 updates [root@centos-01 ~]#
4.三臺都安裝elasticsearch(下載rpm包會慢，能夠提早下載好再用rpm -ivh安裝)
[root@centos-01 ~]# yum install -y elasticsearch
1.查看elasticsearch安裝了哪些文件
[root@centos-01 ~]# rpm -ql elasticsearch
2.es有兩個配置文件
/etc/elasticsearch/elasticsearch.yml （配置集羣的時候設置主節點父節點等配置） /etc/sysconfig/elasticsearch (和服務自己相關的配置)
3.編輯31服務器elasticsearch.yml
[root@centos-01 ~]# vim /etc/elasticsearch/elasticsearch.yml
修改
cluster.name: centos （簇名稱） node.name: centos-01 （節點名稱）
network.host: 192.168.242.131 （定義綁定的ip，須要在哪一個ip上監聽端口，爲了安全咱們這裏設置成本機ip，能夠寫成0.0.0.0表明全部ip）
http.port: 9200
在node配置塊兒添加
node.master: true （是主節點）
node.data: false （不是數據節點）
discovery.zen.ping.unicast.hosts: ["centos-01", "centos-02", "centos-03"] （定義集羣裏面都有哪些角色，也能夠寫IP）
4.將配置好的文件拷貝到另外兩個節點上去
[root@centos-01 ~]# scp /etc/elasticsearch/elasticsearch.yml centos-02:/tmp/ The authenticity of host 'centos-02 (192.168.242.132)' can't be established. ECDSA key fingerprint is 15:ce:01:bd:cf:ed:1c:10:b5:e7:d3:8f:ca:66:a1:19. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'centos-02,192.168.242.132' (ECDSA) to the list of known hosts. root@centos-02's password: elasticsearch.yml 100% 3016 3.0KB/s 00:00 [root@centos-01 ~]# scp /etc/elasticsearch/elasticsearch.yml centos-03:/tmp/ The authenticity of host 'centos-03 (192.168.242.133)' can't be established. ECDSA key fingerprint is 0e:f3:c9:fe:6e:3a:e9:09:ed:b3:47:72:9d:c7:e0:ae. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'centos-03,192.168.242.133' (ECDSA) to the list of known hosts. root@centos-03's password: elasticsearch.yml 100% 3016 3.0KB/s 00:00 [root@centos-01 ~]#
5.到02和03服務器上修改下配置文件
[root@centos-02 ~]# cp /tmp/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml [root@centos-02 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name不變
node.name改變成centos-02
node.master: false （改爲false）
node.data: true （改爲true）
network.host: 192.168.242.132 （改爲132）
[root@centos-03 ~]# cp /tmp/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml
[root@centos-03 ~]# vi /etc/elasticsearch/elasticsearch.yml
node.name: centos-03 node.master: false node.data: true network.host: 192.168.242.133
6.這樣咱們就能夠啓動三臺機器的es服務了（咱們這裏省略xpack安裝，xpack是收費的）
7.先啓動主節點es，而後再啓動其餘兩個節點
[root@centos-01 ~]# systemctl start elasticsearch.service [root@centos-01 ~]# [root@centos-02 ~]# systemctl start elasticsearch.service [root@centos-02 ~]# [root@centos-03 ~]# systemctl start elasticsearch.service [root@centos-03 ~]#
8.查看是否啓動成功，結果沒有啓動成功，查看下日誌狀況，先看var/log/elasticsearch下面的，若是沒有再看/var/log/messages
[root@centos-01 ~]# ps aux|grep elastic root 6640 0.0 0.0 112668 976 pts/0 R+ 15:50 0:00 grep --color=auto elastic [root@centos-01 ~]#
[root@centos-01 ~]# vi /var/log/elasticsearch/
[root@centos-01 ~]# less /var/log/messages
9.原來是配置選項冒號後必定要加空格，重啓master(01)服務，成功
[root@centos-01 local]# systemctl start elasticsearch.service [root@centos-01 local]# ps aux |grep elasticsearch elastic+ 8379 3.4 70.0 3280924 701048 ? Ssl 17:03 0:51 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:
+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.
recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.Tm3xQEXo -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/
lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/
var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribut
ion.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet elastic+ 8507 0.0 0.0 72076 860 ? Sl 17:04 0:00 /usr/share/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller root 9200 0.0 0.0 112668 980 pts/0 R+ 17:29 0:00 grep --color=auto elasticsearch [root@centos-01 local]#
curl查看es
1.查看端口，9200、9300端口也起來了
[root@centos-01 local]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2729/mysqld
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1849/nginx: master
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 2865/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1172/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1173/cupsd
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 1849/nginx: master
tcp6 0 0 192.168.242.131:9200 :::* LISTEN 8379/java tcp6 0 0 192.168.242.131:9300 :::* LISTEN 8379/java
tcp6 0 0 :::22 :::* LISTEN 1172/sshd
tcp6 0 0 ::1:631 :::* LISTEN 1173/cupsd
[root@centos-01 local]#
2.啓動0二、03服務器
[root@centos-02 ~]# systemctl start elasticsearch.service
[root@centos-02 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master
tcp6 0 0 192.168.242.132:9200 :::* LISTEN 2873/java tcp6 0 0 192.168.242.132:9300 :::* LISTEN 2873/java
tcp6 0 0 :::22 :::* LISTEN 1437/sshd
tcp6 0 0 ::1:25 :::* LISTEN 2350/master
[root@centos-02 ~]#
[root@centos-03 ~]# systemctl start elasticsearch
[root@centos-03 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1435/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2570/master
tcp6 0 0 192.168.242.133:9200 :::* LISTEN 2926/java tcp6 0 0 192.168.242.133:9300 :::* LISTEN 2926/java
tcp6 0 0 :::22 :::* LISTEN 1435/sshd
tcp6 0 0 ::1:25 :::* LISTEN 2570/master
[root@centos-03 ~]#
3.三臺機器都關掉防火牆
[root@centos-01 local]# systemctl stop firewalld [root@centos-01 local]# iptables -nvL Chain INPUT (policy ACCEPT 20558 packets, 2482K bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 27470 packets, 11M bytes) pkts bytes target prot opt in out source destination [root@centos-01 local]#
[root@centos-01 local]# systemctl stop iptables
4.檢測，發現只有一個節點，原來是服務器的senlinux沒有關閉，把三臺機器的senlinux都關閉再從新啓動三臺機器的elasticsearch服務，成功。
[root@centos-01 local]# curl '192.168.242.131:9200/_cluster/health?pretty'
{
"cluster_name" : "centos",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[root@centos-01 local]#
[root@centos-01 local]# curl '192.168.242.131:9200/_cluster/health?pretty'
{
"cluster_name" : "centos",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[root@centos-01 local]#
5.查看集羣詳細信息，端口9200是通訊的，9300是數據傳輸用到的
[root@centos-01 local]# curl '192.168.242.131:9200/_cluster/state?pretty'|less
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
"cluster_name" : "centos",
"compressed_size_in_bytes" : 9376,
"version" : 7,
{
"cluster_name" : "centos",
"compressed_size_in_bytes" : 9376,
"version" : 7,
"state_uuid" : "AYXexjAZRhC53TySE-_1ug",
"master_node" : "LuQzu2TISnmztU5O2R5DWw",
"blocks" : { },
"nodes" : {
"rDHWQTRwQRunAWdlalN4Nw" : {
"name" : "centos-02",
"ephemeral_id" : "ERGYa3b9SzewG4Dr4VOAew",
"transport_address" : "192.168.242.132:9300",
"attributes" : {
"ml.machine_memory" : "1025363968",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
}
},
"FNnHSA2rT2m6s7Ez8tYyLg" : {
{
"cluster_name" : "centos",
"compressed_size_in_bytes" : 9376,
"version" : 7,
"state_uuid" : "AYXexjAZRhC53TySE-_1ug",
"master_node" : "LuQzu2TISnmztU5O2R5DWw",
"blocks" : { },
"nodes" : {
"rDHWQTRwQRunAWdlalN4Nw" : {
"name" : "centos-02",
"ephemeral_id" : "ERGYa3b9SzewG4Dr4VOAew",
"transport_address" : "192.168.242.132:9300",
"attributes" : {
"ml.machine_memory" : "1025363968",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
}
},
"FNnHSA2rT2m6s7Ez8tYyLg" : {
"name" : "centos-03",
"ephemeral_id" : "2OUZMwW-TOWMpOBZUY6Erw",
"transport_address" : "192.168.242.133:9300",
"attributes" : {
"ml.machine_memory" : "1025363968",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
}
},
"LuQzu2TISnmztU5O2R5DWw" : {
"name" : "centos-01",
"ephemeral_id" : "jSwXaS5dR0SMy1FI9uPhrQ",
"transport_address" : "192.168.242.131:9300",
"attributes" : {
"ml.machine_memory" : "1025363968",
"xpack.installed" : "true",
安裝kibana
kibana是用nodejs開發的
kibana是用來展現圖像的
1.能夠用yum直接安裝kibana，若是太慢了能夠預先下載rpm包用rmp -ivh安裝包
[root@centos-01 local]# yum install -y kibana
2.編輯kibana配置文件
[root@centos-01 local]# vim /etc/kibana/kibana.yml
server.port: 5601 （端口） server.host: "192.168.242.131" （爲了安裝設置成只監聽內網不讓外網訪問，若是還想讓公網訪問能夠用nginx作一個代理，加一個安全認證） elasticsearch.url: "http://192.168.242.131:9200" （設置elastic服務器的ip，由於kibana須要與elastic通訊）
3.啓動kibana
[root@centos-01 local]# systemctl start kibana [root@centos-01 local]#
4.查看進程和端口
[root@centos-01 local]# ps aux|grep kibana kibana 28148 46.1 19.0 1143756 190284 ? Rsl 10:38 0:15 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml root 28199 0.0 0.0 112668 976 pts/0 R+ 10:38 0:00 grep --color=auto kibana [root@centos-01 local]# [root@centos-01 local]# netstat -lntp|grep node tcp 0 0 192.168.242.131:5601 0.0.0.0:* LISTEN 28148/node [root@centos-01 local]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2729/mysqld tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1849/nginx: master tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 2865/dnsmasq tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1172/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1173/cupsd tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 1849/nginx: master tcp 0 0 192.168.242.131:5601 0.0.0.0:* LISTEN 28148/node tcp6 0 0 192.168.242.131:9200 :::* LISTEN 26052/java tcp6 0 0 192.168.242.131:9300 :::* LISTEN 26052/java tcp6 0 0 :::22 :::* LISTEN 1172/sshd tcp6 0 0 ::1:631 :::* LISTEN 1173/cupsd [root@centos-01 local]#
5.瀏覽器訪問kibana
http://192.168.242.131:5601
6.kibana默認日誌在/var/log/message下
安裝logstash
1.lagstash不支持java9
2.咱們在132服務器上yum安裝logstash，能夠下載rpm包安裝（瀏覽器提早下載好rpm包，而後rz命令上傳到服務器，rpm -ivh 安裝便可）
[root@centos-02 ~]# yum install -y logstash
3.安裝完以後配置logstash，咱們這裏收集系統日誌，input就是咱們的進入源，output就是咱們的輸出源
[root@centos-02 ~]# vim /etc/logstash/conf.d/syslog.conf
[root@centos-02 ~]# cat /etc/logstash/conf.d/syslog.conf
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
stdout {
codec => rubydebug
}
}
[root@centos-02 ~]#
4.檢查配置是否有錯，path.settings指定配置文件所在的目錄，-f指定我配置的logstash相關的配置文件
[root@centos-02 ~]# cd /usr/share/logstash/bin/ [root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf /conf.d/syslog.conf --config.test_and_exit
5.編輯rsyslog.conf文件在RULES下加一行，將日誌輸出到10514端口
[root@centos-02 bin]# vim /etc/rsyslog.conf #### RULES #### *.* @@192.168.242.132:10514
6.啓動logstash
[root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
7.再打開一個132服務器終端，重啓rsyslog服務，第一個打開的終端有日誌輸出了
[root@centos-02 ~]# systemctl restart rsyslog [root@centos-02 ~]#
{
"severity_label" => "Informational",
"facility" => 5,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "syslogd",
"logsource" => "centos-02",
"@version" => "1",
"program" => "rsyslogd",
"priority" => 46,
"message" => "[origin software=\"rsyslogd\" swVersion=\"7.4.7\" x-pid=\"4749\" x-info=\"http://www.rsyslog.com\"] start\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
{
"severity_label" => "Informational",
"facility" => 3,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "system",
"logsource" => "centos-02",
"@version" => "1",
"program" => "systemd",
"priority" => 30,
"message" => "Stopping System Logging Service...\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
{
"severity_label" => "Informational",
"facility" => 3,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "system",
"logsource" => "centos-02",
"@version" => "1",
"program" => "systemd",
"priority" => 30,
"message" => "Starting System Logging Service...\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
{
"severity_label" => "Informational",
"facility" => 3,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "system",
"logsource" => "centos-02",
"@version" => "1",
"program" => "systemd",
"priority" => 30,
"message" => "Started System Logging Service.\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
{
"severity_label" => "Notice",
"facility" => 10,
"severity" => 5,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "polkitd",
"pid" => "1232",
"priority" => 85,
"message" => "Unregistered Authentication Agent for unix-process:4742:8497324 (system bus name :1.71, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale zh_CN.UTF-8)
(disconnected from bus)\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
8.查看10514端口是否啓動
[root@centos-02 ~]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java tcp6 0 0 :::10514 :::* LISTEN 4678/java tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java tcp6 0 0 :::22 :::* LISTEN 1437/sshd tcp6 0 0 ::1:25 :::* LISTEN 2350/master tcp6 0 0 127.0.0.1:9600 :::* LISTEN 4678/java [root@centos-02 ~]#
9.再用03服務器登陸下02服務器看看有日誌輸出麼？輸出了
[root@centos-03 ~]# ssh centos-02
"severity_label" => "Notice",
"facility" => 10,
"severity" => 5,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:53:12",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "unix_chkpwd",
"pid" => "4760",
"priority" => 85,
"message" => "password check failed for user (root)\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:53:12.000Z
}
{
"severity_label" => "Notice",
"facility" => 10,
"severity" => 5,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:53:12",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "sshd",
"pid" => "4758",
"priority" => 85,
"message" => "pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=centos-03 user=root\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:53:12.000Z
}
{
"severity_label" => "Informational",
"facility" => 10,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:53:12",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "sshd",
"pid" => "4758",
"priority" => 86,
"message" => "pam_succeed_if(sshd:auth): requirement \"uid >= 1000\" not met by user \"root\"\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:53:12.000Z
}
{
"severity_label" => "Informational",
"facility" => 10,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:53:14",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "sshd",
"pid" => "4758",
"priority" => 86,
"message" => "Failed password for root from 192.168.242.133 port 60658 ssh2\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:53:14.000Z
}
10.日誌配置成功
配置logstash
怎麼把日誌輸出到es裏面去呢，下面來說解
1.首先把第一個終端退出ctrl+c，編輯配置文件，output改爲elasticsearch
[root@centos-02 bin]# vim /etc/logstash/conf.d/syslog.conf
[root@centos-02 bin]# cat /etc/logstash/conf.d/syslog.conf
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
elasticsearch {
hosts => ["192.168.242.131:9200"]
index => "system-syslog-%{+YYYY.MM}"
}
}
[root@centos-02 bin]#
2.檢查下配置文件語法
[root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
[2018-07-06T21:06:23,517][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-07-06T21:06:29,416][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@centos-02 bin]#
3.以服務的形式啓動logstash。
[root@centos-02 bin]# systemctl start logstash [root@centos-02 bin]# ps aux|grep logstash
4.查看端口，有10514和logstash的監聽端口9600說明啓動成功了，logstash日誌文件以下
[root@centos-02 bin]# ls /var/log/logstash/logstash-plain.log /var/log/logstash/logstash-plain.log [root@centos-02 bin]#
[root@centos-02 bin]# vim /etc/logstash/logstash.yml path.config: /etc/logstash/conf.d/*.conf
[root@centos-02 bin]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java tcp6 0 0 :::22 :::* LISTEN 1437/sshd tcp6 0 0 ::1:25 :::* LISTEN 2350/master [root@centos-02 bin]#
5.一直沒有9600端口，多是權限問題，咱們修改下日誌權限
[root@centos-02 bin]# ll /var/log/logstash/logstash-plain.log -rw-r--r--. 1 root root 2928 7月 6 21:06 /var/log/logstash/logstash-plain.log [root@centos-02 bin]# chown logstash /var/log/logstash/logstash-plain.log [root@centos-02 bin]# systemctl restart logstash [root@centos-02 bin]# tail /var/log/logstash/logstash-plain.log
6.查看日誌提示沒有寫的權限，這是由於咱們以前是用root身份啓動的logstash，因此logstash對應的/var/lib/logstash下的文件都是root權限了
[root@centos-02 bin]# tail /var/log/logstash/logstash-plain.log
[2018-07-06T20:41:06,868][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.242.132:55245"}
[2018-07-06T20:57:05,143][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2018-07-06T20:57:05,692][INFO ][logstash.inputs.syslog ] connection error: stream closed
[2018-07-06T20:57:05,888][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x1868527b run>"}
[2018-07-06T21:06:23,517][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-07-06T21:06:29,416][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[2018-07-06T21:39:23,273][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not
writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:in `validate_value'",
"/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in `block in validate_all'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.
rb:140:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:279:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:
in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/sha
re/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2018-07-06T21:39:23,325][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-07-06T21:40:28,999][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not
writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:in `validate_value'",
"/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in `block in validate_all'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.
rb:140:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:279:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:
in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/sh
are/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2018-07-06T21:40:29,047][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[root@centos-02 bin]#
7.咱們修改下權限
[root@centos-02 bin]# chown -R logstash /var/lib/logstash [root@centos-02 bin]#
[root@centos-02 bin]# systemctl restart logstash [root@centos-02 bin]#
8.9600端口出現了，成功
[root@centos-02 bin]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java tcp6 0 0 :::10514 :::* LISTEN 5828/java tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java tcp6 0 0 :::22 :::* LISTEN 1437/sshd tcp6 0 0 ::1:25 :::* LISTEN 2350/master tcp6 0 0 127.0.0.1:9600 :::* LISTEN 5828/java [root@centos-02 bin]#
kibana上查看日誌
1.在es上看看有沒有日誌，獲取索引（若是獲取不到配置host），有索引說明logstash和es通訊正常了
[root@centos-02 bin]# vim /etc/logstash/logstash.yml http.host: "192.168.242.132"
root@centos-01 local]# curl '192.168.242.131:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open system-syslog-2018.07 Nh8lRsu3T56U1BIE-_mLnw 5 1 1 0 24.9kb 12.4kb green open .kibana n2dgGyqCRs-9KTIs1A-uAw 1 1 1 0 8kb 4kb [root@centos-01 local]#
2.獲取索引的詳細信息
[root@centos-01 local]# curl '192.168.242.131:9200/system-syslog-2018.07?pretty'
{
"system-syslog-2018.07" : {
"aliases" : { },
"mappings" : {
"doc" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"facility" : {
"type" : "long"
},
"facility_label" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"logsource" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"pid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"priority" : {
"type" : "long"
},
"program" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"severity" : {
"type" : "long"
},
"severity_label" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"timestamp" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1530856398824",
"number_of_shards" : "5",
"number_of_replicas" : "1",
"uuid" : "Nh8lRsu3T56U1BIE-_mLnw",
"version" : {
"created" : "6030099"
},
"provided_name" : "system-syslog-2018.07"
}
}
}
}
[root@centos-01 local]#
配置kibana
1.建立索引，將咱們剛看到的索引，粘貼到kibana中（以下圖），這個地方支持正則能夠寫成system-syslog-*
[root@centos-01 local]# curl '192.168.242.131:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open system-syslog-2018.07 Nh8lRsu3T56U1BIE-_mLnw 5 1 1 0 24.9kb 12.4kb
green open .kibana n2dgGyqCRs-9KTIs1A-uAw 1 1 1 0 8kb 4kb
[root@centos-01 local]#
2.成功建立，這個過程就是講es裏面的索引搞到kibana裏面來
3.點擊discover
收集nginx日誌
1.在132服務器，也就是logstash所在的服務器上添加一個nginx配置文件
[root@centos-02 bin]# vim /etc/logstash/conf.d/nginx.conf^C
[root@centos-02 bin]# cat /etc/logstash/conf.d/nginx.conf
input {
file { # 指定一個文件做爲logstash的輸入源
path => "/tmp/elk_access.log" # 指定文件的路徑
start_position => "beginning" # 指定什麼時候開始收集
type => "nginx" # 定義日誌類型，可自定義
}
}
filter { # 配置過濾器
grok {
match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER
:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"} # 定義日誌的輸出格式
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.242.132:9200"]
index => "nginx-test-%{+YYYY.MM.dd}"
}
}
[root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCTh
reads=N
^[[ASending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
[2018-07-06T23:12:09,630][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-07-06T23:12:19,638][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@centos-02 bin]#
1.安裝nginx
https://my.oschina.net/andyfeng/blog/1634805
[root@centos-02 yum.repos.d]# rpm -ivh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm 獲取http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm 警告：/var/tmp/rpm-tmp.RGilFh: 頭V4 RSA/SHA1 Signature, 密鑰 ID 7bd9bf62: NOKEY 準備中... ################################# [100%] 正在升級/安裝... 1:nginx-release-centos-7-0.el7.ngx ################################# [100%] [root@centos-02 yum.repos.d]# yum install -y nginx
2.啓動nginx
[root@centos-02 yum.repos.d]# systemctl start nginx [root@centos-02 yum.repos.d]# ps aux|grep nginx root 6091 1.0 0.0 48524 968 ? Ss 23:33 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 6092 0.3 0.1 48928 1920 ? S 23:33 0:00 nginx: worker process root 6094 0.0 0.0 112660 964 pts/0 R+ 23:33 0:00 grep --color=auto nginx [root@centos-02 yum.repos.d]#
[root@centos-02 yum.repos.d]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 6091/nginx: master
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master
tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java
tcp6 0 0 :::10514 :::* LISTEN 5828/java
tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java
tcp6 0 0 :::22 :::* LISTEN 1437/sshd
tcp6 0 0 ::1:25 :::* LISTEN 2350/master
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 5828/java
[root@centos-02 yum.repos.d]#
3.創建虛擬主機並配置代理ip爲kibana所在的服務器
[root@centos-02 conf.d]# vim elk.conf^C
[root@centos-02 conf.d]# cat elk.conf
server {
listen 80;
server_name elk.test.com;
location / {
proxy_pass http://192.168.242.131:5601;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /tmp/elk_access.log main2;
}
[root@centos-02 conf.d]#
4.編輯nginx的配置文件，在http選項中添加main2日誌配置
[root@centos-02 nginx]# vim nginx.conf
log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$upstream_addr" $request_time';
5.檢查配置文件是否有錯誤
[root@centos-02 usr]# sbin/nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@centos-02 usr]#
6.從新加載配置文件
[root@centos-02 usr]# /usr/sbin/nginx -s reload [root@centos-02 usr]#
[root@centos-02 conf.d]# mv default.conf default.conf.bak [root@centos-02 conf.d]# ls default.conf.bak elk.conf [root@centos-02 conf.d]#
7.將detault.conf重命名再從新加載配置文件
[root@centos-02 conf.d]# /usr/sbin/nginx -s reload [root@centos-02 conf.d]#
http://elk.test.com/app/kibana
8.檢查是否生成了，已經生成了，而且有日誌內容
[root@centos-02 conf.d]# ls /tmp/elk_access.log /tmp/elk_access.log [root@centos-02 conf.d]#
1.再次測試logstash配置文件是否正確
[root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCTh
reads=N Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties [2018-07-07T00:37:57,356][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [2018-07-07T00:38:03,846][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@centos-02 bin]#
2.重啓logstash
[root@centos-02 bin]# systemctl restart logstash [root@centos-02 bin]# ps aux|grep logstash logstash 6250 89.5 31.3 3180544 313724 ? SNsl 00:39 0:31 /bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOcc
upancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /u
sr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-19.0.
jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson
-databind-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/share/logstash/logstash-core/
lib/jars/jruby-complete-9.1.13.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/share/logstash/logstash-
core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/
logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/
lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.reso
urces-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/
logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/
lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.o
sgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings
/etc/logstash root 6276 0.0 0.0 112660 964 pts/0 S+ 00:40 0:00 grep --color=auto logstash [root@centos-02 bin]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 6091/nginx: master tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java tcp6 0 0 :::10514 :::* LISTEN 6250/java tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java tcp6 0 0 :::22 :::* LISTEN 1437/sshd tcp6 0 0 ::1:25 :::* LISTEN 2350/master tcp6 0 0 127.0.0.1:9600 :::* LISTEN 6250/java [root@centos-02 bin]#
3.再次查看有沒有生成nginx-test
[root@centos-01 local]# curl '192.168.242.131:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open system-syslog-2018.07 Nh8lRsu3T56U1BIE-_mLnw 5 1 113 0 914.8kb 484kb green open .kibana n2dgGyqCRs-9KTIs1A-uAw 1 1 2 0 22kb 11kb green open nginx-test-2018.07.06 -Nkch6RJTt2bFVyl-2XpHA 5 1 56 0 227.3kb 121.1kb [root@centos-01 local]#
4.配置kibana的索引，點擊index patterns
5.點擊create index pattern
6.建立新索引
7.點擊discover查看nginx-test
使用beats採集日誌
輕量型數據採集器
1.下載filebeat
[root@centos-03 src]# pwd /usr/local/src [root@centos-03 src]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.1-x86_64.rpm
2.安裝filebeat
[root@centos-03 src]# rpm -ivh filebeat-6.3.1-x86_64.rpm
3.編輯配置文件
# Change to true to enable this input configuration.
enabled: true （改成true）
paths:
- /var/log/messages (修改paths)
#- c:\programdata\elasticsearch\logs\*
# Configure what output to use when sending the data collected by the beat.
output.console: （新增）
enable: true （新增）
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch: （註釋掉）
# Array of hosts to connect to.
# hosts: ["localhost:9200"] （註釋掉）
4.啓動filebeat
[root@centos-03 src]# /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml
5.用02服務器登陸03服務器看看是否有日誌生成，成功生成
[root@centos-02 ~]# ssh centos-03
root@centos-03's password:
Last login: Mon Jul 9 19:29:07 2018 from centos-02
[root@centos-03 ~]#
","offset":1680,"message":"Jul 9 19:29:40 centos-03 systemd-logind: New session 6 of user root.","input":{"type":"log"}}
{"@timestamp":"2018-07-09T11:29:42.186Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.3.1"},"message":"Jul 9 19:29:40 centos-03 systemd: Starting Session 6 of user root.","prospect
or":{"type":"log"},"input":{"type":"log"},"beat":{"name":"centos-03","hostname":"centos-03","version":"6.3.1"},"host":{"name":"centos-03"},"source":"/var/log/messages","offset":1750}
6.filebeat已服務形式啓動
1.編輯配置文件，咱們將paths指定的日誌文件路徑修改日誌路徑
[root@centos-03 src]# ls /var/log/elasticsearch/centos.log /var/log/elasticsearch/centos.log [root@centos-03 src]#
[root@centos-03 src]# vim /etc/filebeat/filebeat.yml
paths:
- /var/log/elasticsearch/centos.log
# Configure what output to use when sending the data collected by the beat. #output.console: （註釋掉） # enable: true （註釋掉） #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: （打開） # Array of hosts to connect to. hosts: ["192.168.242.131:9200"] （打開）
2.啓動filebeat
[root@centos-03 src]# systemctl start filebeat [root@centos-03 src]# ps aux|grep filebeat root 4999 0.2 1.2 360060 12772 ? Ssl 19:49 0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat root 5008 0.0 0.0 112660 964 pts/0 R+ 19:50 0:00 grep --color=auto filebeat [root@centos-03 src]#
3.查看es服務器是否生成了新的索引
[root@centos-01 ~]# curl '192.168.242.131:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open system-syslog-2018.07 Nh8lRsu3T56U1BIE-_mLnw 5 1 34663 0 8.7mb 4.3mb green open .kibana n2dgGyqCRs-9KTIs1A-uAw 1 1 3 0 35.9kb 17.9kb green open nginx-test-2018.07.06 -Nkch6RJTt2bFVyl-2XpHA 5 1 34856 0 8.7mb 4.3mb green open filebeat-6.3.1-2018.07.09 CbdrEwpqTEK1WJdXGZg92g 3 1 412 0 281kb 151.4kb [root@centos-01 ~]#
4.在kibana上創建filebeat日誌配置（和以前作過的配置日誌同樣）