第七章·Logstash深刻-收集NGINX日誌

1.NGINX安裝配置

源碼安裝nginx

由於資源問題，咱們先將nginx安裝在Logstash所在機器

#安裝nginx依賴包 [root@elkstack03 ~]# yum install -y gcc gcc-c++ automake pcre-devel zlib-devel openssl-devel #下載nginx安裝包 [root@elkstack03 ~]# wget http://nginx.org/download/nginx-1.10.3.tar.gz #解壓 [root@elkstack03 ~]# tar xf nginx-1.10.3.tar.gz #進入nginx安裝目錄 [root@elkstack03 ~]# cd nginx-1.10.3/ #生成編譯文件 [root@elkstack03 nginx-1.10.3]# ./configure --prefix=/usr/local/nginx-1.10.3 #編譯 [root@elkstack03 nginx-1.10.3]# make #安裝 [root@elkstack03 nginx-1.10.3]# make install #作軟連接 [root@elkstack03 nginx-1.10.3]# ln -s /usr/local/nginx-1.10.3 /usr/local/nginx #檢測nginx語法 [root@elkstack03 nginx-1.10.3]# /usr/local/nginx/sbin/nginx -t nginx: the configuration file /usr/local/nginx-1.10.3/conf/nginx.conf syntax is ok nginx: configuration file /usr/local/nginx-1.10.3/conf/nginx.conf test is successful #啓動nginx [root@elkstack03 nginx-1.10.3]# /usr/local/nginx/sbin/nginx 

---

配置nginx

#簡化nginx配置文件 [root@elkstack03 ~]# grep -Ev '#|^$' /usr/local/nginx/conf/nginx.conf.default > /usr/local/nginx/conf/nginx.conf #編輯nginx配置文件 [root@elkstack03 ~]# vim /usr/local/nginx/conf/nginx.conf worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location / { root /code/html; index index.html index.htm; } } } #建立nginx站點目錄 [root@elkstack03 ~]# mkdir /code/html #寫測試頁面 [root@elkstack03 ~]# echo zls nginx test page > /code/html/index.html #從新加載nginx [root@elkstack03 ~]# /usr/local/nginx/sbin/nginx -s reload 

打開瀏覽器，訪問：http://10.0.0.53/


---

修改nginx日誌格式爲Json

以前咱們講了tomcat日誌，在企業中，修改格式須要與開發商量，可是nginx咱們不須要，若是須要原來的格式日誌，咱們能夠將日誌輸出兩份，一份 main格式，一份Json格式

#編輯nginx日誌，添加日誌格式，源main格式和Json格式 [root@elkstack03 ~]# vim /usr/local/nginx/conf/nginx.conf worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; #main格式日誌 log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log logs/access.log main; #Json格式日誌 log_format access_json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"url":"$uri",' '"domain":"$host",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"status":"$status"}'; access_log logs/access_json.log access_json; server { listen 80; server_name 10.0.0.53; location / { root /code/html; index index.html index.htm; } } } #檢測nginx配置文件語法 [root@elkstack03 ~]# /usr/local/nginx/sbin/nginx -t nginx: the configuration file /usr/local/nginx-1.10.3/conf/nginx.conf syntax is ok nginx: configuration file /usr/local/nginx-1.10.3/conf/nginx.conf test is successful #從新加載nginx [root@elkstack03 ~]# /usr/local/nginx/sbin/nginx -s reload 

打開瀏覽器，訪問：http://10.0.0.53/ 查看日誌

#進入nginx日誌目錄 [root@elkstack03 ~]# cd /usr/local/nginx/logs/ #查看目錄中日誌 [root@elkstack03 logs]# ll 總用量 24 #修改後的Json格式日誌 -rw-r--r-- 1 root root 1280 4月 8 10:47 access_json.log #源main格式日誌 -rw-r--r-- 1 root root 5286 4月 8 10:47 access.log -rw-r--r-- 1 root root 4218 4月 8 10:46 error.log -rw-r--r-- 1 root root 5 4月 8 10:20 nginx.pid #查看Json格式日誌 [root@elkstack03 logs]# cat access_json.log {"@timestamp":"2019-04-08T10:47:41+08:00","host":"10.0.0.53","clientip":"10.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.53","url":"/index.html","domain":"10.0.0.53","xff":"-","referer":"-","status":"304"} #查看main格式日誌 [root@elkstack03 logs]# cat access.log 10.0.0.1 - - [08/Apr/2019:10:29:11 +0800] "GET / HTTP/1.1" 404 571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 

結果以下：



---

經過Logstash收集nginx日誌輸出到ES中

[root@elkstack03 ~]# cd /etc/logstash/conf.d/ [root@elkstack03 conf.d]# vim nginx_es.conf input { file { path => "/usr/local/nginx/logs/access_json.log" start_position => "end" type => "nginx_access" codec => json } } output { elasticsearch { hosts => ["10.0.0.51:9200"] index => "nginx_access-%{+YYYY.MM.dd}" } } #檢測Logstash語法 [root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx_es.conf -t #啓動Logstash [root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx_es.conf & 

打開瀏覽器，訪問：http://10.0.0.51:9100/



---

將ES中的索引添加到Kibana中

打開瀏覽器，訪問：http://10.0.0.54:5601/ Kibana頁面