Nginx訪問日誌分析

nginx默認的日誌格式

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

字段說明

127.0.0.1 - - [14/May/2017:12:51:13 +0800] "GET /index.html HTTP/1.1" 200 4286 "http://127.0.0.1/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36"
遠程主機IP            請求時間       時區  方法    資源      協議   狀態碼 發送字節    referer      瀏覽器信息

統計訪問IP前十

# awk '{print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
   6958 123.174.51.164
   2307 111.85.34.165
   1617 118.112.143.148
   1489 117.63.146.40
   1404 118.182.116.39
   1352 1.48.219.30
   1132 60.222.231.46
   1129 10.35.1.82
    943 27.227.163.200
    880 58.253.6.133

統計指定某一天的訪問IP

# grep "17/May/2017" /usr/local/nginx/logs/access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -10
# awk '/17\/May\/2017/{print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
   6958 123.174.51.164
   2307 111.85.34.165
   1617 118.112.143.148
   1489 117.63.146.40
   1404 118.182.116.39
   1352 1.48.219.30
   1132 60.222.231.46
   1129 10.35.1.82
    943 27.227.163.200
    880 58.253.6.133

通過測試，在文件較大的時候，先grep再awk速度快不少。

過濾URL

# awk '{print $11}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
  20737 "http://www.adreambox.net/index.php?app=home&mod=User&act=index"
   4155 "-"
   3981 "http://www.adreambox.net/"
   1921 "http://www.adreambox.net/index.php?app=adreambox&mod=Class&act=prensent&id=5&type=2"
   1299 "http://www.adreambox.net/index.php?app=home&mod=Public&act=doLogin"
   1191 "http://www.adreambox.net/index.php?app=group&mod=Group&act=index&gid=1413"
    718 "http://www.adreambox.net/index.php?app=group&mod=Group&act=index&gid=1403"
    657 "http://www.adreambox.net/index.php?app=wap&mod=Index&act=index"
    657 "http://www.adreambox.net/index.php?act=index&app=home&mod=User"
    639 "http://www.adreambox.net/index.php?app=group&mod=Manage&act=index&gid=1413"

統計指定資源

# awk '($7~/\.html$/){print $1 " " $7 " " $9}' /usr/local/nginx/logs/access.log     #處理第7個字段以'.html'結尾的行
11.0.8.5 //ckeditor/notexist_path.html 404
11.0.8.5 //ckeditor/CHANGES.html 404
11.0.8.18 //docs/CHANGELOG.html 404
11.0.8.5 //themes/mall/default/seller_order.confirm.html 404
11.0.8.18 //themes/mall/default/header.html 404
11.0.8.5 //themes/store/default/footer.html 404
11.0.8.5 //templates/admin/index.html 404
11.0.8.5 //system/templates/admin/login.html 404
11.0.8.18 //templates/404.html 404
11.0.8.18 //admin/editor/editor/dialog/fck_about.html 404
11.0.8.5 //fckeditor/_whatsnew.html 404
11.0.8.5 //FCKeditor/_docs/whatsnew.html 404
11.0.8.5 //style/gb/help/index.html 404
10.10.1.11 /Login/login.html 404

過濾指定時間後的日誌並打印IP

# awk '($4>"[15/May/2017:21:16:38"){print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr
 291031 11.0.8.5
 274174 11.0.8.18
   2764 10.10.1.11
   1193 11.0.8.6
      1 127.0.0.1

統計流量

# grep "17/May/2017" /usr/local/nginx/logs/access.log | awk '{sum+=$10}END{print sum}'
95210093059

統計狀態碼

# awk '{print $9}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
1271257 200
 957444 503
  61875 502
  32852 404
  19121 302
  13356 304
   2819 500
   2789 400
    271 499
    203 401