使用goaccess分析nginx日誌

GoAccess簡介

GoAccess 是一款開源(MIT許可證)的且具備交互視圖界面的實時 Web 日誌分析工具，經過你的 Web 瀏覽器或者 *nix 系統下的終端程序便可訪問。

能爲系統管理員提供快速且有價值的 HTTP 統計，並以在線可視化服務器的方式呈現。 GoAccess 解析指定的 Web 日誌文件並將統計結果輸出到 X 終端。功能以下：

• 通用統計: 此面板展現了幾個主要指標，好比：有效和無效請求的數量，分析這些數據所花費的時間，獨立訪客的狀況，請求的文件，靜態文件(CSS, ICO, JPG 等)的完整URL，404錯誤，被解析的日誌文件的大小以及消耗的帶寬。
• 獨立訪客: 此面板按照日期展現了訪問次數，獨立訪客數，以及累計消耗的帶寬等指標。具備相同IP，相同訪問時間，相同的 UserAgent 的 HTTP 請求將會被識別爲獨立訪客。默認狀況下包含了網絡爬蟲。
  您也能夠選擇使用 --date-spec=hr 參數將按照日期分析修改成按照小時，例如：05/Jun/2016:16 。這對於但願在小時級別去跟蹤每日流量很是有幫助。
• 請求的文件: 此面板展現您服務器上被請求次數最多的文件。包含訪問次數，獨立訪客數，百分比，累計消耗帶寬，使用的協議，請求方式。
• 請求的靜態文件: 列出請求頻率最高的靜態文件類型，例如： JPG, CSS, SWF, JS, GIF, 和 PNG , 以及和上一個面板同樣的其餘指標。 另外靜態文件能夠被添加到配置文件中。
• 404 或者文件未找到: 展現內容與以前的面板相似，可是其數據包含了全部未找到的頁面，以及俗稱的 404 狀態碼。
• 主機: 此面板展現主機自身的詳細信息。可以很好的發現不懷好意的爬蟲以及識別出是誰吃掉了你的帶寬。
  擴展面板將向您展現更多信息，好比主機的反向DNS解析結果，主機所在國家和城市。若是開啓了 參數，選擇想查看的 IP 地址並按回車，將會顯示 UserAgent 列表。
• 操做系統: 此面板將顯示主機使用的操做系統的信息。GoAccess 將盡量嘗試爲每一款操做系統提供詳細的信息。
• 瀏覽器: 此面板將顯示來訪主機使用的瀏覽器信息。GoAccess 將盡量嘗試爲每一款瀏覽器提供詳細的信息。
• 訪問次數: 此面板按小時報告。所以將顯示24個數據點，每個均對應每一天的某一個小時。
  使用 --hour-spec=min 參數能夠設定爲按每十分鐘報告，並將以 16:4 的格式顯示時間。這對發現服務器的峯值訪問時段頗有幫助。
• 虛擬主機: 此面板將顯示從訪問日誌中解析出來的不一樣的虛擬主機的狀況。此面板僅在日誌格式中啓用了 %v 參數時顯示。
• 來路URL: 若是問題主機經過其餘的資源訪問了你的站點，以及經過從其餘主機上的連接或者跳轉到你的站點，則這些來路URL將會被顯示在此面板。能夠在配置文件中經過 --ignore-panel 開啓此功能。(默認關閉)
• 來路站點: 此面板將僅顯示主機的部分，而不是完整的URL。
• 關鍵字: 報告支持用在谷歌搜索，谷歌緩存，谷歌翻譯上使用關鍵字。目前僅支持經過 HTTP 使用谷歌搜索。 能夠在配置文件中經過 --ignore-panel 開啓此功能。(默認關閉)
• 地理位置: 根據 IP 地址判斷地理位置。統計數據按照大洲和國家分組。須要地理位置模塊的支持。
• HTTP 狀態碼: 以數字表示的 HTTP 請求的狀態編碼。
• 遠程用戶(HTTP驗證) 經過 HTTP 驗證來肯定訪問文檔的權限。若是文檔沒有被密碼保護起來，這部分將會顯示爲 「-」。此面板默認爲開啓，除非在日誌格式變量中設置了參數 %e 。

GoAccess使用

安裝goaccess

[root@VM_0_26_centos logs]# yum install goaccess
Loaded plugins: fastestmirror, langpacks
Repository epel is listed more than once in the configuration
epel                                                                | 4.7 kB  00:00:00     
extras                                                              | 2.9 kB  00:00:00     
nux-dextop                                                          | 2.9 kB  00:00:00     
os                                                                  | 3.6 kB  00:00:00     
rpmfusion-free-updates                                              | 3.7 kB  00:00:00     
rpmfusion-nonfree-updates                                           | 3.7 kB  00:00:00     
updates                                                             | 2.9 kB  00:00:00     
zabbix                                                              | 2.9 kB  00:00:00     
zabbix-non-supported                                                |  951 B  00:00:00     
(1/2): epel/7/x86_64/updateinfo                                     | 1.0 MB  00:00:00     
(2/2): epel/7/x86_64/primary_db                                     | 6.9 MB  00:00:02     
Loading mirror speeds from cached hostfile
 * nux-dextop: mirror.li.nux.ro
 * rpmfusion-free-updates: mirrors.ustc.edu.cn
 * rpmfusion-nonfree-updates: mirrors.ustc.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package goaccess.x86_64 0:1.3-1.el7 will be installed
--> Processing Dependency: libtokyocabinet.so.9()(64bit) for package: goaccess-1.3-1.el7.x86_64
--> Running transaction check
---> Package tokyocabinet.x86_64 0:1.4.48-3.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================
 Package                 Arch              Version                   Repository       Size
===========================================================================================
Installing:
 goaccess                x86_64            1.3-1.el7                 epel            240 k
Installing for dependencies:
 tokyocabinet            x86_64            1.4.48-3.el7              os              459 k

Transaction Summary
===========================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 699 k
Installed size: 2.0 M
Is this ok [y/d/N]: y
Downloading packages:
(1/2): goaccess-1.3-1.el7.x86_64.rpm                                | 240 kB  00:00:00     
(2/2): tokyocabinet-1.4.48-3.el7.x86_64.rpm                         | 459 kB  00:00:00     
-------------------------------------------------------------------------------------------
Total                                                      1.3 MB/s | 699 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : tokyocabinet-1.4.48-3.el7.x86_64                                        1/2 
  Installing : goaccess-1.3-1.el7.x86_64                                               2/2 
  Verifying  : tokyocabinet-1.4.48-3.el7.x86_64                                        1/2 
  Verifying  : goaccess-1.3-1.el7.x86_64                                               2/2 

Installed:
  goaccess.x86_64 0:1.3-1.el7                                                              

Dependency Installed:
  tokyocabinet.x86_64 0:1.4.48-3.el7

查看使用方式

[root@VM_0_26_centos logs]# goaccess -help

GoAccess - 1.3

Usage: goaccess [filename] [ options ... ] [-c][-M][-H][-S][-q][-d][...]
The following options can also be supplied to the command:

Log & Date Format Options

  --date-format=<dateformat>      - Specify log date format. e.g., %d/%b/%Y
  --log-format=<logformat>        - Specify log format. Inner quotes need to be
                                    escaped, or use single quotes.
  --time-format=<timeformat>      - Specify log time format. e.g., %H:%M:%S

User Interface Options

  -c --config-dialog              - Prompt log/date/time configuration window.
  -i --hl-header                  - Color highlight active panel.
  -m --with-mouse                 - Enable mouse support on main dashboard.
  --color=<fg:bg[attrs, PANEL]>   - Specify custom colors. See manpage for more
                                    details and options.
  --color-scheme=<1|2|3>          - Schemes: 1 => Grey, 2 => Green, 3 => Monokai.
  --html-custom-css=<path.css>    - Specify a custom CSS file in the HTML report.
  --html-custom-js=<path.js>      - Specify a custom JS file in the HTML report.
  --html-prefs=<json_obj>         - Set default HTML report preferences.
  --html-report-title=<title>     - Set HTML report page title and header.
  --json-pretty-print             - Format JSON output w/ tabs & newlines.
  --max-items                     - Maximum number of items to show per panel.
                                    See man page for limits.
  --no-color                      - Disable colored output.
  --no-column-names               - Don't write column names in term output.
  --no-csv-summary                - Disable summary metrics on the CSV output.
  --no-html-last-updated          - Hide HTML last updated field.
  --no-parsing-spinner            - Disable progress metrics and parsing spinner.
  --no-progress                   - Disable progress metrics.
  --no-tab-scroll                 - Disable scrolling through panels on TAB.

Server Options

  --addr=<addr>                   - Specify IP address to bind server to.
  --daemonize                     - Run as daemon (if --real-time-html enabled).
  --fifo-in=<path>                - Path to read named pipe (FIFO).
  --fifo-out=<path>               - Path to write named pipe (FIFO).
  --origin=<addr>                 - Ensure clients send the specified origin header
                                    upon the WebSocket handshake.
  --pid-file=<path>               - Write PID to a file when --daemonize is used.
  --port=<port>                   - Specify the port to use.
  --real-time-html                - Enable real-time HTML output.
  --ssl-cert=<cert.crt>           - Path to TLS/SSL certificate.
  --ssl-key=<priv.key>            - Path to TLS/SSL private key.
  --ws-url=<url>                  - URL to which the WebSocket server responds.

File Options

  -                               - The log file to parse is read from stdin.
  -f --log-file=<filename>        - Path to input log file.
  -S --log-size=<number>          - Specify the log size, useful when piping in logs.
  -l --debug-file=<filename>      - Send all debug messages to the specified
                                    file.
  -p --config-file=<filename>     - Custom configuration file.
  --invalid-requests=<filename>   - Log invalid requests to the specified file.
  --no-global-config              - Don't load global configuration file.

Parse Options

  -a --agent-list                 - Enable a list of user-agents by host.
  -b --browsers-file=<path>       - Use additional custom list of browsers.
  -d --with-output-resolver       - Enable IP resolver on HTML|JSON output.
  -e --exclude-ip=<IP>            - Exclude one or multiple IPv4/6. Allows IP
                                    ranges e.g. 192.168.0.1-192.168.0.10
  -H --http-protocol=<yes|no>     - Set/unset HTTP request protocol if found.
  -M --http-method=<yes|no>       - Set/unset HTTP request method if found.
  -o --output=file.html|json|csv  - Output either an HTML, JSON or a CSV file.
  -q --no-query-string            - Ignore request's query string. Removing the
                                    query string can greatly decrease memory
                                    consumption.
  -r --no-term-resolver           - Disable IP resolver on terminal output.
  --444-as-404                    - Treat non-standard status code 444 as 404.
  --4xx-to-unique-count           - Add 4xx client errors to the unique visitors
                                    count.
  --anonymize-ip                  - Anonymize IP addresses before outputting to report.
  --all-static-files              - Include static files with a query string.
  --crawlers-only                 - Parse and display only crawlers.
  --date-spec=<date|hr>           - Date specificity. Possible values: `date`
                                    (default), or `hr`.
  --double-decode                 - Decode double-encoded values.
  --enable-panel=<PANEL>          - Enable parsing/displaying the given panel.
  --hide-referer=<NEEDLE>         - Hide a referer but still count it. Wild cards
                                    are allowed. i.e., *.bing.com
  --hour-spec=<hr|min>            - Hour specificity. Possible values: `hr`
                                    (default), or `min` (tenth of a min).
  --ignore-crawlers               - Ignore crawlers.
  --ignore-panel=<PANEL>          - Ignore parsing/displaying the given panel.
  --ignore-referer=<NEEDLE>       - Ignore a referer from being counted. Wild cards
                                    are allowed. i.e., *.bing.com
  --ignore-statics=<req|panel>    - Ignore static requests.
                                    req => Ignore from valid requests.
                                    panel => Ignore from valid requests and panels.
  --ignore-status=<CODE>          - Ignore parsing the given status code.
  --num-tests=<number>            - Number of lines to test. >= 0 (10 default)
  --process-and-exit              - Parse log and exit without outputting data.
  --real-os                       - Display real OS names. e.g, Windows XP, Snow
                                    Leopard.
  --sort-panel=PANEL,METRIC,ORDER - Sort panel on initial load. For example:
                                    --sort-panel=VISITORS,BY_HITS,ASC. See
                                    manpage for a list of panels/fields.
  --static-file=<extension>       - Add static file extension. e.g.: .mp3.
                                    Extensions are case sensitive.

GeoIP Options

  -g --std-geoip                  - Standard GeoIP database for less memory
                                    consumption.
  --geoip-database=<path>         - Specify path to GeoIP database file. i.e.,
                                    GeoLiteCity.dat, GeoIPv6.dat ...

Other Options

  -h --help                       - This help.
  -V --version                    - Display version information and exit.
  -s --storage                    - Display current storage method. e.g., B+
                                    Tree, Hash.
  --dcf                           - Display the path of the default config
                                    file when `-p` is not used.

Examples can be found by running `man goaccess`.

For more details visit: http://goaccess.io
GoAccess Copyright (C) 2009-2017 by Gerardo Orellana

獲取Nginx日誌格式
格式轉換腳本在https://github.com/stockrt/nginx2goaccess/blob/master/nginx2goaccess.sh，具體內容以下

[root@VM_0_26_centos logs]# cat nginx2goaccess.sh 
#!/bin/bash
#
# Convert from this:
#   http://nginx.org/en/docs/http/ngx_http_log_module.html
# To this:
#   https://goaccess.io/man
#
# Conversion table:
#   $time_local         %d:%t %^
#   $host               %v
#   $http_host          %v
#   $remote_addr        %h
#   $request_time       %T
#   $request_method     %m
#   $request_uri        %U
#   $server_protocol    %H
#   $request            %r
#   $status             %s
#   $body_bytes_sent    %b
#   $bytes_sent         %b
#   $http_referer       %R
#   $http_user_agent    %u
#
# Samples:
#
# log_format combined '$remote_addr - $remote_user [$time_local] '
# '"$request" $status $body_bytes_sent '
# '"$http_referer" "$http_user_agent"';
#   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
#
# log_format compression '$remote_addr - $remote_user [$time_local] '
# '"$request" $status $bytes_sent '
# '"$http_referer" "$http_user_agent" "$gzip_ratio"';
#   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent" "$gzip_ratio"'
#
# log_format main
# '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\t'
# 'Local:\t$status\t$body_bytes_sent\t$request_time\t'
# 'Proxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\t'
# 'Agent:\t$http_user_agent\t'
# 'Fwd:\t$http_x_forwarded_for';
#   ./nginx2goaccess.sh '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\tLocal:\t$status\t$body_bytes_sent\t$request_time\tProxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\tAgent:\t$http_user_agent\tFwd:\t$http_x_forwarded_for'
#
# log_format main
# '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t'
# '${http_referer}\t${http_x_mobile_group}\t'
# 'Local:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\t'
# 'Proxy:\t${upstream_status}\t${upstream_cache_status}\t'
# '${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\t'
# 'Agent:\t${http_user_agent}\t'
# 'Fwd:\t${http_x_forwarded_for}';
#   ./nginx2goaccess.sh '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t${http_referer}\t${http_x_mobile_group}\tLocal:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\tProxy:\t${upstream_status}\t${upstream_cache_status}\t${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\tAgent:\t${http_user_agent}\tFwd:\t${http_x_forwarded_for}'
#
# Author: Rogério Carvalho Schneider <stockrt@gmail.com>

# Params
log_format="$1"

# Usage
if [[ -z "$log_format" ]]; then
    echo "Usage: $0 '<log_format>'"
    exit 1
fi

# Variables map
conversion_table="time_local,%d:%t_%^
host,%v
http_host,%v
remote_addr,%h
request_time,%T
request_method,%m
request_uri,%U
server_protocol,%H
request,%r
status,%s
body_bytes_sent,%b
bytes_sent,%b
http_referer,%R
http_user_agent,%u"

# Conversion
for item in $conversion_table; do
    nginx_var=${item%%,*}
    goaccess_var=${item##*,}
    goaccess_var=${goaccess_var//_/ }
    log_format=${log_format//\$\{$nginx_var\}/$goaccess_var}
    log_format=${log_format//\$$nginx_var/$goaccess_var}
done
log_format=$(echo "$log_format" | sed 's/${[a-z_]*}/%^/g')
log_format=$(echo "$log_format" | sed 's/$[a-z_]*/%^/g')

# Config output
echo "
- Generated goaccess config:
time-format %T
date-format %d/%b/%Y
log_format $log_format
"

# EOF

注意，其中nginx配置文件的log_format以下，下面轉換時須要與實際狀況保持一致

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $upstream_addr $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';

獲取日誌格式

[root@VM_0_26_centos logs]# sh nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $upstream_addr $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for"'

- Generated goaccess config:
time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %^ %b "%R" "%u" "%^"

設置日誌格式

[root@VM_0_26_centos logs]# cat /etc/goaccess/goaccess.conf 
time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %^ %b "%R" "%u" "%^"

生成分析報告

[root@VM_0_26_centos logs]# goaccess -f ./nginx_access.log -p ./nginxlog.conf -o day-report.html
[root@VM_0_26_centos logs]# ls
day-report.html      nginx_access.log             nginx2goaccess.sh           nginxlog.conf

查看報告效果
瀏覽器打開day-report.html，效果以下