Linux Centos7.4--apache日誌分割，日誌管理分析

時間 2019-11-06

標籤 linux centos7.4 centos apache 日誌分割管理分析欄目 Linux 简体版

原文原文鏈接

Apache日誌分割

日誌分割有兩種方式，第一個是apache自帶的rotatelogs分割工具實現，第二個是第三方的工具cronnolog分割php

apache自帶的分割工具rotatelogs

[root@client ~]# yum install bind httpd -y
[root@client ~]# cd /usr/sbin/
[root@client sbin]# ls rotat*
rotatelogs
//安裝好apache才能在系統能使用的命令底下看到

配置apache主配置文件，開啓服務

[root@client sbin]# vim /etc/httpd/conf/httpd.conf 
Listen 192.168.136.128:80   //監聽你本地的地址
#Listen 80  //把ipv6的監聽端口註釋掉
#If your host doesn't have a registered DNS name, enter its IP address here.
ServerName www.kgc.com:80  //把本來的改爲你定義的域名，並開啓
[root@client httpd]# systemctl stop firewalld.service 
[root@client httpd]# setenforce 0
[root@client httpd]# systemctl start httpd

[root@client httpd]# ls  //日誌文件在服務開啓的時候纔有
access_log  error_log
[root@client httpd]# cat access_log   //訪問日誌是空的
[root@client httpd]# cat error_log    //錯誤日誌文件，這些是PID的進程使用，不表明又一些錯誤
[Wed Oct 23 21:18:24.917418 2019] [core:notice] [pid 4429] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Oct 23 21:18:24.918499 2019] [suexec:notice] [pid 4429] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Oct 23 21:18:24.938959 2019] [lbmethod_heartbeat:notice] [pid 4429] AH02282: No slotmem from mod_heartmonitor
[Wed Oct 23 21:18:24.941240 2019] [mpm_prefork:notice] [pid 4429] AH00163: Apache/2.4.6 (CentOS) configured -- resuming normal operations
[Wed Oct 23 21:18:24.941268 2019] [core:notice] [pid 4429] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

去客戶段去測試一下

回到服務器當中看一下訪問日誌

若是來訪者愈來愈多，日誌就愈來愈多，咱們人看不過來，咱們就須要進行分割

[root@client httpd]# cat access_log 
192.168.136.134 - - [23/Oct/2019:21:24:55 +0800] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240"
192.168.136.134 - - [23/Oct/2019:21:24:55 +0800] "GET / HTTP/1.1" 403 4897 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240"
192.168.136.134 - - [23/Oct/2019:21:24:56 +0800] "GET /noindex/css/bootstrap.min.css HTTP/1.1" 200 19341 "http://192.168.136.128/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240"
192.168.136.134 - - [23/Oct/2019:21:24:56 +0800] "GET /images/apache_pb.gif HTTP/1.1" 200 2326 "http://192.168.136.128/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240"
192.168.136.134 - - [23/Oct/2019:21:24:56 +0800] "GET /noindex/css/open-sans.cs

日誌分割配置

[root@client httpd]# vim /etc/httpd/conf/httpd.conf 
#ErrorLog "logs/error_log"  //咱們把原來的註釋掉，線網上的技巧，萬一錯了能夠恢復。
ErrorLog "| /usr/sbin/rotatelogs -l logs/www.kgc.com.error_%Y%m%dlog 86400" //找到這一行，/輸入管道符號「|」 跟你係統apache命令的絕對路徑，起個名字www.kgc.vom，%Y%m%d固定格式表明年月日，86400表明一天的時間86400秒

    CustomLog "| /usr/sbin/rotatelogs -l logs/www.kgc.com.access_%Y%m%dlog 86400" combined

        //這邊也是同樣，找到這一行

驗證日誌分割

[root@client httpd]# systemctl stop httpd
[root@client httpd]# systemctl start httpd
[root@client httpd]# ls
access_log error_log www.kgc.com.error_20191023log //咱們當天的日誌就被分割出來了
[root@client httpd]# date -s 10/24
2019年 10月 24日星期四 00:00:00 CST
[root@client httpd]# systemctl stop httpd
[root@client httpd]# systemctl start httpd
[root@client httpd]# ls
access_log error_log www.kgc.com.error_20191023log www.kgc.com.error_20191024log //咱們改了一下時間，日誌分割出來了css

第三方工具日誌分割，我這有包，要的私信我

[root@localhost httpd]# smbclient -L //192.168.100.3  //記得你的宿主機Vmnet1網卡要設置成192.168.100.3
Enter SAMBA\root's password: 
OS=[Windows 10 Enterprise 17763] Server=[Windows 10 Enterprise 6.3]

    Sharename       Type      Comment
    ---------       ----      -------
    ADMIN$          Disk      遠程管理
    C$              Disk      默認共享
    D$              Disk      默認共享
    E$              Disk      默認共享
    F$              Disk      默認共享
    G$              Disk      默認共享
    IPC$            IPC       遠程 IPC
    LAMP            Disk      
    LAMP-C7         Disk      
    share           Disk      
    Users           Disk      
Connection to 192.168.100.3 failed (Error NT_STATUS_RESOURCE_NAME_NOT_FOUND)
NetBIOS over TCP disabled -- no workgroup available
[root@localhost httpd]# cd ~
[root@localhost ~]# mkdir /abc  //建立掛載點
[root@localhost ~]# mount.cifs //192.168.100.3/LAMP-C7 /abc/  //把宿主機的文件掛載到咱們的掛載點中
Password for root@//192.168.100.3/LAMP-C7:  
[root@localhost ~]# cd /abc/
[root@localhost abc]# ls
apr-1.6.2.tar.gz                  Discuz_X2.5_SC_UTF8.zip  mysql-5.6.26.tar.gz
apr-util-1.6.0.tar.gz             fiddler.exe              php-5.6.11.tar.bz2
awstats-7.6.tar.gz                httpd-2.4.29.tar.bz2
cronolog-1.6.2-14.el7.x86_64.rpm  LAMP-php5.6.txt
[root@localhost abc]# rpm -ivh cronolog-1.6.2-14.el7.x86_64.rpm //安裝這個第三方分割工具包
警告：cronolog-1.6.2-14.el7.x86_64.rpm: 頭V3 RSA/SHA256 Signature, 密鑰 ID 352c64e5: NOKEY
準備中...                          ################################# [100%]
正在升級/安裝...
   1:cronolog-1.6.2-14.el7            ################################# [100%]
[root@localhost abc]# cd /usr/sbin/
[root@localhost sbin]# ls cronolog*  //到系統命令中查看有沒有這個命令
cronolog

第三方工具日誌分割

[root@localhost sbin]# vim /etc/httpd/conf/httpd.conf 
ErrorLog "| /usr/sbin/cronolog logs/www.kgc.comerror_%Y%m%dlog"  //注意不要加-l 也沒有86400
CustomLog "| /usr/sbin/ cronolog logs/www.kgc.comaccess_%Y%m%dlog" combined
[root@localhost sbin]# systemctl stop httpd
[root@localhost sbin]# systemctl start httpd
[root@localhost sbin]# ls /var/log/httpd/
access_log  error_log  www.kgc.comerror_20191023log
[root@localhost sbin]# date -s 10/24
2019年 10月 24日 星期四 00:00:00 CST
[root@localhost sbin]# ls /var/log/httpd/
access_log  www.kgc.comerror_20191023log
error_log   www.kgc.comerror_20191024log

日誌分析

咱們換一臺虛擬機作，配置DNS主配置文件

[root@localhost ~]# yum install bind httpd -y //安裝DNS,Apache軟件包
[root@localhost ~]# vim /etc/named.conf 

options {
        listen-on port 53 { any; };  //改爲any
        listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        recursing-file  "/var/named/data/named.recursing";
        secroots-file   "/var/named/data/named.secroots";
        allow-query     { any; };  //改爲any

配置DNS區域配置文件

[root@localhost ~]# vim /etc/named.rfc1912.zones
zone "kgc.com" IN { //定義域名
type master;
file "kgc.com.zone"; //定義區域數據配置文件
allow-update { none; };
};
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0html

配置DNS區域數據配置文件

[root@localhost ~]# cd /var/named/
[root@localhost named]# cp -p named.localhost kgc.com.zone
[root@localhost named]# vim kgc.com.zone 
$TTL 1D
@       IN SOA  @ rname.invalid. (
                                        0       ; serial
                                        1D      ; refresh
                                        1H      ; retry
                                        1W      ; expire
                                        3H )    ; minimum
        NS      @
        A       127.0.0.1
www IN  A       192.168.136.135  //加入你本地地址

配置Apache主配置文件

[root@localhost named]# vim /etc/httpd/conf/httpd.conf 
Listen 192.168.136.135:80
#Listen 80
ServerName www.kgc.com:80
[root@localhost named]# systemctl stop firewalld.service 
[root@localhost named]# setenforce 0
[root@localhost named]# systemctl start httpd

去客戶端測試一下

日誌分析又有工具包，須要的私信我，

[root@localhost httpd]# mkdir /abc  //建立掛載點
[root@localhost httpd]# mount.cifs //192.168.100.3/LAMP-C7 /abc //掛載到abc
Password for root@//192.168.100.3/LAMP-C7:  
[root@localhost httpd]# cd /abc/
[root@localhost abc]# ls
apr-1.6.2.tar.gz                  Discuz_X2.5_SC_UTF8.zip  mysql-5.6.26.tar.gz
apr-util-1.6.0.tar.gz             fiddler.exe              php-5.6.11.tar.bz2
awstats-7.6.tar.gz      //這個包就是咱們的日誌分析工具包          httpd-2.4.29.tar.bz2
cronolog-1.6.2-14.el7.x86_64.rpm  LAMP-php5.6.txt

解壓，移動，利用咱們剛纔建立的這個腳本awstats，去配置apache,讓它去加載awstats的日誌分析於統計模塊

[root@localhost abc]# tar zxvf awstats-7.6.tar.gz -C /opt/  //jie'ya解壓到OPT下
awstats-7.6/
awstats-7.6/tools/
awstats-7.6/tools/awstats_buildstaticpages.pl
awstats-7.6/tools/awstats_updateall.pl
[root@localhost abc]# cd /opt/
[root@localhost opt]# ls
awstats-7.6  rh
[root@localhost opt]# mv awstats-7.6/ /usr/local/awstats //把這個數據包移動到/usr/local下起個名字awstats
[root@localhost opt]# ls
rh
[root@localhost opt]# cd /usr/local/  //查看
[root@localhost local]# ls
awstats  bin  etc  games  include  lib  lib64  libexec  sbin  share  src

在工具中作apache的路徑和域名

[root@localhost local]# cd awstats/
[root@localhost awstats]# ls
docs  README.md  tools  wwwroot
[root@localhost awstats]# cd tools/
[root@localhost tools]# ls
awstats_buildstaticpages.pl  dolibarr            maillogconvert.pl   xslt
awstats_configure.pl         geoip_generator.pl  nginx
awstats_exportlib.pl         httpd_conf          urlaliasbuilder.pl
awstats_updateall.pl         logresolvemerge.pl  webmin
[root@localhost tools]# ./awstats_configure.pl 

> /etc/httpd/conf/httpd.conf  //寫上apache路徑
file (required if first install) [y/N] ? y  //要不要創建一個新的apache文件
> www.kgc.com  //輸入你的域名
-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats  //跟你這個域名相關的配置文件，會生成到這個路徑下
Directory path to store config file(s) (Enter for default):
>   //直接回車

Press ENTER to continue...   //直接回車

> http://localhost/awstats/awstats.pl?config=www.kgc.com  //這個路徑就是它給你的日誌分析網頁
Press ENTER to finish...  //直接回車

檢查，配置apache有沒有加載這些模塊

[root@localhost tools]# cd /etc/httpd/conf
[root@localhost conf]# vim httpd.conf 

Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"
Alias /awstatscss "/usr/local/awstats/wwwroot/css/"
Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"
ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"
#This is to permit URL access to scripts/files in AWStats directory.
<Directory "/usr/local/awstats/wwwroot">  //目錄站點
    Options None
    AllowOverride None    //相關的權限控制
   # Order allow,deny   //把這兩行註釋掉
   # Allow from all
     Require all granted   //加入讓全部人能夠訪問

分析出來的數據要放在Apache訪問日誌中

[root@localhost etc]# cd /etc/awstats/
[root@localhost awstats]# ls
awstats.www.kgc.com.conf //剛纔生成的文件就是這個
[root@localhost awstats]# ls /var/log/httpd/ //
access_log error_log
[root@localhost awstats]# vim awstats.www.kgc.com.conf
LogFile="/var/log/httpd/access_log" //把原來的改爲access_log，分析這個日誌
DirData="/var/lib/awstats" //這個不須要改，咱們分析的數據放這個文件下mysql

建立awstats目錄才能讓系統識別把分析數據放進去

[root@localhost awstats]# cd /var/lib
[root@localhost lib]# ls awstats*
ls: 沒法訪問awstats*: 沒有那個文件或目錄
[root@localhost lib]# mkdir awstats
[root@localhost lib]# ls
AccountsService  dav       hyperv      net-snmp        rpcbind         tpm
alsa             dbus      initramfs   NetworkManager  rpm             tuned
alternatives     dhclient  ipa-client  nfs             rpm-state       udisks2
authconfig       dnsmasq   iscsi       ntp             rsyslog         upower
awstats     //這個就是     flatpak   libvirt    
[root@localhost lib]# systemctl restart httpd  //重啓服務

去客戶端測試一下

這只是一個分析頁面

輸入這個網址http://www.kgc.com/awstats/awstats.pl?config=www.kgc.com
nginx

更新數據才能統計

[root@localhost lib]# cd /usr/local/awstats/
[root@localhost awstats]# ls
docs  README.md  tools  wwwroot
[root@localhost awstats]# cd tools/
[root@localhost tools]# ls
awstats_buildstaticpages.pl  dolibarr            maillogconvert.pl   xslt
awstats_configure.pl         geoip_generator.pl  nginx
awstats_exportlib.pl         httpd_conf          urlaliasbuilder.pl
awstats_updateall.pl         logresolvemerge.pl  webmin
[root@localhost tools]# ./awstats_updateall.pl now
 //直接執行就好了，記得加上now最新的
Running '"/usr/local/awstats/wwwroot/cgi-bin/awstats.pl" -update -config=www.kgc.com -configdir="/etc/awstats"' to update config www.kgc.com
Create/Update database for config "/etc/awstats/awstats.www.kgc.com.conf" by AWStats version 7.6 (build 20161204)
From data in log file "/var/log/httpd/access_log"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
Jumped lines in file: 0
Parsed lines in file: 485
 Found 0 dropped records,
 Found 0 comments,
 Found 0 blank records,
 Found 1 corrupted records,
 Found 0 old records,
 Found 484 new qualified records.

回到客戶端測試一下能不能統計數據

全部的數據採集咱們每次都要去執行那個腳本，咱們這裏作個週期性計劃性任務讓它自動執行

[root@localhost tools]# crontab -e
*/5 * * * * /usr/local/awstats/tools/awstats_updateall.pl now
//每個月每週天天每小時每5分種去執行這個腳本
[root@localhost tools]# systemctl start crond //開啓

優化網頁地址

[root@localhost tools]# cd /var/www//html/
您在 /var/spool/mail/root 中有郵件
[root@localhost html]# ls
[root@localhost html]# vim aws.html
<html>
  <head>
    <meta http-equiv=refresh content="0;url=http://www.kgc.com/awstats/awstats.pl?config=www.kgc.com">
  <head>
  <body></body>
</html>