用swoole實現nginx日誌解析

時間 2019-12-01

原文原文鏈接

1.原技術路線解析

在nging配置中將日誌信息交給syslog處理，rsyslog配置中將數據傳遞給了514端口解析，而後將解析好的數據傳入elasticsearch中。php

nginx配置html

server {
        listen 80; 
        listen [::]:80;
        server_name test.86dev.wrddns.com;
        
        # 如下兩行將日誌寫入syslog
        access_log syslog:server=unix:/dev/log,facility=local5,tag=web_1,severity=info main;
        error_log syslog:server=unix:/dev/log,facility=local5,tag=web_1,severity=error warn;
        
        # ....其餘配置
    }

/etc/rsyslog.confnginx

# 配置文件，存放解析規則xxx.conf和ruleBase文件xx.rb
$IncludeConfig /etc/rsyslog.d/*.conf

# 配置將日誌放到哪一個端口解析
local5.* @10.3.19.86:514

在實際應用過程當中有一些問題，不能和php上面的一些配置進行配合記錄，解析規則很差配置，有些內容解析很差，因此探索使用新的技術路線。laravel

2.新技術路線

嘗試使用新技術路線，經過swoole起一個服務，而後監聽9502端口，rsyslog將日誌推向該端口，對日誌進行解析後推入elasticsearch，此時能夠獲取到php端的一些配置。如下是大致思路git

2.1 選擇swoole的server類型。

這裏再也不贅述swoole的安裝，首要考慮的問題是原推向514的協議類型。github

先查看端口的協議類型web

# root @ WENGINE in ~ [9:48:51] 
$ netstat -antup | grep 514 
udp        0      0 0.0.0.0:514                 0.0.0.0:*                               23560/rsyslogd      
udp        0      0 :::514                      :::*                                    23560/rsyslogd

能夠看到是udp協議，因此選用swoole的 upd服務正則表達式

結合laravel的commands來編寫服務端數據庫

<?php

namespace App\Console\Swoole;

use Illuminate\Console\Command;
use swoole_websocket_server;
use swoole_server;
use swoole_process;
use swoole_sock_udp;
use UAParser\Parser;
use GeoIp2\Database\Reader;
use Wrd\Framework\Models\SysConfig;
use Elasticsearch\ClientBuilder;

class SwooleServer extends Command
{

    protected $signature = 'swoole-server start
                            {cmd=start : can use start}
                            {--daemon : set to run in daemonize mode}
                            ';

    protected $description = 'swoole server control';

    public $access_buffer = [];


    public function __construct()
    {
        parent::__construct();
    }

    public function handle()
    {
        $command = $this->argument('cmd');
        $option = $this->option('daemon');
        switch ($command) {
            case 'start':
                $this->initWs($option);
                break;
            default:
                $this->info('請按照下面格式輸入命令：php artisan swoole-server {start}');
                break;
        }

    }
        
    public function initWs($daemonize = false) 
    {
        if ($daemonize) {
            $this->info('Starting Websocket server in daemon mode...');
        } else {
            $this->info('Starting Websocket server in interactive mode...');
        }

        $server = new swoole_server('0.0.0.0', 9502, SWOOLE_PROCESS, SWOOLE_SOCK_UDP);
        $server->set([
            'daemonize' => $daemonize,
            'log_file' => '/var/www/html/storage/logs/websocket.log',
            'worker_num' => 1,
            'task_worker_num' => 1,
        ]);

        $server->on('Packet', function($serv, $data, $clientInfo) 
        {
            $serv->task($data);
        });
        
        $server->on('Task', function ($serv, $task_id, $from_id, $data) {

            //經過正則表達式提取出須要的信息，不一樣的日誌格式須要不一樣的正則，這裏只寫一種狀況
            $rule = '/\<\d*\>.*\d{2}\:\d{2}\:\d{2}\s[^\s]*\s[^\s]*\s(\w*\_\d*)\:\s\[Customize-format\]/';

            preg_match($rule, $data, $matches);

            if (empty($matches)) {
                $this->writeLog($data); //記錄下沒法解析的日誌，更正正則
                return false;
            }
            
            $vhost = $matches[1];
            $ip = $matches[2];
            //...更多參數
            
            $ua = $matches[12];
            //解析UA，這裏使用的解析庫https://github.com/ua-parser/uap-php
            $parser = Parser::create();
            $parser_ua = $parser->parse($ua);
            $browser = $parser_ua->ua->family;
            $os = $parser_ua->os->family;
            $device = $parser_ua->device->family; 

            //解析IP，這裏使用的解析庫https://github.com/maxmind/GeoIP2-php
            $reader = new Reader(public_path().'/geoip2/GeoLite2-City.mmdb');

            try{
                $record = $reader->city($ip);
                $country = $record->country->isoCode;
                $continent = $record->continent->names['zh-CN'];
                $subdivisions = $record->mostSpecificSubdivision->names['zh-CN'];
                $city = $record->city->names['zh-CN'];
                $geoip = array(
                    'location' => array($record->location->longitude, $record->location->latitude)
                );

            } catch (\Exception $e) {
               //若是ip沒有被收錄（項目有不少內網ip），則拿數據庫中的提早配置項，進行解析
            }
            
            $res = array(
                'vhost' => $vhost,
                'ip' => $ip,
                 // ...其它項
                'token' => $token,
                'browser' => $browser,
                'os' => $os,
                'device' => $device,
                'continent' => $continent,
                'country' => $country,
                'subdivisions' => $subdivisions,
                'city' => $city,
                'geoip' => $geoip,
            );

            $this->access_buffer[] = $res;

            //每隔一段時間，寫入到elasticsearch
            if (count($this->access_buffer) > 0 && time() - strtotime($this->access_buffer[0]['@timestamp']) > 10) {
                $insert_data = $this->access_buffer;
                $this->access_buffer = [];

                $this->insertElasticsearch('access', $insert_data);
            }

            //return 數據 給 Finish
            return "Task {$task_id}'s result";
        });
        
        $server->on('Finish', function ($serv,$task_id, $data) {
            echo "Task {$task_id} finish\n";
        });

        $server->start();
                
    }
    
    public function insertElasticsearch($type='access', $data){
        foreach($data as $item){
            $params['body'][] = [
                'index' => [
                    '_index' => $type.'-'.date('Y.m.d', time()),
                    '_type' => 'events',
                ]
            ];
            $params['body'][] = $item;
        }

        extract(\Config::get('app.elastic', [
            'host' => '127.0.0.1',
            'port' => '9200'
        ]));

        //往elasticsearch寫數據，這裏使用的庫https://github.com/elastic/elasticsearch-php
        $helper = ClientBuilder::create()
            ->setHosts([$host.":".$port])
            ->build();

        if (!empty($params['body'])) {
            $response = $helper->bulk($params);
            //var_dump($response);
        }

    }
    
    public function writeLog($info){
        $alert_message = array(
            'error' => '此條信息未能命中日誌格式，未寫入elasticsearch',
            'info' => $info
        );
        \Log::alert($alert_message);
    }

2.2 經過更改nginx的main日誌格式簡化正則表達式

nginx的配置中的apache

log_format main [$proxy_add_x_forwarded_for]-[$remote_user]-[$time_local]-[$request]-[$status]-[$bytes_sent]-[$http_host]-[$http_referer]-[$http_user_agent]-[$cookie_wengine_ticket]-[archer-main];

加了特殊符號，而且最後給了一個標識,這樣能提升命中準確度

2.3 更改rsyslog的端口

local5.* @10.3.19.86:9502

2.4 其它一些問題

能夠經過nc來檢測swoole的udp服務是否通

yum install -y nc

# root @ WENGINE in ~ [10:17:07] C:130
$ nc -u 127.0.0.1 9502
ceshi

能夠寫supervisor的腳原本使swoole服務器一直啓動

[program:swooleserver]
directory = /var/www/html
command=php artisan swoole-server
user=apache
autostart=true
startsecs=2
autorestart=true
redirect_stderr=true
stopsignal=INT
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。