ELK實時分析之php的laravel項目日誌
最近因公司項目增多，且環境也越來複雜，開發人員找我查看錯誤日誌愈來愈頻繁，內心便念想到ELK，心之所想，念之所達，快來get一項新技能吧php
一、系統架構組成
- laravel日誌：日誌源經過filebeat將日誌寫進redis中間件
- logstsh：logstash經過input將redis數據拿來分析，經過其filter模塊分析所須要的語句，而後輸出到elasticsearch
3.elasticsearch 接收logstash發送過來的數據，並提供了一個分佈式多用戶能力的全文搜索引擎- Kibana是一個優秀的前端日誌展現框架，它能夠很是詳細的將日誌轉化爲各類圖表，爲用戶提供強大的數據可視化支持。
2、各個服務的ip地址
laravel: 172.18.109.227 redis: 172.18.215.207 elasticsearch: 172.18.215.207 kibana: 172.18.215.207
3、laravel日誌服務器配置：
- 配置filebeat yum源文件
[elastic-5.x] name=Elastic repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
- 安裝filebeat
yum install filebeat
- 配置filebeat配置文件
... #寫入源 - input_type: log paths: - /var/www/html/*/storage/logs/laravel-2018-12-29.log ... #輸出至redis output.redis: # Array of hosts to connect to. hosts: ["172.18.215.207:6379"] password: "***********" db: 0 timeout: 5 key: "php-01" #protocol: "https" #username: "elastic" #password: "changeme"
4、redis服務器
- 安裝redis
yum install redis
- 配置redis
... # bind 192.168.1.100 10.0.0.1 bind 172.18.215.207 # bind 127.0.0.1 ::1 # ... # are explicitly listed using the "bind" directive. protected-mode yes # Accept connections on the specified port, default is 6379 (IANA #815344). # If port 0 is specified Redis will not listen on a TCP socket. port 6379 # TCP listen() backlog. # # In high requests-per-second environments you need an high backlog in order # to avoid slow clients connections issues. Note that the Linux kernel # will silently truncate it to the value of /proc/sys/net/core/somaxconn so # make sure to raise both the value of somaxconn and tcp_max_syn_backlog # in order to get the desired effect. tcp-backlog 511 ... # use a very strong password otherwise it will be very easy to break. # requirepass *********** # Command renaming. # # It is possible to change the name of dangerous commands in a shared # environment. For instance the CONFIG command may be renamed into something # hard to guess so that it
5、配置logstash服務器
- 配置logstash yum源
[elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
- logstash配置文件
#
# Where to fetch the pipeline configuration for the main pipeline
#
path.config: /etc/logstash/conf.d
#
# Pipeline configuration string for the main pipeline
#
# config.string:
#
...
path.config: /etc/logstash/conf.d
#
...
http.host: "127.0.0.1"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
#
# ------------ Debugging Settings --------------
#
# Options for log.level:
# * fatal
# * error
# * warn
# * info (default)
# * debug
# * trace
#
# log.level: info
path.logs: /var/log/logstash
#
...
vim /etc/logstash/conf.d/nginx.conf
# 從redis將數據取出
input {
redis {
type => "php-01"
host => "172.18.215.207"
port => "6379"
db => "0"
password => "*************"
data_type => "list"
key => "php-01"
}
}
# 格式化laravel日誌
filter {
grok {
match => [ "message","\[%{TIMESTAMP_ISO8601:logtime}\] %{WORD:env}\.(?<level>[A-Z]{4,5})\: %{GREEDYDATA:msg}}" ]
}
}
output {
#過濾level爲ERROR的日誌
if [level] == "ERROR" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "laravellog"
user => "elastic"
password => "changeme"
}
}
}
laravel的實例日誌爲html
[2019-01-02 09:58:00] produce.INFO: {"code":200,"message":"成功","data":{"code":"1000","message":"成功"}}
[2019-01-02 10:00:03] produce.INFO: 不能充值的緣由帳戶金額不夠
[2019-01-02 10:00:03] produce.INFO: 不能充值的緣由帳戶金額不夠
[2019-01-02 10:00:03] produce.INFO: 不能充值的緣由帳戶金額不夠
[2019-01-02 10:00:34] produce.ERROR: cannot find user by this audience {"exception":"[object] (JPush\\Exceptions\\APIRequestException(code: 1011): cannot find user by this audience at /var/www/html/enjoyCarTask/vendor/jpush/jpush/src/JPush/Http.php:123)
[stacktrace]
#0 /var/www/html/enjoyCarTask/vendor/jpush/jpush/src/JPush/Http.php(16): JPush\\Http::proce***esp(Array)
#1 /var/www/html/enjoyCarTask/vendor/jpush/jpush/src/JPush/PushPayload.php(537): JPush\\Http::post(Object(JPush\\Client), 'https://api.jpu...', '{\"platform\":\"al...')
#2 /var/www/html/enjoyCarTask/vendor/ucar/push/Push/Jobs/JPush.php(89): JPush\\PushPayload->send()
#3 [internal function]: Ucar\\Push\\Jobs\\JPush->handle(Object(JPush\\Client))
#4 /var/www/html/enjoyCarTask/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#5 /var/www/html/enjoyCarTask/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#6 /var/www/html/enjoyCarTask/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#7 /var/www/html/enjoyCarTask/vendor/laravel/framework/src/Illuminate/Container/Container.php(564): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
注意：咱們只想把ERROR的信息提取出來，因此logstash的配置文件中把level 爲ERROR的篩選了出來前端
6、配置elasticsearch服務器
- 配置elasticsearch yum源
[elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
- 安裝elasticsearch
yum install -y elasticsearch
- 配置elasticsearch
chown -R elasticsearch:elasticsearch /data/es-data chown -R elasticsearch:elasticsearch /var/log/elstic vim /etc/elasticsearch/elasticsearch.yml # # Use a descriptive name for the node: # node.name: Elstic # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # # 數據路徑 path.data: /data/es-data # # Path to log files: #日誌路徑 path.logs: /var/log/elstic ... # 監聽地址，設置爲127，只保持本機訪問 network.host: 127.0.0.1 # # Set a custom port for HTTP: # # 監聽的端口 http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- #
7、配置kibana服務
vim /etc/kibana/kibana.ym # Kibana is served by a back end server. This setting specifies the port to use. # 監聽的端口 server.port: 5601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. # 監聽地址，使用內網地址，而後用nginx反代 server.host: "127.0.0.1" # Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects # the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests # to Kibana. This setting cannot end in a slash. #server.basePath: "" # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576 # The Kibana server's name. This is used for display purposes. #server.name: "your-hostname" # The URL of the Elasticsearch instance to use for all your queries. elasticsearch.url: "http://localhost:9200" # When this setting's value is true Kibana uses the hostname specified in the server.host # setting. When the value of this setting is false, Kibana uses the hostname of the host # that connects to this Kibana instance. #elasticsearch.preserveHost: true # Kibana uses an index in Elasticsearch to store saved searches, visualizations and # dashboards. Kibana creates a new index if the index doesn't already exist. #kibana.index: ".kibana" # The default application to load. ....
8、使用nginx反代kibana
$ cat /etc/nginx/conf.d/elk.conf
server {
listen 443 http2 ssl;
listen [::]:443 http2 ssl;
server_name *********;
ssl on;
ssl_certificate "**************";
ssl_certificate_key "/usr/local/certificate/************";
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
ssl_prefer_server_ciphers on;
#ssl_dhparam /etc/ssl/certs/dhparam.pem;
########################################################################
# from https://cipherli.st/ #
# and https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html #
########################################################################
# Disable preloading HSTS for now. You can use the commented out header line that includes
# the "preload" directive if you understand the implications.
#add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
##################################
# 這裏是反向代理到kibana服務 走http協議
location / {
proxy_pass http://127.0.0.1:5601;
}
}
#配置80端口重寫443端口
server {
listen 80;
server_name *********;
rewrite ^/(.*)$ https://*********/$1;
}
9、訪問kibana配置索引
啓動因此服務後再次訪問kibana，查看效果。
大功告成~，之後可讓開發本身看錯誤日誌咯node
總結：
- 這次由於服務器數量有限,(不敢在正式服務器上亂搞= =)只使用了兩臺服務器，php的laravel日誌和filebeat服務一臺，做爲寫入源，Redis，Elasticsearch、Logstash、Kibana，一臺做爲日誌處理服務器，(若是作多臺的話，監聽地址須要改變)。跑一段時間確認沒問題再上正式。
- 監聽的地址最好都使用內網地址，防止被***。
喜歡我寫的東西的朋友能夠關注一下個人公衆號：Devops部落nginx
相關文章
- 1. ELK實時分析之php的laravel項目日誌
- 2. ELK 6.4 實時日誌分析系統
- 3. 實時日誌分析ELK概述
- 4. 開源實時日誌分析ELK
- 5. ELK實時日誌分析平臺（一）
- 6. ELK Stack日誌實時收集分析
- 7. ELK實時日誌分析平臺（二）
- 8. ELK日誌分析
- 9. ELK 企業內部日誌分析系統 項目實戰14—ELK 企業內部日誌分析系統
- 10. 項目實戰14—ELK 企業內部日誌分析系統
- 更多相關文章...
- • Thymeleaf項目實踐 - Thymeleaf 教程
- • SQLite 日期 & 時間 - SQLite教程
- • 互聯網組織的未來：剖析GitHub員工的任性之源
- • 適用於PHP初學者的學習線路和建議
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章