專職DBA-MySQL日誌管理1 MySQL日誌種類: 錯誤日誌error log : 當數據庫啓動、運行、中止時產生該日誌。 普通查詢日誌general query log : 客戶端鏈接數據庫後執行語句時產生該日誌。 二進制日誌binary log : 當數據庫內容發生改變時產生該日誌。 中繼日誌relay log : 從庫上收到主庫的數據更新時產生該日誌。 慢查詢日誌slow query log : SQL語句在數據庫查詢超出指定時間時產生該日誌。 DDL日誌metadata log : 執行DDL語句操做元數據時產生該日誌。 對日誌進行輪訓切割常見的命令: mysqladmin flush-logs mysqldump -F/--master-data [root@db01 ~]# mysqld --defaults-file=/data/mysql/3306/my.cnf & [1] 6758 [root@db01 ~]# ps -ef | grep mysql mysql 6758 6711 0 Jul20 pts/0 00:00:01 mysqld --defaults-file=/data/mysql/3306/my.cnf root 6816 6711 0 00:02 pts/0 00:00:00 grep --color=auto mysql [root@db01 ~]# mysql -S /data/mysql/3306/mysql.sock -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.26-log MySQL Community Server (GPL) Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> 1.錯誤日誌error log MySQL錯誤日誌用於記錄MySQL服務進程mysqld在啓動關閉運行過程當中遇到的錯誤信息。 [root@db01 ~]# grep "log_error" /data/mysql/3306/my.cnf log_error = /data/mysql/3306/error.log [root@db01 ~]# ls -l /data/mysql/3306/error.log -rw-r----- 1 mysql mysql 26338 Jul 21 00:00 /data/mysql/3306/error.log mysql> show global variables like "log_error"; +---------------+----------------------------+ | Variable_name | Value | +---------------+----------------------------+ | log_error | /data/mysql/3306/error.log | +---------------+----------------------------+ 1 row in set (0.00 sec) 默認狀況下日誌的文件名部分和系統主機名是一致的,擴展名爲:.err [root@db01 ~]# hostname db01 ==========================》db01.err 錯誤日誌輪詢(按天輪詢) [root@db01 ~]# cd /data/mysql/3306/ [root@db01 /data/mysql/3306]# mv error.log error_$(date +%F).log [root@db01 /data/mysql/3306]# mysqladmin -S /data/mysql/3306/mysql.sock -p flush-logs Enter password: [root@db01 /data/mysql/3306]# ls -l total 44 -rw-r----- 1 mysql mysql 5 Jul 21 00:00 3306.pid drwxr-xr-x 5 mysql mysql 161 Jul 21 00:00 data -rw-r----- 1 mysql mysql 26338 Jul 21 00:00 error_2019-07-21.log -rw-r----- 1 mysql mysql 0 Jul 21 00:10 error.log drwxr-xr-x 2 mysql mysql 173 Jul 21 00:10 logs -rw-r--r-- 1 mysql mysql 1147 Jul 15 05:42 my.cnf srwxrwxrwx 1 mysql mysql 0 Jul 21 00:00 mysql.sock -rw------- 1 mysql mysql 5 Jul 21 00:00 mysql.sock.lock -rw-r----- 1 mysql mysql 1050 Jul 21 00:10 slow.log drwxr-xr-x 2 mysql mysql 6 Jul 21 00:00 tmp 2.普通查詢日誌general query log 普通查詢日誌記錄客戶端鏈接信息,以及執行的SQL語句信息。 [root@db01 ~]# grep "general" /data/mysql/3306/my.cnf # general log general_log = off general_log_file = /data/mysql/3306/general.log mysql> show global variables like "%general%"; +------------------+------------------------------+ | Variable_name | Value | +------------------+------------------------------+ | general_log | OFF | | general_log_file | /data/mysql/3306/general.log | +------------------+------------------------------+ 2 rows in set (0.00 sec) mysql> set global general_log = on; Query OK, 0 rows affected (0.00 sec) mysql> show global variables like "%general%"; +------------------+------------------------------+ | Variable_name | Value | +------------------+------------------------------+ | general_log | ON | | general_log_file | /data/mysql/3306/general.log | +------------------+------------------------------+ 2 rows in set (0.00 sec) [root@db01 ~]# ls -l /data/mysql/3306/ total 48 -rw-r----- 1 mysql mysql 5 Jul 21 00:00 3306.pid drwxr-xr-x 5 mysql mysql 161 Jul 21 00:00 data -rw-r----- 1 mysql mysql 26338 Jul 21 00:00 error_2019-07-21.log -rw-r----- 1 mysql mysql 0 Jul 21 00:10 error.log -rw-r----- 1 mysql mysql 254 Jul 21 00:15 general.log drwxr-xr-x 2 mysql mysql 173 Jul 21 00:10 logs -rw-r--r-- 1 mysql mysql 1147 Jul 15 05:42 my.cnf srwxrwxrwx 1 mysql mysql 0 Jul 21 00:00 mysql.sock -rw------- 1 mysql mysql 5 Jul 21 00:00 mysql.sock.lock -rw-r----- 1 mysql mysql 1050 Jul 21 00:10 slow.log drwxr-xr-x 2 mysql mysql 6 Jul 21 00:00 tmp [root@db01 ~]# cat /data/mysql/3306/general.log mysqld, Version: 5.7.26-log (MySQL Community Server (GPL)). started with: Tcp port: 3306 Unix socket: /data/mysql/3306/mysql.sock Time Id Command Argument 2019-07-20T16:15:45.036658Z 2 Query show global variables like "%general%" 對於general.log在高併發數據庫場景下,應該是關閉的,由於查詢日誌信息量很大,容易致使磁盤I/O性能問題。 當訪問量不是很大,並且企業又有審計執行的SQL語句時,能夠開啓general.log 3.二進制日誌binary log 二進制日誌記錄數據庫裏數據被修改的SQL語句。 通常爲DDL和DML語句:insert/delete/update/create/drop/alter等關鍵字的語句。 做用:(1).記錄MySQL數據的增量數據,用來作增量數據庫恢復。(2).實現主從複製。 [root@db01 ~]# grep "bin" /data/mysql/3306/my.cnf # for binlog binlog_format = row log_bin = /data/mysql/3306/logs/mysql-bin 默認狀況下binlog日誌前綴是"主機名-bin" ======》db01-bin mysql> show global variables like "%log_bin%"; +---------------------------------+---------------------------------------+ | Variable_name | Value | +---------------------------------+---------------------------------------+ | log_bin | ON | | log_bin_basename | /data/mysql/3306/logs/mysql-bin | | log_bin_index | /data/mysql/3306/logs/mysql-bin.index | | log_bin_trust_function_creators | OFF | | log_bin_use_v1_row_events | OFF | +---------------------------------+---------------------------------------+ 5 rows in set (0.00 sec) MySQL5.6臨時不記錄binlog參數是: set session sql_log_bin = off; #臨時中止記錄binlog,session級別不影響其餘會話。 之後你用mysql恢復數據時不但願恢復的數據SQL記錄到binlog裏,就能夠關閉這個參數。 binlog文件刷新爲新文件的條件: (1).數據庫重啓會自動刷新binlog爲新文件。 (2).執行mysqladmin -F (3).執行mysqladmin flush-log (4).binlog文件達到1GB左右時。 (5).人爲配置切割及調整。 mysql> show global variables like "%binlog_size%"; +-----------------+------------+ | Variable_name | Value | +-----------------+------------+ | max_binlog_size | 1073741824 | +-----------------+------------+ 1 row in set (0.01 sec) binlog的索引文件 [root@db01 ~]# ls -l /data/mysql/3306/logs/ total 28 -rw-r----- 1 mysql mysql 177 Jul 15 05:43 mysql-bin.000001 -rw-r----- 1 mysql mysql 461 Jul 15 05:53 mysql-bin.000002 -rw-r----- 1 mysql mysql 241 Jul 15 05:53 mysql-bin.000003 -rw-r----- 1 mysql mysql 217 Jul 15 05:53 mysql-bin.000004 -rw-r----- 1 mysql mysql 241 Jul 21 00:10 mysql-bin.000005 -rw-r----- 1 mysql mysql 194 Jul 21 00:10 mysql-bin.000006 -rw-r----- 1 mysql mysql 234 Jul 21 00:10 mysql-bin.index [root@db01 ~]# cat /data/mysql/3306/logs/mysql-bin.index /data/mysql/3306/logs/mysql-bin.000001 /data/mysql/3306/logs/mysql-bin.000002 /data/mysql/3306/logs/mysql-bin.000003 /data/mysql/3306/logs/mysql-bin.000004 /data/mysql/3306/logs/mysql-bin.000005 /data/mysql/3306/logs/mysql-bin.000006 mysql> show global variables like "%log_bin_index%"; +---------------+---------------------------------------+ | Variable_name | Value | +---------------+---------------------------------------+ | log_bin_index | /data/mysql/3306/logs/mysql-bin.index | +---------------+---------------------------------------+ 1 row in set (0.01 sec) binlog日誌刪除方法: 理論上來說天天數據庫全備時刻之前的binlog都是無用的,可是工做中會根據需求保留3到7天的本地binlog文件。 (1). expire_logs_days #刪除7天前的binlog日誌 能夠寫入到my.cnf配置文件裏 expire_logs_days = 7 mysql> show global variables like "%logs_days%"; +------------------+-------+ | Variable_name | Value | +------------------+-------+ | expire_logs_days | 0 | +------------------+-------+ 1 row in set (0.00 sec) mysql> set global expire_logs_days = 7; Query OK, 0 rows affected (0.00 sec) mysql> show global variables like "%logs_days%"; +------------------+-------+ | Variable_name | Value | +------------------+-------+ | expire_logs_days | 7 | +------------------+-------+ 1 row in set (0.01 sec) (2).從最開始一直刪除到指定的文件位置(不包含指定文件) mysql> show binary logs; +------------------+-----------+ | Log_name | File_size | +------------------+-----------+ | mysql-bin.000001 | 177 | | mysql-bin.000002 | 461 | | mysql-bin.000003 | 241 | | mysql-bin.000004 | 217 | | mysql-bin.000005 | 241 | | mysql-bin.000006 | 194 | +------------------+-----------+ 6 rows in set (0.00 sec) mysql> purge binary logs to 'mysql-bin.000002'; Query OK, 0 rows affected (0.01 sec) mysql> show binary logs; +------------------+-----------+ | Log_name | File_size | +------------------+-----------+ | mysql-bin.000002 | 461 | #序列000002以前的binlog就沒了 | mysql-bin.000003 | 241 | | mysql-bin.000004 | 217 | | mysql-bin.000005 | 241 | | mysql-bin.000006 | 194 | +------------------+-----------+ 5 rows in set (0.00 sec) (3).按照時間刪除binlog日誌 [root@db01 ~]# ls -l --time-style=long-iso /data/mysql/3306/logs/mysql-bin.* -rw-r----- 1 mysql mysql 461 2019-07-15 05:53 /data/mysql/3306/logs/mysql-bin.000002 -rw-r----- 1 mysql mysql 241 2019-07-15 05:53 /data/mysql/3306/logs/mysql-bin.000003 -rw-r----- 1 mysql mysql 217 2019-07-15 05:53 /data/mysql/3306/logs/mysql-bin.000004 -rw-r----- 1 mysql mysql 241 2019-07-21 00:10 /data/mysql/3306/logs/mysql-bin.000005 -rw-r----- 1 mysql mysql 194 2019-07-21 00:10 /data/mysql/3306/logs/mysql-bin.000006 -rw-r----- 1 mysql mysql 195 2019-07-21 00:51 /data/mysql/3306/logs/mysql-bin.index 刪除2019-07-21 00:10之前的binlog文件 mysql> purge master logs before "2019-07-21 00:10"; Query OK, 0 rows affected (0.00 sec) mysql> system ls -l --time-style=long-iso /data/mysql/3306/logs/mysql-bin.*; -rw-r----- 1 mysql mysql 241 2019-07-21 00:10 /data/mysql/3306/logs/mysql-bin.000005 -rw-r----- 1 mysql mysql 194 2019-07-21 00:10 /data/mysql/3306/logs/mysql-bin.000006 -rw-r----- 1 mysql mysql 78 2019-07-21 01:04 /data/mysql/3306/logs/mysql-bin.index (4).清除全部的binlog,並從000001開始記錄。 mysql> reset master; Query OK, 0 rows affected (0.02 sec) mysql> system ls -l --time-style=long-iso /data/mysql/3306/logs/mysql-bin.* -rw-r----- 1 mysql mysql 154 2019-07-21 01:06 /data/mysql/3306/logs/mysql-bin.000001 -rw-r----- 1 mysql mysql 39 2019-07-21 01:06 /data/mysql/3306/logs/mysql-bin.index 下面是binlog相關參數的設置和優化思路 mysql> show global variables like "%binlog%"; +--------------------------------------------+----------------------+ | Variable_name | Value | +--------------------------------------------+----------------------+ | binlog_cache_size | 32768 | | binlog_checksum | CRC32 | | binlog_direct_non_transactional_updates | OFF | | binlog_error_action | ABORT_SERVER | | binlog_format | ROW | | binlog_group_commit_sync_delay | 0 | | binlog_group_commit_sync_no_delay_count | 0 | | binlog_gtid_simple_recovery | ON | | binlog_max_flush_queue_time | 0 | | binlog_order_commits | ON | | binlog_row_image | FULL | | binlog_rows_query_log_events | OFF | | binlog_stmt_cache_size | 32768 | | binlog_transaction_dependency_history_size | 25000 | | binlog_transaction_dependency_tracking | COMMIT_ORDER | | innodb_api_enable_binlog | OFF | | innodb_locks_unsafe_for_binlog | OFF | | log_statements_unsafe_for_binlog | ON | | max_binlog_cache_size | 18446744073709547520 | | max_binlog_size | 1073741824 | | max_binlog_stmt_cache_size | 18446744073709547520 | | sync_binlog | 1 | +--------------------------------------------+----------------------+ 22 rows in set (0.00 sec) binlog_cache_size #binlog日誌緩存是數據庫爲每個客戶端鏈接分配的內存空間。 max_binlog_size #默認是1GB,可是並不能嚴格控制binlog日誌大小,若binlog大小已經接近1GB,而此時又在執行一個較大的事務時,爲了保證事務的完整性,數據庫不會作日誌刷新動做,而是直到該事務的日誌所有記錄進入當前binlog日誌後纔會進行刷新。 sync_binlog #控制binlog何時同步到磁盤。 sync_binlog = 0 表示正在事務提交以後,數據庫不會將binlog_cache中的數據刷新到磁盤,而是讓文件系統自行決定何時來作刷新或者在緩存滿了以後才刷新到磁盤。 sync_binlog = n 表示每進行n次事務提交以後,數據庫都會進行一次將緩存數據強制刷新到磁盤。 mysql> show global variables like "%log_bin%"; +---------------------------------+---------------------------------------+ | Variable_name | Value | +---------------------------------+---------------------------------------+ | log_bin | ON | | log_bin_basename | /data/mysql/3306/logs/mysql-bin | | log_bin_index | /data/mysql/3306/logs/mysql-bin.index | | log_bin_trust_function_creators | OFF | | log_bin_use_v1_row_events | OFF | +---------------------------------+---------------------------------------+ 5 rows in set (0.00 sec) 記錄binlog日誌的三種模式: (1).語句模式statement(MySQL5.6默認的模式) 就是每一條被修改數據的SQL語句都會記錄到binlog中。 (2).行級模式row 將數據被修改的每一行的狀況記錄爲一條語句。 (3).混合模式mixed 默認採用statement模式記錄日誌,在一些特殊狀況下會轉換爲row模式。 線網環境建議使用row模式。 mysql> show global variables like "%binlog_format%"; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | binlog_format | ROW | +---------------+-------+ 1 row in set (0.01 sec) [root@db01 ~]# grep "binlog" /data/mysql/3306/my.cnf # for binlog binlog_format = row --base64-output=decode-rows -v #以row模式解析binlog日誌 [root@db01 ~]# mysqlbinlog --base64-output=decode-rows -v /data/mysql/3306/logs/mysql-bin.000001 /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 4 #190721 1:06:07 server id 113306 end_log_pos 123 CRC32 0x391e6633 Start: binlog v 4, server v 5.7.26-log created 190721 1:06:07 at startup # Warning: this binlog is either in use or was not closed properly. ROLLBACK/*!*/; # at 123 #190721 1:06:07 server id 113306 end_log_pos 154 CRC32 0x5a6e058c Previous-GTIDs # [empty] SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/; DELIMITER ; # End of log file /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/; 4.慢查詢日誌slow.log 慢查詢日誌就是記錄執行時間超出指定值 或 其餘指定條件的SQL語句。 [root@db01 ~]# grep "query" /data/mysql/3306/my.cnf # for slow query log slow_query_log = on slow_query_log_file = /data/mysql/3306/slow.log long_query_time = 1.000000 mysql> show global variables like "%slow%"; +---------------------------+---------------------------+ | Variable_name | Value | +---------------------------+---------------------------+ | log_slow_admin_statements | OFF | | log_slow_slave_statements | OFF | | slow_launch_time | 2 | | slow_query_log | ON | | slow_query_log_file | /data/mysql/3306/slow.log | +---------------------------+---------------------------+ 5 rows in set (0.00 sec) mysql> show global variables like "%long_query_time%"; +-----------------+----------+ | Variable_name | Value | +-----------------+----------+ | long_query_time | 1.000000 | +-----------------+----------+ 1 row in set (0.01 sec) mysql> show global variables like "%log_qu%"; +-------------------------------+-------+ | Variable_name | Value | +-------------------------------+-------+ | log_queries_not_using_indexes | OFF | +-------------------------------+-------+ 1 row in set (0.00 sec) mysql> show global variables like "%row_limit%"; +------------------------+-------+ | Variable_name | Value | +------------------------+-------+ | min_examined_row_limit | 0 | +------------------------+-------+ 1 row in set (0.00 sec) mysql> show global variables like "%log_th%"; +----------------------------------------+-------+ | Variable_name | Value | +----------------------------------------+-------+ | log_throttle_queries_not_using_indexes | 0 | +----------------------------------------+-------+ 1 row in set (0.01 sec) 對於天天所產生的大量慢查詢日誌,如何進行刷新切割呢? 利用定時任務按天對慢查詢日誌進行切割。 [root@db01 ~]# mkdir -p /disk/sh [root@db01 ~]# cd /disk/sh/ [root@db01 /disk/sh]# vim cut_slow_log.sh #!/bin/bash cd /data/mysql/3306/ && \ mv slow.log slow_$(date +%F).log && \ mysqladmin flush-log -S /data/mysql/3306/mysql.sock -p123 >/dev/null 2>&1 [root@db01 /disk/sh]# sh cut_slow_log.sh [root@db01 /disk/sh]# ls -l /data/mysql/3306/ total 56 -rw-r----- 1 mysql mysql 5 Jul 21 00:00 3306.pid drwxr-xr-x 5 mysql mysql 161 Jul 21 00:00 data -rw-r----- 1 mysql mysql 26338 Jul 21 00:00 error_2019-07-21.log -rw-r----- 1 mysql mysql 0 Jul 21 00:10 error.log -rw-r----- 1 mysql mysql 4571 Jul 21 02:05 general.log drwxr-xr-x 2 mysql mysql 125 Jul 21 02:05 logs -rw-r--r-- 1 mysql mysql 1147 Jul 15 05:42 my.cnf srwxrwxrwx 1 mysql mysql 0 Jul 21 00:00 mysql.sock -rw------- 1 mysql mysql 5 Jul 21 00:00 mysql.sock.lock -rw-r----- 1 mysql mysql 175 Jul 21 02:04 slow_2019-07-21.log -rw-r----- 1 mysql mysql 175 Jul 21 02:05 slow.log drwxr-xr-x 2 mysql mysql 6 Jul 21 00:00 tmp 天天0點執行切割任務 [root@db01 ~]# crontab -e # cut mysql slow log by zhouwanchun at 2019-07-20 00 00 * * * /bin/sh /disk/sh/cut_slow_log.sh >/dev/null 2>&1 [root@db01 ~]# cat /var/spool/cron/root # cut mysql slow log by zhouwanchun at 2019-07-20 00 00 * * * /bin/sh /disk/sh/cut_slow_log.sh >/dev/null 2>&1 使用工具分析慢查詢日誌 分析慢查詢日誌的工具備: mysqldumpslow mysqlsla pt-query-diges myprofi mysql-explain-slow-log mysqllogfilter ELK(可視化展現) https://yq.aliyun.com/articles/59260
[root@db01 ~]# mysqldumpslow --help 推薦使用mysqlsla工具 [root@db01 ~]# wget http://hackmysql.com/scripts/mysqlsla-2.03.tar.gz [root@db01 ~]# yum install perl-devel perl-DBI perl-DBD-MySQL -y [root@db01 ~]# rpm -qa perl-devel perl-DBI perl-DBD-MySQL perl-DBI-1.627-4.el7.x86_64 perl-devel-5.16.3-294.el7_6.x86_64 perl-DBD-MySQL-4.023-6.el7.x86_64 [root@db01 /disk]# ls -l total 631080 -rw-r--r-- 1 root root 644869837 Jul 15 05:30 mysql-5.7.26-linux-glibc2.12-x86_64.tar.gz -rw-r--r-- 1 root root 1352558 Jul 21 02:51 mysqlsla.tar.gz drwxr-xr-x 2 root root 29 Jul 21 02:05 sh drwxr-xr-x. 2 root root 6 Jul 12 03:35 soft drwxr-xr-x. 2 root root 6 Jul 12 03:35 tools tar -xf mysqlsla-2.03.tar.gz cd mysqlsla-2.03 perl Makefile.PL make make install [root@db01 /disk]# tar -xf mysqlsla.tar.gz [root@db01 /disk]# cd mysqlsla/ [root@db01 /disk/mysqlsla]# perl Makefile.PL Checking if your kit is complete... Looks good Writing Makefile for mysqlsla [root@db01 /disk/mysqlsla]# make cp lib/mysqlsla.pm blib/lib/mysqlsla.pm cp bin/mysqlsla blib/script/mysqlsla /usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/mysqlsla Manifying blib/man3/mysqlsla.3pm [root@db01 /disk/mysqlsla]# make install Installing /usr/local/share/perl5/mysqlsla.pm Installing /usr/local/share/man/man3/mysqlsla.3pm Installing /usr/local/bin/mysqlsla Appending installation info to /usr/lib64/perl5/perllocal.pod 寫切割日誌的腳本 [root@db01 ~]# cd /disk/sh/ [root@db01 /disk/sh]# vim slow_log_analyze.sh #!/bin/bash Date=`date +%F -d -1day` # cut slow log cd /data/mysql/3306/ && \ mv slow.log slow_$Date.log && \ mysqladmin flush-log -S /data/mysql/3306/mysql.sock -p123 >/dev/null 2>&1 # analyze slow log Time=`date +%F` Path=/usr/local/bin/ cd /data/mysql/3306/ && \ $Path/mysqlsla -lt show slow_$Date.log >/tmp/analyze_slow_$Date.log 2>&1 # rsync analyzed_slow to backup server. # send analyzed slow log to admin on backup server by mail. 寫定時任務 [root@db01 ~]# cat /var/spool/cron/root # analyzed mysql slow log by zhouwanchun at 2019-07-20 00 00 * * * /bin/sh /disk/sh/slow_log_analyze.sh >/dev/null 2>&1