Zabbix DB歷史表由Innodb轉換成Tokudb引擎總結

時間 2020-08-05

標籤 zabbix 歷史 innodb 轉換 tokudb 引擎總結欄目 Zabbix 简体版

原文原文鏈接

線上Zabbix監控服務器DB採用的是MySQL一主一從。監控歷史數據公司要求保留2年，歷史數據相關的表作了表分區，過時數據自動刪除。隨着線上Zabbix監控設備數的不斷增長，服務器磁盤空間增加的很厲害，3.7T的磁盤空間目前已經用了3.3T。嘗試用pt-online工具壓縮數據表，因爲歷史數據沒有主鍵不能壓縮。mysql

最後考慮把Zabbix歷史相關的數據表由Innodb引擎轉換成TokuDB引擎從而來減小磁盤空間。因爲採用TokuDB壓縮歷史數據會損耗必定的CPU資源，申請一臺HP G10服務器，內存64G 8塊600G硬盤作Raid5專門作Zabbix DB服務器。sql

1.Percona數據庫安裝並開啓Tokudb存儲引擎。shell

1）安裝percona數據庫，採用官方最新的RPM進行安裝。數據庫

[root@ZBXTKDB01 software]# wget https://www.percona.com/downloads/Percona-Server-5.6/Percona-Server-5.6.42-84.2/binary/redhat/6/x86_64/Percona-Server-5.6.42-84.2-r6b2b987-el6-x86_64-bundle.tar
[root@ZBXTKDB01 software]# tar -xvf Percona-Server-5.6.42-84.2-r6b2b987-el6-x86_64-bundle.tar
[root@ZBXTKDB01 software]# rpm -ivh Percona-Server-server-56-5.6.42-rel84.2.el6.x86_64.rpm\ Percona-Server-client-56-5.6.42-rel84.2.el6.x86_64.rpm \Percona-Server-shared-56-5.6.42-rel84.2.el6.x86_64.rpm
[root@ZBXTKDB01 software]# service mysql start

2）安裝LIBJEMALLOC庫。TokuDB存儲引擎須要libjemalloc library 3.3.0或者更新，RPM安裝庫文件默認在/usr/lib64/libjemalloc.so.1。服務器

[root@ZBXTKDB01 software]# rpm -ivh jemalloc-3.6.0-1.el6.art.x86_64.rpmapp

3）安裝Tokudb插件，RPM 安裝的軟件包Tokudb插件默認在 /usr/lib64/mysql/plugin/ha_tokudb.so。socket

[root@ZBXTKDB01 software]# rpm -ivh Percona-Server-tokudb-56-5.6.42-rel84.2.el6.x86_64.rpmide

4）加載Tokudb引擎須要修改下面的內核參數並寫到/etc/rc.local裏面。工具

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

5）安裝Tokudb引擎。測試

[root@ZBXTKDB01 ~]# ps_tokudb_admin --enable -uroot
Checking if Percona Server is running with jemalloc enabled...
INFO: Percona Server is running with jemalloc enabled.

Checking transparent huge pages status on the system...
INFO: Transparent huge pages are currently disabled on the system.

Checking if thp-setting=never option is already set in config file...
INFO: Option thp-setting=never is set in the config file.

Checking TokuDB engine plugin status...
INFO: TokuDB engine plugin is not installed.

Installing TokuDB engine...
INFO: Successfully installed TokuDB engine plugin.

6）能夠經過手工的方式加載Tokudb存儲引擎。

INSTALL PLUGIN tokudb SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_file_map SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_fractal_tree_info SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_fractal_tree_block_map SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_trx SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_locks SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_lock_waits SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_background_job_status SONAME 'ha_tokudb.so';

7）驗證MySQL數據庫的Tokudb引擎是否安裝成功。經過show engines或者show plugins命令均可以查看。

mysql> show engines;
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+
| Engine             | Support | Comment                                                                    | Transactions | XA   | Savepoints |
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+
| PERFORMANCE_SCHEMA | YES     | Performance Schema                                                         | NO           | NO   | NO         |
| MRG_MYISAM         | YES     | Collection of identical MyISAM tables                                      | NO           | NO   | NO         |
| CSV                | YES     | CSV storage engine                                                         | NO           | NO   | NO         |
| BLACKHOLE          | YES     | /dev/null storage engine (anything you write to it disappears)             | NO           | NO   | NO         |
| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables                  | NO           | NO   | NO         |
| TokuDB             | YES     | Percona TokuDB Storage Engine with Fractal Tree(tm) Technology             | YES          | YES  | YES        |
| MyISAM             | YES     | MyISAM storage engine                                                      | NO           | NO   | NO         |
| ARCHIVE            | YES     | Archive storage engine                                                     | NO           | NO   | NO         |
| FEDERATED          | NO      | Federated MySQL storage engine                                             | NULL         | NULL | NULL       |
| InnoDB             | DEFAULT | Percona-XtraDB, Supports transactions, row-level locking, and foreign keys | YES          | YES  | YES        |
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+

8）因爲原來Zabbix數據庫版本5.5.20的安裝的，能夠經過下面的命令update數據庫。

[root@ZBXTKDB01 ~]# mysql_upgrade -uroot -p
Enter password:
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
Running 'mysqlcheck' with connection arguments: '--socket=/mysql/3307/mysql.sock' '--port=3307'

9）MySQL my.conf配置文件的參數優化。主要優化了Innodb和Tokubdb參數。若是是源碼安裝的數據庫，Tokudb擴展及LIBJEMALLOC庫須要在my.conf配置文件裏面指定Tokudb插件和LIBJEMALLOC庫的路徑地址。

[root@ZBXTKDB01 ~]# cat /etc/my.cnf
[client]
socket                 = /mysql/3307/mysql.sock
port                   = 3307

[mysql]
no-auto-rehash

[mysqld]
server-id              = 6243
port                   = 3307
user                   = mysql
basedir                = /usr
datadir                = /mysql/3307/data
socket                 = /mysql/3307/mysql.sock
skip-name-resolve
default_storage_engine = innodb 

binlog-do-db           = zabbix   
log-bin                = /mysql/3307/binlog/mysql-bin
binlog_format          = ROW
binlog_cache_size      = 20M
expire_logs_days       = 10

#slow_query_log         = 1
slow_query_log_file    = /mysql/3307/data/mysql-slow.log
long_query_time        = 20
log_error              = /mysql/3307/mysql_error.log
pid-file               = /mysql/3307/mysql.pid

max_connections     = 3000
back_log            = 100
max_connect_errors  = 1000000
max_allowed_packet  = 64M
max_heap_table_size = 300M
sort_buffer_size    = 2M
join_buffer_size    = 2M
thread_cache_size   = 400
query_cache_type    = 1
query_cache_size    = 128M
query_cache_limit   = 2M
tmp_table_size      = 1024M
#table_cache         = 1000

key_buffer_size           = 128M
read_buffer_size          = 8M
read_rnd_buffer_size      = 8M
bulk_insert_buffer_size   = 10M
myisam_sort_buffer_size   = 16M
myisam_max_sort_file_size = 4G
myisam_repair_threads     = 1

innodb_file_per_table           = 1
innodb_buffer_pool_size         = 10G
innodb_use_sys_malloc           = 1 
innodb_data_file_path           = ibdata01:400M:autoextend
innodb_log_buffer_size          = 10M
innodb_log_file_size            = 400M
innodb_log_files_in_group       = 2
innodb_open_files               = 2048
innodb_flush_log_at_trx_commit  = 2
innodb_lock_wait_timeout        = 60

tokudb_directio = 1
tokudb_row_format = tokudb_lzma
tokudb_cache_size=30G
optimizer_switch=index_condition_pushdown=off
tokudb_read_buf_size = 4K
tokudb_commit_sync = 0
tokudb_fsync_log_period = 60000
tokudb_fs_reserve_percent = 2
tokudb-data-dir = /mysql/3307/tokudb
tokudb-log-dir=

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer_size  = 512M
sort_buffer_size = 512M
read_buffer      = 8M
write_buffer     = 8M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
thp-setting = never
open-files-limit = 8192

2.在安裝Tokudb存儲引擎的服務器上面開啓Zabbix數據庫新的從庫。主從同步和MySQL二進制文件有關，和存儲引擎無關，即Innodb存儲引擎的MySQL表的數據能夠同步到Tokudb存儲引擎的數據表裏面。

1）中止該服務器和原從庫MySQL服務，將data目錄拷貝到該服務器上面。

2）根據原來從庫服務器數據庫data目錄裏面/data/mysql/3307/data/master.info記錄主從複製的信息，開啓新服務器的主從複製。

#主庫上面添加主從複製的帳號
grant replication slave on *.* to 'xxx'@'192.168.6.243'identified by 'xxxxxx'
#新服務器上面開啓主從同步
change master to
master_host='92.168.6.244',
master_port=3307,
master_user='xxx',
master_password='xxxxxx’,
master_log_file='msql-bin.xxxx’,  
master_log_pos=xxxx;

3）驗證新從庫的數據和主庫的數據是否正常同步，若是出現數據不一致能夠經過下面的命令解決。

stop slave;
set global sql_slave_skip_counter=1;
start slave ;

set global slave_exec_mode=idempotent;
set global slave_exec_mode=STRICT;

4）Tokudb存儲引擎的轉換，將Zabbix數據庫歷史數據相關的表(history、history_uint、trends、tredns_uint 、history_str、history_log、history_text)所有轉換成Tokudb存儲引擎。

mysql> alter table history_uint engine=TokuDB;

由Innodb存儲引擎轉換成Tokudb存儲引擎應該注意如下事項：

1）在執行alter table history_uint engine=TokuDB 命令以後，首先數據庫庫會執行MDL（Meta Data lock元數據鎖），把Innodb表裏面每條數據拿出來放存放到tokudb-data-dir目錄裏面的臨時文件裏。等全部的數據取出來存儲到臨時文件以後，在合併臨時文件裏面的數據到對應的Tokudb數據表裏面。而後刪除Innodb數據文件並釋放磁盤空間。

mysql> show processlist;
+----+-------------+-----------+--------+---------+-------+-----------------------------------------------------------+----------------------------------------+-----------+---------------+
| Id | User        | Host      | db     | Command | Time  | State                                                     | Info                                   | Rows_sent | Rows_examined |
+----+-------------+-----------+--------+---------+-------+-----------------------------------------------------------+----------------------------------------+-----------+---------------+
|  1 | system user |           | NULL   | Connect | 13776 | Waiting for table metadata lock                           | NULL                                   |         0 |             0 |
|  2 | system user |           | NULL   | Connect | 13936 | Waiting for master to send event                          | NULL                                   |         0 |             0 |
|  7 | root        | localhost | zabbix | Query   | 13767 | Fetched about 8502619000 rows, loading data still remains | alter table history_uint engine=TokuDB |         0 |             0 |
| 37 | root        | localhost | NULL   | Query   |     0 | init                                                      | show processlist                       |         0 |             0 |
+----+-------------+-----------+--------+---------+-------+-----------------------------------------------------------+----------------------------------------+-----------+---------------+

[root@ZBXTKDB01 ~]# ls -l /mysql/3307/tokudb/ |more
total 214125632
-rw------- 1 mysql mysql 0 Dec 20 17:35 __tokudb_lock_dont_delete_me_data
-rw------- 1 mysql mysql 0 Dec 20 17:35 __tokudb_lock_dont_delete_me_temp
-rw------- 1 mysql mysql 1921350 Dec 24 15:07 tokuld102q5N
-rw------- 1 mysql mysql 1948116 Dec 24 13:16 tokuld106Z4m
-rw------- 1 mysql mysql 1936454 Dec 24 14:10 tokuld10AgeQ
-rw------- 1 mysql mysql 1936518 Dec 24 14:42 tokuld10DHEW
-rw------- 1 mysql mysql 1938374 Dec 24 14:42 tokuld10DJqt
-rw------- 1 mysql mysql 1943288 Dec 24 14:51 tokuld10FlFb
-rw------- 1 mysql mysql 1943969 Dec 24 12:42 tokuld10jIQP

mysql> show processlist;
+-----+-------------+-----------+--------+---------+-------+---------------------------------------------------------------------+----------------------------------------+-----------+---------------+
| Id  | User        | Host      | db     | Command | Time  | State                                                               | Info                                   | Rows_sent | Rows_examined |
+-----+-------------+-----------+--------+---------+-------+---------------------------------------------------------------------+----------------------------------------+-----------+---------------+
|   1 | system user |           | NULL   | Connect | 97410 | Waiting for table metadata lock                                     | NULL                                   |         0 |             0 |
|   2 | system user |           | NULL   | Connect | 97570 | Waiting for master to send event                                    | NULL                                   |         0 |             0 |
|   7 | root        | localhost | zabbix | Query   | 97401 | Loading of data t ./zabbix/#sql-2822_7#P#p20180419 about 15.1% done | alter table history_uint engine=TokuDB |         0 |             0 |
| 827 | root        | localhost | NULL   | Query   |     0 | init                                                                | show processlist                       |         0 |             0 |
+-----+-------------+-----------+--------+---------+-------+---------------------------------------------------------------------+----------------------------------------+-----------+---------------+

2）因爲在轉換成TokuDB存儲引擎的時候，會生成成千上萬的臨時文件，確保數據庫分區空間可用性。建議1000G的Innodb轉換成Tokudb的時候至少預留300G磁盤空間。

3）Innodb大表在轉換成TokuDB存儲引擎的時候，特別耗時。經測試history_uint表總共有230億左右的數據，Fetched到臨時表總共花費了10小時30分鐘，Loading合成臨時文件也須要26個小時左右。

建議把命令放在後臺以腳本的方式去執行，若是出錯還能夠查看具體出錯的日誌。

[root@ZBXTKDB01 shell]# nohup ChangeInnodbToTokudb.sh &;

4）在後臺執行alter table history_uint engine=TokuDB的時候，程序忽然中斷報下面的錯誤。須要調大innodb_buffer_pool_size 的數值。

ERROR 1206 (HY000) at line 1: The total number of locks exceeds the lock table size

1. zabbix 數據庫更換 TokuDB 引擎
2. 【總結】InnoDB存儲引擎
3. 【MySQL】InnoDB引擎TokuDB引擎對比測試
4. TokuDB存儲引擎
5. TokuDB引擎筆記
6. TokuDB · 引擎特性 · HybridDB for MySQL高壓縮引擎TokuDB 揭祕
7. 轉換表的引擎
8. MySql 轉換表的引擎
9. [轉]TOKUDB® VS. INNODB FLASH MEMORY
10. 《MySQL + Innodb存儲引擎》三刷總結
更多相關文章...
• RSS 歷史 - RSS 教程
• XSLT - 轉換 - XSLT 教程
• 算法總結-雙指針
• 算法總結-回溯法

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。