第一次發佈博客,發現目錄竟然不會生成,後續慢慢熟悉博客園的設置。回正文~~~mysql
使用innobackupex,在業務空閒期執行,考慮到IO影響及 FLUSH TABLE WITH READ LOCAK 拷貝非INNODB文件的鎖表時間。。正則表達式
常規備份中,使用innobackupex在從庫備份執行,在無從庫的狀況下,容許在業務低峯期對整個實例拷貝。sql
考慮 數據量、磁盤IO狀況、恢復難度問題。數據庫
mysqldump鎖表時間長,備份時間長,可是導入方便,適合數據量小可是表格多 的庫/表級別備份。vim
innobackupex鎖表時間短,備份時間短,可是恢復較複雜,須要discord tablespace及 import TABLESPACE,除非容許備份文件成立單個實例,適合表數據量大但表格數量少的庫/表級別備份。緩存
若是是單表簡單查詢,使用mysqldump,添加where條件,例如:mysqldump -S /tmp/mysql3330.sock -uroot -p --databases db1 --tables tb1 tb2 tb3 -d >/data/backup/3330/mysqldump_20161229.sql 。架構
若是是複雜SQL查詢結果,使用 INTO OUTFILE,以下:app
1 #FIELDS TERMINATED BY ',' 字段間分割符 2 #OPTIONALLY ENCLOSED BY '"' 將字段包圍 對數值型無效 3 #LINES TERMINATED BY '\n' 換行符 4 5 #查詢導出 6 select * into outfile '/tmp/pt.txt' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' from pt where id >3; 7 8 #加載數據 9 load data infile '/tmp/pt1.txt' into table pt FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n'
使用mysqldump,添加-d參數。less
支持功能多且全面,可是鎖表時間是個風險點,使用時注意,同時,如果5.6版本以前的,要充分考慮buffer pool的使用狀況。socket
經過general log查看mysqldump運行原理,詳細流程見代碼塊 mysqldump。
mysqldump運行中,第一步,會檢查數據庫的配置狀況,例如是否設置GTID模式及參數配置;第二步,鎖全部表格,只容許讀操做;第三步,逐個拷貝表格,生成建立表格上SQL(字符集爲binary),再SELECT * FROM 表格 生成數據腳步(字符集爲UTF8);第4步,解鎖。
當導出全實例或者大數據庫時,這裏有2個須要注意到問題:
1 執行SQL:mysqldump -S /tmp/mysql3330.sock -uroot -p --databases zero >/data/backup/3330/mysqldump_20161229.sql 2 3 2016-12-27T14:38:27.782875Z 1732 Connect root@localhost on using Socket 4 2016-12-27T14:38:27.803572Z 1732 Query /*!40100 SET @@SQL_MODE='' */ 5 2016-12-27T14:38:27.804096Z 1732 Query /*!40103 SET TIME_ZONE='+00:00' */ 6 2016-12-27T14:38:27.804528Z 1732 Query SHOW VARIABLES LIKE 'gtid\_mode' #檢查是否設置了GTID 7 2016-12-27T14:38:27.813387Z 1732 Query SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IN (SELECT DISTINCT LOGFILE_GROUP_NAME FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA IN ('zero'))) GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME 8 2016-12-27T14:38:27.816987Z 1732 Query SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA IN('zero')) ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME 9 2016-12-27T14:38:27.819423Z 1732 Query SHOW VARIABLES LIKE 'ndbinfo\_version' 10 2016-12-27T14:38:27.824802Z 1732 Init DB zero 11 2016-12-27T14:38:27.825015Z 1732 Query SHOW CREATE DATABASE IF NOT EXISTS `zero` #生成建立數據庫的的腳步 12 2016-12-27T14:38:27.825381Z 1732 Query show tables #檢查該數據庫裏邊有多少表格,根據這些表格來開始lock table 13 2016-12-27T14:38:27.825969Z 14 1732 Query LOCK TABLES `dsns` READ /*!32311 LOCAL */,`pt` READ 15 /*!32311 LOCAL */,`sbtest20` READ /*!32311 LOCAL */ 16 #鎖表,僅容許讀操做 17 18 ########################################每一個表格重複部分############################################################ 19 2016-12-27T14:38:27.826324Z 1732 Query show table status like 'dsns' 20 2016-12-27T14:38:27.832651Z 1732 Query SET SQL_QUOTE_SHOW_CREATE=1 21 2016-12-27T14:38:27.832930Z 1732 Query SET SESSION character_set_results = 'binary' 22 2016-12-27T14:38:27.833169Z 1732 Query show create table `dsns` 23 #字符集修改成 binary,生成架構SQL 24 2016-12-27T14:38:27.833448Z 1732 Query SET SESSION character_set_results = 'utf8' 25 2016-12-27T14:38:27.833793Z 1732 Query show fields from `dsns` 26 2016-12-27T14:38:27.834697Z 1732 Query show fields from `dsns` 27 2016-12-27T14:38:27.835598Z 1732 Query SELECT /*!40001 SQL_NO_CACHE */ * FROM `dsns` 28 #字符集修改成 utf8,導出數據SQL 29 2016-12-27T14:38:27.836129Z 1732 Query SET SESSION character_set_results = 'binary' 30 2016-12-27T14:38:27.836401Z 1732 Query use `zero` 31 2016-12-27T14:38:27.836644Z 1732 Query select @@collation_database 32 2016-12-27T14:38:27.836949Z 1732 Query SHOW TRIGGERS LIKE 'dsns' 33 2016-12-27T14:38:27.837738Z 1732 Query SET SESSION character_set_results = 'utf8' 34 ########################################每一個表格重複部分############################################################ 35 36 #每一個表格的導出都重複上述部分 37 38 2016-12-27T14:38:28.525530Z 1732 Query SET SESSION character_set_results = 'utf8' 39 2016-12-27T14:38:28.525832Z 1732 Query UNLOCK TABLES 40 #解鎖,容許讀寫
如下參數在使用過程當中,須要留意,根據實際狀況添加:
只導出表結構
主要參數相見代碼模 mysqldump主要參數,並不是全部參數內容,這些參數較常使用。
1 [root@localhost zero]# mysqldump --help 2 Dumping structure and contents of MySQL databases and tables. 3 Usage: mysqldump [OPTIONS] database [tables] 4 OR mysqldump [OPTIONS] --databases [OPTIONS] DB1 [DB2 DB3...] 5 OR mysqldump [OPTIONS] --all-databases [OPTIONS] 6 7 8 --no-defaults Don't read default options from any option file, 9 except for login file. 10 --defaults-file=# Only read default options from the given file #. 11 12 13 -A, --all-databases Dump all the databases. This will be same as --databases 14 with all databases selected. 15 -Y, --all-tablespaces 16 Dump all the tablespaces. 17 -y, --no-tablespaces 18 Do not dump any tablespace information. 19 --add-drop-database Add a DROP DATABASE before each create. 20 --add-drop-table Add a DROP TABLE before each create. 21 (Defaults to on; use --skip-add-drop-table to disable.) 22 --add-drop-trigger Add a DROP TRIGGER before each create. 23 --add-locks Add locks around INSERT statements. 24 (Defaults to on; use --skip-add-locks to disable.) 25 26 27 --apply-slave-statements 28 Adds 'STOP SLAVE' prior to 'CHANGE MASTER' and 'START 29 SLAVE' to bottom of dump. 30 31 32 -B, --databases Dump several databases. Note the difference in usage; in 33 this case no tables are given. All name arguments are 34 regarded as database names. 'USE db_name;' will be 35 included in the output. 36 37 38 --master-data[=#] This causes the binary log position and filename to be 39 appended to the output. If equal to 1, will print it as a 40 CHANGE MASTER command; if equal to 2, that command will 41 be prefixed with a comment symbol. This option will turn 42 --lock-all-tables on, unless --single-transaction is 43 specified too (in which case a global read lock is only 44 taken a short time at the beginning of the dump; don't 45 forget to read about --single-transaction below). In all 46 cases, any action on logs will happen at the exact moment 47 of the dump. Option automatically turns --lock-tables 48 off. 49 -n, --no-create-db Suppress the CREATE DATABASE ... IF EXISTS statement that 50 normally is output for each dumped database if 51 --all-databases or --databases is given. 52 -t, --no-create-info 53 Don't write table creation info. 54 -d, --no-data No row information. 55 -p, --password[=name] 56 Password to use when connecting to server. If password is 57 not given it's solicited on the tty. 58 -P, --port=# Port number to use for connection. 59 60 61 --replace Use REPLACE INTO instead of INSERT INTO. 62 63 64 --set-gtid-purged[=name] 65 Add 'SET @@GLOBAL.GTID_PURGED' to the output. Possible 66 values for this option are ON, OFF and AUTO. If ON is 67 used and GTIDs are not enabled on the server, an error is 68 generated. If OFF is used, this option does nothing. If 69 AUTO is used and GTIDs are enabled on the server, 'SET 70 @@GLOBAL.GTID_PURGED' is added to the output. If GTIDs 71 are disabled, AUTO does nothing. If no value is supplied 72 then the default (AUTO) value will be considered. 73 --single-transaction 74 Creates a consistent snapshot by dumping all tables in a 75 single transaction. Works ONLY for tables stored in 76 storage engines which support multiversioning (currently 77 only InnoDB does); the dump is NOT guaranteed to be 78 consistent for other storage engines. While a 79 --single-transaction dump is in process, to ensure a 80 valid dump file (correct table contents and binary log 81 position), no other connection should use the following 82 statements: ALTER TABLE, DROP TABLE, RENAME TABLE, 83 TRUNCATE TABLE, as consistent snapshot is not isolated 84 from them. Option automatically turns off --lock-tables. 85 86 87 --tables Overrides option --databases (-B). 88 --triggers Dump triggers for each dumped table. 89 (Defaults to on; use --skip-triggers to disable.) 90 -u, --user=name User for login if not current user.
語法主要有如下三類:
Usage: mysqldump [OPTIONS] database [tables] OR mysqldump [OPTIONS] --databases [OPTIONS] DB1 [DB2 DB3...] OR mysqldump [OPTIONS] --all-databases [OPTIONS]
#實例備份 mysqldump -S /tmp/mysql3330.sock -uroot -p --all-datqabases >/data/backup/3330/mysqldump_20161229.sql #實例恢復 #新建實例後,導入腳本 mysql --socket=/tmp/mysql3306.sock -uroot -p < /data/backup/3330/mysqldump_20161229.sql
1 #指定單個或者多個DB備份 2 mysqldump -S /tmp/mysql3330.sock -uroot -p db1 db2 db3 >/data/backup/3330/mysqldump_20161229.sql 3 mysqldump -S /tmp/mysql3330.sock -uroot -p --databases db1 db2 db3 >/data/backup/3330/mysqldump_20161229.sql 4 5 #指定單個或者多個表格備份 6 mysqldump -S /tmp/mysql3330.sock -uroot -p --databases db1 --tables tb1 tb2 tb3 >/data/backup/3330/mysqldump_20161229.sql 7 mysqldump -S /tmp/mysql3330.sock -uroot -p db1 tb1 tb2 tb3 >/data/backup/3330/mysqldump_20161229.sql 8 9 #只導出單個表格的某些行數據 10 mysqldump -S /tmp/mysql3330.sock -uroot -pycf.com zero pt --where='1=1 limit 2' >/data/backup/3330/mysqldump_20161229.sql 11 12 #只備份表結構,不要表數據 13 mysqldump -S /tmp/mysql3330.sock -uroot -p --databases db1 --tables tb1 tb2 tb3 -d >/data/backup/3330/mysqldump_20161229.sql 14 15 #只備份表數據,不要表結構 16 mysqldump -S /tmp/mysql3330.sock -uroot -pycf.com zero pt --where='id>3' --no-create-info >/data/backup/3330/mysqldump_20161229.sql 17 18 #恢復數據 19 source /data/backup/3330/mysqldump_20161229.sql
PerconaXtraBackup軟件中,含有xtrabackup跟innobackupex,xtrabackup中不備份表結構,innobackupex調用xtrabackup子線程後再備份表結構,故經常使用innobackupex,xtraback不作平常使用。目前支持 Myisam,innodb,能夠備份 .frm, .MRG, .MYD, .MYI, .MAD, .MAI, .TRG, .TRN, .ARM, .ARZ, .CSM, CSV, .opt, .par, innoDB data 及innobdb log 文件。
對數據庫文件進行copy操做,同時創建多一個xtrabackup log 同步mysql的redo線程,copy數據文件結束時,flush table with read lock,拷貝非innodb數據文件的文件,拷貝結束後解鎖。原理圖見下圖(圖片來自知數堂)。經過general log查看mysqldump運行原理,詳細流程見代碼塊 innobackupex。
這裏須要注意2個點:
innobackupex鎖表時間是 data文件及log文件copy結束時,才鎖表,鎖表時長爲拷貝non-InnoDB tables and files的時長,相對時間較短,對業務影響小。
copy數據文件的過程當中,因爲是不鎖表,容許數據進行DML操做,這裏須要注意,若是這個時候,拷貝的過程當中有大事務一直沒有提交,界面顯示log scanned up,持續copy binlog追上數據庫的binlog文件,而且該時間點恰好全部事務已提交(這裏測試的時候,若是是單條 insert ,delete,update的大事務,則是要等待單條完成才提交,可是若是是begin事務裏邊的,不用等待是否commit or rollback,begin裏邊的單條事務執行結束,則就開始提交,恢復的時候,看成是undo 事務,不會提交該事物,回滾該事務)。大事務容易致使備份時長加長,IO佔用。
1 2016-12-26T15:18:39.627366Z 1659 Connect root@localhost on using Socket 2 2016-12-26T15:18:39.627789Z 1659 Query SET SESSION wait_timeout=2147483 3 2016-12-26T15:18:39.628193Z 1659 Query SHOW VARIABLES 4 #記錄LSN號碼,開始copy ibd文件 5 2016-12-26T15:18:55.673740Z 1659 Query SET SESSION lock_wait_timeout=31536000 6 2016-12-26T15:18:55.674281Z 1659 Query FLUSH NO_WRITE_TO_BINLOG TABLES 7 #強制把沒有 還沒寫入binlog 磁盤文件的緩存 強制刷新到磁盤 8 #開始拷貝數據庫文件,這裏須要注意,若是這個時候,拷貝的過程當中有大事務一直沒有提交,則會一直拷貝其產生的 ,界面顯示log scanned up,直到copy binlog追上數據庫的binlog文件,而且該時間點恰好全部事務已提交(這裏測試的時候,若是是單條 insert ,delete,update的大事務,則是要等待單條完成才提交,可是若是是begin事務裏邊的,不用等待是否commit or rollback,begin裏邊的單條事務執行結束,則就開始提交,恢復的時候,看成是undo 事務,不會提交該事物,回滾該事務。 ) 9 2016-12-26T15:18:55.676345Z 1659 Query FLUSH TABLES WITH READ LOCK 10 #鎖表,只容許讀,不容許寫及其餘架構修改操做 11 #拷貝除innodb 數據文件外的其餘全部文件,包括表結構等,Starting to backup non-InnoDB tables and files 12 2016-12-26T15:18:59.691409Z 1659 Query SHOW MASTER STATUS 13 #記錄 備份到的 binlog文件及position位置,這個記錄在 xtrabackup_binlog_info 文件,可提供複製使用 14 2016-12-26T15:18:59.734418Z 1659 Query SHOW VARIABLES 15 2016-12-26T15:18:59.754530Z 1659 Query FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS 16 2016-12-26T15:18:59.968452Z 1659 Query UNLOCK TABLES 17 #解鎖,表格恢復可寫,架構可修改 18 2016-12-26T15:18:59.991046Z 1659 Query SELECT UUID() 19 2016-12-26T15:19:00.005980Z 1659 Query SELECT VERSION()
根據 BACKUP-DIR/xtrabackup_logfile建立新的logfile,xtrabackup爲子進程,不鏈接數據庫服務.
1 #全量備份 實例備份及恢復 2 #備份 3 innobackupex --defaults-file=/data/mysql/mysql3330.cnf --user=root --password=ycf.com --no-timestamp /data/backup/3330/20161229 4 innobackupex --apply-log /data/backup/3330/20161229 5 6 #恢復 7 innobackupex --copy-back --datadir=/data/mysql/mysql3350/data /data/backup/3330/20161229
1 #增量備份 2 innobackupex --defaults-file=/data/mysql/mysql3376.cnf --user=root --password=ycf.com --no-timestamp --incremental-basedir=/data/backup/3330/20161229 --incremental /data/backup/mysql3376/20161230diff 3 4 innobackupex --defaults-file=/data/mysql/mysql3376.cnf --user=root --password=ycf.com --no-timestamp --incremental-basedir=/data/backup/3330/20161230diff --incremental /data/backup/mysql3376/20161231diff 5 6 #增量恢復 7 #如今完整備份文件中中應用redo日誌,記得是redo-only, redo-only, redo-only, redo-only, 不是readonly,打死記得,不要亂來!!!!!! 8 innobackupex --apply-log --redo-only /data/backup/3330/20161229 9 10 #應用第一個增量備份文件的redo日誌到完整備份文件夾中 11 innobackupex --apply-log --redo-only /data/backup/3330/20161229 --incremental-dir=/data/backup/mysql3376/20161230diff 12 13 #應用最後一個增量備份文件的redo日誌到完整備份文件夾中,能夠直接apply-log 14 innobackupex --apply-log /data/backup/3330/20161229 --incremental-dir=/data/backup/mysql3376/20161231diff
1 #部分備份 2 #指定數據庫備份 3 innobackupex --defaults-file=/data/mysql/mysql3330.cnf --databases='zero mysql' --user=root --password=ycf.com --no-timestamp /data/backup/3330/20161202 4 5 #指定表格備份 6 #3.1 --include 使用正則表達式 7 8 #3.2 --table-file 備份的完整表名寫在file文件中 9 vim /tmp/backupfile #每行寫一個庫名,或者一個表的全名(database.table),寫完庫名或者表名後,千萬不要有空格或者其餘空白符號,會致使識別不了該表格或者庫名,從而致使跳過 10 innobackupex --defaults-file=/data/mysql/mysql3330.cnf --tables-file=/tmp/backupfile --user=root --password=ycf.com --no-timestamp /data/backup/3330/20161204 11 12 #3.3 --databases 完整庫名和表名寫在一塊兒,用空格隔開 13 innobackupex --defaults-file=/data/mysql/mysql3330.cnf --user=root --password=ycf.com --no-timestamp --databases=zero.s1 /data/backup/3330/20161229 14 15 #指定表格恢復(開啓獨立表空間) 16 #首先要本身如今須要恢復的數據庫上,建立該表格,而後discard tablespace,拷貝ibd文件過來,chown 文件全部者及用戶組爲mysql,再 import tablespace。 17 #若是有大量表格,用這個操做就比較麻煩,須要一個個來建立,包括指定數據庫,也是這樣處理,整個數據庫先建立以後,在一個個表格discard,再import。 18 ALTER TABLE S1 DISCARD TABLESPACE; 19 ALTER TABLE S1 import TABLESPACE;