服務器異常宕機引起的mysql故障

時間 2019-11-17

原文原文鏈接

1：服務器異常宕機、mysql、mysql_safe都沒法啓動
查看mysql報以下錯誤：
2018-03-30 06:25:12 1649 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future r
elease. Please use the full name instead.
2018-03-30 06:25:12 1649 [Note] Plugin 'FEDERATED' is disabled.
2018-03-30 06:25:12 1649 [ERROR] Function 'innodb' already exists
2018-03-30 06:25:12 1649 [Warning] Couldn't load plugin named 'innodb' with soname 'ha_innodb.so'.
2018-03-30 06:25:12 1649 [ERROR] Function 'federated' already exists
2018-03-30 06:25:12 1649 [Warning] Couldn't load plugin named 'federated' with soname 'ha_federated.so'.
2018-03-30 06:25:12 1649 [ERROR] Function 'blackhole' already exists
2018-03-30 06:25:12 1649 [Warning] Couldn't load plugin named 'blackhole' with soname 'ha_blackhole.so'.
2018-03-30 06:25:12 1649 [ERROR] Function 'archive' already exists
2018-03-30 06:25:12 1649 [Warning] Couldn't load plugin named 'archive' with soname 'ha_archive.so'.
2018-03-30 06:25:12 1649 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-03-30 06:25:12 1649 [Note] InnoDB: The InnoDB memory heap is disabled
2018-03-30 06:25:12 1649 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-03-30 06:25:12 1649 [Note] InnoDB: Memory barrier is not used
2018-03-30 06:25:12 1649 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-03-30 06:25:12 1649 [Note] InnoDB: Using Linux native AIO
2018-03-30 06:25:12 1649 [Note] InnoDB: Using CPU crc32 instructions
2018-03-30 06:25:12 1649 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-03-30 06:25:12 1649 [Note] InnoDB: Completed initialization of buffer pool
2018-03-30 06:25:35 1649 [ERROR] InnoDB: Tried to read 1048576 bytes at offset 1048576. Was only able to read 913408.
InnoDB: Fatal error: cannot read from file. OS error number 17.
2018-03-30 06:25:35 7fcc5aafc780 InnoDB: Assertion failure in thread 140515671525248 in file os0file.cc line 2694
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
22:25:35 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.html

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=400
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 175093 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.mysql

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x555930e2e94c]
/usr/sbin/mysqld(handle_fatal_signal+0x3c2)[0x555930b84e62]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fcc59c81330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fcc590c2c37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fcc590c6028]
/usr/sbin/mysqld(+0x815583)[0x555930f3f583]
/usr/sbin/mysqld(_Z28buf_dblwr_init_or_load_pagesiPcb+0x272)[0x5559310105a2]
/usr/sbin/mysqld(+0x883b1c)[0x555930fadb1c]
/usr/sbin/mysqld(+0x7c5bbc)[0x555930eefbbc]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x555930ab3b28]
/usr/sbin/mysqld(+0x4f19e0)[0x555930c1b9e0]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x928)[0x555930c21608]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xa92)[0x555930aac9b2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fcc590adf45]
/usr/sbin/mysqld(+0x375c2d)[0x555930a9fc2d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.linux

解決辦法：
刪除掉數據目錄下ibdata*、ib_logfile*文件,sql

重啓後可啓動數據庫，因數據庫是非正常關閉引發的
****全部這相似的狀況必須先備份數據庫，避免人爲操做形成的損壞
tar -zcf mysqlback20180329.tar.gz mysql
正常啓動後mysql 能夠正常訪問，可是日誌報錯
2018-03-30 10:59:59 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 10:59:59 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.數據庫

因此直接退出，查看整個全部庫大概有多少張表損壞
mysqlcheck -A -uroot -ppassworkd > /tmp/unnormaltable
**(查單個庫和表：mysqlcheck zabbix users -uroot -ppassword)socket

經過上述命令輸出而後在分析大概有哪些表須要作恢復
grep -v ":" /tmp/unnormaltable |awk -F"." '{print $1,$2}'|sort -rn |uniq -c測試

恢復zabbix庫
grep -v ":" /tmp/unnormaltable|grep zabbix |awk -F"." '{print $2}' > tablezabbix優化

把zabbix庫中的文件所有mv到其它目錄，導入空表結構ui

mysql -uroot -ppassword < zabbix.sql

到出表空間，清空idb文件，只剩下frm文件
while read line;do mysql -uroot -ppassword -e "use zabbix;SET FOREIGN_KEY_CHECKS = 0 ;ALTER TABLE $line DISCARD TABLESPACE";done < tablezabbix

ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails ()
SET FOREIGN_KEY_CHECKS = 0

把以前備份的ibd文件從新導入到zabbix目錄下
for i in `find myzabbix/ -name *.ibd `;do mv $i ./zabbix;done
把導入的ibd文件導入表空間
while read line;do mysql -uroot -ppassword -e "use zabbix;ALTER TABLE $line IMPORT TABLESPACE";done < tablezabbix

Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
導入的過程當中若是有的表空間過大的話可能會報錯，能夠適當優化兩個參數
wait_timeout = x 超時時間如600秒
max_allowed_packet = y 最大容許數據量

總體過程：一、備份二、肯定哪些表損壞三、移除損壞表的數據至備份目錄四、建立這些表五、導出表空間六、移入剛移出的這些表的ibd文件七、導入表空間八、測試