昨天晚上zabbix服務器報一臺主機的磁盤空間超過利用率,剛好我電腦開着,因此登陸上去看了下,排查過程以下:
bash
由於知道這臺服務器的具體業務功能,正常狀況下是不會當即達到監控線的,經過top查看了下進程,發現sytemd-journal這個進程,懷疑日誌文件被大量的寫入。服務器
systemd-journald是一個改進型日誌管理服務,能夠收集來自內核、系統早期啓動階段的日誌、系統守護進程在啓動和運行中的標準輸出和錯誤信息,還有syslog的日誌。該日誌服務僅僅把日誌集中保存在單一結構的日誌文件/run/log中,因爲日誌是經歷過壓縮和格式化的二進制數據,因此在查看和定位的時候很迅速。默認狀況下並不會持久化保存日誌,只會保留一個月的日誌。另外,一些rsyslog沒法收集的日誌也會被journal記錄到.ide
[root@ops-monitor-01 /var/log/journal] # du -sh /var/log/journal/ 1.5G /var/log/journal/ 說明:果真,這個目錄比以前多了不少東西,原本經過刪除清理下里面的日誌,忽然想到journactl有個高級的功能,能夠作,因而有了下面的操做 [root@ops-monitor-01 /var/log/journal] # journalctl --vacuum-size=500M Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000000001-000550f7deac7403.journal (64.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-00000000000047fb-000551307d518b3e.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000013fcd-0005535c3692ab03.journal (48.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-00000000000140be-0005535fdff3ea1d.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-000000000001f484-000555c082ebcdf7.journal (64.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-000000000001fb4d-000555d51cf51bc6.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-00000000000315f3-00055824d9a1b26c.journal (96.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-0000000000032b7f-0005585387ff50f0.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-000000000004ac87-00055a8926aba5d2.journal (104.1M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-0000000000051652-00055b1a3b4b15ab.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000069762-00055ced73056d1d.journal (112.1M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-000000000006a14c-00055cf8d0f034cb.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000089211-00055e2ef50709d8.journal (128.1M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-000000000008b23a-00055e319abb77e8.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-00000000000bc349-00055e8eb47900a7.journal (128.1M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-00000000000d57a3-00055e8ef8591843.journal (8.0M). Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-00000000000f99cd-00055e8f57466708.journal (128.1M). Vacuuming done, freed 937.0M of archived journals on disk. [root@ops-monitor-01 /var/log/journal] # du -sh /var/log/journal/ 513M /var/log/journal/ [root@ops-monitor-01 /var/log/journal] # df -h Filesystem Size Used Avail Use% Mounted on /dev/vda1 99G 76G 18G 82% / devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 100K 1.9G 1% /dev/shm tmpfs 1.9G 488K 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/vdb 197G 5.9G 181G 4% /war /dev/vdc1 100G 60G 41G 60% /mnt/yum tmpfs 380M 0 380M 0% /run/user/1000
到此,問題已經接近,須要作進一步瞭解的朋友能夠查看下journalctl的幫助命令,相信會有收穫。
spa