經過文件系統檢查(fsck)確保數據完整性
經過文件系統檢查(fsck)確保數據完整性java
做者：尹正傑node
版權聲明：原創做品，謝絕轉載！不然將追究法律責任。api
一.HDFS文件系統檢查概述app
Linux系統管理員運行fsck命令以修復ext3或者ext4文件系統上的損壞文件。fsck命令檢查磁盤結構，若是它們已損壞，能夠修復它們。HDFS提供了相似的fsck命令，來幫助肯定文件丟失快或損壞的塊。 與Linux fsck命令不一樣，hdfs fsck命令不會嘗試修復錯誤，但提供了修復損壞塊的選項。 [root@hadoop101.yinzhengjie.com ~]# hdfs fsck Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks | -replicaDetails | -upgradedomains]]]] [-includeSnapshots] [-stor agepolicies] [-maintenance] [-blockId <blk_Id>] <path> start checking from this path -move move corrupted files to /lost+found -delete delete corrupted files -files print out files being checked -openforwrite print out files opened for write -includeSnapshots include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it -list-corruptfileblocks print out list of missing blocks and files they belong to -files -blocks print out block report -files -blocks -locations print out locations for every block -files -blocks -racks print out network topology for data-node locations -files -blocks -replicaDetails print out each replica details -files -blocks -upgradedomains print out upgrade domains for every block -storagepolicies print out storage policy summary for the blocks -maintenance print out maintenance state node details -blockId print out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc) Please Note: 1. By default fsck ignores files opened for write, use -openforwrite to report such files. They are usually tagged CORRUPT or HEALTHY depending on their block allocation status 2. Option -includeSnapshots should not be used for comparing stats, should be used only for HEALTH check, as this may contain duplicates if the same file present in both original fs tree and inside snapshots. Generic options supported are: -conf <configuration file> specify an application configuration file -D <property=value> define a value for a given property -fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations. -jt <local|resourcemanager:port> specify a ResourceManager -files <file1,...> specify a comma-separated list of files to be copied to the map reduce cluster -libjars <jar1,...> specify a comma-separated list of jar files to be included in the classpath -archives <archive1,...> specify a comma-separated list of archives to be unarchived on the compute machines The general command line syntax is: command [genericOptions] [commandOptions] [root@hadoop101.yinzhengjie.com ~]#
二.hdfs fsck經常使用命令選項案例dom
1>.執行HDFS文件系統檢查ide
[root@hadoop101.yinzhengjie.com ~]# hdfs fsck / Connecting to namenode via http://hadoop101.yinzhengjie.com:50070/fsck?ugi=root&path=%2F FSCK started by root (auth:SIMPLE) from /172.200.6.101 for path / at Fri Aug 21 18:18:07 CST 2020 ...............Status: HEALTHY #很明顯，當前的狀態是"HEALTHY" Total size: 392718079 B #HDFS文件系統使用的實際大小，並不包含副本因子。 Total dirs: 12 #總的目錄數是12個 Total files: 15 #總的文件數是15個 Total symlinks: 0 #總的符號鏈接數0個 Total blocks (validated): 16 (avg. block size 24544879 B) #總的塊數16個(平均快大小是24MB左右) Minimally replicated blocks: 16 (100.0 %) #最小副本塊數 Over-replicated blocks: 0 (0.0 %) #過分複製的副本塊數 Under-replicated blocks: 0 (0.0 %) #正在複製的副本塊數 Mis-replicated blocks: 0 (0.0 %) #錯誤副本塊數 Default replication factor: 3 #默認的副本因子 Average block replication: 2.0 #平均的複製塊 Corrupt blocks: 0 #損壞的塊 Missing replicas: 0 (0.0 %) #缺乏的副本 Number of data-nodes: 5 #DataNode的數量 Number of racks: 3 #機架的數量 FSCK ended at Fri Aug 21 18:18:07 CST 2020 in 1 milliseconds The filesystem under path '/' is HEALTHY [root@hadoop101.yinzhengjie.com ~]#
[root@hadoop101.yinzhengjie.com ~]# hdfs fsck / | grep -v "replicated" #檢測某路徑時，您也能夠按關鍵字來進行過濾 Connecting to namenode via http://hadoop101.yinzhengjie.com:50070/fsck?ugi=root&path=%2F FSCK started by root (auth:SIMPLE) from /172.200.6.101 for path / at Fri Aug 21 18:17:37 CST 2020 ...............Status: HEALTHY Total size: 392718079 B Total dirs: 12 Total files: 15 Total symlinks: 0 Total blocks (validated): 16 (avg. block size 24544879 B) Default replication factor: 3 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 5 Number of racks: 3 FSCK ended at Fri Aug 21 18:17:37 CST 2020 in 1 milliseconds The filesystem under path '/' is HEALTHY [root@hadoop101.yinzhengjie.com ~]#
2>.顯示正在檢查的文件oop
[root@hadoop101.yinzhengjie.com ~]# hdfs fsck / -files Connecting to namenode via http://hadoop101.yinzhengjie.com:50070/fsck?ugi=root&files=1&path=%2F FSCK started by root (auth:SIMPLE) from /172.200.6.101 for path / at Fri Aug 21 18:31:30 CST 2020 / <dir> /bigdata <dir> /system <dir> /user <dir> /user/root <dir> /user/root/.Trash <dir> /user/root/.Trash/200821150442 <dir> /user/root/.Trash/200821150442/hosts 316968 bytes, 1 block(s): OK /user/root/.Trash/Current <dir> /user/root/.Trash/Current/yinzhengjie <dir> /user/root/a.txt 0 bytes, 0 block(s): OK /user/root/hadoop-2.10.0.tar.gz 392115733 bytes, 3 block(s): OK /user/root/messages 271339 bytes, 1 block(s): OK /user/root/test <dir> /yinzhengjie <dir> /yinzhengjie/hosts 371 bytes, 1 block(s): OK /yinzhengjie/wc.txt.gz 69 bytes, 1 block(s): OK /yinzhengjie/yum.repos.d <dir> /yinzhengjie/yum.repos.d/CentOS-Base.repo 1664 bytes, 1 block(s): OK /yinzhengjie/yum.repos.d/CentOS-CR.repo 1309 bytes, 1 block(s): OK /yinzhengjie/yum.repos.d/CentOS-Debuginfo.repo 649 bytes, 1 block(s): OK /yinzhengjie/yum.repos.d/CentOS-Media.repo 630 bytes, 1 block(s): OK /yinzhengjie/yum.repos.d/CentOS-Sources.repo 1331 bytes, 1 block(s): OK /yinzhengjie/yum.repos.d/CentOS-Vault.repo 5701 bytes, 1 block(s): OK /yinzhengjie/yum.repos.d/CentOS-fasttrack.repo 314 bytes, 1 block(s): OK /yinzhengjie/yum.repos.d/epel-testing.repo 1050 bytes, 1 block(s): OK /yinzhengjie/yum.repos.d/epel.repo 951 bytes, 1 block(s): OK Status: HEALTHY Total size: 392718079 B Total dirs: 12 Total files: 15 Total symlinks: 0 Total blocks (validated): 16 (avg. block size 24544879 B) Minimally replicated blocks: 16 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 5 Number of racks: 3 FSCK ended at Fri Aug 21 18:31:30 CST 2020 in 1 milliseconds The filesystem under path '/' is HEALTHY [root@hadoop101.yinzhengjie.com ~]#
3>.顯示正在檢查的文件及其對應的塊信息ui
[root@hadoop101.yinzhengjie.com ~]# hdfs fsck / -files -blocks Connecting to namenode via http://hadoop101.yinzhengjie.com:50070/fsck?ugi=root&files=1&blocks=1&path=%2F FSCK started by root (auth:SIMPLE) from /172.200.6.101 for path / at Fri Aug 21 18:35:37 CST 2020 / <dir> /bigdata <dir> /system <dir> /user <dir> /user/root <dir> /user/root/.Trash <dir> /user/root/.Trash/200821150442 <dir> /user/root/.Trash/200821150442/hosts 316968 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741838_1892 len=316968 Live_repl=2 /user/root/.Trash/Current <dir> /user/root/.Trash/Current/yinzhengjie <dir> /user/root/a.txt 0 bytes, 0 block(s): OK /user/root/hadoop-2.10.0.tar.gz 392115733 bytes, 3 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741885_1915 len=134217728 Live_repl=1 1. BP-110925747-172.200.6.101-1597215213173:blk_1073741886_1916 len=134217728 Live_repl=1 2. BP-110925747-172.200.6.101-1597215213173:blk_1073741887_1917 len=123680277 Live_repl=1 /user/root/messages 271339 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741881_1911 len=271339 Live_repl=3 /user/root/test <dir> /yinzhengjie <dir> /yinzhengjie/hosts 371 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741898_1928 len=371 Live_repl=3 /yinzhengjie/wc.txt.gz 69 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741899_1929 len=69 Live_repl=3 /yinzhengjie/yum.repos.d <dir> /yinzhengjie/yum.repos.d/CentOS-Base.repo 1664 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741839_1015 len=1664 Live_repl=2 /yinzhengjie/yum.repos.d/CentOS-CR.repo 1309 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741840_1016 len=1309 Live_repl=2 /yinzhengjie/yum.repos.d/CentOS-Debuginfo.repo 649 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741841_1017 len=649 Live_repl=2 /yinzhengjie/yum.repos.d/CentOS-Media.repo 630 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741842_1018 len=630 Live_repl=2 /yinzhengjie/yum.repos.d/CentOS-Sources.repo 1331 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741843_1019 len=1331 Live_repl=2 /yinzhengjie/yum.repos.d/CentOS-Vault.repo 5701 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741844_1020 len=5701 Live_repl=2 /yinzhengjie/yum.repos.d/CentOS-fasttrack.repo 314 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741845_1021 len=314 Live_repl=2 /yinzhengjie/yum.repos.d/epel-testing.repo 1050 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741846_1022 len=1050 Live_repl=2 /yinzhengjie/yum.repos.d/epel.repo 951 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741847_1023 len=951 Live_repl=2 Status: HEALTHY Total size: 392718079 B Total dirs: 12 Total files: 15 Total symlinks: 0 Total blocks (validated): 16 (avg. block size 24544879 B) Minimally replicated blocks: 16 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 5 Number of racks: 3 FSCK ended at Fri Aug 21 18:35:37 CST 2020 in 1 milliseconds The filesystem under path '/' is HEALTHY [root@hadoop101.yinzhengjie.com ~]#
4>.顯示正在檢查的文件，塊及其位置信息this
[root@hadoop101.yinzhengjie.com ~]# hdfs fsck / -files -blocks -locations Connecting to namenode via http://hadoop101.yinzhengjie.com:50070/fsck?ugi=root&files=1&blocks=1&locations=1&path=%2F FSCK started by root (auth:SIMPLE) from /172.200.6.101 for path / at Fri Aug 21 18:39:21 CST 2020 / <dir> /bigdata <dir> /system <dir> /user <dir> /user/root <dir> /user/root/.Trash <dir> /user/root/.Trash/200821150442 <dir> /user/root/.Trash/200821150442/hosts 316968 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741838_1892 len=316968 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ca481097-2f96-4656-86ce-7c508ebc862a,DISK], DatanodeIn foWithStorage[172.200.6.102:50010,DS-08774d77-a5d0-4442-8b19-91924ee8f413,DISK]] /user/root/.Trash/Current <dir> /user/root/.Trash/Current/yinzhengjie <dir> /user/root/a.txt 0 bytes, 0 block(s): OK /user/root/hadoop-2.10.0.tar.gz 392115733 bytes, 3 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741885_1915 len=134217728 Live_repl=1 [DatanodeInfoWithStorage[172.200.6.102:50010,DS-08774d77-a5d0-4442-8b19-91924ee8f413,DISK]] 1. BP-110925747-172.200.6.101-1597215213173:blk_1073741886_1916 len=134217728 Live_repl=1 [DatanodeInfoWithStorage[172.200.6.102:50010,DS-db3faa5c-87ee-4df3-ad65-ae44d1239bc3,DISK]] 2. BP-110925747-172.200.6.101-1597215213173:blk_1073741887_1917 len=123680277 Live_repl=1 [DatanodeInfoWithStorage[172.200.6.102:50010,DS-08774d77-a5d0-4442-8b19-91924ee8f413,DISK]] /user/root/messages 271339 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741881_1911 len=271339 Live_repl=3 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ae05d8da-dc8f-42a1-8965-bf1033c4802e,DISK], DatanodeIn foWithStorage[172.200.6.104:50010,DS-ff7e2587-7848-474e-84e4-b086e04af40b,DISK], DatanodeInfoWithStorage[172.200.6.102:50010,DS-db3faa5c-87ee-4df3-ad65-ae44d1239bc3,DISK]] /user/root/test <dir> /yinzhengjie <dir> /yinzhengjie/hosts 371 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741898_1928 len=371 Live_repl=3 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ae05d8da-dc8f-42a1-8965-bf1033c4802e,DISK], DatanodeInfoW ithStorage[172.200.6.104:50010,DS-ff7e2587-7848-474e-84e4-b086e04af40b,DISK], DatanodeInfoWithStorage[172.200.6.102:50010,DS-db3faa5c-87ee-4df3-ad65-ae44d1239bc3,DISK]] /yinzhengjie/wc.txt.gz 69 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741899_1929 len=69 Live_repl=3 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ca481097-2f96-4656-86ce-7c508ebc862a,DISK], DatanodeInfoWi thStorage[172.200.6.104:50010,DS-3ae3a2ae-f716-4bec-b113-4b92243a0592,DISK], DatanodeInfoWithStorage[172.200.6.102:50010,DS-08774d77-a5d0-4442-8b19-91924ee8f413,DISK]] /yinzhengjie/yum.repos.d <dir> /yinzhengjie/yum.repos.d/CentOS-Base.repo 1664 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741839_1015 len=1664 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ae05d8da-dc8f-42a1-8965-bf1033c4802e,DISK], DatanodeInfo WithStorage[172.200.6.102:50010,DS-08774d77-a5d0-4442-8b19-91924ee8f413,DISK]] /yinzhengjie/yum.repos.d/CentOS-CR.repo 1309 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741840_1016 len=1309 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ae05d8da-dc8f-42a1-8965-bf1033c4802e,DISK], DatanodeInfo WithStorage[172.200.6.102:50010,DS-08774d77-a5d0-4442-8b19-91924ee8f413,DISK]] /yinzhengjie/yum.repos.d/CentOS-Debuginfo.repo 649 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741841_1017 len=649 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ae05d8da-dc8f-42a1-8965-bf1033c4802e,DISK], DatanodeInfoW ithStorage[172.200.6.102:50010,DS-db3faa5c-87ee-4df3-ad65-ae44d1239bc3,DISK]] /yinzhengjie/yum.repos.d/CentOS-Media.repo 630 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741842_1018 len=630 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ae05d8da-dc8f-42a1-8965-bf1033c4802e,DISK], DatanodeInfoW ithStorage[172.200.6.102:50010,DS-db3faa5c-87ee-4df3-ad65-ae44d1239bc3,DISK]] /yinzhengjie/yum.repos.d/CentOS-Sources.repo 1331 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741843_1019 len=1331 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ca481097-2f96-4656-86ce-7c508ebc862a,DISK], DatanodeInfo WithStorage[172.200.6.102:50010,DS-08774d77-a5d0-4442-8b19-91924ee8f413,DISK]] /yinzhengjie/yum.repos.d/CentOS-Vault.repo 5701 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741844_1020 len=5701 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ca481097-2f96-4656-86ce-7c508ebc862a,DISK], DatanodeInfo WithStorage[172.200.6.102:50010,DS-db3faa5c-87ee-4df3-ad65-ae44d1239bc3,DISK]] /yinzhengjie/yum.repos.d/CentOS-fasttrack.repo 314 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741845_1021 len=314 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.104:50010,DS-ff7e2587-7848-474e-84e4-b086e04af40b,DISK], DatanodeInfoW ithStorage[172.200.6.102:50010,DS-08774d77-a5d0-4442-8b19-91924ee8f413,DISK]] /yinzhengjie/yum.repos.d/epel-testing.repo 1050 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741846_1022 len=1050 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ca481097-2f96-4656-86ce-7c508ebc862a,DISK], DatanodeInfo WithStorage[172.200.6.102:50010,DS-db3faa5c-87ee-4df3-ad65-ae44d1239bc3,DISK]] /yinzhengjie/yum.repos.d/epel.repo 951 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741847_1023 len=951 Live_repl=2 [DatanodeInfoWithStorage[172.200.6.103:50010,DS-ae05d8da-dc8f-42a1-8965-bf1033c4802e,DISK], DatanodeInfoW ithStorage[172.200.6.102:50010,DS-db3faa5c-87ee-4df3-ad65-ae44d1239bc3,DISK]] Status: HEALTHY Total size: 392718079 B Total dirs: 12 Total files: 15 Total symlinks: 0 Total blocks (validated): 16 (avg. block size 24544879 B) Minimally replicated blocks: 16 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 5 Number of racks: 3 FSCK ended at Fri Aug 21 18:39:21 CST 2020 in 1 milliseconds The filesystem under path '/' is HEALTHY [root@hadoop101.yinzhengjie.com ~]#
5>.顯示正在檢查的文件，塊，位置和所在機架信息spa
[root@hadoop101.yinzhengjie.com ~]# hdfs fsck / -files -blocks -locations -racks Connecting to namenode via http://hadoop101.yinzhengjie.com:50070/fsck?ugi=root&files=1&blocks=1&locations=1&racks=1&path=%2F FSCK started by root (auth:SIMPLE) from /172.200.6.101 for path / at Fri Aug 21 18:44:43 CST 2020 / <dir> /bigdata <dir> /system <dir> /user <dir> /user/root <dir> /user/root/.Trash <dir> /user/root/.Trash/200821150442 <dir> /user/root/.Trash/200821150442/hosts 316968 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741838_1892 len=316968 Live_repl=2 [/rack002/172.200.6.103:50010, /rack001/172.200.6.102:50010] /user/root/.Trash/Current <dir> /user/root/.Trash/Current/yinzhengjie <dir> /user/root/a.txt 0 bytes, 0 block(s): OK /user/root/hadoop-2.10.0.tar.gz 392115733 bytes, 3 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741885_1915 len=134217728 Live_repl=1 [/rack001/172.200.6.102:50010] 1. BP-110925747-172.200.6.101-1597215213173:blk_1073741886_1916 len=134217728 Live_repl=1 [/rack001/172.200.6.102:50010] 2. BP-110925747-172.200.6.101-1597215213173:blk_1073741887_1917 len=123680277 Live_repl=1 [/rack001/172.200.6.102:50010] /user/root/messages 271339 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741881_1911 len=271339 Live_repl=3 [/rack002/172.200.6.103:50010, /rack002/172.200.6.104:50010, /rack001/172.200.6.102:50010] /user/root/test <dir> /yinzhengjie <dir> /yinzhengjie/hosts 371 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741898_1928 len=371 Live_repl=3 [/rack002/172.200.6.103:50010, /rack002/172.200.6.104:50010, /rack001/172.200.6.102:50010] /yinzhengjie/wc.txt.gz 69 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741899_1929 len=69 Live_repl=3 [/rack002/172.200.6.103:50010, /rack002/172.200.6.104:50010, /rack001/172.200.6.102:50010] /yinzhengjie/yum.repos.d <dir> /yinzhengjie/yum.repos.d/CentOS-Base.repo 1664 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741839_1015 len=1664 Live_repl=2 [/rack002/172.200.6.103:50010, /rack001/172.200.6.102:50010] /yinzhengjie/yum.repos.d/CentOS-CR.repo 1309 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741840_1016 len=1309 Live_repl=2 [/rack002/172.200.6.103:50010, /rack001/172.200.6.102:50010] /yinzhengjie/yum.repos.d/CentOS-Debuginfo.repo 649 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741841_1017 len=649 Live_repl=2 [/rack002/172.200.6.103:50010, /rack001/172.200.6.102:50010] /yinzhengjie/yum.repos.d/CentOS-Media.repo 630 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741842_1018 len=630 Live_repl=2 [/rack002/172.200.6.103:50010, /rack001/172.200.6.102:50010] /yinzhengjie/yum.repos.d/CentOS-Sources.repo 1331 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741843_1019 len=1331 Live_repl=2 [/rack002/172.200.6.103:50010, /rack001/172.200.6.102:50010] /yinzhengjie/yum.repos.d/CentOS-Vault.repo 5701 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741844_1020 len=5701 Live_repl=2 [/rack002/172.200.6.103:50010, /rack001/172.200.6.102:50010] /yinzhengjie/yum.repos.d/CentOS-fasttrack.repo 314 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741845_1021 len=314 Live_repl=2 [/rack002/172.200.6.104:50010, /rack001/172.200.6.102:50010] /yinzhengjie/yum.repos.d/epel-testing.repo 1050 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741846_1022 len=1050 Live_repl=2 [/rack002/172.200.6.103:50010, /rack001/172.200.6.102:50010] /yinzhengjie/yum.repos.d/epel.repo 951 bytes, 1 block(s): OK 0. BP-110925747-172.200.6.101-1597215213173:blk_1073741847_1023 len=951 Live_repl=2 [/rack002/172.200.6.103:50010, /rack001/172.200.6.102:50010] Status: HEALTHY Total size: 392718079 B Total dirs: 12 Total files: 15 Total symlinks: 0 Total blocks (validated): 16 (avg. block size 24544879 B) Minimally replicated blocks: 16 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 5 Number of racks: 3 FSCK ended at Fri Aug 21 18:44:43 CST 2020 in 1 milliseconds The filesystem under path '/' is HEALTHY [root@hadoop101.yinzhengjie.com ~]#
6>.列出缺乏的塊及其所述的文件
[root@hadoop101.yinzhengjie.com ~]# hdfs fsck -list-corruptfileblocks #若是你的集羣是健康的，就不會有損壞塊的消息提示。若是你的集羣是不健康的(好比有丟失的塊)，則可能以下圖所示。 Connecting to namenode via http://hadoop101.yinzhengjie.com:50070/fsck?ugi=root&listcorruptfileblocks=1&path=%2F The filesystem under path '/' has 0 CORRUPT files [root@hadoop101.yinzhengjie.com ~]#
三.處理HDFS損壞文件的案例
1>.列出缺乏的塊及其所屬的文件
[root@hadoop101.yinzhengjie.org.cn ~]# hdfs fsck -list-corruptfileblocks Connecting to namenode via http://hadoop101.yinzhengjie.org.cn:50070/fsck?ugi=root&listcorruptfileblocks=1&path=%2F The list of corrupt files under path '/' are: blk_1073742846 /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660 The filesystem under path '/' has 1 CORRUPT files [root@hadoop101.yinzhengjie.org.cn ~]#
2>.處理HDFS損壞文件的方法
有兩種方法處理損壞的文件，即move或者delete(hdfs dfs -rm)。 使用"-move"選項容許fsck命令將損壞的文件移動到"/lost+found"目錄。這比使用"hdfs dfs -rm"命令從HDFS文件系統刪除損壞的文件更好(須要注意的是，當使用rm命令從HDFS刪除損壞的文件時，損壞的文件顯示爲已損壞，若是配置了回收站功能，它將進入trash目錄)。
刪除損壞的文件更好的方法時使用fsck命令並指定"-delete"選項。
3>.嘗試修復已損壞的塊
以下圖所示，咱們能夠嘗試找到損壞塊在集羣中哪一個節點上，而後可使用塊號到NameNode上檢查其日誌以查看塊處於的位置。若是發現問題是因爲DataNode已關閉引發的，則一旦修復該問題，將再也不標記爲損壞塊。
但我下面的這種狀況下是沒法恢復的文件，所以建議使用"-delete"選項進行刪除該塊。
[root@hadoop101.yinzhengjie.org.cn ~]# hdfs fsck -list-corruptfileblocks Connecting to namenode via http://hadoop101.yinzhengjie.org.cn:50070/fsck?ugi=root&listcorruptfileblocks=1&path=%2F The list of corrupt files under path '/' are: blk_1073742846 /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660 The filesystem under path '/' has 1 CORRUPT files [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# hdfs fsck /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660 -blocks -locations Connecting to namenode via http://hadoop101.yinzhengjie.org.cn:50070/fsck?ugi=root&blocks=1&locations=1&path=%2Ftmp%2Flogs%2Froot%2Flogs%2Fapplication_15933 89957653_0002%2Fhadoop101.yinzhengjie.org.cn_11660FSCK started by root (auth:SIMPLE) from /172.200.4.101 for path /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660 at Fri Aug 21 03:27:50 CST 2020. /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660: CORRUPT blockpool BP-536854561-172.200.4.101-1589326197709 block blk_ 1073742846 /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660: MISSING 1 blocks of total size 6148 B.Status: CORRUPT Total size: 6148 B Total dirs: 0 Total files: 1 Total symlinks: 0 Total blocks (validated): 1 (avg. block size 6148 B) ******************************** UNDER MIN REPL'D BLOCKS: 1 (100.0 %) dfs.namenode.replication.min: 1 CORRUPT FILES: 1 MISSING BLOCKS: 1 MISSING SIZE: 6148 B CORRUPT BLOCKS: 1 ******************************** Minimally replicated blocks: 0 (0.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 0.0 Corrupt blocks: 1 Missing replicas: 0 Number of data-nodes: 6 Number of racks: 1 FSCK ended at Fri Aug 21 03:27:50 CST 2020 in 4 milliseconds The filesystem under path '/tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660' is CORRUPT [root@hadoop101.yinzhengjie.org.cn ~]#
4>.使用"hdfs fsck -move"命令將全部損壞的塊移動到/lost+found
[root@hadoop101.yinzhengjie.org.cn ~]# hdfs dfs -ls / #注意觀察夏安有目錄結構 Found 5 items drwxr-xr-x - root supergroup 0 2020-05-30 11:57 /hbase drwxr-xr-x - root supergroup 0 2020-05-29 18:55 /input_fruit drwx------ - root supergroup 0 2020-06-29 08:50 /tmp drwxr-xr-x - root supergroup 0 2020-06-29 08:22 /user drwxr-xr-x - root supergroup 0 2020-07-03 04:33 /yinzhengjie [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# hdfs fsck -move Connecting to namenode via http://hadoop101.yinzhengjie.org.cn:50070/fsck?ugi=root&move=1&path=%2F FSCK started by root (auth:SIMPLE) from /172.200.4.101 for path / at Fri Aug 21 03:43:00 CST 2020 .................................................................................................... .................................................................................................... ...................................................... /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742288_1480. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742289_1481. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742278_1470. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742282_14 74. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742279_1471. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742281_1473 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742269_1461 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737422 70_1462. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 273_1465. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742285_1 477. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742274_146 6. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742268 _1460. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742276_14 68. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2284_1476. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742277_1469 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742286_1478. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374227 2_1464. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2283_1475. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742280_1 472. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742271_1463. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742267_1459 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742275_146 7. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742287_1479. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742319_1511. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742320_1512. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742309_1501. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742313_15 05. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742310_1502. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742312_1504 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742300_1492 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737423 01_1493. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 304_1496. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742316_1 508. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742305_149 7. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742299 _1491. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742307_14 99. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2315_1507. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742308_1500 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742317_1509. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374230 3_1495. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2314_1506. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742311_1 503. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742302_1494. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742298_1490 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742306_149 8. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742318_1510. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742353_1545. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742354_1546. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742343_1535. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742347_15 39. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742344_1536. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742346_1538 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742334_1526 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737423 35_1527. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 338_1530. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742350_1 542. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742339_153 1. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742333 _1525. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742341_15 33. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2349_1541. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742342_1534 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742351_1543. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374233 7_1529. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2348_1540. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742345_1 537. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742336_1528. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742332_1524 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742340_153 2. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742352_1544. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742405_1606. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742406_1607. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742395_1596. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742399_16 00. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742396_1597. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742398_1599 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742386_1587 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737423 87_1588. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 390_1591. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742402_1 603. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742391_159 2. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742385 _1586. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742393_15 94. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2401_1602. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742394_1595 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742403_1604. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374238 9_1590. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2400_1601. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742397_1 598. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742388_1589. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742384_1585 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742392_159 3. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742404_1605. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742454_1663. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742455_1664. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742444_1653. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742448_16 57. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742445_1654. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742447_1656 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742435_1644 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737424 36_1645. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 439_1648. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742451_1 660. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742440_164 9. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742434 _1643. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742442_16 51. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2450_1659. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742443_1652 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742452_1661. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374243 8_1647. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2449_1658. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742446_1 655. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742437_1646. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742433_1642 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742441_165 0. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742453_1662. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)... .......... /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660: CORRUPT blockpool BP-536854561-172.200.4.101-1589326197709 block blk_1073742846 /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660: MISSING 1 blocks of total size 6148 B.......Status: CORRUPT Total size: 434686501 B Total dirs: 273 Total files: 416 Total symlinks: 0 Total blocks (validated): 367 (avg. block size 1184431 B) ******************************** UNDER MIN REPL'D BLOCKS: 1 (0.27247956 %) dfs.namenode.replication.min: 1 CORRUPT FILES: 1 MISSING BLOCKS: 1 MISSING SIZE: 6148 B CORRUPT BLOCKS: 1 ******************************** Minimally replicated blocks: 366 (99.72752 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 115 (31.33515 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.93188 Corrupt blocks: 1 Missing replicas: 460 (24.134314 %) Number of data-nodes: 6 Number of racks: 1 FSCK ended at Fri Aug 21 03:43:01 CST 2020 in 262 milliseconds The filesystem under path '/' is CORRUPT [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# hdfs dfs -ls / Found 6 items drwxr-xr-x - root supergroup 0 2020-05-30 11:57 /hbase drwxr-xr-x - root supergroup 0 2020-05-29 18:55 /input_fruit drw-r--r-- - root supergroup 0 2020-08-21 03:43 /lost+found drwx------ - root supergroup 0 2020-06-29 08:50 /tmp drwxr-xr-x - root supergroup 0 2020-06-29 08:22 /user drwxr-xr-x - root supergroup 0 2020-07-03 04:33 /yinzhengjie [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# hdfs dfs -ls /lost+found #不難發現執行"-move"命令後，會自動生成"/lost+found"目錄 Found 1 items drwxr----- - root supergroup 0 2020-08-21 03:43 /lost+found/tmp [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# hdfs dfs -ls -R /lost+found drwxr----- - root supergroup 0 2020-08-21 03:43 /lost+found/tmp drwxr----- - root supergroup 0 2020-08-21 03:43 /lost+found/tmp/logs drwxr----- - root supergroup 0 2020-08-21 03:43 /lost+found/tmp/logs/root drwxr----- - root supergroup 0 2020-08-21 03:43 /lost+found/tmp/logs/root/logs drwxr----- - root supergroup 0 2020-08-21 03:43 /lost+found/tmp/logs/root/logs/application_1593389957653_0002 drw-r----- - root supergroup 0 2020-08-21 03:43 /lost+found/tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660 [root@hadoop101.yinzhengjie.org.cn ~]#
5>.使用"hdfs fsck -delete"命令刪除損壞文件
[root@hadoop101.yinzhengjie.org.cn ~]# hdfs fsck -list-corruptfileblocks Connecting to namenode via http://hadoop101.yinzhengjie.org.cn:50070/fsck?ugi=root&listcorruptfileblocks=1&path=%2F The list of corrupt files under path '/' are: blk_1073742846 /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660 The filesystem under path '/' has 1 CORRUPT files [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# hdfs fsck -delete Connecting to namenode via http://hadoop101.yinzhengjie.org.cn:50070/fsck?ugi=root&delete=1&path=%2F FSCK started by root (auth:SIMPLE) from /172.200.4.101 for path / at Fri Aug 21 03:53:25 CST 2020 .................................................................................................... .................................................................................................... ...................................................... /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742288_1480. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742289_1481. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742278_1470. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742282_14 74. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742279_1471. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742281_1473 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742269_1461 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737422 70_1462. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 273_1465. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742285_1 477. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742274_146 6. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742268 _1460. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742276_14 68. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2284_1476. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742277_1469 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742286_1478. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374227 2_1464. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2283_1475. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742280_1 472. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742271_1463. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742267_1459 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742275_146 7. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0001/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742287_1479. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742319_1511. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742320_1512. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742309_1501. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742313_15 05. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742310_1502. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742312_1504 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742300_1492 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737423 01_1493. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 304_1496. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742316_1 508. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742305_149 7. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742299 _1491. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742307_14 99. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2315_1507. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742308_1500 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742317_1509. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374230 3_1495. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2314_1506. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742311_1 503. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742302_1494. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742298_1490 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742306_149 8. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0002/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742318_1510. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742353_1545. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742354_1546. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742343_1535. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742347_15 39. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742344_1536. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742346_1538 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742334_1526 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737423 35_1527. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 338_1530. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742350_1 542. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742339_153 1. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742333 _1525. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742341_15 33. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2349_1541. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742342_1534 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742351_1543. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374233 7_1529. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2348_1540. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742345_1 537. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742336_1528. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742332_1524 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742340_153 2. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590748317397_0003/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742352_1544. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742405_1606. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742406_1607. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742395_1596. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742399_16 00. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742396_1597. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742398_1599 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742386_1587 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737423 87_1588. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 390_1591. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742402_1 603. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742391_159 2. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742385 _1586. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742393_15 94. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2401_1602. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742394_1595 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742403_1604. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374238 9_1590. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2400_1601. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742397_1 598. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742388_1589. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742384_1585 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742392_159 3. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590749076365_0001/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742404_1605. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/job.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742454_1663. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/job.split: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742455_1664. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)........ /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/commons-lang3-3.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742444_1653. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hadoop-common-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742448_16 57. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hadoop-mapreduce-client-core-2.10.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:bl k_1073742445_1654. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-client-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742447_1656 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-common-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742435_1644 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-hadoop-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_10737424 36_1645. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-hadoop2-compat-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742 439_1648. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-mapreduce-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742451_1 660. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-metrics-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742440_164 9. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-metrics-api-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742434 _1643. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-protocol-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742442_16 51. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-protocol-shaded-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2450_1659. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-server-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742443_1652 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-shaded-miscellaneous-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1 073742452_1661. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-shaded-netty-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374243 8_1647. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-shaded-protobuf-2.2.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_107374 2449_1658. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/hbase-zookeeper-2.2.4.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742446_1 655. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/htrace-core4-4.1.0-incubating.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073 742437_1646. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/metrics-core-3.0.1.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742433_1642 . Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/protobuf-java-2.5.0.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742441_165 0. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).. /tmp/hadoop-yarn/staging/root/.staging/job_1590781316348_0001/libjars/zookeeper-3.4.9.jar: Under replicated BP-536854561-172.200.4.101-1589326197709:blk_1073742453_1662. T arget Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)... .......... /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660: CORRUPT blockpool BP-536854561-172.200.4.101-1589326197709 block blk_1073742846 /tmp/logs/root/logs/application_1593389957653_0002/hadoop101.yinzhengjie.org.cn_11660: MISSING 1 blocks of total size 6148 B.......Status: CORRUPT Total size: 434686501 B Total dirs: 280 Total files: 416 Total symlinks: 0 Total blocks (validated): 367 (avg. block size 1184431 B) ******************************** UNDER MIN REPL'D BLOCKS: 1 (0.27247956 %) dfs.namenode.replication.min: 1 CORRUPT FILES: 1 MISSING BLOCKS: 1 MISSING SIZE: 6148 B CORRUPT BLOCKS: 1 ******************************** Minimally replicated blocks: 366 (99.72752 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 115 (31.33515 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.93188 Corrupt blocks: 1 Missing replicas: 460 (24.134314 %) Number of data-nodes: 6 Number of racks: 1 FSCK ended at Fri Aug 21 03:53:25 CST 2020 in 53 milliseconds The filesystem under path '/' is CORRUPT [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# hdfs fsck -list-corruptfileblocks Connecting to namenode via http://hadoop101.yinzhengjie.org.cn:50070/fsck?ugi=root&listcorruptfileblocks=1&path=%2F The filesystem under path '/' has 0 CORRUPT files [root@hadoop101.yinzhengjie.org.cn ~]#
四.處理HDFS沒法恢復的文件
若是數據塊的全部副本丟失或損壞，則意味着沒有可讀的副本，即數據塊上的沒法被讀取。缺乏的數據塊沒有活副本。這多是因爲損壞或由於塊所在的DataNode離線引發的。
若是同時看到許多塊的文件系統損壞信息，則意味着幾個DataNode不能同時使用。
若是存儲的數據的複製因子爲1，那麼即便一個節點丟失或運行不正確，也會看到丟失的塊信息。
經過運行"fsck -delete"命令，可確保從HDFS元數據中永久刪除丟失的數據。若是不刪除它們，則可能發生一個進程卡在那裏試圖讀取丟失的文件的狀況。
就像咱們上面的案例，當咱們執行"fsck -move"命令時，對於已經損壞(沒法恢復)的文件是很差使的，這個時候必須使用"fsck -delete"來永久刪除損壞塊在NameNode所對應的元數據信息。
舒適提示:
要主動捕獲潛在的HDFS換壞信息，請按期運行hdfs fsck明朗了，就像cron做業運行HDFS平衡器命令同樣。
fsck檢查文件的開銷並不重要，可是在繁忙的集羣上，也能夠在非高峯時間調度做業，就像調度平衡器做業同樣。
五.處理HDFS副本不足的文件
Hadoop的在複製塊(Under-replicated blocks)以知足配置的文件複製因子方面作的很好。當DataNode崩潰時，該節點上的文件一般在短期內被複制。 Hadoop能夠感知丟失的DataNode，由於它們再也不從DataNode獲取心跳。在DataNode崩潰後的十多分鐘內開始複製。
相關文章
- 1. fsck檢查和修復文件系統
- 2. 使用Python腳本檢驗文件系統數據完整性
- 3. 檢查 Linux 文件系統中的錯誤：經過案例學習 FSCK 命令
- 4. 系統完整性檢查(轉)
- 5. 正確數據確保數據完整性
- 6. ORACLE數據完整性檢查
- 7. 數據庫完整性檢查
- 8. 確保事務完整性
- 9. 使用fsck命令檢查文件系統錯誤
- 10. 保證數據完整性
- 更多相關文章...
- • C# Windows 文件系統的操作 - C#教程
- • 查看 XML 文件 - XML 教程
- • Docker容器實戰(七) - 容器眼光下的文件系統
- • Flink 數據傳輸及反壓詳解
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. 在windows下的虛擬機中,安裝華爲電腦的deepin操作系統
- 2. 強烈推薦款下載不限速解析神器
- 3. 【區塊鏈技術】孫宇晨：區塊鏈技術帶來金融服務的信任變革
- 4. 搜索引起的鏈接分析-計算網頁的重要性
- 5. TiDB x 微衆銀行 | 耗時降低 58%，分佈式架構助力實現普惠金融
- 6. 《數字孿生體技術白皮書》重磅發佈（附完整版下載）
- 7. 雙十一「避坑」指南：區塊鏈電子合同爲電商交易保駕護航！
- 8. 區塊鏈產業，怎樣「鏈」住未來？
- 9. OpenglRipper使用教程
- 10. springcloud請求一次好用一次不好用zuul Name or service not known
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. fsck檢查和修復文件系統
- 2. 使用Python腳本檢驗文件系統數據完整性
- 3. 檢查 Linux 文件系統中的錯誤：經過案例學習 FSCK 命令
- 4. 系統完整性檢查(轉)
- 5. 正確數據確保數據完整性
- 6. ORACLE數據完整性檢查
- 7. 數據庫完整性檢查
- 8. 確保事務完整性
- 9. 使用fsck命令檢查文件系統錯誤
- 10. 保證數據完整性