指定用於在DataNode間傳輸block數據的最大線程數node
dfs.datanode.max.transfer.threads = 4096apache
(裝有HBASE時可適當提高此值,老版本爲dfs.datanode.max.xcievers)bash
指定DataNode用於balancer的帶寬,默認是1048576(1MB)單位爲Byte,一般可設置爲10MB;若機器性能優異能夠設定爲30-50MB;若是機器的網卡和交換機的帶寬有限,能夠設爲1-10MBide
dfs.datanode.balance.bandwidthPerSec = 31457280oop
指定DataNode上同時用於balance待移動block的最大線程個數,這個值默認是5性能
dfs.datanode.balance.max.concurrent.moves = 50優化
1) 從namenode獲取datanode磁盤的使用狀況
2) 計算須要把哪些數據移動到哪些節點
3) 分別移動,完成後刪除舊的block信息
4) 循環執行,直到達到平衡標準this
#臨時設置帶寬(10MB) sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth 10240000 #各節點存儲的浮動比例10%上下浮動 sudo -u hdfs hdfs balancer -threshold 10
1)出現報錯日誌spa
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby.
此報錯信息一般出如今配置了HA的HDFS中,balancer執行時不知道哪一個namenode爲active哪一個爲standby,當鏈接到standby的namenode時會出現如下報錯。無須擔憂只要集羣中爲一個active狀態的namenode,進程會嘗試鏈接active狀態的namenode,程序可以正常執行。線程
2)程序繼續正常執行
INFO - Jul 28, 2017 3:32:04 PM 32 23.39 GB 22.80 GB 30 GB 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.27:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.22:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.23:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.25:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.34:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.29:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.39:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.32:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.33:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.28:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.26:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.37:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.24:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.35:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.38:50010 17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.36:50010 17/07/28 15:32:13 INFO balancer.Balancer: 1 over-utilized: [192.168.111.37:50010:DISK] 17/07/28 15:32:13 INFO balancer.Balancer: 3 underutilized: [192.168.111.25:50010:DISK, 192.168.111.35:50010:DISK, 192.168.111.36:50010:DISK] 17/07/28 15:32:13 INFO balancer.Balancer: Need to move 22.58 GB to make the cluster balanced. 17/07/28 15:32:13 INFO balancer.Balancer: Decided to move 10 GB bytes from 192.168.111.37:50010:DISK to 192.168.111.35:50010:DISK 17/07/28 15:32:13 INFO balancer.Balancer: Decided to move 10 GB bytes from 192.168.111.27:50010:DISK to 192.168.111.25:50010:DISK 17/07/28 15:32:13 INFO balancer.Balancer: Decided to move 7.83 GB bytes from 192.168.111.39:50010:DISK to 192.168.111.36:50010:DISK 17/07/28 15:32:13 INFO balancer.Balancer: Decided to move 2.17 GB bytes from 192.168.111.32:50010:DISK to 192.168.111.36:50010:DISK 17/07/28 15:32:13 INFO balancer.Balancer: Will move 30 GB in this iteration 17/07/28 15:32:13 INFO balancer.Dispatcher: Successfully moved blk_1079961644_6305194 with size=41 from 192.168.111.39:50010:DISK to 192.168.111.36:50010:DISK through 192.168.111.39:50010 17/07/28 15:32:13 INFO balancer.Dispatcher: Successfully moved blk_1079963221_6306771 with size=41 from 192.168.111.39:50010:DISK to 192.168.111.36:50010:DISK through 192.168.111.39:50010 17/07/28 15:32:13 INFO balancer.Dispatcher: Successfully moved blk_1075009112_1268291 with size=99980 from 192.168.111.27:50010:DISK to 192.168.111.25:50010:DISK through 192.168.111.27:50010 17/07/28 15:32:13 INFO balancer.Dispatcher: Successfully moved blk_1079960720_6304270 with size=5234467 from 192.168.111.39:50010:DISK to 192.168.111.36:50010:DISK through 192.168.111.39:50010