管理Hadoop的配額
管理Hadoop的配額ide
做者：尹正傑oop
版權聲明：原創做品，謝絕轉載！不然將追究法律責任。ui
一.Hadoop的配額概述spa
能夠在HDFS目錄上配置配額，由此能夠限制用戶或應用程序消耗的HDFS空間。
HDFS的空間分配與底層Linux文件系統上的空間分配沒有直接關係。
Hadoop容許設置兩種類型的配額，即空間配額和名稱配額。
名稱配額:
指定根目錄樹中的文件和目錄的最大數量。
空間配額:
爲單個目錄使用的空間設置上限。
舒適提示:
若是建立了用戶的主(家)目錄但未向用戶授予名稱配額或空間配額，則用戶在HDFS中具備無限存儲空間，這是很很差的操做。
名稱配額和空間配額不是特定於用戶的，而是特定於目錄的。
二.管理名稱配額3d
能夠經過指定名稱配額來限制任何目錄中的文件數和目錄數。若是用戶嘗試建立超出指定配額的文件或目錄，則文件或目錄將建立失敗。 咱們能夠經過下面的命令檢查配額信息(此時咱們還爲給HDFS配置任何配額)： [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -q -v -h /user/root #使用"-q"選項能夠查看到空間配額和名稱配額相關信息喲~ QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf none inf 15 18 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]# 相關術語解釋以下: QUOTA: 名稱配額相關信息，即文件和目錄的限制。 REM_QUOTA: 此用戶能夠建立的配額中剩餘文件和目錄數。 SPACE_QUOTA: 授予此用戶的空間配額。 REM_SPACE_QUOTA: 此用戶剩餘空間配額。 DIR_COUNT: 目錄數。 FILE_COUNT: 文件數。 CONTENT_SIZE: 文件大小 PATHNAME: 路徑名稱。
1>.爲HDFS設置名稱配額code
[root@hadoop101.yinzhengjie.com ~]# hdfs dfsadmin -help setQuota #使用"dfsadmin -setQuota"命令設置目錄的HDFS名稱配額 -setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each directory <dirName>. The directory quota is a long integer that puts a hard limit on the number of names in the directory tree For each directory, attempt to set the quota. An error will be reported if 1. quota is not a positive integer, or 2. User is not an administrator, or 3. The directory does not exist or is a file. Note: A quota of 1 would force the directory to remain empty. [root@hadoop101.yinzhengjie.com ~]#
[root@hadoop101.yinzhengjie.com ~]# hdfs dfs -ls -R /user/root drwx------ - root admingroup 0 2020-08-15 08:00 /user/root/.Trash drwx------ - root admingroup 0 2020-08-14 19:32 /user/root/.Trash/200814193733 -rw-r--r-- 3 root admingroup 490 2020-08-14 19:31 /user/root/.Trash/200814193733/fstab -rw-r--r-- 3 root admingroup 10779 2020-08-14 19:32 /user/root/.Trash/200814193733/sysctl.conf drwxr-xr-x - root admingroup 0 2020-08-14 19:04 /user/root/.Trash/200814193733/test2 drwxr-xr-x - root admingroup 0 2020-08-14 19:04 /user/root/.Trash/200814193733/test2/sub1 drwxr-xr-x - root admingroup 0 2020-08-14 19:04 /user/root/.Trash/200814193733/test2/sub1/sub2 drwx------ - root admingroup 0 2020-08-14 19:21 /user/root/.Trash/200814193733/yinzhengjie drwx------ - root admingroup 0 2020-08-15 00:04 /user/root/.Trash/200815080000 -rw-r--r-- 3 root admingroup 0 2020-08-14 22:47 /user/root/.Trash/200815080000/a.txt -rw-r--r-- 3 root admingroup 392115733 2020-08-14 23:25 /user/root/.Trash/200815080000/hadoop-2.10.0.tar.gz -rw-r--r-- 3 root admingroup 0 2020-08-14 22:58 /user/root/.Trash/200815080000/hdfs2020.log -rw-r--r-- 3 root admingroup 26 2020-08-14 23:42 /user/root/.Trash/200815080000/hostname -rw-r--r-- 3 root admingroup 371 2020-08-14 23:49 /user/root/.Trash/200815080000/hosts2020 -rw-r--r-- 3 root admingroup 69 2020-08-14 23:14 /user/root/.Trash/200815080000/wc.txt.gz drwx-w-r-x - jason admingroup 0 2020-08-14 21:46 /user/root/.Trash/200815080000/yinzhengjie drwx-w-r-x - jason admingroup 0 2020-08-14 07:07 /user/root/.Trash/200815080000/yinzhengjie/data drwx-w-r-x - jason admingroup 0 2020-08-14 07:07 /user/root/.Trash/200815080000/yinzhengjie/data/hadoop drwx-w-r-x - jason admingroup 0 2020-08-14 07:07 /user/root/.Trash/200815080000/yinzhengjie/data/hadoop/hdfs drwx-w-r-x - jason admingroup 0 2020-08-14 21:46 /user/root/.Trash/200815080000/yinzhengjie/softwares drwxr-xr-x - root admingroup 0 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020 -rw-r--r-- 3 root admingroup 69 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/wc.txt.gz drwxr-xr-x - root admingroup 0 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d -rw-r--r-- 3 root admingroup 1664 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d/CentOS-Base.repo -rw-r--r-- 3 root admingroup 1309 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d/CentOS-CR.repo -rw-r--r-- 3 root admingroup 649 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d/CentOS-Debuginfo.repo -rw-r--r-- 3 root admingroup 630 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d/CentOS-Media.repo -rw-r--r-- 3 root admingroup 1331 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d/CentOS-Sources.repo -rw-r--r-- 3 root admingroup 5701 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d/CentOS-Vault.repo -rw-r--r-- 3 root admingroup 314 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d/CentOS-fasttrack.repo -rw-r--r-- 3 root admingroup 1050 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d/epel-testing.repo -rw-r--r-- 3 root admingroup 951 2020-08-14 23:48 /user/root/.Trash/200815080000/yinzhengjie2020/yum.repos.d/epel.repo [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -ls -R /user/root | wc -l #"/user/root"下有32個文件，包含"/user/root"目錄共計33個目錄和文件 32 [root@hadoop101.yinzhengjie.com ~]#
[root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -q -v -h /user/root QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf none inf 15 18 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfsadmin -setQuota 35 /user/root #咱們爲"/user/root"目錄設置名稱配額大小爲35，注意觀察"count"命令的統計信息喲~ [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -q -v -h /user/root QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME 35 2 none inf 15 18 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]#
2>.驗證名稱配額是否生效blog
[root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -q -v -h /user/root QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME 35 2 none inf 15 18 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -ls /user/root #接下來咱們在已配置的名稱配額的目錄下建立文件和目錄進行驗證是否當即生效。 Found 1 items drwx------ - root admingroup 0 2020-08-15 08:00 /user/root/.Trash [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -touchz /user/root/a.txt [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -mkdir /user/root/test [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -q -v -h /user/root QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME 35 0 none inf 16 19 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -touchz /user/root/b.txt touchz: The NameSpace quota (directories and files) of directory /user/root is exceeded: quota=35 file count=36 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -mkdir /user/root/test02 mkdir: The NameSpace quota (directories and files) of directory /user/root is exceeded: quota=35 file count=36 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -ls /user/root Found 3 items drwx------ - root admingroup 0 2020-08-15 08:00 /user/root/.Trash -rw-r--r-- 3 root admingroup 0 2020-08-19 18:32 /user/root/a.txt drwxr-xr-x - root admingroup 0 2020-08-19 18:32 /user/root/test [root@hadoop101.yinzhengjie.com ~]#
3>.清除當前名稱配額 hadoop
[root@hadoop101.yinzhengjie.com ~]# hdfs dfsadmin -help clrQuota #使用"dfsadmin clrQuota"命令能夠清除當前的名稱配額 -clrQuota <dirname>...<dirname>: Clear the quota for each directory <dirName>. For each directory, attempt to clear the quota. An error will be reported if 1. the directory does not exist or is a file, or 2. user is not an administrator. It does not fault if the directory has no quota. [root@hadoop101.yinzhengjie.com ~]#
[root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -q -v -h /user/root QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME 35 0 none inf 16 19 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfsadmin -clrQuota /user/root #使用該命令成功清除名稱配額 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -q -v -h /user/root QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf none inf 16 19 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]#
三.管理空間配額ci
能夠對分配給HDFS下特定目錄的存儲設置限制，此配額是目錄中全部文件可使用的字節數。一旦目錄用完其分配的空間配額，用戶和應用程序將沒法在目錄中建立文件。
空間配額對HDFS目錄樹中的全部文件可使用的磁盤空間設置硬性限制。能夠經過設置用戶的主目錄或用戶與其它用戶共享的其它目錄來限制用戶的空間消耗。若是不在目錄上設置空間配額，則意味着該目錄的磁盤空間配額不受限制，它可使用整個HDFS。
在配置空間配額時，重要的是要理解，在HDFS中，必須有足夠的空間配額來容納整個塊。若是用戶在分配的配額中有200MB的空閒空間，先不論副本因子等因素，無論你要存儲的文件大小如何，若是HDFS塊大小大於200MB(如256MB)，則沒法建立新文件。
舒適提示:
空間配額包括全部複製的數據。若是用戶設置了30GB的配額，則該用戶能夠經過在其HDFS目錄中存儲10GB的實際數據(使用默認複製因子3，HDFS存儲10GB x 3 = 30GB的數據)來消耗配額。
1>.爲HDFS設置空間配額rem
[root@hadoop101.yinzhengjie.com ~]# hdfs dfsadmin -help setSpaceQuota -setSpaceQuota <quota> [-storageType <storagetype>] <dirname>...<dirname>: Set the space quota <quota> for each directory <dirName>. The space quota is a long integer that puts a hard limit on the total size of all the files under the directory tree. The extra space required for replication is also counted. E.g. a 1GB file with replication of 3 consumes 3GB of the quota. Quota can also be specified with a binary prefix for terabytes, petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB). For each directory, attempt to set the quota. An error will be reported if 1. quota is not a positive integer or zero, or 2. user is not an administrator, or 3. the directory does not exist or is a file. The storage type specific quota is set when -storageType option is specified. Available storageTypes are - RAM_DISK - DISK - SSD - ARCHIVE [root@hadoop101.yinzhengjie.com ~]#
[root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -h -v -q /user/root #注意觀察空間配額信息 QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf none inf 16 19 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfsadmin -setSpaceQuota 2g /user/root #此處我僅爲"/user/root"目錄設置2G的空間配額 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -h -v -q /user/root QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf 2 G 926.1 M 16 19 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]#
舒適提示:
(1)爲目錄設置空間配額其包括副本因子的容量;
(2)咱們能夠同時爲多個目錄設置空間配額;
2>.驗證空間配額是否生效
[root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -h -v -q /user/root #觀察剩餘空間配額的容量爲926.1MB QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf 2 G 926.1 M 16 19 374.0 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# ll -h total 375M -rw-r--r-- 1 root root 374M Aug 10 15:42 hadoop-2.10.0.tar.gz -rw------- 1 root root 265K Aug 20 16:49 messages [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -put messages /user/root #請思考爲何上傳一個265K的文件會拋出異常說配額不足？ put: The DiskSpace quota of /user/root is exceeded: quota = 2147483648 B = 2 GB but diskspace consumed = 2787036144 B = 2.60 GB [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -D dfs.blocksize=32m -put messages /user/root #爲何如今有能夠成功上傳啦？ [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -h -v -q /user/root QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf 2 G 925.3 M 16 20 374.2 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -D dfs.blocksize=128m -put hadoop-2.10.0.tar.gz /user/root #爲何這裏指定了塊大小依舊上傳失敗呢？ put: The DiskSpace quota of /user/root is exceeded: quota = 2147483648 B = 2 GB but diskspace consumed = 2385196977 B = 2.22 GB [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -D dfs.blocksize=128m -D dfs.replication=1 -put hadoop-2.10.0.tar.gz /user/root #爲何這樣配置又上傳成功啦？ [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -h -v -q /user/root #請思考爲何剩餘空間配額還有551.3MB呢？ QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf 2 G 551.3 M 16 21 748.2 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -ls -h /user/root Found 5 items drwx------ - root admingroup 0 2020-08-15 08:00 /user/root/.Trash -rw-r--r-- 3 root admingroup 0 2020-08-19 18:32 /user/root/a.txt -rw-r--r-- 1 root admingroup 374.0 M 2020-08-20 17:13 /user/root/hadoop-2.10.0.tar.gz -rw-r--r-- 3 root admingroup 265.0 K 2020-08-20 17:12 /user/root/messages drwxr-xr-x - root admingroup 0 2020-08-19 18:32 /user/root/test [root@hadoop101.yinzhengjie.com ~]#
3>.清除當前空間配額
[root@hadoop101.yinzhengjie.com ~]# hdfs dfsadmin -help clrSpaceQuota -clrSpaceQuota [-storageType <storagetype>] <dirname>...<dirname>: Clear the space quota for each directory <dirName>. For each directory, attempt to clear the quota. An error will be reported if 1. the directory does not exist or is a file, or 2. user is not an administrator. It does not fault if the directory has no quota. The storage type specific quota is cleared when -storageType option is specified. Available storageTypes are - RAM_DISK - DISK - SSD - ARCHIVE [root@hadoop101.yinzhengjie.com ~]#
[root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -h -v -q /user/root #注意觀察空間配額信息 QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf 2 G 551.3 M 16 21 748.2 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfsadmin -clrSpaceQuota /user/root #清除空間配額 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# hdfs dfs -count -h -v -q /user/root QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf none inf 16 21 748.2 M /user/root [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]#
相關文章
- 1. Kubernetes1.3：Quota配額管理
- 2. quota 磁盤配額管理
- 3. redhat5.3 磁盤配額管理
- 4. Linux--磁盤配額管理
- 5. Linux中的磁盤管理和配額
- 6. K8S-磁盤配額管理-整理
- 7. Linux系統的磁盤管理（分區管理，配額）
- 8. Kafka多租戶（配額）管理
- 9. linux中磁盤配額管理
- 10. LVM與磁盤配額管理
- 更多相關文章...
- • Swarm 集羣管理 - Docker教程
- • Maven 依賴管理 - Maven教程
- • Docker 清理命令
- • IntelliJ IDEA 代碼格式化配置和快捷鍵
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. Kubernetes1.3：Quota配額管理
- 2. quota 磁盤配額管理
- 3. redhat5.3 磁盤配額管理
- 4. Linux--磁盤配額管理
- 5. Linux中的磁盤管理和配額
- 6. K8S-磁盤配額管理-整理
- 7. Linux系統的磁盤管理（分區管理，配額）
- 8. Kafka多租戶（配額）管理
- 9. linux中磁盤配額管理
- 10. LVM與磁盤配額管理