Yarn配置kerberos認證

一、環境說明

系統環境:

• 操做系統：centos6.6
• Hadoop版本：CDH5.5
• JDK版本：1.7.0_67

集羣各節點Yarn的角色規劃爲：

172.16.57.74   bd-ops-test-74    ResourceManager  NodeManager
172.16.57.75   bd-ops-test-75    ResourceManager  NodeManager
172.16.57.76   bd-ops-test-76    NodeManager JobHistoryServer yarn-proxyserver
172.16.57.77   bd-ops-test-77    NodeManager

二、生成keytab

在 74 節點，即 KDC server 節點上執行下面命令：

cd /var/kerberos/krb5kdc/

kadmin.local -q "addprinc -randkey yarn/bd-ops-test-74@BIGDATA.COM "
kadmin.local -q "addprinc -randkey yarn/bd-ops-test-75@BIGDATA.COM "
kadmin.local -q "addprinc -randkey yarn/bd-ops-test-76@BIGDATA.COM "
kadmin.local -q "addprinc -randkey yarn/bd-ops-test-77@BIGDATA.COM "

kadmin.local -q "addprinc -randkey mapred/bd-ops-test-74@BIGDATA.COM "
kadmin.local -q "addprinc -randkey mapred/bd-ops-test-75@BIGDATA.COM "
kadmin.local -q "addprinc -randkey mapred/bd-ops-test-76@BIGDATA.COM "
kadmin.local -q "addprinc -randkey mapred/bd-ops-test-77@BIGDATA.COM "

kadmin.local -q "xst  -k yarn.keytab  yarn/bd-ops-test-74@BIGDATA.COM"
kadmin.local -q "xst  -k yarn.keytab  yarn/bd-ops-test-75@BIGDATA.COM "
kadmin.local -q "xst  -k yarn.keytab  yarn/bd-ops-test-76@BIGDATA.COM "
kadmin.local -q "xst  -k yarn.keytab  yarn/bd-ops-test-77@BIGDATA.COM "

kadmin.local -q "xst  -k mapred.keytab  mapred/bd-ops-test-74@BIGDATA.COM "
kadmin.local -q "xst  -k mapred.keytab  mapred/bd-ops-test-75@BIGDATA.COM "
kadmin.local -q "xst  -k mapred.keytab  mapred/bd-ops-test-76@BIGDATA.COM "
kadmin.local -q "xst  -k mapred.keytab  mapred/bd-ops-test-77@BIGDATA.COM "

拷貝 yarn.keytab 和 mapred.keytab 文件到其餘節點的 /etc/hadoop/conf 目錄

scp yarn.keytab mapred.keytab bd-ops-test-xx:/etc/hadoop/conf

並設置權限，在各節點上執行：

# cd /etc/hadoop/conf/;chown yarn:hadoop yarn.keytab;chown mapred:hadoop mapred.keytab ;chmod 400 *.keytab

因爲 keytab 至關於有了永久憑證，不須要提供密碼(若是修改 kdc 中的 principal 的密碼，則該 keytab 就會失效)，因此其餘用戶若是對該文件有讀權限，就能夠冒充 keytab 中指定的用戶身份訪問 hadoop，因此 keytab 文件須要確保只對 owner 有讀權限(0400)

3. 修改 YARN 配置文件

修改 yarn-site.xml，添加下面配置：

<property>
  <name>yarn.resourcemanager.keytab</name>
  <value>/etc/hadoop/conf/yarn.keytab</value>
</property>
<property>
  <name>yarn.resourcemanager.principal</name> 
  <value>yarn/_HOST@BIGDATA.COM</value>
</property>

<property>
  <name>yarn.nodemanager.keytab</name>
  <value>/etc/hadoop/conf/yarn.keytab</value>
</property>
<property>
  <name>yarn.nodemanager.principal</name> 
  <value>yarn/_HOST@BIGDATA.COM</value>
</property> 
<property>
  <name>yarn.nodemanager.container-executor.class</name>  
  <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property> 
<property>
  <name>yarn.nodemanager.linux-container-executor.group</name>
  <value>yarn</value>
</property>

<property>
   <name>yarn.web-proxy.keytab</name>
   <value>/etc/hadoop/conf/yarn.keytab</value>
</property>

<property>
    <name>yarn.web-proxy.principal</name>
    <value>yarn/_HOST@BIGDATA.COM</value>
</property>

修改 mapred-site.xml，添加以下配置：

<property>
  <name>mapreduce.jobhistory.keytab</name>
  <value>/etc/hadoop/conf/mapred.keytab</value>
</property> 
<property>
  <name>mapreduce.jobhistory.principal</name> 
  <value>mapred/_HOST@BIGDATA.COM</value>
</property>

在 /etc/hadoop/conf 目錄下建立 container-executor.cfg 文件，內容以下：

#configured value of yarn.nodemanager.linux-container-executor.group
yarn.nodemanager.linux-container-executor.group=yarn
#comma separated list of users who can not run applications
banned.users=bin
#Prevent other super-users
min.user.id=0
#comma separated list of system users who CAN run applications
allowed.system.users=root,nobody,impala,hive,hdfs,yarn

設置該文件權限：

# chown root:yarn container-executor.cfg
# chmod 400 container-executor.cfg

# ll container-executor.cfg
-r-------- 1 root yarn 354 11-05 14:14 container-executor.cfg

注意：

• container-executor.cfg 文件讀寫權限需設置爲 400，全部者爲 root:yarn。
• yarn.nodemanager.linux-container-executor.group 要同時配置在 yarn-site.xml 和 container-executor.cfg，且其值須要爲運行 NodeManager 的用戶所在的組，這裏爲 yarn。
• banned.users 不能爲空，默認值爲 hfds,yarn,mapred,bin
• min.user.id 默認值爲 1000，在有些 centos 系統中，用戶最小 id 爲500，則須要修改該值
• 確保 yarn.nodemanager.local-dirs 和 yarn.nodemanager.log-dirs 對應的目錄權限爲 755 。

設置 /usr/lib/hadoop-yarn/bin/container-executor 讀寫權限爲 6050 以下：

# chown root:yarn /usr/lib/hadoop-yarn/bin/container-executor
# chmod 6050 /usr/lib/hadoop-yarn/bin/container-executor

# ll /usr/lib/hadoop-yarn/bin/container-executor
---Sr-s--- 1 root yarn 333 11-04 19:11 container-executor

測試是否配置正確：

# /usr/lib/hadoop-yarn/bin/container-executor --checksetup

若是提示錯誤，則查看 NodeManger 的日誌，而後對照 YARN ONLY: Container-executor Error Codes 查看錯誤對應的問題說明。

關於 LinuxContainerExecutor 的詳細說明，能夠參考 http://hadoop.apache.org/docs/r2.5.0/hadoop-project-dist/hadoop-common/SecureMode.html#LinuxContainerExecutor。

記住將修改的上面文件同步到其餘節點，並再次一一檢查權限是否正確。

# cd /etc/hadoop/conf/

# scp yarn-site.xml mapred-site.xml container-executor.cfg  bd-ops-test-xx:/etc/hadoop/conf/

# cd /etc/hadoop/conf/; chown root:yarn container-executor.cfg ; chmod 400 container-executor.cfg
# cd /etc/hadoop/conf/; chown root:yarn container-executor.cfg ; chmod 400 container-executor.cfg

四、啓動服務

啓動 ResourceManager

resourcemanager 是經過 yarn 用戶啓動的，故在 74,75 上先獲取 yarn 用戶的 ticket 再啓動服務：

$ kinit -k -t /etc/hadoop/conf/yarn.keytab yarn/bd-ops-test-xx@BIGDATA.COM
# service hadoop-yarn-resourcemanager start

而後查看日誌，確認是否啓動成功。

2016-09-05 13:40:11,190 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=yarn	OPERATION=transitionToActive	TARGET=RMHAProtocolService	RESULT=SUCCESS

啓動 NodeManager

resourcemanager 是經過 yarn 用戶啓動的，故在各節點上先獲取 yarn 用戶的 ticket 再啓動服務：

kinit -k -t /etc/hadoop/conf/yarn.keytab yarn/bd-ops-test-xx@BIGDATA.COM ;service hadoop-yarn-nodemanager start

觀察日誌成功信息：

2016-09-05 13:50:37,869 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user yarn/bd-ops-test-76@BIGDATA.COM using keytab file /etc/hadoop/conf/yarn.keytab

啓動 MapReduce Job History Server

resourcemanager 是經過 mapred 用戶啓動的，故在 76節點 上先獲取 mapred 用戶的 ticket 再啓動服務：

$ kinit -k -t /etc/hadoop/conf/mapred.keytab mapred/bd-ops-test-75@BIGDATA.COM
# service hadoop-mapreduce-historyserver start

觀察日誌成功信息：

16/09/05 13:55:49 INFO security.UserGroupInformation: Login successful for user mapred/bd-ops-test-76@BIGDATA.COM using keytab file /etc/hadoop/conf/mapred.keytab

啓動 yarn-proxyserver

# service hadoop-yarn-proxyserver start

觀察日誌成功信息：

2016-09-05 16:28:24,569 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user yarn/bd-ops-test-76@BIGDATA.COM using keytab file /etc/hadoop/conf/yarn.keytab

5. 測試

運行一個 mapreduce 的例子：

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 10000

若是沒有報錯，則說明配置成功。最後運行的結果爲：

Job Finished in 45.007 seconds
Estimated value of Pi is 3.14120000000000000000