配置兩個不一樣kerberos認證中心的集羣間的互信

兩個Hadoop集羣開啓Kerberos驗證後，集羣間不可以相互訪問，須要實現Kerberos之間的互信，使用Hadoop集羣A的客戶端訪問Hadoop集羣B的服務（實質上是使用Kerberos Realm A上的Ticket實現訪問Realm B的服務）。
先決條件：
1）兩個集羣(XDF.COM和HADOOP.COM)均開啓Kerberos認證
2）Kerberos的REALM分別設置爲XDF.COM和HADOOP.COM
步驟以下：

1 配置KDC之間的信任ticket

實現DXDF.COM和HADOOP.COM之間的跨域互信，例如使用XDF.COM的客戶端訪問HADOOP.COM中的服務，兩個REALM須要共同擁有名爲krbtgt/HADOOP.COM@XDF.COM的principal，兩個Keys須要保證密碼，version number和加密方式一致。默認狀況下互信是單向的， HADOOP.COM的客戶端訪問XDF.COM的服務，兩個REALM須要有krbtgt/XDF.COM@HADOOP.COM的principal。
向兩個集羣中添加krbtgt principal

#XDF CLUSTER
  kadmin.local: addprinc –e 「aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal 」 krbtgt/HADOOP.COM@XDF.COM
  kadmin.local: addprinc –e 「aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal 」  krbtgt/XDF.COM@HADOOP.COM

  #HADOOP CLUSTER
   kadmin.local: addprinc –e 「aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal 」 krbtgt/HADOOP.COM@XDF.COM
   kadmin.local: addprinc –e 「aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal 」  krbtgt/XDF.COM@HADOOP.COM

要驗證兩個entries具備匹配的kvno和加密type，查看命令使用getprinc <principal_name>

kadmin.local:  getprinc  krbtgt/XDF.COM@HADOOP.COM
Principal: krbtgt/XDF.COM@HADOOP.COM
Expiration date: [never]
Last password change: Wed Jul 05 14:18:11 CST 2017
Password expiration date: [none]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 30 days 00:00:00
Last modified: Wed Jul 05 14:18:11 CST 2017 (admin/admin@XDF.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 7
Key: vno 1, aes128-cts-hmac-sha1-96
Key: vno 1, des3-cbc-sha1
Key: vno 1, arcfour-hmac
Key: vno 1, camellia256-cts-cmac
Key: vno 1, camellia128-cts-cmac
Key: vno 1, des-hmac-sha1
Key: vno 1, des-cbc-md5
MKey: vno 1
Attributes:
Policy: [none]
kadmin.local:  getprinc  addprinc krbtgt/HADOOP.COM@XDF.COM
usage: get_principal [-terse] principal
kadmin.local:  getprinc  krbtgt/HADOOP.COM@XDF.COM
Principal: krbtgt/HADOOP.COM@XDF.COM
Expiration date: [never]
Last password change: Wed Jul 05 14:17:47 CST 2017
Password expiration date: [none]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 30 days 00:00:00
Last modified: Wed Jul 05 14:17:47 CST 2017 (admin/admin@XDF.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 7
Key: vno 1, aes128-cts-hmac-sha1-96
Key: vno 1, des3-cbc-sha1
Key: vno 1, arcfour-hmac
Key: vno 1, camellia256-cts-cmac
Key: vno 1, camellia128-cts-cmac
Key: vno 1, des-hmac-sha1
Key: vno 1, des-cbc-md5
MKey: vno 1
Attributes:
Policy: [none]

2 在core-site中配置principal和user的映射RULES

 

 

Paste_Image.png

設置hadoop.security.auth_to_local參數，該參數用於將principal轉變爲user，一個須要注意的問題是SASL RPC客戶端須要遠程Server的Kerberos principal在自己的配置中匹配該principal。相同的pricipal name須要分配給源和目標cluster的服務，例如Source Cluster中的NameNode的kerbeors principal name爲nn/h@XDF.COM，在Destination cluster中NameNode的pricipal設置爲nn/h@HADOOP.COM（不能設置爲nn2/h***@HADOOP.COM），例如：
在XDF Cluster和 HADOOP Cluster的core-site中增長：

 

<property>
<name>hadoop.security.auth_to_local</name>
<value>
RULE:[1:$1@$0](^.*@HADOOP\.COM$)s/^(.*)@HADOOP\.COM$/$1/g
RULE:[2:$1@$0](^.*@HADOOP\.COM$)s/^(.*)@HADOOP\.COM$/$1/g
RULE:[1:$1@$0](^.*@XDF\.COM$)s/^(.*)@XDF\.COM$/$1/g
RULE:[2:$1@$0](^.*@XDF\.COM$)s/^(.*)@XDF\.COM$/$1/g 
DEFAULT             
</value>
</property>

使用hadoop org.apache.hadoop.security.HadoopKerberosName <principal-name>來實現驗證，例如:

[root@node1a141 ~]#  hadoop org.apache.hadoop.security.HadoopKerberosName hdfs/nodea1a141@XDF.COM
Name: hdfs/nodea1a141@XDF.COM to hdfs

3 在krb5.conf中配置信任關係

3.1 配置capaths

第一種方式是配置shared hierarchy of names，這個是默認及比較簡單的方式，第二種方式是在krb5.conf文件中改變capaths，複雜可是比較靈活，這裏採用第二種方式。
在兩個集羣的節點的/etc/krb5.conf文件配置domain和realm的映射關係，例如：在XDF cluster中配置：

[capaths]
       XDF.COM = {
              HADOOP.COM = .
       }

在HADOOP Cluster中配置：

[capaths]
       HADOOP.COM = {
              XDF.COM = .
       }

配置成'.'是表示沒有intermediate realms

3.2 配置realms

爲了是XDF 能夠訪問HADOOP的KDC，須要將HADOOP的KDC Server配置到XDF cluster中，以下，反之相同：

[realms]
  XDF.COM = {
    kdc = {host}.XDF.COM:88
    admin_server = {host}.XDF.COM:749
    default_domain = XDF.COM
  }
  HADOOP.COM = {
    kdc = {host}.HADOOP.COM:88
    admin_server = {host}.HADOOP.COM:749
    default_domain = HADOOP.COM
  }

3.3 配置domain_realm

在domain_realm中，通常配置成'.XDF.COM'和'XDF.COM'的格式，'.'前綴保證kerberos將全部的XDF.COM的主機均映射到XDF.COM realm。可是若是集羣中的主機名不是以XDF.COM爲後綴的格式，那麼須要在domain_realm中配置主機與realm的映射關係，例XDF.nn.local映射爲XDF.COM，須要增長XDF.nn.local = XDF.COM。

[domain_realm]
.hadoop.com=HADOOP.COM
 hadoop.com=HADOOP.COM
 .xdf.com=XDF.COM
 xdf.com=XDF.COM
 node1a141 = XDF.COM
 node1a143 = XDF.COM
 node1a210 = HADOOP.COM
 node1a202 = HADOOP.COM
 node1a203 = HADOOP.COM

重啓kerberos服務

3.4 配置hdfs-site.xml

在hdfs-site.xml，設置容許的realms
在hdfs-site.xml中設置dfs.namenode.kerberos.principal.pattern爲"*"

 

Paste_Image.png

 

這個是客戶端的匹配規則用於控制容許的認證realms，若是該參數不配置，會有下面的異常：

java.io.IOException: Failed on local exception: java.io.IOException:
java.lang.IllegalArgumentException:
       Server has invalid Kerberosprincipal:nn/ HADOOP.COM@ XDF.COM;
       Host Details : local host is: "host1.XDF.COM/10.181.22.130";
                        destination host is: "host2.HADOOP.COM":8020;

4 測試

1)使用hdfs命令測試XDF 和HADOOP 集羣間的數據訪問，例如在XDF Cluster中kinit admin@XDF.CON，而後運行hdfs命令：

[root@node1a141 ~]# kdestroy
[root@node1a141 ~]# kinit admin
Password for admin@XDF.COM: 
[root@node1a141 ~]# hdfs dfs -ls /
Found 3 items
drwxrwxrwx+  - hdfs supergroup          0 2017-06-13 15:13 /tmp
drwxrwxr-x+  - hdfs supergroup          0 2017-06-22 15:55 /user
drwxrwxr-x+  - hdfs supergroup          0 2017-06-14 14:11 /wa
[root@node1a141 ~]# hdfs dfs -ls hdfs://node1a202:8020/
Found 9 items
drwxr-xr-x   - root  supergroup          0 2017-05-27 18:55 hdfs://node1a202:8020/cdtest
drwx------   - hbase hbase               0 2017-05-22 18:51 hdfs://node1a202:8020/hbase
drwx------   - hbase hbase               0 2017-07-05 19:16 hdfs://node1a202:8020/hbase1
drwxr-xr-x   - hbase hbase               0 2017-05-11 10:46 hdfs://node1a202:8020/hbase2
drwxr-xr-x   - root  supergroup          0 2016-12-01 17:30 hdfs://node1a202:8020/home
drwxr-xr-x   - mdss  supergroup          0 2016-12-13 18:30 hdfs://node1a202:8020/idfs
drwxr-xr-x   - hdfs  supergroup          0 2017-05-22 18:51 hdfs://node1a202:8020/system
drwxrwxrwt   - hdfs  supergroup          0 2017-05-31 17:37 hdfs://node1a202:8020/tmp
drwxrwxr-x+  - hdfs  supergroup          0 2017-05-04 15:48 hdfs://node1a202:8020/user

在HADOOP.COM中進行相同的操做
2）運行distcp程序將XDF的數據複製到HADOOP集羣，命令以下：

[root@node1a141 ~]# hadoop distcp hdfs://node1a141:8020/tmp/test.sh  hdfs://node1a202:8020/tmp/

5 附錄

兩集羣的/etc/krb5.conf完整文件內容以下：

[root@node1a141 xdf]# cat /etc/krb5.conf
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = XDF.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 7d
 renew_lifetime = 30
 forwardable = true
 renewable=true
 #default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 HADOOP.COM = {
   kdc = node1a198
   admin_server = node1a198
   default_realm = HADOOP.COM
   supported_enctypes = aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }
 XDF.COM = {
   kdc = node1a141
   admin_server = node1a141
   default_realm = XDF.COM
   supported_enctypes = aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }

[domain_realm]
 .hadoop.com=HADOOP.COM
 hadoop.com=HADOOP.COM
 .xdf.com=XDF.COM
 xdf.com=XDF.COM
 node1a141 = XDF.COM
 node1a143 = XDF.COM
 node1a210 = HADOOP.COM
 node1a202 = HADOOP.COM
 node1a203 = HADOOP.COM

[capaths]
XDF.COM = {
 HADOOP.COM = .
}