0027-如何在CDH集羣啓用Kerberos
1.文檔編寫目的javascript
本文檔講述如何在CDH集羣啓用及配置Kerberos，您將學習到如下知識：java
1.如何安裝及配置KDC服務node
2.如何經過CDH啓用Kerberosshell
3.如何登陸Kerberos並訪問Hadoop相關服務數據庫
文檔主要分爲如下幾步：apache
1.安裝及配置KDC服務vim
2.CDH集羣啓用Kerberos安全
3.Kerberos使用服務器
這篇文檔將重點介紹如何在CDH集羣啓用及配置Kerberos，並基於如下假設：app
1.CDH集羣運行正常
2.集羣未啓用Kerberos
3.MySQL 5.1.73
如下是本次測試環境，但不是本操做手冊的必需環境：
1.操做系統：CentOS 6.5
2.CDH和CM版本爲5.12.0
3.採用root用戶進行操做
2.KDC服務安裝及配置
本文檔中將KDC服務安裝在Cloudera Manager Server所在服務器上（KDC服務可根據本身須要安裝在其餘服務器）
1.在Cloudera Manager服務器上安裝KDC服務
[root@ip-172-31-6-148~]# yum -y install krb5-serverkrb5-libs krb5-auth-dialog krb5-workstation
2.修改/etc/krb5.conf配置
[root@ip-172-31-6-148 fayson_r]# vim /etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = FAYSON.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
[realms]
FAYSON.COM = {
kdc = ip-172-31-6-148.fayson.com
admin_server = ip-172-31-6-148.fayson.com
}
[domain_realm]
.ip-172-31-6-148.fayson.com = FAYSON.COM
ip-172-31-6-148.fayson.com = FAYSON.COM
標紅部分爲須要修改的信息。
3.修改/var/kerberos/krb5kdc/kadm5.acl配置
[root@ip-172-31-6-148~]# vim /var/kerberos/krb5kdc/kadm5.acl */admin@FAYSON.COM *
4.修改/var/kerberos/krb5kdc/kdc.conf配置
[root@ip-172-31-6-148 ~]# vim /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88
[realms]
FAYSON.COM= {
#master_key_type = aes256-cts
max_renewable_life= 7d 0h 0m 0s
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:n
ormal des-cbc-md5:normal des-cbc-crc:normal
}
標紅部分爲須要修改的配置。
5.建立Kerberos數據庫
[root@ip-172-31-6-148 ~]# kdb5_util create –r FAYSON.COM -s Loading random data Initializing database '/var/kerberos/krb5kdc/principal' for realm 'FAYSON.COM', master key name 'K/M@FAYSON.COM' You will be prompted for the database Master Password. It is important that you NOT FORGET this password. Enter KDC database master key: Re-enter KDC database master key to verify:
此處須要輸入Kerberos數據庫的密碼。
6.建立Kerberos的管理帳號
[root@ip-172-31-6-148 ~]# kadmin.local Authenticating as principal fayson/admin@CLOUDERA.COM with password. kadmin.local: addprinc admin/admin@FAYSON.COM WARNING: no policy specified for admin/admin@FAYSON.COM; defaulting to no policy Enter password for principal "admin/admin@FAYSON.COM": Re-enter password for principal "admin/admin@FAYSON.COM": Principal "admin/admin@FAYSON.COM" created. kadmin.local: exit [root@ip-172-31-6-148 ~]#
標紅部分爲Kerberos管理員帳號，須要輸入管理員密碼。
7.將Kerberos服務添加到自啓動服務，並啓動krb5kdc和kadmin服務
[root@ip-172-31-6-148~]# chkconfig krb5kdc on [root@ip-172-31-6-148 ~]# chkconfig kadmin on [root@ip-172-31-6-148 ~]# service krb5kdc start Starting Kerberos 5 KDC: [ OK ] [root@ip-172-31-6-148 ~]# service kadmin start Starting Kerberos 5 Admin Server: [ OK ] [root@ip-172-31-6-148 ~]#
8.測試Kerberos的管理員帳號
[root@ip-172-31-6-148 ~]# kinit admin/admin@FAYSON.COM
Password for admin/admin@FAYSON.COM:
[root@ip-172-31-6-148 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin/admin@FAYSON.COM
Valid starting Expires Service principal
09/05/17 16:39:17 09/06/17 16:39:17 krbtgt/FAYSON.COM@FAYSON.COM
renew until 09/12/17 16:39:17
[root@ip-172-31-6-148 ~]#
9.爲集羣安裝全部Kerberos客戶端，包括Cloudera Manager
[root@ip-172-31-6-148 cdh-shell-master]# yum -y install krb5-libs krb5-workstation
10.在Cloudera Manager Server服務器上安裝額外的包
[root@ip-172-31-6-148cdh-shell-master]# yum -y install openldap-clients
11.將KDC Server上的krb5.conf文件拷貝到全部Kerberos客戶端
[root@ip-172-31-6-148cdh-shell-master]# scp -r /etc/krb5.conf root@172.31.5.190:/etc/
此處使用腳本進行拷貝
[root@ip-172-31-6-148cdh-shell-master]# sh b.sh node.list /etc/krb5.conf /etc/ krb5.conf 100% 451 0.4KB/s 00:00 krb5.conf 100% 451 0.4KB/s 00:00 krb5.conf 100% 451 0.4KB/s 00:00 krb5.conf 100% 451 0.4KB/s 00:00 [root@ip-172-31-6-148 cdh-shell-master]#
3.CDH集羣啓用Kerberos
1.在KDC中給Cloudera Manager添加管理員帳號
[root@ip-172-31-6-148 cdh-shell-bak]# kadmin.local Authenticating as principal admin/admin@FAYSON.COM with password. kadmin.local: addprinc cloudera-scm/admin@FAYSON.COM WARNING: no policy specified for cloudera-scm/admin@FAYSON.COM; defaulting to no policy Enter password for principal "cloudera-scm/admin@FAYSON.COM": Re-enter password for principal "cloudera-scm/admin@FAYSON.COM": Principal "cloudera-scm/admin@FAYSON.COM" created. kadmin.local: exit [root@ip-172-31-6-148 cdh-shell-bak]#
2.進入Cloudera Manager的「管理」-> 「安全」界面
3.選擇「啓用Kerberos」，進入以下界面
確保以下列出的全部檢查項都已完成
4.點擊「繼續」，配置相關的KDC信息，包括類型、KDC服務器、KDC Realm、加密類型以及待建立的Service Principal（hdfs，yarn,，hbase，hive等）的更新生命期等
5.點擊「繼續」
6.不建議讓Cloudera Manager來管理krb5.conf, 點擊「繼續」
7.輸入Cloudera Manager的Kerbers管理員帳號，必須和以前建立的帳號一致，點擊「繼續」
8.等待啓用Kerberos完成，點擊「繼續」
9.點擊「繼續」
10.勾選重啓集羣，點擊「繼續」
11.等待集羣重啓成功，點擊「繼續」
至此已成功啓用Kerberos。
4.Kerberos使用
使用fayson用戶運行MapReduce任務及操做Hive，須要在集羣全部節點建立fayson用戶。
1.使用kadmin建立一個fayson的principal
[root@ip-172-31-6-148 cdh-shell-bak]# kadmin.local Authenticating as principal admin/admin@FAYSON.COM with password. kadmin.local: addprinc fayson@FAYSON.COM WARNING: no policy specified for fayson@FAYSON.COM; defaulting to no policy Enter password for principal "fayson@FAYSON.COM": Re-enter password for principal "fayson@FAYSON.COM": Principal "fayson@FAYSON.COM" created. kadmin.local: exit [root@ip-172-31-6-148 cdh-shell-bak]#
2.使用fayson用戶登陸Kerberos
[root@ip-172-31-6-148 cdh-shell-bak]# kdestroy
[root@ip-172-31-6-148 cdh-shell-bak]# kinit fayson
Password for fayson@FAYSON.COM:
[root@ip-172-31-6-148 cdh-shell-bak]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: fayson@FAYSON.COM
Valid starting Expires Service principal
09/05/17 17:19:08 09/06/17 17:19:08 krbtgt/FAYSON.COM@FAYSON.COM
renew until 09/12/17 17:19:08
[root@ip-172-31-6-148 cdh-shell-bak]#
3.運行MapReduce做業
[root@ip-172-31-6-148~]# hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 1 ... Starting Job 17/09/02 20:10:43 INFO mapreduce.Job: Running job: job_1504383005209_0001 17/09/02 20:10:56 INFO mapreduce.Job: Job job_1504383005209_0001 running in ubermode : false 17/09/02 20:10:56 INFO mapreduce.Job: map0% reduce 0% 17/09/02 20:11:09 INFO mapreduce.Job: map20% reduce 0% 17/09/02 20:11:12 INFO mapreduce.Job: map40% reduce 0% 17/09/02 20:11:13 INFO mapreduce.Job: map50% reduce 0% 17/09/02 20:11:15 INFO mapreduce.Job: map60% reduce 0% 17/09/02 20:11:16 INFO mapreduce.Job: map70% reduce 0% 17/09/02 20:11:19 INFO mapreduce.Job: map80% reduce 0% 17/09/02 20:11:21 INFO mapreduce.Job: map100% reduce 0% 17/09/02 20:11:26 INFO mapreduce.Job: map100% reduce 100% 17/09/02 20:11:26 INFO mapreduce.Job: Job job_1504383005209_0001 completedsuccessfully
4.使用beeline鏈接hive進行測試
[root@ip-172-31-6-148 cdh-shell-bak]# beeline Beeline version 1.1.0-cdh5.12.1 by Apache Hive beeline> !connect jdbc:hive2://localhost:10000/;principal=hive/ip-172-31-6-148.fayson.com@FAYSON.COM ... Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://localhost:10000/> show tables; ... INFO : OK +-------------+--+ | tab_name | +-------------+--+ | test_table | +-------------+--+ 1 row selected (0.194 seconds) 0: jdbc:hive2://localhost:10000/> select * from test_table; ... INFO : OK +----------------+----------------+--+ | test_table.s1 | test_table.s2 | +----------------+----------------+--+ | 4 | lisi | | 1 | test | | 2 | fayson | | 3 | zhangsan | +----------------+----------------+--+ 4 rows selected (0.144 seconds) 0: jdbc:hive2://localhost:10000/>
運行Hive MapReduce做業
0: jdbc:hive2://localhost:10000/> select count(*) from test_table; ... INFO : OK +------+--+ | _c0 | +------+--+ | 4 | +------+--+ 1 row selected (35.779 seconds) 0: jdbc:hive2://localhost:10000/>
5.常見問題
1.使用Kerberos用戶身份運行MapReduce做業報錯
main : run as user is fayson
main : requested yarn user is fayson
Requested user fayson is not whitelisted and has id 501,whichis below the minimum allowed 1000
Failing this attempt. Failing the application.
17/09/02 20:05:04 INFO mapreduce.Job: Counters: 0
Job Finished in 6.184 seconds
java.io.FileNotFoundException: File does not exist:hdfs://ip-172-31-6-148:8020/user/fayson/QuasiMonteCarlo_1504382696029_1308422444/out/reduce-out
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1258)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1258)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1820)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1844)
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
atorg.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
問題緣由：是因爲Yarn限制了用戶id小於10000的用戶提交做業；
解決方法：修改Yarn的min.user.id來解決
推薦關注Hadoop實操，第一時間，分享更多Hadoop乾貨，歡迎轉發和分享。
相關文章
- 1. 0027-如何在CDH集羣啓用Kerberos
- 2. CDH(6.2.2)集羣禁用kerberos
- 3. CDH集羣開啓Kerberos安全認證
- 4. 如何在CDH5.12集羣中啓用Kerberos認證
- 5. 在CDH集羣中啓用HDFS的HA
- 6. 爲CDH 5集羣添加Kerberos支持
- 7. 0013-如何在Kerberos與非Kerberos的CDH集羣BDR不可用時複製數據
- 8. 如何在CDH 6.3.2 啓用Kerberos 中 使用sentry限制 用戶讀寫
- 9. CDH禁用kerberos
- 10. CDH 禁用 Kerberos
- 更多相關文章...
- • XSD 如何使用? - XML Schema 教程
- • Swarm 集羣管理 - Docker教程
- • TiDB 在摩拜單車在線數據業務的應用和實踐
- • ☆技術問答集錦（13）Java Instrument原理
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
