kafka ranger integration issuse

kafka-ranger

ranger-1.0.0 kafka-1.0.0(confluent-4.0.0)

安裝ranger-1.0.0-kafka-plugin

具體安裝步驟能夠參考： https://www.jianshu.com/p/cda2ef3b56e2 https://blog.csdn.net/sudaxhh/article/details/74390413

下面是安裝過程當中遇到的一些問題

• 下載並解壓 ranger-1.0.0-kafka-plugin.tar.gz
• 修改配置文件install.properties

COMPONENT_INSTALL_DIR_NAME=/usr/local/confluent/
POLICY_MGR_URL=http://192.168.206.144:6080
REPOSITORY_NAME=kafkadev
CUSTOM_USER=kafka
CUSTOM_GROUP=hadoop

• 增長kafka的configs和libs的軟鏈接

ln -s /usr/local/confluent/etc/kafka /usr/local/confluent/config

ln -s /usr/local/confluent/share/java/kafka /usr/local/confluent/libs

• 把kafka的配置文件目錄加到CLASSPATH

reason: the program will only load server.properties when kafka starting, so we need to do this that program could find the configuration files of ranger-kafka.

export CLASSPATH=/usr/local/confluent/etc/kafka

• ERROR: Server not found in Kerberos database

one reason: the kafka-host must be in advertised.listeners

Server not found in Kerberos database

[2018-07-05 15:48:03,763] DEBUG Accepted connection from /172.17.0.15:38950 on /172.17.0.15:9093 and assigned it to processor 0, sendBu
fferSize [actual|requested]: [102400|102400] recvBufferSize [actual|requested]: [102400|102400] (kafka.network.Acceptor)
[2018-07-05 15:48:03,770] DEBUG Processor 0 listening to new connection from /172.17.0.15:38950 (kafka.network.Processor)
[2018-07-05 15:48:03,771] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslC
lientAuthenticator)
[2018-07-05 15:48:03,774] DEBUG Creating SaslClient: client=kafka/master.mesos@LINKTIME.CLOUD;service=kafka;serviceHostname=e318e3a9e22
c;mechs=[GSSAPI] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2018-07-05 15:48:03,783] DEBUG [Controller id=2, targetBrokerId=2] Created socket with SO_RCVBUF = 530904, SO_SNDBUF = 1313280, SO_TIM
EOUT = 0 to node 2 (org.apache.kafka.common.network.Selector)
[2018-07-05 15:48:03,796] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.S
aslClientAuthenticator)
[2018-07-05 15:48:03,798] DEBUG [Controller id=2, targetBrokerId=2] Completed connection to node 2. Ready. (org.apache.kafka.clients.Ne
tworkClient)
[2018-07-05 15:48:03,803] DEBUG Set SASL server state to HANDSHAKE_OR_VERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.
SaslServerAuthenticator)
[2018-07-05 15:48:03,803] DEBUG Handling Kafka request API_VERSIONS (org.apache.kafka.common.security.authenticator.SaslServerAuthentic
ator)
[2018-07-05 15:48:03,816] DEBUG Set SASL server state to HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslServerAu
thenticator)
[2018-07-05 15:48:03,827] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslCli
entAuthenticator)
[2018-07-05 15:48:03,829] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.Sas
lClientAuthenticator)
[2018-07-05 15:48:03,829] DEBUG Handling Kafka request SASL_HANDSHAKE (org.apache.kafka.common.security.authenticator.SaslServerAuthent
icator)
[2018-07-05 15:48:03,830] DEBUG Using SASL mechanism 'GSSAPI' provided by client (org.apache.kafka.common.security.authenticator.SaslSe
rverAuthenticator)
[2018-07-05 15:48:03,831] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticato
r)
[2018-07-05 15:48:03,835] DEBUG Creating SaslServer for kafka/master.mesos@LINKTIME.CLOUD with mechanism GSSAPI (org.apache.kafka.commo
n.security.authenticator.SaslServerAuthenticator)
[2018-07-05 15:48:03,847] DEBUG Set SASL server state to AUTHENTICATE (org.apache.kafka.common.security.authenticator.SaslServerAuthent
icator)
[2018-07-05 15:48:03,869] DEBUG [Controller id=2, targetBrokerId=2] Connection with e318e3a9e22c/172.17.0.15 disconnected due to authen
tication exception (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.Sas
lException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos d
atabase (7) - LOOKING_UP_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]

• schema-registry restart fail

[2018-07-06 04:01:58,149] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry:719)
[2018-07-06 04:01:58,152] ERROR Server died unexpectedly:  (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51)
java.lang.NullPointerException
	at io.confluent.kafka.schemaregistry.storage.KafkaStore.close(KafkaStore.java:366)
	at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.close(KafkaSchemaRegistry.java:720)
	at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.onShutdown(SchemaRegistryRestApplication.java:111)
	at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:66)
	at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:42)
	at io.confluent.rest.Application.createServer(Application.java:157)
	at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)

kafka error log:

...
[2018-07-06 04:01:58,070] ERROR Unsupported access type. operation=DescribeConfigs (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)
[2018-07-06 04:01:58,070] FATAL Unsupported access type. session=Session(User:schemaRegistry,/172.17.0.1), operation=DescribeConfigs, resource=Topic:__schemas (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)
[2018-07-06 04:01:58,070] ERROR Unsupported access type. operation=DescribeConfigs, request=RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={topic=__schemas; } }} accessType={_any} user={schemaRegistry} userGroups={} accessTime={Fri Jul 06 04:01:58 CST 2018} clientIPAddress={172.17.0.1} forwardedAddresses={} remoteIPAddress={null} clientType={null} action={null} requestData={__schemas} sessionId={null} resourceMatchingScope={SELF} clusterName={} context={} } (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)
...

solution: update ranger to 1.1.0 refrence: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+1.1.0+-+Release+Notes https://issues.apache.org/jira/browse/RANGER-2117

notice

Must add all permissions of topics(*) for user kafka that is the same as sasl.kerberos.service.name.