hive0.11 hiveserver custom認證bug
最近在測試hive0.11 hiveserver時遇到的一個關於認證的bug，具體表現：
在配置中指定了custom的認證方式時，經過beeline鏈接hiveserver2,發現鏈接hang住。
hive配置：
java
<property> <name>hive.server2.authentication</name>a <value>CUSTOM</value> </property> <property> <name>hive.server2.custom.authentication.class</name> <value>com.vipshop.hive.service.AuthWithPasswd</value> </property>
查看hiveserver的日誌，發現有以下報錯：
apache
15/01/08 17:54:59 ERROR server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hive.service.auth.PasswdAuthenticationProvider.<init>() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hive.service.auth.CustomAuthenticationProviderImpl.<init>(CustomAuthenticationProviderImpl.java:52) at org.apache.hive.service.auth.AuthenticationProviderFactory.getAuthenticationProvider(AuthenticationProviderFactory.java:62) at org.apache.hive.service.auth.PlainSaslHelper$PlainServerCallbackHandler.handle(PlainSaslHelper.java:73) at org.apache.hive.service.auth.PlainSaslServer.evaluateResponse(PlainSaslServer.java:102) at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:509) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:264) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NoSuchMethodException: org.apache.hive.service.auth.PasswdAuthenticationProvider.<init>() at java.lang.Class.getConstructor0(Class.java:2706) at java.lang.Class.getDeclaredConstructor(Class.java:1985) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 12 more
註釋掉hive.server2.authentication的配置（即便用默認的none）後正常。這實際上是hive0.11.0中的bug，在hive0.13.0中fix，bug id:ide
https://issues.apache.org/jira/browse/HIVE-4778
下面分析下具體涉及的類：
org.apache.hive.service.auth.HiveAuthFactory中的AuthTypes定義了多種認證方法（NONE,LDAP,KERBEROS,CUSTOM,PAM(hive0.11不支持)等）這裏咱們用到了CUSTOM，基於user和password的認證方式，涉及到PasswdAuthenticationProvider和CustomAuthenticationProviderImpl類
1）org.apache.hive.service.auth.PasswdAuthenticationProvider是一個基於用戶名和密碼驗證的接口，主要定義了一個抽象方法Authenticate （參數就是用戶名和密碼）
2）org.apache.hive.service.auth.AnonymousAuthenticationProviderImpl類實現了PasswdAuthenticationProvider接口，提供了一個空的Authenticate方法（直接return）
3）org.apache.hive.service.auth.AuthenticationProviderFactory定義了一個AuthMethods的enum類，定義了有效的認證方法（hive0.13:LDAP/PAM/CUSTOM/NONE,hive0.11:LDAP/CUSTOM/NONE），並提供了對應的封裝,同時定義了一個getAuthenticationProvider方法，用來返回對應的 AuthMethods的具體實現類:
函數
public static PasswdAuthenticationProvider getAuthenticationProvider(AuthMethods authMethod)
throws AuthenticationException {
if (authMethod.equals(AuthMethods.LDAP)) {
return new LdapAuthenticationProviderImpl();
}
else if (authMethod.equals(AuthMethods.PAM)) {
return new PamAuthenticationProviderImpl();
}
else if (authMethod.equals(AuthMethods.CUSTOM)) {
return new CustomAuthenticationProviderImpl();
}
else if (authMethod.equals(AuthMethods.NONE)) {
return new AnonymousAuthenticationProviderImpl();
}
else {
throw new AuthenticationException("Unsupported authentication method");
}
}
4）org.apache.hive.service.auth.CustomAuthenticationProviderImpl
PasswdAuthenticationProvider接口的具體實現類
在hive0.13中的實現:
oop
public class CustomAuthenticationProviderImpl implements PasswdAuthenticationProvider {
Class<? extends PasswdAuthenticationProvider> customHandlerClass;
PasswdAuthenticationProvider customProvider;
@SuppressWarnings("unchecked")
CustomAuthenticationProviderImpl () {
HiveConf conf = new HiveConf();
this.customHandlerClass = (Class<? extends PasswdAuthenticationProvider>)
conf.getClass(
HiveConf.ConfVars.HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS.varname, //即hive.server2.custom.authentication.class
PasswdAuthenticationProvider.class);
//HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS("hive.server2.custom.authentication.class", null),
this.customProvider =
ReflectionUtils.newInstance(this.customHandlerClass, conf); //反射調用，生成class的實例
}
@Override
public void Authenticate(String user, String password) //實現具體的Authenticate方法，用於實現的驗證
throws AuthenticationException {
this.customProvider.Authenticate(user, password);
}
}
hive0.11中的實現:
測試
public class CustomAuthenticationProviderImpl
implements PasswdAuthenticationProvider {
Class<? extends PasswdAuthenticationProvider> customHandlerClass;
PasswdAuthenticationProvider customProvider;
@SuppressWarnings("unchecked")
CustomAuthenticationProviderImpl () {
HiveConf conf = new HiveConf();
this.customHandlerClass = (Class<? extends PasswdAuthenticationProvider>)
conf.getClass(
HiveConf.ConfVars.HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS.name(), //即HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS
PasswdAuthenticationProvider.class);
this.customProvider =
ReflectionUtils.newInstance(this.customHandlerClass, conf);
}
@Override
public void Authenticate(String user, String password)
throws AuthenticationException {
this.customProvider.Authenticate(user, password);
}
}
不一樣的就是customHandlerClass 的獲取方法，這裏咱們手動測試下：
this
import java.lang.*;
public class HiveConf{
public static enum ConfVars {
PLAN_SERIALIZATION("hive.plan.serialization.format","kryo"),
HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS("hive.server2.custom.authentication.class", null),
;
public final String varname;
public final String defaultVal;
ConfVars(String varname, String defaultVal) {
this.varname = varname;
this.defaultVal = defaultVal;
}
public String toString() {
return varname;
}
}
public static void main(String args[]) {
System.out.println("ConfVars List:");
for(ConfVars c:ConfVars.values()){ //ConfVars.values()是全部enum的值
System.out.println(c + " is: " + c);
//hive.server2.custom.authentication.class is: hive.server2.custom.authentication.class
System.out.println(c + " varname: " + c.varname); //能夠看到這裏varname是enum元素定義的值的名稱，而name()是enum元素的名稱
//hive.server2.custom.authentication.class varname: hive.server2.custom.authentication.class
System.out.println(c + " name(): " + c.name());
//hive.server2.custom.authentication.class name(): HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS
}
System.out.println(HiveConf.ConfVars.HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS.name()); //HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS
System.out.println(HiveConf.ConfVars.HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS.varname); //hive.server2.custom.authentication.class
}
}
能夠看到使用HiveConf.ConfVars.HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS.varname時，this.customHandlerClass的結果爲具體的實現類（即hive.server2.custom.authentication.class設置的類）好比class com.vipshop.hive.service.AuthWithPasswd，而使用HiveConf.ConfVars.HIVE_SERVER2_CUSTOM_AUTHENTICATION_CLASS.name()時，返回的是接口，即interface org.apache.hive.service.auth.PasswdAuthenticationProvider
再來看ReflectionUtils.newInstance方法:
lua
private static final Class<?>[] EMPTY_ARRAY = new Class[]{};
....
public static <T> T newInstance(Class<T> theClass, Configuration conf) {
T result;
try {
Constructor<T> meth = (Constructor<T>) CONSTRUCTOR_CACHE.get(theClass);
if (meth == null) {
meth = theClass.getDeclaredConstructor(EMPTY_ARRAY); //返回類的構造函數對象
meth.setAccessible(true);
CONSTRUCTOR_CACHE.put(theClass, meth);
}
result = meth.newInstance();
} catch (Exception e) {
throw new RuntimeException(e);
}
setConf(result, conf);
return result;
}
在hive0.11時這裏theClass爲org.apache.hive.service.auth.PasswdAuthenticationProvider，org.apache.hive.service.auth.PasswdAuthenticationProvider是一個接口，沒有定義構造函數，所以會拋出異常。
再來看認證的配置是在何時加載的？咱們經過指定一個錯誤的配置來看其報錯堆棧：
使用以下命令開啓hiveserver的debug log:
debug
bin/hiveserver2 -hiveconf hive.root.logger=DEBUG,console start
15/01/08 17:56:55 INFO service.AbstractService: Service:ThriftBinaryCLIService is started. 15/01/08 17:56:55 INFO service.AbstractService: Service:HiveServer2 is started. 15/01/08 17:56:55 ERROR thrift.ThriftCLIService: Error: javax.security.auth.login.LoginException: Unsupported authentication type CUSTEM at org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(HiveAuthFactory.java:148) at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:43) at java.lang.Thread.run(Thread.java:662)
在hiveserver2正常啓動時默認會啓動ThriftBinaryCLIService服務：
在ThriftBinaryCLIService 類的run方法：
日誌
public void run() {
try {
hiveAuthFactory = new HiveAuthFactory(); //run方法中首先會聲明一個HiveAuthFactory對象
TTransportFactory transportFactory = hiveAuthFactory.getAuthTransFactory(); //調用HiveAuthFactory的getAuthTransFactory方法獲取
對應的TTransportFactory對象
TProcessorFactory processorFactory = hiveAuthFactory.getAuthProcFactory(this);
.....
而在HiveAuthFactory的構造函數中會解析hive的配置，獲取對應的hiveserver的認證設置：
public HiveAuthFactory() throws TTransportException {
conf = new HiveConf();
transportMode = conf.getVar(HiveConf.ConfVars.HIVE_SERVER2_TRANSPORT_MODE);
//HIVE_SERVER2_TRANSPORT_MODE("hive.server2.transport.mode", "binary",new StringsValidator("binary", "http")),
authTypeStr = conf.getVar(HiveConf.ConfVars.HIVE_SERVER2_AUTHENTICATION);
//HIVE_SERVER2_AUTHENTICATION("hive.server2.authentication", "NONE", new StringsValidator("NOSASL", "NONE", "LDAP", "KERBEROS", "PAM", "CUSTOM")),
默認爲null，有效值爲"NOSASL", "NONE", "LDAP", "KERBEROS", "PAM", "CUSTOM"
// In http mode we use NOSASL as the default auth type
if (transportMode.equalsIgnoreCase("http")) {
if (authTypeStr == null) {
authTypeStr = AuthTypes.NOSASL.getAuthName();
}
}
else {
if (authTypeStr == null) {
authTypeStr = AuthTypes.NONE.getAuthName();
}
if (authTypeStr.equalsIgnoreCase(AuthTypes.KERBEROS.getAuthName())
&& ShimLoader.getHadoopShims().isSecureShimImpl()) {
saslServer = ShimLoader.getHadoopThriftAuthBridge().createServer(
conf.getVar(ConfVars.HIVE_SERVER2_KERBEROS_KEYTAB),
conf.getVar(ConfVars.HIVE_SERVER2_KERBEROS_PRINCIPAL)
);
// start delegation token manager
try {
saslServer.startDelegationTokenSecretManager(conf, null);
} catch (IOException e) {
throw new TTransportException("Failed to start token manager", e);
}
}
}
}
getAuthTransFactory方法會判斷authTypeStr是否爲有效值，不然拋出異常，退出啓動
public TTransportFactory getAuthTransFactory() throws LoginException {
TTransportFactory transportFactory;
if (authTypeStr.equalsIgnoreCase(AuthTypes.KERBEROS.getAuthName())) {
try {
transportFactory = saslServer.createTransportFactory(getSaslProperties());
} catch (TTransportException e) {
throw new LoginException(e.getMessage());
}
} else if (authTypeStr.equalsIgnoreCase(AuthTypes.NONE.getAuthName())) {
transportFactory = PlainSaslHelper.getPlainTransportFactory(authTypeStr);
} else if (authTypeStr.equalsIgnoreCase(AuthTypes.LDAP.getAuthName())) {
transportFactory = PlainSaslHelper.getPlainTransportFactory(authTypeStr);
} else if (authTypeStr.equalsIgnoreCase(AuthTypes.PAM.getAuthName())) {
transportFactory = PlainSaslHelper.getPlainTransportFactory(authTypeStr);
} else if (authTypeStr.equalsIgnoreCase(AuthTypes.NOSASL.getAuthName())) {
transportFactory = new TTransportFactory();
} else if (authTypeStr.equalsIgnoreCase(AuthTypes.CUSTOM.getAuthName())) {
transportFactory = PlainSaslHelper.getPlainTransportFactory(authTypeStr);
} else {
throw new LoginException("Unsupported authentication type " + authTypeStr);
}
return transportFactory;
}
相關文章
- 1. HiveServer 與 HiveServer2
- 2. HTTP認證：基本認證、摘要認證、HTTPS+表單認證
- 3. 034.認證方式 | 基本認證 、Token認證、 AK/SK認證
- 4. hive0.11元數據表彙總說明
- 5. Windows認證 | 域認證
- 6. HTTP認證-摘要認證
- 7. Windows認證 | 網絡認證
- 8. Java 經過jdbc鏈接HIVESERVER
- 9. libcurl使用認證證書 https認證
- 10. HiveServer2集成LDAP作用戶認證
- 更多相關文章...
- • MySQL默認值（DEFAULT） - MySQL教程
- • XML 驗證 - XML 教程
- • Spring Cloud 微服務實戰(三) - 服務註冊與發現
- • 再有人問你分佈式事務，把這篇扔給他
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
