【原創】大數據基礎之Ambari（4）經過Ambari部署Impala

ambari2.7.3（hdp3.1） 安裝 impala2.12（自動安裝最新）

ambari的hdp中原生不支持impala安裝，下面介紹如何經過mpack方式使ambari支持impala安裝：

一 安裝Service

1 下載

# wget https://github.com/cas-bigdatalab/ambari-impala-service/raw/master/ambari-impala-mpack-2.6.0-0816.tar.gz

2 安裝

 # ambari-server install-mpack --mpack=/path/to/ambari-impala-mpack-2.6.0-0816.tar.gz --verbose

3 重啓

 # ambari-server restart

訪問ambari頁面發現services中並無impala，檢查mpack.json發現問題：

/var/lib/ambari-server/resources/mpacks/ambari-impala-mpack-2.6.0-0816/mpack.json

{"name": "ambari-impala-mpack", "prerequisites": {"max-ambari-version": "", "min-ambari-version": "2.4.0.0"}, "artifacts": [{"type": "service-definitions", "name": "ambari-common-services", "source_dir": "common-services"}, {"service_versions_map": [{"service_name": "IMPALA", "applicable_stacks": [{"stack_name": "HDP", "stack_version": "2.6"}], "service_version": "2.6.0"}], "type": "stack-addon-service-definitions", "name": "ambari-addon-services", "source_dir": "addon-services"}], "version": "2.6.0-0816", "type": "full-release", "description": "Ambari Management Pack ambari-impala-mpack"}

修改hdp版本，增長3.1

{"stack_name": "HDP", "stack_version": "2.6"}

從新打包ambari-impala-mpack-2.6.0-0816.tar.gz

注意：impala版本不用修改，默認會安裝最新的；

4 卸載

ambari-server uninstall-mpack --mpack-name=ambari-impala-mpack

5 安裝（同上）

6 重啓（同上）

二 安裝Impala

按照頁面提示進行，要增長repo

# cat /etc/yum.repos.d/impala.repo
[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat or CentOS 7 x86_64
name=Cloudera's Distribution for Hadoop, Version 5
baseurl=https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/
gpgkey =https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1

注意這裏是redhat7，按照須要將7改成6也能夠；

三 啓動Impala

 impala-state-store、impala-catalog、impala-server所有啓動失敗，OMG

下面看各類報錯：

1 看日誌報錯

# more /var/log/impala/impala-catalog.log
find: '/usr/local/jdk/': No such file or directory
find: '/usr/local/jdk/': No such file or directory
find: '/usr/local/jdk/': No such file or directory

/usr/lib/impala/sbin/impalad: error while loading shared libraries: libjsig.so: cannot open shared object file: No such file or directory

 增長連接

ln -s /data/jdk1.8.0_191 /usr/local/jdk

 此次impala-state-store能夠啓動，可是impala-catalog和impala-server仍是報錯；

2 各類ClassNotFoundException

ClassNotFoundException: org.apache.hadoop.conf.Configurationjava.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration

         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

 檢查lib目錄：/usr/lib/impala/lib/* 

lrwxrwxrwx 1 root root 43 Jan 19 07:34 hadoop-common.jar -> /usr/hdp/None/hadoop/hadoop-common.jar

發現不少連接有問題，指向一個不存在的目錄/usr/hdp/None，實際爲/usr/hdp/3.1.0.0-78，將這些連接所有替換爲正確的地址：

cd /usr/lib/impala/lib
# ls -l|grep None|awk '{print "/bin/rm "$9}'
# ls -l|grep None|awk '{print "ln -s "$9" "$11}'|sed 's/None/3.1.0.0-78/g'

而後換了個錯：

ClassNotFoundException: com.ctc.wstx.io.InputBootstrapperjava.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper

Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper

         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

這個類在包woodstox-core-5.0.3.jar中，從hadoop2.9版本開始引入，由於hdp3.1內置的hadoop是3.1.1，嘗試將這個jar包放到/usr/lib/impala/lib/，結果錯誤依舊，同時考慮到這個目錄下hadoop相關的jar都沒有版本號，猜想impala可能直接根據一些寫死的jar名字來加載，因此後添加的jar識別不到，只能降hadoop版本

hadoop2.7.2

hadoop-auth.jar

hadoop-common.jar
hadoop-hdfs.jar
hadoop-mapreduce-client-core.jar

hive2.1.0

hive-common.jar
hive-metastore.jar

hbase1.2.6

hbase-common.jar
hbase-client.jar

將以上版本的jar更新到/usr/lib/impala/lib/下，ClassNotFoundException消失；

3 權限

E0119 18:16:24.230655 29797 impala-server.cc:285] Invalid short-circuit reads configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path

查看hdfs配置dfs.domain.socket.path對應的目錄

ls -l /var/lib
drwxr-x--x 3 hdfs hadoop 36 Jan 19 17:05 hadoop-hdfs

因爲impala是impala用戶啓動的，因此沒有權限

# usermod -G hadoop impala

將impala加到hadoop組解決

4 NoSuchMethodError

# cat /var/log/impala/catalogd.ERROR

F0119 18:21:35.781538 16153 catalog.cc:90] java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Lorg/apache/hadoop/hive/conf/HiveConf;Lorg/apache/hadoop/hive/metastore/HiveM

etaHookLoader;Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient;

         at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:93)

         at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:72)

         at org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:168)

         at org.apache.impala.catalog.Catalog.<init>(Catalog.java:100)

         at org.apache.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:263)

         at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:121)

報錯的類是hive metastore相關的類，查看

# javap -cp hive-metastore.jar org.apache.hadoop.hive.metastore.RetryingMetaStoreClient

Compiled from "RetryingMetaStoreClient.java"

public class org.apache.hadoop.hive.metastore.RetryingMetaStoreClient implements java.lang.reflect.InvocationHandler {

  protected org.apache.hadoop.hive.metastore.RetryingMetaStoreClient(org.apache.hadoop.hive.conf.HiveConf, java.lang.Class<?>[], java.lang.Object[], java.util.concurrent.ConcurrentHashMap<java.lang.String, java.lang.Long>, java.lang.Class<? extends org.apache.hadoop.hive.metastore.IMetaStoreClient>) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.hive.conf.HiveConf, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.hive.conf.HiveConf, org.apache.hadoop.hive.metastore.HiveMetaHookLoader, java.lang.String) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.hive.conf.HiveConf, org.apache.hadoop.hive.metastore.HiveMetaHookLoader, java.util.concurrent.ConcurrentHashMap<java.lang.String, java.lang.Long>, java.lang.String, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.hive.conf.HiveConf, java.lang.Class<?>[], java.lang.Object[], java.lang.String) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.hive.conf.HiveConf, java.lang.Class<?>[], java.lang.Object[], java.util.concurrent.ConcurrentHashMap<java.lang.String, java.lang.Long>, java.lang.String) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public java.lang.Object invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) throws java.lang.Throwable;

  static {};

}

方法簽名是存在的，再仔細分析一下NoSuchMethodError，這個錯誤並非說方法不存在，不然編譯都過不了，而是由於jar包衝突有兩個相同的類，一個有這個方法，一個沒有，運行時一旦加載第二個類的方法就會報錯NoSuchMethodError，再檢查其餘jar果真發現問題

# javap -cp hive-exec.jar org.apache.hadoop.hive.metastore.RetryingMetaStoreClient

Compiled from "RetryingMetaStoreClient.java"

public class org.apache.hadoop.hive.metastore.RetryingMetaStoreClient implements java.lang.reflect.InvocationHandler {

  protected org.apache.hadoop.hive.metastore.RetryingMetaStoreClient(org.apache.hadoop.conf.Configuration, java.lang.Class<?>[], java.lang.Object[], java.util.concurrent.ConcurrentHashMap<java.lang.String, java.lang.Long>, java.lang.Class<? extends org.apache.hadoop.hive.metastore.IMetaStoreClient>) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.conf.Configuration, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.conf.Configuration, org.apache.hadoop.hive.metastore.HiveMetaHookLoader, java.lang.String) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.conf.Configuration, org.apache.hadoop.hive.metastore.HiveMetaHookLoader, java.util.concurrent.ConcurrentHashMap<java.lang.String, java.lang.Long>, java.lang.String, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.conf.Configuration, java.lang.Class<?>[], java.lang.Object[], java.lang.String) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public static org.apache.hadoop.hive.metastore.IMetaStoreClient getProxy(org.apache.hadoop.conf.Configuration, java.lang.Class<?>[], java.lang.Object[], java.util.concurrent.ConcurrentHashMap<java.lang.String, java.lang.Long>, java.lang.String) throws org.apache.hadoop.hive.metastore.api.MetaException;

  public java.lang.Object invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) throws java.lang.Throwable;

  static org.apache.hadoop.hive.metastore.IMetaStoreClient access$000(org.apache.hadoop.hive.metastore.RetryingMetaStoreClient);

  static {};

}

在hive-exec.jar中也有一個RetryingMetaStoreClient ，繼續替換jar

hive2.1.0

hive-exec.jar

5 NoClassDefFoundError

F0119 18:59:40.175907 17009 catalog.cc:90] java.lang.IllegalStateException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

         at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:99)

         at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:72)

         at org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:168)

         at org.apache.impala.catalog.Catalog.<init>(Catalog.java:100)

         at org.apache.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:263)

         at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:121)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

         at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1627)

         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)

         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)

         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)

         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)

         at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:93)

         ... 5 more

Caused by: java.lang.reflect.InvocationTargetException

         at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)

         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

         at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1625)

         ... 10 more

Caused by: java.lang.NoClassDefFoundError: org/datanucleus/PersistenceNucleusContext

         at java.lang.Class.forName0(Native Method)

         at java.lang.Class.forName(Class.java:348)

         at org.apache.hadoop.hive.metastore.MetaStoreUtils.getClass(MetaStoreUtils.java:1592)

         at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:64)

         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:581)

         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:546)

         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:608)

         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:398)

         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)

         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)

         at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6396)

         at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236)

         ... 14 more

查看/usr/lib/impala/lib/下有datanucleus相關jar：

datanucleus-api-jdo-3.2.6.jar
datanucleus-core-3.2.12.jar
datanucleus-rdbms-3.2.12.jar

經過javap檢查這幾個jar包都沒有org.datanucleus.PersistenceNucleusContext，再看hive2.1.0依賴的版本是

datanucleus-core-4.1.6.jar

並且不能經過添加jar到/usr/lib/impala/lib/來解決，只能降hive版本到1.2.1，由於這個版本的hive用是的datanucleus3，至此問題所有解決，impala-catalog和impala-server都成功啓動；

6）經過hue訪問impala server報錯

NoClassDefFoundError: org/apache/hive/service/cli/thrift/TGetInfoReq CAUSED BY: ClassNotFoundException: org.apache.hive.service.cli.thrift.TGetInfoReq

這個類在hive-service-1.2.1.jar中，因此還要替換

hive1.2.0

hive-service.jar

注意hue中鏈接impala必定要使用21050端口，不能像impala-shell同樣使用21000端口，不然會報錯

Invalid method name 'OpenSession'

最後彙總一下須要替換的jar

hadoop2.7.2

hadoop-auth.jar
hadoop-common.jar
hadoop-hdfs.jar
hadoop-mapreduce-client-core.jar

hive1.2.0

hive-common.jar
hive-metastore.jar
hive-exec.jar
hive-service.jar

hbase1.2.6

hbase-common.jar
hbase-client.jar

 

參考：

https://github.com/cas-bigdatalab/ambari-impala-service