Ubuntu 14.10 下Hive配置

時間 2019-12-07

標籤 ubuntu 14.10 hive 配置欄目 Ubuntu 简体版

原文原文鏈接

1 系統環境

Ubuntu 14.10 html

JDK-7java

Hadoop 2.6.0mysql

2 安裝步驟

2.1 下載Hive

我第一次安裝的時候，下載的是Hive-1.2.1，配置好以後，老是報錯web

[ERROR] Terminal initialization failed; falling back to unsupported
java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
    at jline.TerminalFactory.create(TerminalFactory.java:101)
    at jline.TerminalFactory.get(TerminalFactory.java:158)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:229)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:209)
    at org.apache.hadoop.hive.cli.CliDriver.getConsoleReader(CliDriver.java:773)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:715)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:230)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:209)
    at org.apache.hadoop.hive.cli.CliDriver.getConsoleReader(CliDriver.java:773)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:715)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

對於這個錯誤，不少人的作法都是將hadoop/share/hadoop/yarn/lib中的jline 替換成高版本的，如hive中的2.×sql

1.Delete jline from the Hadoop lib directory (it's only pulled in transitively from ZooKeeper).
2.export HADOOP_USER_CLASSPATH_FIRST=true

可是我替換以後，仍是報錯，後來換了一個低版本的HIve-1.0.1，就行了，因此若是出現如上錯誤，不必定非的是jar問題，也多是版本問題。數據庫

下載地址：http://mirrors.hust.edu.cn/apache/hive/hive-1.0.1/apache-hive-1.0.1-bin.tar.gzapache

2.2 配置hive

2.2.1 拷貝一份hive-env.sh，設置HADOOP_HOME

2.2.2 拷貝一份hive-site.xml

主要修改以下參數session

<property> 
   <name>javax.jdo.option.ConnectionURL </name> 
   <value>jdbc:mysql://localhost:3306/hive </value> 
</property> 
 
<property> 
   <name>javax.jdo.option.ConnectionDriverName </name> 
   <value>com.mysql.jdbc.Driver </value> 
</property>

<property> 
   <name>javax.jdo.option.ConnectionPassword </name> 
   <value>hive </value> 
</property> 
 
<property> 
   <name>hive.hwi.listen.port </name> 
   <value>10000</value> 
   <description>This is the port the Hive Web Interface will listen on </descript ion> 
</property> 

<property> 
   <name>datanucleus.autoCreateSchema </name> 
   <value>true</value> 
</property> 
 
<property> 
   <name>datanucleus.fixedDatastore </name> 
   <value>false</value> 
</property> 
</property> 

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>Username to use against metastore database</description>
  </property>

  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/home/hdpsrc/hive/iotmp</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/home/hdpsrc/hive/iotmp</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
  <property>
    <name>hive.querylog.location</name>
    <value>/home/hdpsrc/hive/iotmp</value>
    <description>Location of Hive run time structured log file</description>
  </property>

2.3 拷貝一份hive-log4j.properties

這裏我只是設置了log文件的路徑app

2.4 創建相關文件夾，就是在2.2中設置的

2.5 在HDFS中創建相關文件夾

  $ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
  $ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

2.3 安裝MySQL

由於這裏的關係型數據庫是用的mysql，那麼須要安裝mysql，同時按照上面配置的新建hive用戶，新建數據庫。ide

安裝mysql參考：http://www.cnblogs.com/liuchangchun/p/4099003.html

新建用戶參考：http://www.cnblogs.com/liuchangchun/p/4431426.html

由於以前安裝mysql後將編碼設置成了utf8的，在啓動hive的時候報了以下錯：

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes

這個錯誤，須要修改hive數據庫的編碼：

alter database hive character set latin1;

2.4 拷貝mysql的驅動jar包到hive/lib下面

若是不拷貝，那麼會報錯：找不到jar包

Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
    at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)
    at org.datanucleus.store.rdbms.connectionpool.BoneCPConnectionPoolFactory.createConnectionPool(BoneCPConnectionPoolFactory.java:54)
    at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:238)
    ... 67 more

2.5 啓動hive

啓動以前要確保hdfs已經起來。

在命令行執行：hive，便可進入hive命令行界面，若是沒有錯誤，恭喜你，OK了。

若是HDFS沒起來，那麼可能報錯.。。

2.6 啓動hwi WEB客戶端

hive提供了幾種交互方式，有命令行，還有web界面的。若是要啓動web服務，能夠經過以下命令

hive --service hwi

可是以前須要設置一些屬性

<property>
    <name>hive.hwi.listen.host</name>
    <value>192.168.1.102</value>
    <description>This is the host address the Hive Web Interface will listen on</description>
  </property>
  <property>
    <name>hive.hwi.listen.port</name>
    <value>10000</value>
    <description>This is the port the Hive Web Interface will listen on</description>
  </property>
  <property>
    <name>hive.hwi.war.file</name>
    <value>lib/hive-hwi-1.0.1.war</value>
    <description>This sets the path to the HWI war file, relative to ${HIVE_HOME}. </description>
  </property>

啓動以後，進入http://hostname:10000/hwi/。可是，我這裏遇到了幾個問題

a. 找不到war包

ls: cannot access /home/grid2/apache-hive-0.13.1-bin/lib/hive-hwi-*.war: No such file or directory
14/09/14 21:07:10 INFO hwi.HWIServer: HWI is starting up
14/09/14 21:07:11 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
14/09/14 21:07:11 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead
14/09/14 21:07:11 FATAL hwi.HWIServer: HWI WAR file not found at /home/grid2/apache-hive-0.13.1-bin/lib/hive-hwi-@VERSION@.war

由於在hive/lib下面確實沒有hive-hwi-*.war包，那咱們能夠本身下載hive的源碼，而後打包成war放大hive/lib下面。

能夠參考：

http://blog.csdn.net/bluishglc/article/details/41652111

http://blog.csdn.net/wulantian/article/details/38271803

b. 啓動hwi後沒有錯誤，可是訪問http://hostname:10000/hwi/就會報錯

15/08/27 14:13:40 ERROR mortbay.log: /hwi/
java.lang.IllegalStateException: No Java compiler available
    at org.apache.jasper.JspCompilationContext.createCompiler(JspCompilationContext.java:225)
    at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:560)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:299)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:315)
    …………………………………………………………………………

這個問題目前沒解決o(╯□╰)o，個人替代方案是使用HUE

a. HUE中配置HIVE

找到beeswax標籤，不叫hive，配置以下屬性，其中端口號要和hive-site.xml中的保持一致

hive-site.xml中配置thrift端口號

<property>
    <name>hive.server2.thrift.port</name>
    <value>19999</value>
    <description>Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'binary'.</description>
  </property>

hive_server_host=192.168.1.102

# Port where HiveServer2 Thrift server runs on.
hive_server_port=19999

# Hive configuration directory, where hive-site.xml is located
hive_conf_dir=/home/hadoop/software/cloud/apache-hive-1.0.1-bin/conf

使用hue以前須要先啓動hive thrift server

hive --service hiveserver2

我這裏配置好以後遇到一個錯誤，費了好大勁才找到問題所在

java.lang.RuntimeException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:84)
    at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
    at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
    at com.sun.proxy.$Proxy21.fetchResults(Unknown Source)
    at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:450)
    at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:587)
    at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
    at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:312)
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:442)
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:588)
    at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:561)
    at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:138)
    at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1621)
    at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:337)
    at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:248)
    at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:654)
    at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
    ... 19 more

如上錯誤，訪問錯誤，我找到com.google.common.base.Stopwatch這個類的jar包，是guava.jar，這個jar包在hadoop和hive中都有，版本是11.0.2，同時我在JAVA_HOME中也有一個在jre/lib/ext中，可是版本是18.0(這個估計是我當時安裝ambari時候放進去的，由於一個問題，要將guava換成最新版)，這樣估計形成了版本不一致，因此出現瞭如上的錯誤。解決辦法就是刪了那個jar包，或是換成11.02，因此這裏也要考慮一個程序加載順序。

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

參考：

https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallationandConfiguration

http://blog.csdn.net/an342647823/article/details/46048403

http://blog.csdn.net/keljony/article/details/43371995

http://sunjia-704471770-qq-com.iteye.com/blog/1631430

http://blog.csdn.net/jdplus/article/details/46493553