impala 1.2.3升級遇到的問題

升級impala 1.2.3時遇到的一些問題，記錄下。

1.catalog默認的jvm參數問題

catalog啓動一段時間後，報"OutOfMemoryError: GC overhead limit exceeded"錯誤

/var/run/impala/hs_err_pidxxxx.log file:
 The java heap info:
Heap
PSYoungGen      total 904768K, used 402833K [0x00000007ad2b0000, 0x0000000800000000, 0x0000000800000000)
 eden space 452416K, 89% used [0x00000007ad2b0000,0x00000007c5c14448,0x00000007c8c80000)
 from space 452352K, 0% used [0x00000007c8c80000,0x00000007c8c80000,0x00000007e4640000)
 to   space 452352K, 0% used [0x00000007e4640000,0x00000007e4640000,0x0000000800000000)
PSOldGen        total 2714304K, used 2714303K [0x0000000707800000, 0x00000007ad2b0000, 0x00000007ad2b0000)
 object space 2714304K, 99% used [0x0000000707800000,0x00000007ad2affe8,0x00000007ad2b0000)
PSPermGen       total 38848K, used 38569K [0x0000000702600000, 0x0000000704bf0000, 0x0000000707800000)
 object space 38848K, 99% used [0x0000000702600000,0x0000000704baa608,0x0000000704bf0000)

the jstat info:
 jstat -gcutil 8589 1000 1000
 S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT  
 0.00   0.00 100.00 100.00  99.09    115    9.170    92  540.636  549.806
 0.00   0.00 100.00 100.00  99.09    115    9.170    92  540.636  549.806
 0.00   0.00  96.98 100.00  99.02    115    9.170    92  546.807  555.977
 0.00   0.00 100.00 100.00  99.02    115    9.170    93  546.807  555.977
 0.00   0.00 100.00 100.00  99.02    115    9.170    93  546.807  555.977
 0.00   0.00 100.00 100.00  99.02    115    9.170    93  546.807  555.977
0.00   0.00 100.00 100.00  99.02    115    9.170    93  546.807  555.977
 0.00   0.00 100.00 100.00  99.02    115    9.170    93  546.807  555.977

You can pass JVM arguments (including changes to the heap size) to catalogd using the "JAVA_TOOL_OPTIONS" environment variable. If you are using CM, you can set this environment variable using the "Catalog Server Environment Safety Valve".
We have also made a number of improvements to the catalog memory footprint in the upcoming Impala v1.2.4 release (which should be out next week if all goes well).Hope this helps.

經過設置下面變量能夠解決

export JAVA_TOOL_OPTIONS="-Xmx8000m -Xms8000m  -Xmn1024m -XX:PermSize=256m -XX:PermSize=256m -XX:SurvivorRatio=8 -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -Dcom.sun.management.jmxremote  -Xnoclassgc -Xloggc:/apps/logs/jvm/catalog-$(date +%Y%m%d-%H%M%S).log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8060 -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=xxxxxxx"

2.catalog加載table metadata問題

ERROR: AnalysisException: This Impala daemon is not ready to accept user requests. Status: Waiting for catalog update from the StateStore.impala server:
I0214 17:02:01.526229 36166 Frontend.java:443] analyze query use viplog
I0214 17:02:01.576381 36166 jni-util.cc:154] com.cloudera.impala.common.AnalysisException: This Impala daemon is not ready to accept user requests. Status: Waiting for catalog update from the StateStore.
       at com.cloudera.impala.analysis.Analyzer.getCatalog(Analyzer.java:650)
       at com.cloudera.impala.analysis.Analyzer.getDb(Analyzer.java:1326)
       at com.cloudera.impala.analysis.UseStmt.analyze(UseStmt.java:44)
       at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:318)
       at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:444)
       at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:114)

I0214 17:02:01.604887 36166 status.cc:44] AnalysisException: This Impala daemon is not ready to accept user requests. Status: Waiting for catalog update from the StateStore.

bug id:

https://issues.cloudera.org/browse/IMPALA-737

This is a known issue fixed in 1.2.4. After this patch, catalogd loads the metadata lazily instead of loading everything at startup.

Impala 1.2.4 will introduce lazy loading, so you will not see this problem. Your error indicates that the catalog has not loaded all the metadata yet. The best way to determine this is from the impalad's metrics URL. Specifically, you want to look for the value "catalog.ready" to be set to 1 on the impalad debug webpage. You can find the metrics in <impala_host>:25000/metrics.

（impala-server.ready:1，catalog.ready:1）

3.udf問題

We currently don't support String as the input and return types. You'll instead have to  use Text or BytesWritable.

I've filed IMPALA-791 to track fixing this. https://issues.cloudera.org/browse/IMPALA-791

能夠經過org.apache.hadoop.io.Text類代替String類解決

Text類api:

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/Text.html

4.catalog內存問題，不知道是否是內存泄露,一段時間後old區就滿了，致使OOM

7G的old區：

root@GD6G12S190-logserver impala]# jstat -gcutil 10454 1000 1000

 S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT  

 0.00 100.00 100.00 100.00  16.24    572   63.441    22  196.127  259.568

 0.00 100.00 100.00 100.00  16.24    572   63.441    22  196.127  259.568

 0.00 100.00 100.00 100.00  16.24    572   63.441    22  196.127  259.568

 0.00 100.00 100.00 100.00  16.24    572   63.441    22  196.127  259.568

 0.00 100.00 100.00 100.00  16.24    572   63.441    22  196.127  259.568

 0.00 100.00 100.00 100.00  16.24    572   63.441    22  196.127  259.568

 0.00 100.00 100.00 100.00  16.24    572   63.441    22  196.127  259.568

 0.00 100.00 100.00 100.00  16.24    572   63.441    22  196.127  259.568