hive Metastore contains multiple versions

凌晨接到hive做業異常，hive版本爲1.2.1，hadoop版本apache 2.7.1，元數據存儲在mysql中，異常信息以下：

Logging initialized using configuration in jar:file:/opt/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.
at
at org.apache.hadoop.util.RunJar.run(RunJar.
at org.apache.hadoop.util.RunJar.main(RunJar.
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.
... 8 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.
at
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.
... 14 more
Caused by: MetaException(message:Metastore contains multiple versions (2) [ version = 1.2.0, comment = Set by MetaStore *@*.*.*.* ] [ version = 1.2.0, comment = Set by MetaStore *@*.*.*.* ])
at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.
at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.
at
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.
at $Proxy9.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient

異常是由於在啓動hive命令時會檢查hive源數據中有一張VERSION表，若是元數據版本信息獲取不到（緣由多是元數據庫異常||網絡異常||短時間內做業量較多操做都會形成查詢不到版本信息），這種狀況下會判斷hive.metastore.schema.verification屬性是true仍是false，爲true時直接拋出MetaException，爲false時打出warn警告而後插入一條version數據（這種狀況下會形成多條version記錄後面的做業會受影響），下面爲hive-metastore包中ObjectStore類代碼，標紅處爲形成多條version記錄的代碼；

 private synchronized void checkSchema() throws MetaException {
    // recheck if it got verified by another thread while we were waiting
    if (isSchemaVerified.get()) {
      return;
    }

    boolean strictValidation =
      HiveConf.getBoolVar(getConf(), HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION);
    // read the schema version stored in metastore db
    String schemaVer = getMetaStoreSchemaVersion();
    if (schemaVer == null) {
      if (strictValidation) {
        throw new MetaException("Version information not found in metastore. ");
      } else {
        LOG.warn("Version information not found in metastore. "
            + HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
            " is not enabled so recording the schema version " +
            MetaStoreSchemaInfo.getHiveSchemaVersion());
        setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
          "Set by MetaStore " + USER + "@" + HOSTNAME);
      }
    } else {
      // metastore schema version is different than Hive distribution needs
      if (schemaVer.equalsIgnoreCase(MetaStoreSchemaInfo.getHiveSchemaVersion())) {
        LOG.debug("Found expected HMS version of " + schemaVer);
      } else {
        if (strictValidation) {
          throw new MetaException("Hive Schema version "
              + MetaStoreSchemaInfo.getHiveSchemaVersion() +
              " does not match metastore's schema version " + schemaVer +
              " Metastore is not upgraded or corrupt");
        } else {
          LOG.error("Version information found in metastore differs " + schemaVer +
              " from expected schema version " + MetaStoreSchemaInfo.getHiveSchemaVersion() +
              ". Schema verififcation is disabled " +
              HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION + " so setting version.");
          setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
            "Set by MetaStore " + USER + "@" + HOSTNAME);
        }
      }
    }
    isSchemaVerified.set(true);
    return;
  }

解決方案：

 

hive安裝好後將hive-site.xml中hive.metastore.schema.verification設置爲true，version獲取不到時報出異常，不去插入version信息，這樣本做業執行失敗不會影響下游做業；

開啓metastore服務，hive統一鏈接metastore，由守護進程啓動metastore，避免大量hive腳本初始化元數據信息時獲取不到版本信息；

優化hive元數據庫；

 

修改觀察幾天經過grep "Version information not found in metastore"  hive.log發現沒有再報找不到version的異常了；