以前在mac上調試hadoop程序(mac以前配置過hadoop環境)一直都是正常的。由於工做須要,須要在windows上先調試該程序,而後再轉到linux下。程序運行的過程當中,報Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable
null
\bin\winutils.exe in the Hadoop binaries.
java
經過斷點調試、查看源碼發現程序須要根據HADOOP_HOME找到winutils.exe,因爲win機器並無配置該環境變量,因此程序報 null\bin\winutils.exe。linux
private static String checkHadoopHome() { // first check the Dflag hadoop.home.dir with JVM scope String home = System.getProperty("hadoop.home.dir"); // fall back to the system/user-global env variable if (home == null) { home = System.getenv("HADOOP_HOME"); } try { // couldn't find either setting for hadoop's home directory if (home == null) { throw new IOException("HADOOP_HOME or hadoop.home.dir are not set."); } if (home.startsWith("\"") && home.endsWith("\"")) { home = home.substring(1, home.length()-1); } // check that the home setting is actually a directory that exists File homedir = new File(home); if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) { throw new IOException("Hadoop home directory " + homedir + " does not exist, is not a directory, or is not an absolute path."); } home = homedir.getCanonicalPath(); } catch (IOException ioe) { if (LOG.isDebugEnabled()) { LOG.debug("Failed to detect a valid hadoop home directory", ioe); } home = null; } return home; }
private static String HADOOP_HOME_DIR = checkHadoopHome();git
public static final String getQualifiedBinPath(String executable) throws IOException { // construct hadoop bin path to the specified executable String fullExeName = HADOOP_HOME_DIR + File.separator + "bin" + File.separator + executable; File exeFile = new File(fullExeName); if (!exeFile.exists()) { throw new IOException("Could not locate executable " + fullExeName + " in the Hadoop binaries."); } return exeFile.getCanonicalPath(); } /** a Windows utility to emulate Unix commands */ public static final String WINUTILS = getWinUtilsPath(); public static final String getWinUtilsPath() { String winUtilsPath = null; try { if (WINDOWS) { winUtilsPath = getQualifiedBinPath("winutils.exe"); } } catch (IOException ioe) { LOG.error("Failed to locate the winutils binary in the hadoop binary path", ioe); } return winUtilsPath; }
找到緣由後就去網上問了度娘,找到了解決方案,很簡單,以下:github
1.下載winutils的windows版本windows
GitHub上,有人提供了winutils的windows的版本,項目地址是:https://github.com/srccodes/hadoop-common-2.2.0-bin,直接下載此項目的zip包,下載後是文件名是hadoop-common-2.2.0-bin-master.zip,隨便解壓到一個目錄oop
2.配置環境變量spa
增長用戶變量HADOOP_HOME,值是下載的zip包解壓的目錄,而後在系統變量path裏增長$HADOOP_HOME\bin 便可。 debug
再次運行程序,正常執行。調試