Windows下編譯 Hadoop

時間 2019-11-08

原文原文鏈接

Windows下編譯 Hadoop-2.9.2

系統環境

系統： Windows 10 10.0_x64
maven： Apache Maven 3.6.0
jdk: jdk_1.8.0_201
ProtocolBuffer： portoc-2.5.0
zlib： 1.2.3-lib
OpenSSL： 1_0_2r
cmake： 3.14.3-win64-x64
Cygwin： 2.897_x86_64
Visual Studio： Visual Studio 2010 Professional
hadoop： hadoop-2.9.2
複製代碼

Hadoop源碼包大家的的編譯環境要求

Building on Windows

----------------------------------------------------------------------------------
Requirements:

* Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
  tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')

Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK 7.1 is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

The Windows SDK 8.1 is available to download at:

http://msdn.microsoft.com/en-us/windows/bg162891.aspx

Cygwin is neither required nor supported.

複製代碼

編譯必需要設置的環境變量

1.JAVA_HOME必需要設置這個環境變量指向jdk的安裝目錄html
2.Platform這個環境變量也必須設置32位系統爲Platform=Win3264位系統爲Platform=x64git
3.ZLIB_HOME爲你zlib的安裝目錄例如ZLIB_HOME=C:\zlib-1.2.7github

編譯須要注意的點

1.保證你hadoop源碼的路徑儘可能短，好比放在某個盤的根目錄apache
2.保證你的maven倉庫的位置儘可能短，好比放在某個盤的根目錄（注意設置maven的conf文件）windows

編譯過程當中可能出現的問題

1.Command execution failed. Cannot run program "msbuild" (in directory "C:\hadoop-2.9.2-src\hadoop-common-project\hadoop-common")

解決方法：將C:\Windows\Microsoft.NET\Framework\v4.0.30319放入PATH環境變量bash

2.(compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]

解決方法：這個問題是因爲Visual Studio沒有安裝或安裝的版本不對，換成對應的版本就行了網絡

3.(compile-ms-native-dll) on project hadoop-common: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]

解決方法：這個問題是因爲zlib環境的問題，可能缺乏ZLIB_HOME或者你下載的zlib的版本不正確或是你的zlib環境變量大家沒有包含zlib.h或缺乏unistd.h和getopt.h這兩個文件可到GitHub上下載這兩個文件.maven

4.(dist) on project hadoop-kms: An Ant BuildException has occured: exec returned: 2

解決方法：這個問題是因爲在編譯hadoop-kms這個模塊的時候下載Tomcat是因爲網絡問題出現網絡斷開鏈接從新編譯也會出現這個問題，能夠刪除源碼從新下載對應的源碼進行編譯（簡單粗暴）oop

編譯

下載hadoop源碼選擇對應的版本，解壓源碼後放在某個盤的根目錄，進入源碼的根目錄執行mvn package -Pdist,native-win -DskipTests -Dtar後maven就開始去遠程倉庫下載須要的依賴這個過程可能須要好久跟帶寬速度和牆有很大關係，過程當中可能會出現各類問題可是本身都完美解決了，hadoop2.x的版本中我編譯了hadoop2.7.七、hadoop-2.8.八、hadoop-2.9.2這三個版本都是編譯成功的，可是在編譯hadoop3.x版本就會失敗，可能3.x對環境的需求相對於2.x有比較大的改變，後續再研究。。。fetch

通過漫長的等待終於出現了久違的畫面

[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [  0.928 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.579 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.780 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.413 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.278 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.491 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.365 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  3.223 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  4.977 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.304 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:08 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.788 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  9.379 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.052 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 23.895 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 48.088 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 25.830 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 25.398 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 10.032 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.641 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.092 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.053 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.054 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 19.308 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 53.206 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  3.731 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.056 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 16.246 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 20.362 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  2.566 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  6.348 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  3.742 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 25.768 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  1.535 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  6.465 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  3.396 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  3.146 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  3.784 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  4.994 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  1.584 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.336 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  2.052 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.535 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.052 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.050 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  7.322 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.128 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 32.088 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 19.070 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  2.889 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  8.415 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  5.543 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  4.486 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.283 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.521 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  2.765 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.277 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  3.957 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.462 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.628 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 14.629 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.915 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.689 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.238 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.851 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.052 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.016 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  6.852 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  4.349 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  2.039 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  6.595 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.961 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  7.219 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  3.102 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  5.460 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.486 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.075 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 51.638 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.581 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.055 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11:01 min
[INFO] Finished at: 2019-05-08T11:42:52+08:00
[INFO] ------------------------------------------------------------------------

C:\Users\Andy\Downloads\hadoop-2.9.2-src>
複製代碼