Windows 10 上編譯 Hadoop

下載源碼

源碼下載地址（Source download）：https://hadoop.apache.org/releases.html

這裏以 2.9.2 爲例，查看源碼中的編譯說明文件 BUILDING.txt，截取 windows 部分

Requirements:

* Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')

Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK 7.1 is free to download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

The Windows SDK 8.1 is available to download at:

http://msdn.microsoft.com/en-us/windows/bg162891.aspx

Cygwin is neither required nor supported.

 

環境準備

1.JDK

https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

# 設置系統環境變量
setx /M JAVA_HOME "D:\hadoop\jdk1.8.0_192"
setx /M Path "%Path%;%JAVA_HOME%\bin;%JAVA_HOME%\jre\bin"

 

2.Maven

http://maven.apache.org/download.cgi

# 設置系統環境變量
setx /M M2_HOME "D:\hadoop\apache-maven-3.6.1"
setx /M Path "%Path%;%M2_HOME%\bin"

conf\settings.xml 倉庫配置

<!-- 本地倉庫路徑 -->
<localRepository>D:\hadoop\repo</localRepository>

<!-- 網絡倉庫地址 -->
<mirrors>
    <mirror>
        <id>central</id>
        <mirrorOf>central</mirrorOf>
        <name>aliyunmaven</name>
        <url>https://maven.aliyun.com/repository/central</url>
    </mirror>
    <mirror>
        <id>apache.snapshots.https</id>
        <mirrorOf>apache.snapshots.https</mirrorOf>
        <name>aliyunmaven</name>
        <url>https://maven.aliyun.com/repository/apache-snapshots</url>
    </mirror>
</mirrors>

 

3.ProtocolBuffer

https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0

注：除了下載 protobuf 源碼外，還須要下載相應版本的編譯過的用於 Windows 平臺的 protoc 命令（protoc-2.5.0-win32.zip）,該命令用於將 .proto 文件轉化爲 Java 或 C++ 源文件。

將兩個壓縮包解壓，而後將 protoc.exe 複製到 protobuf-2.5.0\src 目錄下

# 安裝 ProtocolBuffer
cd D:\hadoop\protobuf-2.5.0\java
mvn test 
mvn install

# 設置系統環境變量
setx /M Path "%Path%;D:\hadoop\protobuf-2.5.0\src"

# 查看版本
protoc --version

 

4.CMake

https://cmake.org/download/

# 設置系統環境變量
setx /M Path "%Path%;D:\hadoop\cmake-3.14.3-win64-x64\bin"

 

5.Visual Studio 2010 Professional

文件名     cn_visual_studio_2010_professional_x86_dvd_532145.iso
SHA1　　　 33D323446131AB9565082D65C9C380BBD7FF228F
文件大小   2.41GB
發佈時間   2010-05-26

ed2k://|file|cn_visual_studio_2010_professional_x86_dvd_532145.iso|2591844352|6001253431AFE573E4344F5A0B1D9CAC|/

 

6.Zlib

http://www.zlib.net/

使用 MSVC 方式編譯，打開開始菜單，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理員身份運行

cd D:\hadoop\zlib-1.2.11
nmake -f win32/Makefile.msc

# 編譯完成後在該目錄下可看到 zlib1.dll 文件

設置系統環境變量

setx /M ZLIB_HOME "D:\hadoop\zlib-1.2.11"

 

7.Git（須要 bash 命令）

https://git-scm.com/download/

# 設置系統環境變量
setx /M Path "%Path%;D:\hadoop\PortableGit\bin"

 

編譯

1.設置 Platform

setx /M Platform "x64"

 

2.解壓源碼，開始編譯

打開開始菜單，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理員身份運行

cd D:\hadoop\hadoop-2.9.2-src
mvn package -Pdist,native-win -DskipTests -Dtar

# 編譯完成後，編譯好的文件在目錄 hadoop-2.9.2-src\hadoop-dist\target\ 中

編譯失敗時多編譯幾回，或本身更換下 Maven 源再試，一些依賴的下載不是很穩定。附上編譯成功的日誌

[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 39.854 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 35.664 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 13.814 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 13.440 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.205 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 28.307 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 26.019 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [07:47 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [03:03 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.259 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [10:30 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.421 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:23 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.064 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:11 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:45 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  3.279 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.455 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 23.094 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.839 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 12.846 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.044 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.044 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.375 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [04:12 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  3.536 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.050 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 32.678 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 10.878 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  2.167 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 14.856 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 21.482 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.589 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  3.674 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  2.209 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  1.911 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  3.163 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [02:35 min]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.045 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  1.792 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.213 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.042 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.045 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  5.462 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.134 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 23.516 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 13.747 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  2.323 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  6.160 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  4.593 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  2.615 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.280 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.254 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  2.127 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 13.367 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  5.503 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.260 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.315 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.355 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.324 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.477 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.219 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.615 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.040 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  2.513 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [06:23 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 46.260 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [01:14 min]
[INFO] Apache Hadoop Client ............................... SUCCESS [  5.923 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.596 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.338 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [01:43 min]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 30.751 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 13.557 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.047 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 47.214 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.554 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.049 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  55:18 min
[INFO] Finished at: 2019-04-25T22:47:26+08:00
[INFO] ------------------------------------------------------------------------

View Code

 

---

Windows 10編譯Hadoop 2.6.0源碼 - 做業部落 Cmd Markdown 編輯閱讀器 （快照）

http://www.javashuo.com/article/p-ajcuuxbw-q.html