Mac 下編譯 Hadoop

Mac 下編譯 Hadoop-2.9.2

系統環境

系統： Mac OS_10.14.4
maven： Apache Maven 3.6.0
jdk: jdk_1.8.0_201
ProtocolBuffer： portoc-2.5.0
zlib： 1.2.3-lib
OpenSSL： 1_0_2r
cmale： 3.14.3-win64-x64
Cygwin： 2.897_x86_64
hadoop： hadoop-2.9.2
snappy:

搭建編譯環境

1.java環境在oracle官網下載安裝安裝好配置好環境變量

2.maven我用的是brew直接安裝的`brew install maven`

3.cmake 使用brew安裝 `brew install cmake`

4.openssl 使用brew安裝 `brew install openssl`

5.snappy 使用brew安裝 `brew install snappy`

6.zlib 使用brew 安裝 `brew install zlib`

7.protobuf因爲必須使用2.5.0版本須要自行[下載](https://github.com/protocolbuffers/protobuf/releases)安裝

下載後解壓後進入protobuf目錄

# 配置
$ ./configure --prefix=/usr/local/Cellow/protoc/2.5/

# 編譯
$ make

# 安裝
$ make install

# 配置環境變量 `~/.bash_profile` 或 `~/.zshrc`
export PATH="/usr/local/Cellow/protoc/2.5/bin:$PATH"

# 驗證protoc版本
$ protoc --version

過程當中可能遇到的問題

• 1.[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1

[ERROR] around Ant part ...<exec dir="/Users/Downloads/hadoop-2.9.2/hadoop-tools/hadoop-pipes/target/native" executable="cmake" failonerror="true">... @ 5:153 in /Users/Downloads/hadoop-2.9.2/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

解決辦法： 因爲這個xml中的編譯指令中須要用到一個環境變量OPENSSL_ROOT_DIR因爲咱們前面沒有配置因此他在執行的時候就會報錯，須要在~/.bash_profile 或 ~/.zshrc中指定OPENSSL_ROOT_DIR的安裝目錄個人配置是export OPENSSL_ROOT_DIR="/usr/local/Cellor/openssl/1.0.2r"

• 2.因爲OpenSSL爆出有重大的漏洞Mac OS沒有使用OpenSSL而是使用了LibreSSl能夠在你的Mac下執行which openssl 而後找到/usr/bin/openssl 執行/usr/bin/openssl version 就會出現LibreSSl 2.6.4，可是編譯hadoop必需要使用openssl，能夠吧openssl的引用改一個名稱，可是Mac是不容許你關閉Mac的保護模式可是不建議這樣作，咱們能夠修改系統的shell的環境變量使本身配置的環境變量覆蓋系統的環境變量，例如：export PATH="/usr/local/Cellor/openssl/1.0.2r/bin:$PATH"

編譯hadoop

# 下載[hadoop源碼](https://hadoop.apache.org/releases.html)

# 解壓縮後進入hadoop源碼路徑

# 執行一條maven命令（你的maven須要配置到環境變量中）
$ mvn package -Pdist,native -DskipTests -Dtar

# 編譯過程不會太長，可是下載各類依賴就會浪費很長時間，耐心等待！

# 過程當中可能會出現各各樣的問題我只是列舉了我花費比較久時間解決的問題，若是沒法解決能夠留言。

編譯完成

通過漫長的編譯終於出現了使人興奮的畫面

[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.679 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.158 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.071 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.897 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.297 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.699 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  4.280 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.940 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  4.245 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.548 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:05 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.707 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  9.596 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.046 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 18.270 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 46.683 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  7.154 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 14.156 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  3.215 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.857 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.849 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.100 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.073 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 10.318 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 23.624 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  3.413 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.091 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  8.550 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 26.322 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  2.491 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  4.873 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  3.360 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 16.652 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.993 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  4.276 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  2.371 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  2.464 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  3.747 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  5.310 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  1.894 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.053 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  2.099 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.366 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.061 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.098 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  7.639 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.157 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 15.248 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  9.780 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  2.561 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  8.028 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  4.123 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  4.353 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.349 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  3.550 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  3.609 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  2.987 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  3.530 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.456 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.654 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.363 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.833 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.550 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.378 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.101 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  6.900 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  2.804 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  7.562 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  4.022 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  9.670 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  5.619 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.638 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.075 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 10.540 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 12.875 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.670 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.058 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:02 min]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.473 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.042 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  09:17 min
[INFO] Finished at: 2019-05-06T05:03:29-07:00
[INFO] ------------------------------------------------------------------------
MacBookPro:hadoop-2.9.2-src leone$

大功告成編譯後的hadoop放在hadoop-2.9.2-src\hadoop-dist\target目錄下。