ubuntu18.04編譯hadoop2.7.6支持snappy

時間  2019-11-19
標籤 ubuntu18.04 ubuntu 編譯 hadoop2.7.6 hadoop 支持 snappy 欄目 Ubuntu 简体版
原文   原文鏈接

官網下載

http://hadoop.apache.org/releases.htmlhtml

解壓

hadoop解壓

#讀取 BUILDING.txt文件 上圖中用紅色框選中的文件，是官方的編譯步走java

Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements:

* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
* Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

----------------------------------------------------------------------------------
Installing required packages for clean install of Ubuntu 14.04 LTS Desktop:

* Oracle JDK 1.7 (preferred)
  $ sudo apt-get purge openjdk*
  $ sudo apt-get install software-properties-common
  $ sudo add-apt-repository ppa:webupd8team/java
  $ sudo apt-get update
  $ sudo apt-get install oracle-java7-installer
* Maven
  $ sudo apt-get -y install maven
* Native libraries
  $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev
* ProtocolBuffer 2.5.0 (required)
  $ sudo apt-get -y install libprotobuf-dev protobuf-compiler

Optional packages:

* Snappy compression
  $ sudo apt-get install snappy libsnappy-dev
* Bzip2
  $ sudo apt-get install bzip2 libbz2-dev
* Jansson (C Library for JSON)
  $ sudo apt-get install libjansson-dev
* Linux FUSE
  $ sudo apt-get install fuse libfuse-dev

安轉依賴

jdk、maven就不說了。不會的話你也別學hadoop了。linux

本地依賴

  • sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev snappy libsnappy-dev bzip2 libbz2-dev libjansson-dev fuse libfuse-dev

注意 libssl-dev安裝換成 libssl1.0-dev版本。 我以前就是默認安轉，在編譯OpenSSL功能的時候一直卡主編譯不經過。具體錯誤形式以下git

project/hadoop-common/src/main/native/src -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/src -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native -I/opt/jdk1.8.0_171/include -I/opt/jdk1.8.0_171/include/linux -I/usr/local/include -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util  -g -Wall -O2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64   -o CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c.o   -c /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
     [exec] [ 19%] Building C object CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o
     [exec] /usr/bin/cc  -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native/javah -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/src -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native -I/opt/jdk1.8.0_171/include -I/opt/jdk1.8.0_171/include/linux -I/usr/local/include -I/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util  -g -Wall -O2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64   -o CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o   -c /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
     [exec] CMakeFiles/hadoop_static.dir/build.make:230: recipe for target 'CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o' failed
     [exec] make[2]: 離開目錄「/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native」
     [exec] CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/hadoop_static.dir/all' failed
     [exec] make[1]: 離開目錄「/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native」
     [exec] Makefile:83: recipe for target 'all' failed
     [exec] /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c: In function ‘check_update_max_output_len’:
     [exec] /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14: error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct evp_cipher_ctx_st}’
     [exec]    if (context->flags & EVP_CIPH_NO_PADDING) {
     [exec]               ^~
     [exec] /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:275:1: warning: control reaches end of non-void function [-Wreturn-type]
     [exec]  }
     [exec]  ^
     [exec] /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c: In function ‘check_doFinal_max_output_len’:
     [exec] /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:320:1: warning: control reaches end of non-void function [-Wreturn-type]
     [exec]  }
     [exec]  ^
     [exec] make[2]: *** [CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o] Error 1
     [exec] make[1]: *** [CMakeFiles/hadoop_static.dir/all] Error 2
     [exec] make: *** [all] Error 2
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.675 s
[INFO] Finished at: 2018-07-19T20:27:17+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part ...<exec failonerror="true" dir="/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native" executable="make">... @ 7:140 in /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2
around Ant part ...<exec failonerror="true" dir="/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native" executable="make">... @ 7:140 in /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 2
around Ant part ...<exec failonerror="true" dir="/home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/native" executable="make">... @ 7:140 in /home/benny/software/hadoop-2.7.6-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
    at org.apache.maven.plugin.antrun.AntRunMojo.execute (AntRunMojo.java:355)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.tools.ant.BuildException: exec returned: 2
    at org.apache.tools.ant.taskdefs.ExecTask.runExecute (ExecTask.java:646)
    at org.apache.tools.ant.taskdefs.ExecTask.runExec (ExecTask.java:672)
    at org.apache.tools.ant.taskdefs.ExecTask.execute (ExecTask.java:498)
    at org.apache.tools.ant.UnknownElement.execute (UnknownElement.java:291)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.apache.tools.ant.dispatch.DispatchUtils.execute (DispatchUtils.java:106)
    at org.apache.tools.ant.Task.perform (Task.java:348)
    at org.apache.tools.ant.Target.execute (Target.java:390)
    at org.apache.tools.ant.Target.performTasks (Target.java:411)
    at org.apache.tools.ant.Project.executeSortedTargets (Project.java:1399)
    at org.apache.tools.ant.Project.executeTarget (Project.java:1368)
    at org.apache.maven.plugin.antrun.AntRunMojo.execute (AntRunMojo.java:327)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:

安裝protobuf

下載

  1. googletest下載,protobuf編譯須要依賴它， 下載地址：https://github.com/google/googletest/tree/release-1.7.0 最好選擇1.7.0版本，1.7.0版本後目錄結構有變化。github

  2. protobuf下載 官方文檔中說能夠經過下面命令安裝，可是我發現libprotobuf-dev沒有，只好手動安裝web

$ sudo apt-get -y install libprotobuf-dev protobuf-compiler

protobuf下載地址 https://github.com/google/protobuf/tree/v2.5.0 最好選擇2.5.0版本與官方一致， 解壓後，把前面下載的googletest複製到protobuf的跟目錄，而且重命名爲gtestshell

編譯

  1. 運行autogen.sh 生成configure
  2. 運行configure
  3. make
  4. sudo make install 驗證protobuf是否安裝成功
benny@benny-ubuntu:~/software/protobuf-2.5.0$ protoc --version
libprotoc 2.5.0

可看到安轉成功了apache

安裝snappy

  1. 下載地址 https://github.com/google/snappy
  2. 編譯安轉和protobuf同樣。 configure、 make、 sudo make install 安裝完成能夠驗證
-rw-r--r-- 1 root root 616634 7月  19 17:06 /usr/local/lib/libsnappy.a
-rwxr-xr-x 1 root root    945 7月  19 17:06 /usr/local/lib/libsnappy.la*
lrwxrwxrwx 1 root root     18 7月  19 17:06 /usr/local/lib/libsnappy.so -> libsnappy.so.1.3.0*
lrwxrwxrwx 1 root root     18 7月  19 17:06 /usr/local/lib/libsnappy.so.1 -> libsnappy.so.1.3.0*
-rwxr-xr-x 1 root root 304936 7月  19 17:06 /usr/local/lib/libsnappy.so.1.3.0*

#編譯hadoop 編譯以前，刷新一下動態連接庫。ubuntu

sudo ldconfig

到如今，準備工做算是作完了。maven命令編譯。oracle

進入hadoop的根目錄執行下面的maven命令

mvn clean package -DskipTests -Pdist,native -Dtar -Dsnappy.lib -Drequire.snappy -Dbundle.snappy -Drequire.openssl -Dbundle.openssl -Dopenssl.lib

編譯完成後./hadoop-dist/target/ 目錄下就是最終生成的文件了。

benny@benny-ubuntu:~/software/hadoop-2.7.6-src$ ll ./hadoop-dist/target/
總用量 585192
drwxr-xr-x 10 benny benny      4096 7月  20 14:57 ./
drwx------  4 benny benny      4096 7月  20 14:57 ../
drwxr-xr-x  2 benny benny      4096 7月  20 14:57 antrun/
drwxr-xr-x  3 benny benny      4096 7月  20 14:57 classes/
-rw-r--r--  1 benny benny      1877 7月  20 14:57 dist-layout-stitching.sh
-rw-r--r--  1 benny benny       650 7月  20 14:57 dist-tar-stitching.sh
drwxr-xr-x  9 benny benny      4096 7月  20 17:25 hadoop-2.7.6/
-rw-r--r--  1 benny benny 199256146 7月  20 14:57 hadoop-2.7.6.tar.gz
-rw-r--r--  1 benny benny     26522 7月  20 14:57 hadoop-dist-2.7.6.jar
-rw-r--r--  1 benny benny 399837851 7月  20 14:57 hadoop-dist-2.7.6-javadoc.jar
-rw-r--r--  1 benny benny     24051 7月  20 14:57 hadoop-dist-2.7.6-sources.jar
-rw-r--r--  1 benny benny     24051 7月  20 14:57 hadoop-dist-2.7.6-test-sources.jar
drwxr-xr-x  2 benny benny      4096 7月  20 14:57 javadoc-bundle-options/
drwxr-xr-x  2 benny benny      4096 7月  20 14:57 maven-archiver/
drwxr-xr-x  3 benny benny      4096 7月  20 14:57 maven-shared-archive-resources/
-rw-r--r--  1 benny benny        30 7月  20 14:57 .plxarc
drwxr-xr-x  3 benny benny      4096 7月  20 14:57 test-classes/
drwxr-xr-x  2 benny benny      4096 7月  20 14:57 test-dir/
benny@benny-ubuntu:~/software/hadoop-2.7.6-src$

支持snappy的本地動態連接庫位於./hadoop-dist/target/hadoop-2.7.6/lib/native/

benny@benny-ubuntu:~/software/hadoop-2.7.6-src$ ll ./hadoop-dist/target/hadoop-2.7.6/lib/native/*
-rw-r--r-- 1 benny benny 1521778 7月  20 14:57 ./hadoop-dist/target/hadoop-2.7.6/lib/native/libhadoop.a
-rw-r--r-- 1 benny benny 1752088 7月  20 14:57 ./hadoop-dist/target/hadoop-2.7.6/lib/native/libhadooppipes.a
lrwxrwxrwx 1 benny benny      18 7月  20 14:57 ./hadoop-dist/target/hadoop-2.7.6/lib/native/libhadoop.so -> libhadoop.so.1.0.0*
-rwxr-xr-x 1 benny benny  866704 7月  20 14:57 ./hadoop-dist/target/hadoop-2.7.6/lib/native/libhadoop.so.1.0.0*
-rw-r--r-- 1 benny benny  751080 7月  20 14:57 ./hadoop-dist/target/hadoop-2.7.6/lib/native/libhadooputils.a
-rw-r--r-- 1 benny benny  464116 7月  20 14:57 ./hadoop-dist/target/hadoop-2.7.6/lib/native/libhdfs.a
lrwxrwxrwx 1 benny benny      16 7月  20 14:57 ./hadoop-dist/target/hadoop-2.7.6/lib/native/libhdfs.so -> libhdfs.so.0.0.0*
-rwxr-xr-x 1 benny benny  286408 7月  20 14:57 ./hadoop-dist/target/hadoop-2.7.6/lib/native/libhdfs.so.0.0.0*
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程