大數據之----部署安裝編譯打包hadoop終極篇

時間 2021-02-15

標籤 java linux ios c++ apache vim 網絡 app 負載均衡 maven 欄目 Hadoop 简体版

原文原文鏈接

------------------------------軟件版本--------------------------------------java

RHEL6.8 hadoop2.8.1 apache-maven-3.3.9

findbugs-1.3.9 protobuf-2.5.0.tar.gz jdk-8u45

------------------------------軟件版本---------------------------------------linux

大數據概述能夠用「5V + 1C」來歸納：ios

Variety 多樣化

Volume 海量

Velocity 快速

Vitality 靈活

Value 價值性

Complexity 複雜

1.Hadoop生態
宏觀: Hadoop爲主的生態圈 hadoop flume........
狹義: apache hadoop hadoop.apache.orgc++

2.Hadoop(存儲+計算+資源和做業調度)
hadoop1.x
    HDFS      存儲
    MapReduce 計算+資源和做業調度apache

hadoop2.x 企業正在用
    HDFS      存儲
    MapReduce 計算
    YARN      資源和做業調度平臺計算組件都會on yarnvim

hadoop3.x ???網絡

EC技術：Erasure Encoding 簡稱EC，是Hadoop3給HDFS拓展的一種新特性，用來解決存儲空間文件。app
YARN：提供YARN的時間軸服務V.2，以便用戶和開發人員能夠對其進行測試，並提供反饋意見。負載均衡
優化Hadoop Shell腳本maven
重構Hadoop Client Jar包
支持隨機Container
MapReduce任務級本地優化
支持多個NameNode
部分默認服務端口被改變
支持文件系統鏈接器
DataNode內部添加了負載均衡
重構後臺程序和任務對管理

2.Maven部署

blog
2.1解壓

[root@hadoop1 softwore]# pwd
/opt/softwore
[root@hadoop1 softwore]# ls
apache-maven-3.3.9-bin.zip hadoop-2.8.1-src.tar.gz jdk-8u45-linux-x64.gz
findbugs-1.3.9.zip hadoop-2.8.1.tar.gz protobuf-2.5.0.tar.gz
[root@hadoop1 softwore]# unzip apache-maven-3.3.9-bin.zip

--配置環境變量

[root@hadoop1 softwore]#

vi /etc/profile
export MAVEN_HOME=/opt/software/apache-maven-3.3.9
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 apache-maven-3.3.9]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"

Maven部署爲二進制部署，解壓配置環境變量

2.2配置mavne目錄
2.3查看配置文件和解壓咱們準備好的倉庫文件

3.Hadoop編譯

編譯hadoop 經過Requirements我看能夠知道須要安裝jdk maven findbugs protocolBuffer cmake Zlib openssl-devel 等軟件
3.1解壓
3.2查看pom.xml
3.3查看BUILDING.txt
Requirements:編譯軟件環境要求

* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

3.4 JDK部署

[root@hadoop1 softwore]# tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java[root@hadoop1 softwore]# ls -ld /usr/java/*
drwxr-xr-x 8 root root 4096 12月 15 2016 /usr/java/djdk1.7.0_79
drwxr-xr-x 8 uucp 143 4096 4月 11 2015 /usr/java/jdk1.8.0_45
drwxr-xr-x 8 uucp 143 4096 10月 7 2015 /usr/java/jdk1.8.0_65
[root@hadoop1 softwore]# vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_45
export JRE_HOME=/usr/java/jdk1.8.0_45/jre

[root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 softwore]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

二進制版本解壓，配置環境變量便可

3.5 MAVEN

上面咱們已經安裝maven和jdk檢測版本是否知足咱們編譯軟件要求
[root@hadoop1 softwore]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
[root@hadoop1 softwore]#

3.6 FINDBUGS

[root@hadoop1 softwore]# unzip findbugs-1.3.9.zip
[root@hadoop1 softwore]# vi /etc/profile
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH [root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 softwore]# findbugs -version
1.3.9

二進制安裝解壓，配置環境變量便可

3.7 PROTOCBUF編譯安裝

[root@hadoop1 software]# tar xvf protobuf-2.5.0.tar.gz
[root@hadoop1 yum.repos.d]# yum install -y gcc gcc-c++ make cmake ---編譯環境
[root@hadoop1 yum.repos.d]#cd /opt/software/protobuf-2.5.0
[root@hadoop1 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf
[root@hadoop1 protobuf-2.5.0]# make && make install
[root@hadoop1 protobuf-2.5.0]# vim /etc/profile
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop1 software]# source /etc/profile
[root@hadoop1 software]# protoc --version
libprotoc 2.5.0
[root@hadoop000 local]#

編譯安裝完成，配置環境變量

3.8 OTHER 依賴包

yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
yum install -y snappy snappy-devel bzip2 bzip2-devel \n
zo lzo-devel lzop autoconf automake

其餘依賴包在系統安裝ios都包含，執行yum安裝便可

---------------------------------------------------------------------------------------------------------------------------------------

至此，編譯hadoop須要的編譯環境所有準備好，下面開啓編譯之路

---------------------------------------------------------------------------------------------------------------------------------------

3.9 編譯

[root@hadoop1 software]# tar xvf hadoop-2.8.1-src.tar.gz
[root@hadoop1 software]# cd hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]#
mvn clean package -Pdist,native -DskipTests –Dtar

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

----編譯好軟件包保存位置

直接編譯須要聯網找依賴包，通常網絡狀況較好，須要幾個小時

提早準備好依賴包編譯方式

[root@hadoop1 software]# tar xvf hadoop-2.8.1-src.tar.gz
[root@hadoop1 software]# cd hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src

[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# ls pom.xml ---根據這個文件下載jar
pom.xml
[root@hado

-rw-r--r-- 1 root root 96721446 4月 25 09:57 .m2.tar.gz ---提早下好的倉庫軟件包

[root@hadoop1 software]#tar xvf .m2.tar.gz –C /root ---不想download，使用提早準備好的倉庫軟件--->解壓----->root家目錄

[root@hadoop1 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests –Dtar

……..

[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.562 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 13.396 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.969 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 6.050 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.110 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 3.862 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.029 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:01 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:45 min
[INFO] Finished at: 2018-04-25T14:03:20+08:00
[INFO] Final Memory: 161M/479M
[INFO] ------------------------------------------------------------------------

[root@hadoop1 hadoop-2.8.1-src]# 編譯成功後的軟件保存路徑

[root@hadoop1 hadoop-2.8.1-src]# ls /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-*
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-javadoc.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-sources.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-test-sources.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1:
bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share
[root@hadoop1 hadoop-2.8.1-src]# du -sh /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
186M /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

總結：完成源碼的編譯好打包，獲得安裝hadoop安裝包，難點在於軟件包的裝備和倉庫軟件的準備，接下來準備開始hadoop安裝之路。