ubuntu14.04下編譯安裝ambari-2.4.2.0

時間 2019-11-19

標籤 ubuntu14.04 ubuntu 編譯安裝 ambari 2.4.2.0 欄目 Ubuntu 简体版

原文原文鏈接

ubuntu14.04下編譯安裝ambari-2.4.2.0

編譯前的準備工做

準備工做有：html

下面是具體配置java

系統參數

一、fs.inotify.max_user_watches = 8192 調整到524288
echo fs.inotify.max_user_watches=524288 >> /etc/sysctl.conf && sysctl -pnode

二、將每一個進程能夠打開的文件數目加大到65535，缺省爲1024python

ulimit -SHn 65535
sudo tee /etc/security/limits.d/90-nproc.conf <<-'EOF'
* hard nproc 64000
* soft nproc 64000
root hard nproc 65535
root soft nproc 65535
EOF

ulimit -a

系統依賴（編譯環境）

sudo apt-get install build-essential cmake automake autoconf bison libboost-all-dev libtool check libmcrypt-dev libeditline-dev libssl-devlinux

離線安裝包

1. httpserver

sh starthttpd.shgit

#/usr/bin
cd /media/gordon/DataDisk/軟件包
/usr/bin/python -m SimpleHTTPServer 8000 >/dev/null 2>&1 &
exit 0

2. 佈置本地軟件源

http://192.168.56.1:8000/ambari-HDP/
-rwxr-x--- 1 gitlab root 23667150 3月 31 16:26 grafana-2.6.0.linux-x64.tar.gz
-rwxr-x--- 1 gitlab root 210185315 3月 31 17:15 hadoop-2.7.1.2.3.4.0-3347.tar.gz
-rwxr-x--- 1 gitlab root 101551599 3月 31 16:26 hbase-1.1.2.2.3.4.0-3347.tar.gz
-rwxr-x--- 1 gitlab root 251655846 3月 31 16:26 phoenix-4.4.0.2.3.4.0-3347.tar.gzgithub

java環境

vim /etc/profileweb

#set for java
export JAVA_HOME=/opt/jdk1.8.0_111
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"

maven環境

vim /etc/profileshell

#set maven environment
M2_HOME=/opt/apache-maven-3.3.9/
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$M2_HOME/bin:$PATH

maven有2個位置能夠選擇配置M2_HOME/conf/setting.xml和~/.m2/setting.xml
前者是全局性的，後者是針對用戶的。
查詢資料是說用戶配置優先級高於全局配置apache

設置maven-repo本地源

vim /opt/apache-maven-3.3.9/conf/settings.xml
  <localRepository>/opt/maven-repo</localRepository>

gord@gord:/opt$ mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/apache-maven-3.3.9
Java version: 1.8.0_111, vendor: Oracle Corporation
Java home: /opt/jdk1.8.0_111/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-24-generic", arch: "amd64", family: "unix"

Nodejs環境

今天是2017.04.01 https://nodejs.org/en/官網推薦v6.10.1版本 #實驗證實這個版本安裝ambari存在很大問題，切換到0.10.44版本的nodejs
可在淘寶下載各個版本：https://npm.taobao.org/mirrors/node

tar xfp node-v0.10.44-linux-x64.tar.gz -C /opt
vim /etc/profile
#set for nodejs
export NODE_HOME=/opt/node-v0.10.44-linux-x64
#export NODE_HOME=/opt/node-v4.4.4-linux-x64
#export NODE_HOME=/opt/node-v6.10.1-linux-x64
export PATH=$PATH:$NODE_HOME/bin


source /etc/profile

配置淘寶源

kylin@Ubuntu:~$ npm config set registry https://registry.npm.taobao.org  
kylin@Ubuntu:~$ npm info underscor  （若是上面配置正確這個命令會有字符串response）
gord@gord:~$ npm --registry https://registry.npm.taobao.org info underscore

修改源地址爲官方源
npm config set registry https://registry.npmjs.org/

安裝指定版本的brunch

npm root -g
cd ...
npm install -g brunch@1.7.20  #務必要裝這個版本
brunch -V
1.7.20

若是存在問題，執行卸載brunch
npm remove -g brunc

git環境

root@gord:~# git --version
git version 1.9.1

ant環境

root@gord:~# apt-get install ant
root@gord:~# ant -version
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Apache Ant(TM) version 1.9.3 compiled on April 8 2014

python環境

apt install python-pip

準備工做總結：

profile配置總結

#set for java
export JAVA_HOME=/opt/jdk1.8.0_111
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"
#set for maven
M2_HOME=/opt/apache-maven-3.3.9/
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$M2_HOME/bin:$PATH
#set for nodejs
export NODE_HOME=/opt/node-v0.10.44-linux-x64
#export NODE_HOME=/opt/node-v4.4.4-linux-x64
#export NODE_HOME=/opt/node-v6.10.1-linux-x64
export PATH=$PATH:$NODE_HOME/bin

至此編譯ambari所依賴的環境配置完畢，並可做爲通用的編譯環境推廣使用。
推廣到其它編譯環境中，注意nodejs的版本可能須要更換！
準備工做相當重要，關係到編譯的成敗和時間的消耗！

編譯安裝ambari

主要步驟：

參考

https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.4.2

版本控制

wget http://www.apache.org/dist/ambari/ambari-2.4.2/apache-ambari-2.4.2-src.tar.gz (use the suggested mirror from above)
tar xfvz apache-ambari-2.4.2-src.tar.gz
cd apache-ambari-2.4.2-src
mvn versions:set -DnewVersion=2.4.2.0.0
 
root@gord:/opt/apache-ambari-2.4.2-src# cd ambari-metrics
/opt/apache-ambari-2.4.2-src/ambari-metrics /opt/apache-ambari-2.4.2-src

mvn versions:set -DnewVersion=2.4.2.0.0

root@gord:/opt/apache-ambari-2.4.2-src/ambari-metrics# cd ..
/opt/apache-ambari-2.4.2-src

編譯

-B非交互
mvn -B clean install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6"
-X -e debug
mvn -X -e clean install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6"

若是存在手工下載軟件包到指定位置的狀況，上述命令去掉clean參數，防止被clean

編譯成功顯示

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Ambari Main ........................................ SUCCESS [  2.741 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [  0.155 s]
[INFO] Ambari Web ......................................... SUCCESS [01:00 min]
[INFO] Ambari Views ....................................... SUCCESS [  1.475 s]
[INFO] Ambari Admin View .................................. SUCCESS [ 19.390 s]
[INFO] ambari-metrics ..................................... SUCCESS [  0.770 s]
[INFO] Ambari Metrics Common .............................. SUCCESS [  0.732 s]
[INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [  4.208 s]
[INFO] Ambari Metrics Flume Sink .......................... SUCCESS [  2.511 s]
[INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [  1.676 s]
[INFO] Ambari Metrics Storm Sink .......................... SUCCESS [  4.837 s]
[INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [  3.420 s]
[INFO] Ambari Metrics Collector ........................... SUCCESS [ 23.296 s]
[INFO] Ambari Metrics Monitor ............................. SUCCESS [  1.944 s]
[INFO] Ambari Metrics Grafana ............................. SUCCESS [  2.544 s]
[INFO] Ambari Metrics Assembly ............................ SUCCESS [02:26 min]
[INFO] Ambari Server ...................................... SUCCESS [04:24 min]
[INFO] Ambari Functional Tests ............................ SUCCESS [  1.385 s]
[INFO] Ambari Agent ....................................... SUCCESS [ 31.964 s]
[INFO] Ambari Client ...................................... SUCCESS [  0.084 s]
[INFO] Ambari Python Client ............................... SUCCESS [  1.168 s]
[INFO] Ambari Groovy Client ............................... SUCCESS [  8.157 s]
[INFO] Ambari Shell ....................................... SUCCESS [  0.105 s]
[INFO] Ambari Python Shell ................................ SUCCESS [  1.422 s]
[INFO] Ambari Groovy Shell ................................ SUCCESS [  3.714 s]
[INFO] ambari-logsearch ................................... SUCCESS [  0.041 s]
[INFO] Ambari Logsearch Appender .......................... SUCCESS [ 31.627 s]
[INFO] Ambari Logsearch Solr Client ....................... SUCCESS [01:38 min]
[INFO] Ambari Logsearch Portal ............................ SUCCESS [06:10 min]
[INFO] Ambari Logsearch Log Feeder ........................ SUCCESS [06:03 min]
[INFO] Ambari Logsearch Assembly .......................... SUCCESS [  0.323 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:14 min
[INFO] Finished at: 2017-04-05T18:44:18+08:00
[INFO] Final Memory: 274M/850M
[INFO] ------------------------------------------------------------------------

整理deb包

root@gord:/opt/apache-ambari-2.4.2-src# find ./ -name "*.deb" |awk '{print "cp",$0,"/media/sf_DataDisk/軟件包/ambari-repo/Compiled/"}'|sh
./target/ambari_2.4.2.0.0_all.deb
./ambari-metrics/target/ambari-metrics_2.4.2.0.0_all.deb
./ambari-metrics/ambari-metrics-assembly/target/ambari-metrics-assembly_2.4.2.0.0_all.deb
./ambari-agent/target/ambari-agent_2.4.2.0-0.deb
./ambari-shell/target/ambari-shell_2.4.2.0.0_all.deb
./ambari-shell/ambari-python-shell/target/ambari-python-shell_2.4.2.0-0.deb
./ambari-project/target/ambari-project_2.4.2.0.0_all.deb
./ambari-server/target/ambari-server_2.4.2.0-0-dist.deb
./ambari-client/target/ambari-client_2.4.2.0.0_all.deb
./ambari-client/python-client/target/python-client_2.4.2.0-0.deb
./ambari-client/groovy-client/target/groovy-client_2.4.2.0.0_all.deb

安裝

服務端安裝

##gdebi ambari-server*   #This should also pull in postgres packages as well.
dpkg -i ambari-server_2.4.2.0-0-dist.deb
apt-get -f install

設置與啓動
ambari-server setup #默認便可

ambari-server start
若是遇到報錯：/usr/sbin/ambari-server: line 33: buildNumber: unbound variable
打開ambari-server腳本，把${buildNumber}這行換成 HASH="${VERSION}"

相關的命令有如下：

ambari-server status
ambari-server stop

客戶端安裝

dpkg -i ambari-agent*
apt-get -f install

編輯ambari-agent配置文件，設置以前安裝過Ambari-server 的主機名：

vi /etc/ambari-agent/conf/ambari-agent.ini

[server]

hostname=<your.ambari.server.hostname>

url_port=8440

secured_url_port=8441

ambari-agent start
若是遇到報錯：/usr/sbin/ambari-agent: line 23: buildNumber: unbound variable
打開ambari-agent腳本，把${buildNumber}這行換成 HASH="${VERSION}"

相關的命令有如下：

ambari-agent status # 檢查agent服務運行狀態
ambari-agent stop   # 中止agent服務

訪問

http://<ambari-server-host>:8080.
Log in with username admin and password admin

問題解決

一、Caused by: org.vafer.jdeb.PackagingException: 「/opt/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana/src/main/package/deb/control」 is not a valid ‘control’ directory)
/opt/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana# vim pom.xml

<plugin>
         <groupId>org.vafer</groupId>
         <artifactId>jdeb</artifactId>
         <version>1.0.1</version>
         <executions>
           <execution>
             <!--Stub execution on direct plugin call - workaround for ambari deb build process-->
             <id>stub-execution</id>
             <phase>none</phase>
             <goals>
               <goal>jdeb</goal>
             </goals>
           </execution>
         </executions>
         <configuration>
           <skip>true</skip>
           <attach>false</attach>
           <submodules>false</submodules>
           <controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
         </configuration>
       </plugin>

二、若是中間斷掉，建議去掉clean，繼續編譯。

三、安裝Nodejs 6.x.x版本不能編譯，建議卸載，安裝0.10.44。
[INFO] Ambari Web ......................................... FAILURE [ 13.325 s]
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (compile-brunch) on project ambari-web: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
版本檢查：

root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# brunch --version
2.10.9
root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# npm --version
3.10.10
root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# node --version
v6.10.1

改爲安裝node 0.10.44，brunch1.7.20後正常

四、編譯過程當中出現版本不一致問題，請參考http://blog.csdn.net/royma_1990/article/details/51749981。

五、下載慢問題處理
修改以下配置文件：
cat ambari-metrics/pom.xml

<!--TODO change to HDP URL-->
    <hbase.tar>http://192.168.56.1:8000/ambari-HDP/hbase-1.1.2.2.3.4.0-3347.tar.gz</hbase.tar>
    <hbase.folder>hbase-1.1.2.2.3.4.0-3347</hbase.folder>
    <hadoop.tar>http://192.168.56.1:8000/ambari-HDP/hadoop-2.7.1.2.3.4.0-3347.tar.gz</hadoop.tar>
    <hadoop.folder>hadoop-2.7.1.2.3.4.0-3347</hadoop.folder>
    <hbase.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hbase-0.98.4.2.2.4.2-0002-hadoop2.winpkg.zip</hbase.winpkg.zip>
    <hbase.winpkg.folder>hbase-0.98.4.2.2.4.2-0002-hadoop2</hbase.winpkg.folder>
    <hadoop.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hadoop-2.6.0.2.2.4.2-0002.winpkg.zip</hadoop.winpkg.zip>
    <hadoop.winpkg.folder>hadoop-2.6.0.2.2.4.2-0002</hadoop.winpkg.folder>
    <grafana.folder>grafana-2.6.0</grafana.folder>
    <grafana.tar>http://192.168.56.1:8000/ambari-HDP/grafana-2.6.0.linux-x64.tar.gz</grafana.tar>
    <phoenix.tar>http://192.168.56.1:8000/ambari-HDP/phoenix-4.4.0.2.3.4.0-3347.tar.gz</phoenix.tar>
    <phoenix.folder>phoenix-4.4.0.2.3.4.0-3347</phoenix.folder>

六、警告：（可忽略）
[INFO] npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.

root@gord:/opt/apache-ambari-2.4.2-src/ambari-web# npm ls graceful-fs
Ambari@2.4.0 /opt/apache-ambari-2.4.2-src/ambari-web
├─┬ karma@0.11.14
│ └── graceful-fs@2.0.3 
├─┬ karma-coverage@0.2.7
│ └─┬ dateformat@1.0.12
│   └─┬ meow@3.7.0
│     └─┬ read-pkg-up@1.0.1
│       └─┬ read-pkg@1.1.0
│         ├─┬ load-json-file@1.1.0
│         │ └── graceful-fs@4.1.11 
│         └─┬ path-type@1.1.0
│           └── graceful-fs@4.1.11 
├─┬ mocha-phantomjs@3.1.6
│ └─┬ mocha@1.13.0
│   └─┬ glob@3.2.3
│     └── graceful-fs@2.0.3 
└─┬ phantomjs@1.9.20
  └─┬ fs-extra@0.26.7
    └── graceful-fs@4.1.11

七、找不到org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT
[INFO] Ambari Metrics Storm Sink .......................... FAILURE [ 4.111 s]
[ERROR] Failed to execute goal on project ambari-metrics-storm-sink: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-storm-sink:jar:2.4.2.0.0: Could not find artifact org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT in apache-hadoop (http://repo.hortonworks.com/content/repositories/releases/org/apache/) -> [Help 1]
vim ambari-metrics/ambari-metrics-storm-sink/pom.xml

<properties>
    <!--storm.version>1.1.0-SNAPSHOT</storm.version-->
    <storm.version>1.1.0</storm.version>
  </properties>

八、ambari-funtest
[ERROR] Failed to create debian package /home/kylin/apache-ambari-2.4.2-src/ambari-funtest/target/ambari-funtest_2.4.2.0.0_all.deb
org.vafer.jdeb.PackagingException: 「/home/kylin/apache-ambari-2.4.2-src/ambari-funtest/src/main/package/deb/control」 is not a valid ‘control’ directory)
解決辦法：在上面添加一個元素
/opt/apache-ambari-2.4.2-src/ambari-funtest$ vi pom.xml

<plugins>
      ...           
      <plugin>
         <groupId>org.vafer</groupId>
         <artifactId>jdeb</artifactId>
         <version>1.0.1</version>
         <executions>
           <execution>
             <!--Stub execution on direct plugin call - workaround for ambari deb build process-->
             <id>stub-execution</id>
             <phase>none</phase>
             <goals>
               <goal>jdeb</goal>
             </goals>
           </execution>
         </executions>
         <configuration>
           <skip>true</skip>
           <attach>false</attach>
           <submodules>false</submodules>
           <controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
         </configuration>
       </plugin>
    </plugins>
    <resources>
      <resource>

十、[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /opt/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: java.net.UnknownHostException: www.w3.org: Unknown host www.w3.org -> [Help 1]

解決辦法：從新執行
mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true

十一、findbugsXml.html
[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /opt/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: The entity name must immediately follow the ‘&’ in the entity reference. -> [Help 1]
解決辦法1：從新執行
mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true
解決辦法2: 若是網絡狀況很差，這一步就很難過去。那隻好修改pom.xml文件，將findbugs內容註釋掉：
/opt/apache-ambari-2.4.2-src/ambari-server$ vim pom.xml

<plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>findbugs-maven-plugin</artifactId>
        <version>3.0.3</version>
        <configuration>
          <failOnError>false</failOnError>
          <threshold>Low</threshold>
          <findbugsXmlOutputDirectory>${project.basedir}/target/findbugs</findbugsXmlOutputDirectory>
        </configuration>
        <executions>
          <!--  
          <execution>
            <phase>verify</phase>
            <goals>
              <goal>check</goal>
            </goals>
          </execution>
          -->
        </executions>
      </plugin>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>xml-maven-plugin</artifactId>
        <version>1.0</version>
        <executions>
           <!--
           <execution>
            <phase>verify</phase>
            <goals>
              <goal>transform</goal>
            </goals>
          </execution>
          -->
        </executions>

十二、[ERROR] Failed to execute goal org.vafer:jdeb:1.0.1:jdeb (default-cli) on project ambari-logsearch: Failed to create debian package /opt/apache-ambari-2.4.2-src/ambari-logsearch/target/ambari-logsearch_2.4.2.0.0_all.deb: "/opt/apache-ambari-2.4.2-src/ambari-logsearch/src/main/package/deb/control" is not a valid 'control' directory) -> [Help 1]

解決辦法：在上面添加一個元素

/opt/apache-ambari-2.4.2-src/ambari-logsearch$ vim pom.xml

<plugin>
         <groupId>org.vafer</groupId>
         <artifactId>jdeb</artifactId>
         <version>1.0.1</version>
         <executions>
           <execution>
             <!--Stub execution on direct plugin call - workaround for ambari deb build process-->
             <id>stub-execution</id>
             <phase>none</phase>
             <goals>
               <goal>jdeb</goal>
             </goals>
           </execution>
         </executions>
         <configuration>
           <skip>true</skip>
           <attach>false</attach>
           <submodules>false</submodules>
           <controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
         </configuration>
       </plugin>

    </plugins>
  </build>

參考資源：

https://github.com/apache/ambari
https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.4.2
http://blog.csdn.net/chengyuqiang/article/details/53788351
http://blog.csdn.net/royma_1990/article/details/51749981

@firecontainer April 6, 2017 4:53 PM

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。