ambari本地源自動化安裝hortonworks hadoop(轉)

時間 2019-11-09

標籤 ambari 本地自動化安裝 hortonworks hadoop 欄目測試自動化简体版

原文原文鏈接

原文地址：http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=26230811&id=4023821php

環境配置html

Hostnamejava	IPnode	OSpython
Ambarilinux	172.16.235.164ios	Centos X64 base installsql
Maste.hadoopvim	172.16.235.165centos	Centos X64 base install
Datanode1.hadoop	172.16.235.166	Centos X64 base install

操做系統建議以software server 方式安裝，base 方式安裝後，須要安裝Development tools，本文以base舉例。

一：製做ambari，hadoop本地源

首先須要有一臺可以聯網的linux機器

1：製做ambari（1.4.1.25）本地源

[root@controller ~]# wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.4.1.25/ambari.repo

[root@controller ~]# cp ambari.repo /etc/yum.repos.d/

將網絡源中的包同步到本地：

reposync -r HDP-UTILS-1.1.0.16

reposync -r Updates-ambari-1.4.1.25

reposync -r ambari-1.x

會獲得相應的三個目錄，每一個目錄中有相應的rpm包

準備一臺http服務器，本文以172.16.235.164做爲http服務器發佈安裝源

mkdir /var/www/html/ambari

將 HDP-UTILS-1.1.0.16 Updates-ambari-1.4.1.25 ambari-1.x複製到 /var/www/html/ambari下

cd /var/www/html/ambari

createrepo ./ #沒有createrepo命令本身去安裝

這樣ambari安裝源就作好了，下載好jdk-6u31-linux-x64.bin備用

2：準備hortonworks hadoop （HDP）2.0.6本地源

準備HDP網絡源

vim /etc/yum.repos.d/HDP.repo，添加如下內容

[HDP-2.0.6]

name=HDP

baseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.0.6.0

enabled=1

gpgcheck=0

同步網絡源到本地：

reposync -r HDP-2.0.6 #將近1G

講 HDP-2.0.6 複製到http server的/var/www/html/hdp下

cd /var/www/html/hdp

createrepo ./

這樣hadoop的本地源就作好了

3：製做系統本地源（安裝ambari和hadoop時，可能會依賴些系統包，這時須要有系統的本地源來安裝）

http server 掛載系統光盤

mount /dev/cdrom /var/www/html/centos/

二：安裝ambari

系統準備：關閉iptables、selinux,啓動ntpd服務

[root@ambari ]vim /etc/hosts

172.16.235.164 ambari

172.16.235.165 master.hadoop

172.16.235.166 datanode1.hadoop

配置與全部主機的無密碼登入

[root@ambari ~]# ssh-keygen #一路回車（全部主機都要執行）

[root@ambari ~]# cd .ssh/

[root@ambari .ssh]# ls

id_rsa id_rsa.pub

[root@ambari .ssh]# scp id_rsa.pub master.hadoop:/root/.ssh/1

The authenticity of host 'master.hadoop (172.16.235.165)' can't be established.

RSA key fingerprint is fc:33:53:5f:f2:20:86:92:ac:96:09:e2:dc:e7:33:ea.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'master.hadoop,172.16.235.165' (RSA) to the list of known hosts.

root@master.hadoop's password:

id_rsa.pub 100% 393 0.4KB/s 00:00

[root@ambari .ssh]# scp id_rsa.pub datanode1.hadoop:/root/.ssh/1

The authenticity of host 'datanode1.hadoop (172.16.235.166)' can't be established.

RSA key fingerprint is fc:33:53:5f:f2:20:86:92:ac:96:09:e2:dc:e7:33:ea.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'datanode1.hadoop,172.16.235.166' (RSA) to the list of known hosts.

root@datanode1.hadoop's password:

id_rsa.pub 100% 393 0.4KB/s 00:00

登入master.hadoop

[root@master ~]# cd .ssh/

[root@master .ssh]# cat 1> authorized_keys

登入datanode1.hadoop

[root@datanode1 ~]# cd .ssh/

[root@datanode1 .ssh]# cat 1> authorized_keys

回到ambari 測試無密碼登入

[root@ambari .ssh]# ssh master.hadoop

Last login: Wed Dec 4 11:11:15 2013 from 172.16.38.48

[root@master ~]# exit

logout

Connection to master.hadoop closed.

[root@ambari .ssh]# ssh datanode1.hadoop

Last login: Wed Dec 4 10:38:06 2013 from 172.16.38.48

[root@datanode1 ~]# exit

logout

Connection to datanode1.hadoop closed.

[root@ambari .ssh]#

安裝與配置ambari服務：

[root@ambari ]vim /etc/yum.repos.d/ambari.repo 添加如下內容

[ambari-1.x]

name=Ambari 1.x

baseurl=http://172.16.235.164/ambari/

gpgcheck=0

enabled=1

[HDP-UTILS-1.1.0.16]

name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.16

baseurl=http://172.16.235.164/ambari/

gpgcheck=0

enabled=1

[Updates-ambari-1.4.1.25]

name=ambari-1.4.1.25 - Updates

baseurl=http://172.16.235.164/ambari/

gpgcheck=

enabled=1

[root@ambari ]vim /etc/yum.repos.d/centos.repo 添加如下內容

[centos]

baseurl=http://172.16.235.164/centos/

enabled=1

gpgcheck=0

執行安裝ambari：

[root@ambari yum.repos.d]# yum -y install ambari-server

Loaded plugins: fastestmirror, security

Repository 'Updates-ambari-1.4.1.25': Error parsing config: Error parsing "gpgcheck = ''": invalid boolean value

Repository 'centos' is missing name in configuration, using id

Loading mirror speeds from cached hostfile

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package ambari-server.noarch 0:1.4.1.25-1 will be installed

--> Processing Dependency: postgresql-server >= 8.1 for package: ambari-server-1.4.1.25-1.noarch

--> Running transaction check

---> Package postgresql-server.x86_64 0:8.4.13-1.el6_3 will be installed

--> Processing Dependency: postgresql-libs(x86-64) = 8.4.13-1.el6_3 for package: postgresql-server-8.4.13-1.el6_3.x86_64

--> Processing Dependency: postgresql(x86-64) = 8.4.13-1.el6_3 for package: postgresql-server-8.4.13-1.el6_3.x86_64

--> Processing Dependency: libpq.so.5()(64bit) for package: postgresql-server-8.4.13-1.el6_3.x86_64

--> Running transaction check

---> Package postgresql.x86_64 0:8.4.13-1.el6_3 will be installed

---> Package postgresql-libs.x86_64 0:8.4.13-1.el6_3 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================

Package Arch Version Repository Size

====================================================================================================================================

Installing:

ambari-server noarch 1.4.1.25-1 HDP-UTILS-1.1.0.16 37 M

Installing for dependencies:

postgresql x86_64 8.4.13-1.el6_3 centos 2.8 M

postgresql-libs x86_64 8.4.13-1.el6_3 centos 200 k

postgresql-server x86_64 8.4.13-1.el6_3 centos 3.4 M

Transaction Summary

====================================================================================================================================

Install 4 Package(s)

Total download size: 43 M

Installed size: 88 M

Downloading Packages:

(1/4): ambari-server-1.4.1.25-1.noarch.rpm | 37 MB 00:00

(2/4): postgresql-server-8.4.13-1.el6_3.x86_64.rpm | 3.4 MB 00:00

(3/4): postgresql-libs-8.4.13-1.el6_3.x86_64.rpm | 200 kB 00:00

(4/4): postgresql-8.4.13-1.el6_3.x86_64.rpm | 2.8 MB 00:00

------------------------------------------------------------------------------------------------------------------------------------

Total 43 MB/s | 43 MB 00:00

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

Installing : postgresql-libs-8.4.13-1.el6_3.x86_64 1/4

Installing : postgresql-8.4.13-1.el6_3.x86_64 2/4

Installing : postgresql-server-8.4.13-1.el6_3.x86_64 3/4

Installing : ambari-server-1.4.1.25-1.noarch 4/4

Verifying : postgresql-server-8.4.13-1.el6_3.x86_64 1/4

Verifying : postgresql-libs-8.4.13-1.el6_3.x86_64 2/4

Verifying : ambari-server-1.4.1.25-1.noarch 3/4

Verifying : postgresql-8.4.13-1.el6_3.x86_64 4/4

Installed:

ambari-server.noarch 0:1.4.1.25-1

Dependency Installed:

postgresql.x86_64 0:8.4.13-1.el6_3 postgresql-libs.x86_64 0:8.4.13-1.el6_3 postgresql-server.x86_64 0:8.4.13-1.el6_3

Complete!

開始配置ambari 服務：

首先上傳jdk文件

[root@ambari ]cd /var/lib/ambari-server/resources 將jdk-6u31-linux-x64.bin上傳到該目錄

[root@ambari resources]# chmod +x jdk-6u31-linux-x64.bin

[root@ambari yum.repos.d]# ambari-server setup

Using python /usr/bin/python2.6

Initializing...

Setup ambari-server

Checking SELinux...

SELinux status is 'enabled'

SELinux mode is 'permissive'

WARNING: SELinux is set to 'permissive' mode and temporarily disabled.

OK to continue [y/n] (y)?

Customize user account for ambari-server daemon [y/n] (n)?

Adjusting ambari-server permissions and ownership...

Checking iptables...

WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.

OK to continue [y/n] (y)?

Checking JDK...

JDK already exists, using /var/lib/ambari-server/resources/jdk-6u31-linux-x64.bin

Installing JDK to /usr/jdk64

Successfully installed JDK to /usr/jdk64/jdk1.6.0_31

Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-6.zip to /var/lib/ambari-server/resources/jce_policy-6.zip

To download the JCE Policy files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.htmlNot accepting will result in errors when configuring Kerberos security.

Do you accept the Oracle Binary Code License Agreement [y/n] (y)?

Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-6.zip

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)?

Default properties detected. Using built-in database.

Checking PostgreSQL...

Running initdb: This may take upto a minute.

About to start PostgreSQL

Configuring local database...

Connecting to the database. Attempt 1...

Configuring PostgreSQL...

Restarting PostgreSQL

Ambari Server 'setup' completed successfully.

[root@ambari ~]# ambari-server start

Using python /usr/bin/python2.6

Starting ambari-server

Ambari Server running with 'root' privileges.

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Ambari Server 'start' completed successfully.

到此ambari安裝好

三：本地安裝hadoop

給ambari配置本地hadoo源地址：

[root@ambari ~]# cd /var/lib/ambari-server/resources/stacks/HDPLocal/2.0.6/repos/

[root@ambari repos]# vim repoinfo.xml 修改紅色部分爲本地源地址

< os type="centos6">

< repo>

< baseurl>http://172.16.235.164/hdp/</baseurl>

< repoid>HDP-2.0.6</repoid>

< reponame>HDP</reponame>

< /repo>

< /os>

這樣在ambari執行hadoop安裝時，會將本地源地址配置到全部主機上，在爲全部主機安裝ambari-agent時，會將ambari server下的/etc/yum.repo/ambari.repo文件複製到全部主機

主機準備（master.hadoop、datanode1.hadoop）配置相同，只舉例master.hadoop

關閉iptables、selinux、啓動ntpd服務（略）

[root@master ~]# vim /etc/hosts 添加如下

172.16.235.164 ambari

172.16.235.165 master.hadoop

172.16.235.166 datanode1.hadoop

[root@master ~]# vim /etc/yum.repos.d/centos.repo 添加如下內容

[root

[centos]

baseurl=http://172.16.235.164/centos/

enabled=1

gpgcheck=0

@master ~]# yum -y groupinstall "Development tools"

登入ambari管理頁面

http://172.16.235.164:8080/ 帳戶：admin 密碼：admin

配置集羣名稱

選擇HDP版本

選擇目標主機，一行一個主機。要讓ambar server自動在因此主機安裝ambari-agent，須要提供ambari server root帳戶的私鑰，即/root/.ssh/ id_rsa文件，選中使用本地源安裝hadoop，而後下一步

確認而且選擇主機

確認OK

下一步選擇要安裝的組件，選好後下一步

選擇組件分配在哪些主機，都是master 的服務

選擇安裝client，主機有限，master同時安裝client

配置服務，其中nagios中要填寫密碼及報警接受郵箱必填，其餘選擇性配置

顯示配置信息

開始安裝，過程稍微有點長

安裝過程

Google瀏覽器反應比較慢，用另外一某瀏覽器已經顯示succes了

Ganglia：

nagios：

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。