ambari介紹及安裝

Ambari簡介

Ambari概述

Apache Ambari是一種基於Web的工具,支持Apache Hadoop集羣的建立、管理和監控。Ambari已支持大多數Hadoop組件,包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等;除此以外,Ambari還支持Spark、Storm等計算框架及資源調度平臺YARN。javascript

Apache Ambari 從集羣節點和服務收集大量信息,並把它們表現爲容易使用的,集中化的接口:Ambari Web.css

Ambari Web顯示諸如服務特定的摘要、圖表以及警報信息。可經過Ambari Web對Hadoop集羣進行建立、管理、監視、添加主機、更新服務配置等;也能夠利用Ambari Web執行集羣管理任務,例如啓用 Kerberos 安全以及執行Stack升級。任何用戶均可以查看Ambari Web特性。擁有administrator-level 角色的用戶能夠訪問比 operator-level 或 view-only 的用戶能訪問的更多選項。例如,Ambari administrator 能夠管理集羣安全,一個 operator 用戶能夠監控集羣,而 view-only 用戶只能訪問系統管理員授予他的必要的權限。html

Ambari體系結構

Ambari 自身也是一個分佈式架構的軟件,主要由兩部分組成:Ambari Server 和 Ambari Agent。簡單來講,用戶經過Ambari Server通知 Ambari Agent 安裝對應的軟件;Agent 會定時地發送各個機器每一個軟件模塊的狀態給 Ambari Server,最終這些狀態信息會呈如今 Ambari 的 GUI,方便用戶瞭解到集羣的各類狀態,並進行相應的維護。java

Ambari Server 從整個集羣上收集信息。每一個主機上都有 Ambari Agent, Ambari Server 經過 Ambari Agent 控制每部主機。node

Ambari安裝

http://doc.ailinux.net/docs/data/data-1ardi0l817b5qpython

#Ambari操做指南
https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-operations/content/ch_Overview_hdp-ambari-user-guide.htmlmysql

 

安裝準備階段:

本文介紹ambari安裝及經常使用大數據組件安裝配置等

HDP版本兼容性

CentOS 7 Python 2.7.x JDK1.8+ Mariadb 5.5 Ambari2.6.1.5 HDP 2.6.4

查看一下地址獲取軟件版本兼容性支持
https://supportmatrix.hortonworks.com/
因爲嚴格依賴版本,因此要先肯定安裝的版本。linux

系統初始化

更新yum源

yum install -y wget wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo yum clean all yum makecache

修改主機名

sudo hostnamectl set-hostname test-bg-xx sudo hostname test-bg-xx

修改時區:

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

同步集羣時間

yum install –y ntp
ntpdate -u cn.pool.ntp.org
或:
ntpdate time1.aliyun.com
### 硬件時間和系統時間同步: clock -w 配置自動同步:/etc/crontab寫入 */1 * * * * root ntpdate -u cn.pool.ntp.org >> /dev/null 2>&1

關閉防火牆和selinux

#關閉防火牆 systemctl stop firewalld systemctl disable firewalld #關閉selinux 修改配置 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux 臨時關閉 setenforce 0

配置免密鑰登陸

爲了讓Ambari服務器在全部集羣主機上自動安裝Ambari代理,您必須在Ambari服務器主機和集羣中的全部其餘主機之間設置無密碼SSH鏈接。Ambari服務器主機使用SSH公鑰認證來遠程訪問和安裝Ambari代理
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub root@node1 ssh-copy-id -i ~/.ssh/id_rsa.pub root@node2 ssh-copy-id -i ~/.ssh/id_rsa.pub root@node3 注意:您能夠選擇在每一個集羣主機上手動安裝Ambari代理。在這種狀況下,您不須要生成和分發SSH密鑰 可使用非root SSH賬戶,若是該賬戶能夠執行sudo而不輸入密碼 # #若是ssh端口不是默認,能夠在.ssh下建立config文件,修改文件權限 配置以下: touch config chmod 600 config [jt_ops@test-bg-m01 .ssh]$ cat config Host test-bg-m01 HostName test-bg-m01 User jt_ops Port xx Host test-bg-w01 HostName test-bg-w01 User jt_ops Port xx 也能夠修改/etc/sshd/ssh_config配置文件中的端口

系統初始化參數

cat >> /etc/sysctl.conf << EOF fs.file-max=1000000 net.ipv4.tcp_max_tw_buckets = 6000 net.ipv4.tcp_sack = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_rmem = 4096 87380 4194304 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.tcp_max_syn_backlog = 16384 net.core.netdev_max_backlog = 32768 net.core.somaxconn = 32768 net.core.wmem_default = 8388608 net.core.rmem_default = 8388608 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_fin_timeout = 20 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 net.ipv4.tcp_syncookies = 1 #net.ipv4.tcp_tw_len = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_mem = 94500000 915000000 927000000 net.ipv4.tcp_max_orphans = 3276800 net.ipv4.ip_local_port_range = 1024 65000 net.nf_conntrack_max = 6553500 net.netfilter.nf_conntrack_max = 6553500 net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60 net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120 net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120 net.netfilter.nf_conntrack_tcp_timeout_established = 3600 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 kernel.numa_balancing = 0 kernel.shmmax = 68719476736 kernel.printk = 5 kernel.sysrq = 1 vm.overcommit_memory = 0 vm.swappiness = 0 EOF

文件描述符和用戶最大進程數

### 調整文件描述符 cat >> /etc/security/limits.conf <<EOF * soft nproc 65535 * hard nproc 65535 * soft nofile 65535 * hard nofile 65535 EOF ##用戶進程限制 cat >> /etc/security/limits.d/20-nproc.conf <<EOF * soft nproc 8192 root soft nproc unlimited EOF

安裝

安裝有兩種方法:
      在線安裝和搭建本地yum源倉庫安裝

在線安裝

下載yum源配置文件repo

#若是作了免祕鑰登錄,則只須要在主節點下載yum源  sudo wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari.repo -O /etc/yum.repos.d/ambari.repo sudo wget –nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/hdp.repo -O /etc/yum.repos.d/hdp.repo sudo wget -nv http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.4.0/hdp.gpl.repo -O /etc/yum.repos.d/hdp.gpl.repo

離線安裝

製做本地源

製做本地鏡像源安裝HDP,能夠在主節點或者單獨的一臺服務器(建議)
選擇一臺能夠訪問internet的服務器配置本地源
apache httpd
提供離線倉庫的下載服務
#安裝apache yum install httpd –y 下載軟件源 在httpd網站根目錄,默認是即/var/www/html/,建立目錄ambari, 而且將下載的壓縮包解壓到/var/www/html/ambari目錄 cd /var/www/html/ mkdir ambari cd /var/www/html/ambari/ http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari-2.6.1.5-centos7.tar.gz http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos6-rpm.tar.gz http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos6/HDP-UTILS-1.1.0.22-centos6.tar.gz http://public-repo-1.hortonworks.com/HDP-GPL/centos6/2.x/updates/2.6.4.0/HDP-GPL-2.6.4.0-centos6-rpm.tar.gz 解壓下載的tar包 [root@dev-bg-m01 html]# cd /var/www/html/ [root@dev-bg-m01 html]# pwd /var/www/html [root@dev-bg-m01 html]# [root@dev-bg-m01 html]# tree -L 2 ambari/ ambari/ ├── ambari │ └── centos7 ├── HDP │ └── centos7 ├── HDP-GPL │ └── centos7 └── HDP-UTILS-1.1.0.22 ├── hdp-utils.repo ├── openblas ├── repodata ├── RPM-GPG-KEY └── snappy 11 directories, 1 file [root@dev-bg-m01 html]#

編輯ambari yum源文件

[root@dev-bg-m01 yum.repos.d]# cat ambari.repo #VERSION_NUMBER=2.6.1.5-3 [ambari-2.6.1.5] name=ambari Version - ambari-2.6.1.5 baseurl=http://192.168.103.3/ambari/ambari/centos7/2.6.1.5-3/ gpgcheck=1 gpgkey=http://192.168.103.3/ambari/ambari/centos7/2.6.1.5-3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [root@dev-bg-m01 yum.repos.d]# [root@dev-bg-m01 yum.repos.d]# cat hdp.repo #VERSION_NUMBER=2.6.4.0-91 [HDP-2.6.4.0] name=HDP Version - HDP-2.6.4.0 baseurl=http://192.168.103.3/ambari/HDP/centos7/2.6.4.0-91/ gpgcheck=1 gpgkey=http://192.168.103.3/ambari/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.22] name=HDP-UTILS Version - HDP-UTILS-1.1.0.22 baseurl=http://192.168.103.3/ambari/HDP-UTILS-1.1.0.22/ gpgcheck=1 gpgkey=http://192.168.103.3/ambari/HDP-UTILS-1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [root@dev-bg-m01 yum.repos.d]# [root@dev-bg-m01 yum.repos.d]# cat hdp.gpl.repo [HDP-2.6-GPL-repo-1] name=HDP-2.6-GPL-repo-1 baseurl=http://192.168.103.3/ambari/HDP-GPL/centos7/2.6.4.0-91/ gpgkey=http://192.168.103.3/ambari/HDP-GPL/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins path=/ enabled=1 gpgcheck=0 [root@dev-bg-m01 yum.repos.d]# 更新yum 元數據 yum clean all yum makecache yum repolist

安裝Ambari-server

在主節點安裝ambari-server
sudo yum install -y ambari-server

配置Ambari-server

配置mysql鏈接器
因爲ambari要訪問數據庫,因此要添加lib包,請自行去mysql官網下載
sudo mkdir -p /usr/share/java
sudo cp /opt/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/share/java/ 在ambari-server配置文件中添加mysql lib包路徑地址 執行一下命令配置jdbc環境 ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-5.1.47-bin.jar vim /etc/ambari-server/conf/ambari.properties ###在jdbc區下面找個位置添加以下以下語句 server.jdbc.driver.path=/usr/share/java/mysql-connector-java-5.1.47-bin.jar 檢查環境並配置 執行如下命令 sudo ambari-server setup 按提示操做 會檢查selinux,是否自定義用戶,檢查防火牆,設置JDK路徑,數據庫配置等,按提示操做便可 具體操做以下: [root@dev-bg-m01 opt]# vim /etc/ambari-server/conf/ambari.properties [root@dev-bg-m01 opt]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-5.1.47-bin.jar Using python /usr/bin/python Setup ambari-server Copying /usr/share/java/mysql-connector-java-5.1.47-bin.jar to /var/lib/ambari-server/resources If you are updating existing jdbc driver jar for mysql with mysql-connector-java-5.1.47-bin.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts. JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully. [root@dev-bg-m01 opt]# vim /etc/ambari-server/conf/ambari.properties [root@dev-bg-m01 opt]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root):ambari Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... Do you want to change Oracle JDK [y/n] (n)? y [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /opt/jdk1.8.0_161 Validating JDK on Ambari Server...done. Checking GPL software agreement... Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ========================================== Enter choice (1): 3 Hostname (localhost): 192.168.103.3 Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): Re-enter password: Configuring ambari database... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? n WARNING: Remote database setup aborted. Ambari Server 'setup' completed with warnings. [root@dev-bg-m01 opt]#

將Ambari數據庫腳本導入到數據庫

mysql -uambari -pxxx
use ambari; source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql 啓動ambari server ambari-server start 在瀏覽器中訪問ip:8080,默認登陸名:admin 密碼:admin

添加服務組件
https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-administration/content/ambari_admin_overview.htmlnginx

spark配置

因爲spark要訪問數據庫,因此要添加mysql lib包
cp /opt/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/hdp/2.6.4.0-91/spark/lib/

安裝數據庫

使用現有數據庫或者本身安裝sql

建立ambari和hive庫
create database ambari character set utf8 ; 添加權限 GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'ambari'; FLUSH PRIVILEGES; 刪除權限: revoke all privileges on ambari.* from ambari@'%';

程序安裝目錄

程序默認安裝目錄
[root@test-bg-m01 2.6.4.0-91]# pwd /usr/hdp/2.6.4.0-91 [root@test-bg-m01 2.6.4.0-91]# ls atlas hadoop hadoop-mapreduce hbase hive2 kafka pig ranger-hdfs-plugin ranger-kafka-plugin ranger-yarn-plugin spark storm tez usr etc hadoop-hdfs hadoop-yarn hive hive-hcatalog livy ranger-hbase-plugin ranger-hive-plugin ranger-storm-plugin slider spark2 storm-slider-client tez_hive2 zookeeper [root@test-bg-m01 2.6.4.0-91]# PID目錄 /var/run/ 日誌目錄 /var/logs/ 通常修改log目錄,在安裝大數據組件的時候,修改配置文件便可。

刪除服務

先中止服務
Service Actions > Delete Service
yum remove xx https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-administration/content/ambari_admin_overview.html

ambari監控

ambari安裝默認用戶和用戶組

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-administration/content/defining_service_users_and_groups_for_a_hdp_2x_stack.html

常見問題

問題1:Centos 7安裝 ambari-agent報錯NetUtil.py-[SSL: CERTIFICATE_VERIFY_FAILED]

Centos 7安裝 ambari經過Ambari部署系統時,ambari-agent註冊失敗,在/var/log/ambari-agent的log中能夠看到日誌顯示:
ERROR 2018-03-08 15:02:25,223 NetUtil.py:88 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) 
ERROR 2018-03-08 15:02:25,223 NetUtil.py:89 - SSLError: Failed to connect. Please check openssl library versions. 解決方法: 修改/etc/python/cert-verification.cfg配置文件: # vim /etc/python/cert-verification.cfg [https] verify=platform_default ###(這是默認配置) 修改成verify=disable 編輯 /etc/ambari-agent/conf/ambari-agent.ini 配置文件,在 [security] 節部分,確保設置以下兩個值,其它值保持不變: [root@ambari ~]# vim /etc/ambari-agent/conf/ambari-agent.ini [security] ssl_verify_cert=0 force_https_protocol=PROTOCOL_TLSv1_2 以上配置修改完後,重試便可經過ambari-agent註冊,執行便可下一步

參考文檔
官方文檔安裝文檔

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-installation/content/determine_product_interop.html

未完待續~

相關文章
相關標籤/搜索