HDP集羣搭建

下載包:

http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.0.0/ambari-2.6.0.0-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/HDP-2.6.5.0-centos7-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz

//若是出現如下結果說明啓動了THP

[root@node-41 local]# cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never

[root@node-41 local]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

//永久關閉THP

[root@node-41 local]# vi /etc/rc.d/rc.local 

#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
        echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
        echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

修改文件測執行權限：

[root@node-41 ~]# chmod +x /etc/rc.d/rc.local

//檢查是否生效

reboot

重啓後查看

[root@node-41 ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@node-41 ~]# cat /sys/kernel/mm/transparent_hugepage/defrag 
always madvise [never]

說明關閉成功

2、克隆機器及其配置

在VMware中克隆5臺及其：node-4二、node-4三、node-4四、node-4五、node-46

點擊每臺機器中的「網絡適配器」->「高級」->"生成"（mac地址）->「肯定」->"肯定"

開機，修改IP地址： vi /etc/sysconfig/network-scripts/ifcfg-ens33 

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=62acf16a-30fd-4909-a68b-92c63408b5b2
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.194.46
GATEWAY=192.168.194.2
NETMASK=255.255.255.0
DNS1=8.8.8.8

修改hostname：

[root@localhost ~]# hostname
localhost.localdomain
[root@localhost ~]# 
[root@localhost ~]# hostnamectl set-hostname node-46
[root@localhost ~]# hostname
node-46

關閉防火牆：

//臨時關閉

[root@node-42 ~]# systemctl stop firewalld

//禁止開機啓動
[root@node-42 ~]# systemctl disable firewalld

//查看防火牆狀態
[root@node-42 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

安裝Mysql，具體可見 https://my.oschina.net/u/3491962/blog/1934088

修改mysql配置文件：[root@node-41 local]# vi /etc/my.cnf 

[mysqld] 下添加：

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
collation_server=utf8_general_ci
character_set_server=utf8
default-storage-engine=INNODB

[client]
default-character-set=utf8

重啓MySQL服務：
[root@node-41 local]# systemctl restart mysqld

鏈接Mysql查看編碼是否替換：
mysql> show variables like '%character_set_%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.01 sec)

建立ambri相應的DB以及其用戶名：
mysql> create database ambari character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> create user 'ambari'@'%' IDENTIFIED BY 'Ambari123'; 
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

建立HiveDB 以及用戶：

mysql> create database hive character set utf8;        
Query OK, 1 row affected (0.00 sec)

mysql> create user 'hive'@'%' IDENTIFIED BY 'Hive123';             
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';            
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

安裝Mysql-java

[root@node-41 local]# yum install mysql-connector-java

配置免密登錄：

在node-41 中：

[root@node-41 local]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Ri8XZHtAAQUTWADi5zcW2qRUsDEqJKto/V0MXJH2ULk root@node-41
The key's randomart image is:
+---[RSA 2048]----+
|... =oo+*O@o.    |
|oo o =o .B +     |
|o o + ooo = o    |
|o..+ = oo. E     |
|o. .+ = Soo      |
|.   .o.o.o       |
|     . .         |
|                 |
|                 |
+----[SHA256]-----+

將公鑰拷貝至node-4一、node-4二、node-4三、node-4四、node-45

ssh-copy-id node-4一、ssh-copy-id node-4二、ssh-copy-id node-4三、ssh-copy-id node-4四、ssh-copy-id node-45

在node-42 中：執行一樣的操做（生成公鑰、私鑰，而後將公鑰拷貝至各個機器上）

2、配置本地的yum源

在本地yum源上安裝相關yum工具

[root@yum ~]# yum install yum-utils -y

[root@yum ~]# yum repolist

[root@yum ~]# yum install createrepo -y

安裝Apache httpd

[root@yum ~]# yum install httpd -y

安裝完成後會自動生成目錄：[root@yum ~]# /var/www/html/，至關於Tomcat 的 /webapps目錄，進入該目錄，建立ambari和hdp文件夾 

[root@yum html]# cd /var/www/html/

[root@yum html]# mkdir ambari
[root@yum html]# mkdir hdp
[root@yum html]# mkdir hdp/HDP-UTILS-1.1.0.21

解壓相應的文件：

[root@yum html]# tar -zxvf /root/ambari-2.6.0.0-centos7.tar.gz -C /var/www/html/ambari/

[root@yum html]# tar -zxvf /root/HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS-1.1.0.21/

[root@yum html]# tar -zxvf /root/HDP-2.6.5.0-centos7-rpm.tar.gz -C /var/www/html/hdp/

解壓完成後啓動 httpd 服務：

[root@yum html]# systemctl start httpd

啓動後能夠訪問連接：http://192.168.194.40/

設置開機自啓動：

[root@yum html]# systemctl enable httpd 

配置ambari yum 源：

[root@yum html]# wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.2/ambari.repo -O /etc/yum.repos.d/ambari.repo

[root@yum yum.repos.d]# vi ambari.repo 

#VERSION_NUMBER=2.6.0.0-267
[ambari-2.6.0.0]
name=ambari Version - ambari-2.6.0.0
baseurl=http://192.168.194.40/ambari/ambari/centos7/2.6.0.0-267/
gpgcheck=1
gpgkey=http://192.168.194.40/ambari/ambari/centos7/2.6.0.0-267/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[root@yum yum.repos.d]# vi HDP.repo 

#VERSION_NUMBER=2.6.5.0-1
[HDP-2.6.5.0]
name=HDP Version - HDP-2.6.5.0
baseurl=http://192.168.194.40/hdp/HDP/centos7/
gpgcheck=1
gpgkey=http://192.168.194.40/hdp/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.21]
name=HDP Version - HDP-UTILS-1.1.0.21
baseurl=http://192.168.194.40/hdp/HDP-UTILS-1.1.0.21/
gpgcheck=1
gpgkey=http://192.168.194.40/hdp/HDP-UTILS-1.1.0.21/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

分發HDP.repo ambari.repo 到各個節點

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-41:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-42:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-43:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-44:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-45:$PWD

生成本地yum源

使用createrepo命令，生成本地源

【createrepo用以建立yum源（軟件倉庫），即爲存放於本機特定位置的衆多rpm包創建索引描述各包所需以來信息，並造成源數據】

[root@yum yum.repos.d]# createrepo /var/www/html/hdp/HDP/centos7/

[root@yum yum.repos.d]# createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.21/

 

3、安裝 Ambari Server

一、安裝服務

[root@node-41 yum.repos.d]# yum install ambari-server

二、設置服務

[root@node-41 yum.repos.d]# ambari-server setup

Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk1.8.0_161/
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? Y^H
input not recognized, please try again: 
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): node-41
Port (3306): 
Database name (ambari): 
Username (ambari): 
Enter Database Password (bigdata): 
Re-enter password: 
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.6.0.0.267.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root

[root@node-41 yum.repos.d]# ambari-server setup

4、設置ambari用戶遠程登錄權限

[root@node-41 yum.repos.d]# mysql -uroot -p123456

mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'localhost' IDENTIFIED BY 'Ambari123';
Query OK, 0 rows affected, 1 warning (0.17 sec)

mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%' IDENTIFIED BY 'Ambari123';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.09 sec)

mysql> exit
Bye

使用ambari帳號登錄MySQL
[root@node-41 yum.repos.d]# mysql -uambari -pAmbari123 -hnode-41

mysql> use ambari;
Database changed
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

啓動Ambari

[root@node-41 yum.repos.d]# ambari-server start

[root@node-41 yum.repos.d]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.......................................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

訪問Ambari服務：

http://192.168.194.41:8080

安裝Agent

在全部節點上安裝

yum install ambari-agent

安裝成功後可進入Ambari Web端，初始化設置

默認帳號密碼：

admin

admin

設置集羣，點擊Launch Install Wizard ：

輸入集羣名字：