大數據平臺CDH搭建

時間 2019-12-01

標籤數據平臺 cdh 搭建简体版

原文原文鏈接

1. CDH簡介

來自官網簡介: html

CDH is Cloudera’s 100% open source platform distribution, including Apache Hadoop and built specifically to meet enterprise demands. CDH delivers everything you need for enterprise use right out of the box. By integrating Hadoop with more than a dozen other critical open source projects, Cloudera has created a functionally advanced system that helps you perform end-to-end Big Data workflows.前端

www.cloudera.com/java

2. 環境準備

2.1 機器準備

三臺虛擬機便可，博主這裏使用的一臺xenserver虛擬服務器分出來的三臺虛擬機。node

| IP地址       |   主機名           | 系統版本   |  軟件                |
| 192.168.1.80 | devcdh1.juejin.org| centos7.6 |  ClouderaManager Mysql  
| 192.168.1.81 | devcdh1.juejin.org| centos7.6 |  hadoopNameNode  HBaseMaster NodeManagre
| 192.168.1.82 | devcdh1.juejin.org| centos7.6 |  hadoopNode HRegionServer sparkSlave複製代碼

2.2 軟件準備(離線安裝)

因公網安裝速度很慢，而常常會下載斷線，最好的方式是離線安裝，事前將rpm包下載到服務器上.mysql

2.2.1 下圖全部rpm軟件下載到本地git

archive.cloudera.com/cm6/6.1.1/r…
sql

2.2.2 下圖中紅框內rpm軟件下載到本地數據庫

archive.cloudera.com/cdh6/6.1.1/…
vim

2.3 系統初始化(三臺機器都執行)

1. 安裝經常使用軟件
yum install wget iptables-services telnet net-tools git curl unzip sysstat lsof ntpdate lrzsz vim  -y
2. 安裝ntp時間服務器並啓動
yum install ntp -y
systemctl start ntpd
systemctl enable ntpd
3. 爲了後續方便操做，先關閉防火牆並禁止開機啓動(iptables爲上圖安裝,firewalld爲centos7默認防火牆)
systemctl stop iptables
systemctl disable iptables
systemctl stop firewalld
systemctl disable firewalld
複製代碼

2.4 安裝JDK(三臺機器都執行)

將下載好的oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm 上傳到服務器centos

執行命令安裝jdk:

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm複製代碼

寫入系統全局環境文件，並聲明:

echo "export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera"  >> /etc/profile
echo "export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib"  >> /etc/profileecho "export PATH=$PATH:$JAVA_HOME/bin"  >> /etc/profile

source /etc/profile   #聲明
java -version         #查看版本複製代碼

2.5 設置hosts(三臺機器都執行)

服務器主機名看上去和二級域名同樣。只是沒有解析，這是CDH官方文檔要求:

docs.cloudera.com/documentati…

注* 主機名稱儘可能使用自已公司域名

echo "192.168.1.80 devcdh1.juejin.org" >> /etc/hosts
echo "192.168.1.81 devcdh2.juejin.org" >> /etc/hostsecho "192.168.1.82 devcdh3.juejin.org" >> /etc/hosts複製代碼

2.6 安裝mysql(官方支持多種數據庫，博主mysql用得比較多，這裏選擇mysql)

支持的數據庫:

docs.cloudera.com/documentati…

執行下面命令安裝mysql:

wget http://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm  #下載官方rpm包
yum localinstall mysql57-community-release-el7-11.noarch.rpm -y            #安裝rpm包
yum repolist enabled | grep "mysql.*-community.*"  -y                      
yum install mysql-community-server -y                                      #安裝mysql-server
systemctl start mysqld                                                     #啓動mysql
grep 'temporary password' /var/log/mysqld.log                              #查看mysql 初始密碼倉庫複製代碼

建立cdh集羣所須要的庫和帳密(帳密請自行設定)

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'XXXXXXX#####';

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'XXXXXXX#####';複製代碼

2.7 安裝cloudera-manager

將下載好的 cloudera-manager 軟件上傳到服務器上

dev-cdh1.juejin.org  安裝  ->   cloudera-manager-daemons cloudera-manager-agent cloudera-manager-serverdev-cdh2.juejin.org  安裝  ->   cloudera-manager-daemons cloudera-manager-agent dev-cdh3.juejin.org  安裝  ->    cloudera-manager-daemons cloudera-manager-agent 複製代碼

登錄到dev-cdh1.juejin.org 執行命令:

yum localinstall cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server -y複製代碼

登錄到dev-cdh2.juejin.org 執行命令:

yum localinstall cloudera-manager-daemons cloudera-manager-agent -y複製代碼

登錄到dev-cdh3.juejin.org 執行命令:

yum localinstall cloudera-manager-daemons cloudera-manager-agent -y複製代碼

注* yum localinstall 爲安裝本地rpm包文件 yum install 是下載yum倉庫軟件再安裝

2.8 啓動manager

在dev-cdh1.juejin.org 上執行啓動命令:

service cloudera-scm-server restart複製代碼

本地綁一下hosts, 博主這裏使用的SwitchHosts:

192.168.1.80 devcdh1.juejin.org
192.168.1.81 devcdh2.juejin.org
192.168.1.82 devcdh3.juejin.org複製代碼

訪問:

devchd1.juejin.org:7180 打開管理頁面，CDH默認帳密爲: admin admin

3. 集羣安裝

3.1 將主機組成集羣，安裝cdh agent

3.1.1 打開管理後臺，登錄後並接受協議點擊繼續

3.1.2 選擇Cloudera Express版本點擊繼續

3.1.3 Specify Hosts 將三臺主機名填入，搜索後選擇所有點擊繼續

3.1.4 選擇存儲庫默認既可點擊繼續

3.1.5 JDK 安裝選項因咱們已經安裝jdk 這裏不用勾選點擊繼續

3.1.6 提供 SSH 登陸憑據。將服務器密碼填入既可點擊繼續

3.1.7 Install Agents 點擊繼續

3.1.8 Install Parcels 點擊繼續

3.1.8 檢查主機正確性若有報錯根據提示修正後再從新檢查既可點擊繼續

3.2 配置CDH服務部署

3.2.1 選擇安裝的服務自定義安裝, 選擇hadoop hbase spark hive hue zookeepr 既可，其餘或後續添加

3.2.2 自定義角色分配，選擇安裝在那個節點上須要根據服務器配置來選擇，這裏由於cdh安裝在devcdh1上。因此這裏將hadoopNamenode hbaseMaster sparkMaster等所有安裝在devcdh2上

3.2.3 數據設置。將前端設置好的帳密和主機地址填入，點擊測試便可

3.2.4 後續一路默認，直到安裝完成

參考:

官方安裝文檔:

docs.cloudera.com/documentati…

官方對服務器要求文檔:

docs.cloudera.com/documentati…

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。