Cloudera Manager 大數據集羣搭建離線安裝詳細過程

時間  2019-11-21
標籤 cloudera manager 數據 集羣 搭建 離線 安裝 詳細 過程 欄目 雲服務 简体版
原文   原文鏈接

軟件環境

本教程搭建5個CDH節點，分別爲html

cdh1 192.168.10.101
cdh2 192.168.10.102
cdh3 192.168.10.103
cdh4 192.168.10.104
cdh5 192.168.10.105
複製代碼

步驟

安裝虛擬機

用vsphere client新建cdh1一臺虛擬機。 注意：虛擬機安裝選擇上海時區，並修改時間。 修改配置：操做系統ip、防火牆、seliux、hosts、內核參數。java

IP配置

BOOTPROTO=static
IPADDR=192.168.10.101
NETMASK=255.255.255.0
GATEWAY=192.168.10.1
DNS1=8.8.8.8
DNS2=114.114.114.114
ONBOOT=yes
複製代碼

關閉防火牆和禁止開機自啓

systemctl stop firewalld & systemctl disable firewalld
複製代碼

關閉防火牆和禁止開機自啓

seliux配置(重啓生效)

getenforce Enforcing
Enforcing
複製代碼

修改SELINUX爲permissive或disabled。此配置重啓才生效。mysql

vi /etc/selinux/config
SELINUX=disabled
複製代碼

修改本地域名

若是採用局域網內域名服務器，只須要在第一步將DNS服務器配置爲內網域名服務器便可。此步驟能夠忽略。linux

vi /etc/hosts
192.168.10.101 cdh1
192.168.10.102 cdh2
192.168.10.103 cdh3
192.168.10.104 cdh4
192.168.10.105 cdh5
複製代碼

文件打開數(重啓生效)

vi /etc/security/limits.confsql

* soft nofile 65536 
* hard nofile 65536 
* soft nproc 131072 
* hard nproc 131072
複製代碼

修改系統整體文件打開數shell

echo  6553560 > /proc/sys/fs/file-max
複製代碼

修改swappiness (重啓生效)

echo 'vm.swappiness=10'>> /etc/sysctl.conf數據庫

關閉大葉內存(重啓生效)

cat << EOF >> /etc/rc.d/rc.local 
echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo never > /sys/kernel/mm/transparent_hugepage/enabled 
EOF
複製代碼

重啓

以上全部配置完畢，關機並驗證以上配置設置無誤。json

複製虛擬機

因爲以上配置對全部虛擬機都適用，用vsphere client 複製4個虛擬機。 處處OVF文件前關機vim

導出 OVF文件

導出 OVF文件

再部署OVF文件，依次修改各虛擬機IP。centos

chd1-5

修改主機名(每一個節點)

vi /etc/sysconfig/network
cdhx
複製代碼

免密登陸

（全部節點執行）

ssh-keygen   #（全部節點執行）
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdh1 #（其餘非cdh1節點執行）
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys #（cdh1執行)
scp ~/.ssh/authorized_keys root@cdhx:/root/.ssh/ #(cdh1執行)
複製代碼

集羣腳本(cdh1執行)

集羣分發命令

yum install -y rsync #(全部節點執行)
複製代碼
vi /usr/local/bin/xsync
#!/bin/bash
# $#：表示傳遞給腳本或函數的參數個數。
#1 獲取輸入參數個數，若是沒有參數，直接退出
pcount=$#
if((pcount==0)); then
echo no args;
exit;
fi
 
#2 獲取文件名稱
p1=$1
fname=`basename $p1`
echo fname=$fname
 
#3 獲取上級目錄到絕對路徑
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$pdir
 
#4 獲取當前用戶名稱
user=`whoami`
 

#5 循環
for host in {cdh2,cdh3,cdh4,cdh5}; do
        #echo $pdir/$fname $user@$host:$pdir
        echo --------------- $host ----------------
        rsync -rvl $pdir/$fname $user@$host:$pdir
done
複製代碼
集羣調用命令

vi /usr/local/bin/xcall

##!/bin/sh
pcount=$#
if((pcount==0));then
        echo no args...;
        exit;
fi
echo ==================cdh1==================
$@
for host in {cdh2,cdh3,cdh4,cdh5}; do
        echo ==================$host==================
        ssh $host $@
done
複製代碼

賦權

chmod 777 xsync.sh
chmod 777 xcall.sh
複製代碼

時鐘同步

查看時區

[root@cdh1 ~]# xcall 'timedatectl '
==================cdh1==================
      Local time: Fri 2019-10-11 17:24:57 CST
  Universal time: Fri 2019-10-11 09:24:57 UTC
        RTC time: Fri 2019-10-11 09:24:57
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: no
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a
複製代碼

若是時區不對，先調整時區

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
複製代碼

安裝ntp (全部節點)

xcall 'yum install -y ntp'
複製代碼

修改ntp配置文件(cdh1)

vim /etc/ntp.conf
# 容許內網其餘機器同步時間
restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap
server 0.cn.pool.ntp.org iburst
server 1.cn.pool.ntp.org iburst
server 2.cn.pool.ntp.org iburst
server 3.cn.pool.ntp.org iburst

複製代碼

修改ntp配置文件(cdh2-5)

vim /etc/ntp.conf
# 容許內網其餘機器同步時間
server cdh1
複製代碼

重啓ntpd

xcall systemctl start ntpd
xcall systemctl enable ntpd
複製代碼

本地yum庫

安裝httpd,並啓動

yum install -y httpd
systemctl start httpd
systemctl enable httpd
複製代碼

建立/var/www/html/pub/cm/5.12.1/RPMS/x86_64/目錄，將如下文件上傳

mkdir -p /var/www/html/pub/cm/5.12.1/RPMS/x86_64/
複製代碼
cloudera-manager-agent-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
cloudera-manager-daemons-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
cloudera-manager-server-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
cloudera-manager-server-db-2-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
enterprise-debuginfo-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
複製代碼

建立/var/www/html/pub/parcels/5.12.1/目錄，將如下文件上傳

mkdir -p /var/www/html/pub/parcels/5.12.1/
複製代碼
CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel
CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel.sha1
manifest.json
複製代碼

建立/var/www/html/pub/kudu/1.4.0目錄，將如下文件上傳

mkdir -p /var/www/html/pub/kudu/1.4.0
複製代碼
KUDU-1.4.0-1.cdh5.12.1.p0.10-el7.parcel  
KUDU-1.4.0-1.cdh5.12.1.p0.10-el7.parcel.sha1  
manifest.json
複製代碼

製做元數據

yum -y install createrepo
createrepo  /var/www/html/pub/
複製代碼

會生成/var/www/html/pub/repodata/repomd.xml文件

建立/etc/yum.repos.d/cm.repo

[cm] 
name=cm5.12 
baseurl=http://cdh1/pub/
enabled=1 
gpgcheck=0 
priority=1 
複製代碼

yum 更新(全部節點)

xcall yum clean all
xcall yum makecache
複製代碼

搜索 cloudera 肯定本地源配置無誤

yum search cloudera
複製代碼

分發cm.repo文件

xsync /etc/yum.repos.d/cm.repo
複製代碼

jdk安裝

xsync /opt/jdk-8u131-linux-x64.tar.gz
xcall 'tar zxf /opt/jdk-8u131-linux-x64.tar.gz -C /opt/'
xcall  'mv /opt/jdk1.8.0_131 /opt/jdk'
複製代碼

編輯環境變量並分發

vim /etc/profile
export JAVA_HOME=/opt/jdk
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

xsync /etc/profile
source /etc/profile #(各節點執行)
xcall java -version #(各節點執行)

#連接jdk 該CDH版本即便系統配置了JAVA_HOME也找不到
xcall 'mkdir /usr/lib/jvm/'
xcall 'ln -s /opt/jdk/ /usr/lib/jvm/java-8-openjdk'
複製代碼

mariadb 安裝

yum install -y mariadb-server
#安裝完成MariaDB首先啓動MariaDB
systemctl start mariadb
#設置開機啓動
systemctl enable mariadb

update user set host='%' where user='root';
update user set password=password('123456') where user='root'; 
flush privileges;
複製代碼

建立數據庫

create database  cmf default character set utf8;
grant all privileges on cmf.* to 'cmf'@'%' identified by 'Cmf_123!';

create database  metastore default character set latin1;
grant all privileges on metastore.* to 'hive'@'%' identified by 'Hive_123!';

create database  amon default character set utf8;
grant all privileges on amon.* to 'amon'@'%' identified by 'Amon_123!';


create database  rman default character set utf8;
grant all privileges on rman.* to 'rman'@'%' identified by 'Rman_123!';

create database  nas default character set utf8;
grant all privileges on nas.* to 'nas'@'%' identified by 'Nas_123!';

create database  nms default character set utf8;
grant all privileges on nms.* to 'nms'@'%' identified by 'Nms_123!';

create database  oozie default character set utf8;
grant all privileges on oozie.* to 'oozie'@'%' identified by 'Oozie_123!';

create database  hue default character set utf8;
grant all privileges on hue.* to 'hue'@'%' identified by 'Hue_123!';

flush privileges;
複製代碼

mysql 驅動安裝

xcall 'mkdir -p /usr/share/java'
cd /usr/share/java
cp /opt/mysql-connector-java-5.1.48-bin.jar /usr/share/java
mv /usr/share/java/mysql-connector-java-5.1.48-bin.jar /usr/share/java/mysql-connector-java.jar 
xsync  /usr/share/java/mysql-connector-java.jar
xcall 'chmod a+x /usr/share/java/mysql-connector-java.jar'
複製代碼

CDM部署

安裝

yum install -y cloudera-manager-server 

複製代碼

修改db.properties

vim /etc/cloudera-scm-server/db.properties
# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
com.cloudera.cmf.db.host=localhost

# The database name
com.cloudera.cmf.db.name=cmf

# The database user
com.cloudera.cmf.db.user=cmf

# The database user's password
com.cloudera.cmf.db.password=Cmf_123!

# The db setup type
# By default, it is set to INIT
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL
com.cloudera.cmf.db.setupType=EXTERNAL
複製代碼

啓動

/etc/init.d/cloudera-scm-server restart 
/etc/init.d/cloudera-scm-server status 
複製代碼

打開網址 http://cdh1:7180 帳戶名密碼都是admin

部署cdm-agent

一、agent安裝 (時間長，能夠不用xcall)

xcall 'yum -y install cloudera-manager-daemons cloudera-manager-agent' 
#二、修改客戶端配置文件config.ini 
xcall 'cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.inibak' 
sed -i '/server_host/s/localhost/cdh1/' /etc/cloudera-scm-agent/config.ini
#三、啓動客戶端 
xcall '/etc/init.d/cloudera-scm-agent restart'
xcall '/etc/init.d/cloudera-scm-agent status'
複製代碼

填寫parcel的地址

下載 分配

安裝服務

依次安裝服務

問題

若是有目錄權限問題按照如下命令修改

xcall chown  -R hdfs:hdfs /var/lib/hadoop-hdfs
xcall chown  -R httpfs.httpfs /var/lib/hadoop-httpfs
xcall chown  -R kms.kms /var/lib/hadoop-kms
xcall chown  -R mapred:mapred /var/lib/hadoop-mapreduce
xcall chown  -R yarn:yarn /var/lib/hadoop-yarn
xcall chmod -R 755 /var/lib/hadoop-*
複製代碼
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程