大數據平臺CDH6.1.0 安裝配置

時間 2019-11-06

標籤數據平臺 cdh6.1.0 cdh 安裝配置简体版

原文原文鏈接

大數據平臺 CDH6.1.0 安裝配置

標籤（空格分隔）：大數據平臺構建html

一：關於CDH6.1.0 的介紹

二：系統環境初始化

三：安裝CDH6.1.0所需環境

四：配置CDH6.1.0的cloudera-manager安裝

一：關於CDH6.1.0 的介紹

Cloudera在前天12月19日，對外宣佈正式發佈Cloudera Enterprise 6.1包含大數據生態版本以下:
1.組件版本：Kafka 2.0, Spark 2.4, HBase 2.1.1, Accumulo 1.9.2, SOLR 7.4
2.CDH免費版限制100個節點內
3.HDFS的糾刪碼徹底支持，包括Hive，Navigator(auditing)，BDR，MR，Spark
4.對象存儲支持：On-prem (Ceph), Cloud (ADLS gen 2)
5.Impala的多個distinct語句支持
6.Sqoop的HiveServer2與導入Parquet文件支持
7.OpenJDK支持

##二：系統環境初始化java

2.1: 系統環境介紹

系統：CentOS7.5X64

cat /etc/hosts 

192.168.20.111  NN01.yl.com    NN01 
192.168.20.112  SN01.yl.com    SN01 
192.168.20.113  DN01.yl.com    DN01
192.168.20.114  DN02.yl.com    DN02

2.2：無密鑰登陸配置

作root用戶無密鑰認證

 ssh-keygen ---一直敲回車到最後

 cat id_rsa.pub >> authorized_keys 

 將全部的公鑰導入authorized_keys  分發到 全部的 機器的.ssh/ 下面

 而後測試

2.3:關閉 firewalld，iptables 與 selinux

全部節點都要配置：

yum install -y firewall* iptable* 

  (1) 關閉firewalld

 systemctl stop firewalld.service
 systemctl disable firewalld.service
 firewall-cmd --state

 (2) 關閉iptables 
 iptables -F
 systemctl stop iptables.service 
 service iptables save
 systemctl disable iptables.service 

 (3) 禁用 selinux 

  sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config  
  setenforce 0
  getenforce 0
  sestatus

  重啓機器

  reboot

2.4：全部機器更改CentOS7.5x64的文件句柄數

vim /etc/security/limits.conf
---

*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited

在其它 節點上面執行命令：

rsync -arvzP root@192.168.20.111:/etc/security/limits.conf /etc/security/

同步limits.conf  文件

2.5 時間同步：

全部機器 yum install chrony*
1. NN01.yl.com:

vim /etc/chrony.conf 增長
----
註釋掉相關默認的server
server ntp1.aliyun.com iburst

allow 192.168.20.111/16

service chronyd start
chkconfig chronyd on 

2. SN01.yl.com---DN01.yl.com 同步NN01.yl.com

vim /etc/chrony.conf
----
註釋掉默認的server
server 192.168.20.111 iburst
----

service chronyd stop 
service chronyd start 
chkconfig chronyd on

2.6: cdh6.1.0 安裝所需文件下載

cdh6.1.0 離線包：
CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha256
manifest.json

下載： https://archive.cloudera.com/cdh6/6.1.0/parcels/

cdh6 的CM 包:
cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
cloudera-manager-server-db-2-6.1.0-769885.el7.x86_64.rpm
oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
allkeys.asc

下載地址：

https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/

jdbc 驅動：
mysql-connector-java-8.0.15-1.el7.noarch.rpm

下載：mysql官網下載

2.7:安裝jdk

所用大數據節點都要配置：jdk 

rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

vim /etc/profile

---
export JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
---
source /etc/profile 

java -version

2.8 安裝mariadb-server

yum install -y mariadb-server 

service mariadb start

chkconfig mariadb on 

mysql_secure_installation

mysql -uroot -pQingchun123.com

2.9配置 httpd 分發服務器

NN01.yl.com:

yum install -y httpd

vim /etc/httpd/conf/httpd.conf
----
修改/etc/httpd/conf/httpd.conf配置文件，在<IfModule mime_module>中修改如下內容

AddType application/x-gzip .gz .tgz .parcel
----
service httpd start 
chkconfig httpd start

作CM6.1 的 分發源

yum install -y createrepo

cd /root/cm6.1/
createrepo .

mv /root/cm6.1/ /var/www/html/
mv /root/cdh6.1/ /var/www/html/

vim /etc/yum.repos.d/cm.repo

------
[cmrepo]
name = cm_repo
baseurl =http://192.168.20.111/cm6.1
enable = true
gpgcheck = false

-----
yum makecache

全部主機執行命令同步：
rsync -avrzP root@192.168.20.111:/etc/yum.repos.d/cm.repo /etc/yum.repos.d/

yum makecache

2.10 配置mysql-jdbc

rpm -ivh mysql-connector-java-8.0.15-1.el7.noarch.rpm

cd /usr/share/java/

cp -p mysql-connector-java-8.0.15.jar  mysql-connector-java.jar

三：安裝CDH6.1所需環境

3.1:安裝所需依賴包

全部大數據節點都要安裝：

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python

3.2 安裝cloudera-server

yum install -y cloudera-manager-server

3.3 在 mysql 當中建立 CM 須要的註冊庫

# mysql -uroot -pQingchun123.com
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to 'cmf'@'%' identified by 'cmf_1234' with grant option;
mysql> grant all privileges on cmf.* to 'cmf'@'NN01.yl.com' identified by 'cmf_1234' with grant option;

mysql> flush privileges;

3.4 注入CM-server庫與 CM-agent端的server

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql cmf cmf cmf_1234

四：配置CDH6.1的cloudera-manager安裝

4.1 安裝啓動CM

service cloudera-scm-server start 
cd /var/log/cloudera-scm-server/
tail -f cloudera-scm-server.log

選擇更多parcel

全部節點執行：

echo vm.swappiness = 10 >> /etc/sysctl.conf
 echo 10 >  /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

將下面 兩條命令

mysql 受權的配置

數據庫受權：
mysql -uroot -pQingchun123.com
---
建立hive 所須要的庫並受權：
mysql >create database cdh_hive default character set latin1; 
字符必須爲拉丁(latin1),若是uft8可能會報錯Specified key was too long; max key length is 767 bytes 
mysql >grant all privileges on cdh_hive.* to 'hive'@'%' identified by 'Hive_123' with grant option;
mysql >grant all privileges on cdh_hive.* to 'hive'@'NN01.yl.com' identified by 'Hive_123' with grant option;
mysql >flush privileges; 

建立oozie 所須要的庫
mysql >create database cdh_oozie default character set utf8; 
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'%' identified by 'Oozie_123' with grant option;
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'NN01.yl.com' identified by 'Oozie_123' with grant option;
mysql >flush privileges; 

建立hue所須要的庫：
mysql >create database cdh_hue default character set utf8; 
mysql >grant all privileges on cdh_hue.* to 'hue'@'%' identified by 'Hue_123' with grant option;
mysql >grant all privileges on cdh_hue.* to 'hue'@'NN01.yl.com' identified by 'Hue_123' with grant option;
mysql >flush privileges; 

建立CM 的 監控端Activity Monitor 的庫
Activity Monitor:
mysql >create database cdh_am default character set utf8; 
mysql >grant all privileges on cdh_am.* to 'am'@'%' identified by 'Am_123' with grant option;
mysql >grant all privileges on cdh_am.* to 'am'@'NN01.yl.com' identified by 'Am_123' with grant option;
mysql >flush privileges;

建立RM 的 監控端 Reports Manager 的庫
Reports Manager
mysql >create database cdh_rm default character set utf8; 
mysql >grant all privileges on cdh_rm.* to 'rm'@'%' identified by 'Rm_123' with grant option;
mysql >grant all privileges on cdh_rm.* to 'rm'@'NN01.yl.com' identified by 'Rm_123' with grant option;
mysql >flush privileges;