大數據平臺 CDH5.12.2 安裝部署

時間  2021-01-19
標籤 java node python mysql linux web sql 數據庫 json vim 欄目 Java 简体版
原文   原文鏈接
  • 一：初始化環境部署配置
  • 二： CDH 5.12.2 部署處理

一： 初始化環境部署處理

1.1 環境準備：

Centos7.3x64 全部主機名配置 配置好 全部機器的無密鑰登錄

1.2 主機名配置

172.17.100.131  node01.yangyang.com node01
172.17.100.132  node02.yangyang.com node02
172.17.100.133  node03.yangyang.com node03
172.17.100.134  node04.yangyang.com node04
172.17.100.135  node05.yangyang.com node05
172.17.100.136  node06.yangyang.com node06

1.3 無密鑰登陸配置

作root用戶無密鑰認證

 ssh-keygen ---一直敲回車到最後

 cat id_rsa.pub >> authorized_keys 

 將全部的公鑰導入authorized_keys  分發到 全部的 機器的.ssh/ 下面

 而後測試

1.4 關閉 firewalld，iptables 與 selinux

yum install -y firewall* iptable* 

  (1) 關閉firewalld

 systemctl stop firewalld.service
 systemctl disable firewalld.service
 firewall-cmd --state

 (2) 關閉iptables 
 iptables -F
 systemctl stop iptables.service 
 service iptables save
 systemctl disable iptables.service 

 (3) 禁用 selinux 

  sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config  
  setenforce 0
  getenforce 0
  sestatus

  重啓機器

1.4 全部機器更改centos7.3x64的文件句柄數

vim /etc/security/limits.conf
*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited

1.5 全部機器 配置好jdk環境 同步全部機器

(1) 卸載原有jdk
rpm -e java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64 java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64 copy-jdk-configs-2.2-3.el7.noarch java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-devel-1.7.0.141-2.6.10.5.el7.x86_64 java-1.8.0-openjdk-devel-1.8.0.131-11.b12.el7.x86_64 java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el7_3.x86_64 tzdata-java-2017b-1.el7.noarch python-javapackages-3.4.1-11.el7.noarch javapackages-tools-3.4.1-11.el7.noarch --nodeps

(2) 安裝jdk環境
rpm -ivh jdk-8u151-linux-x64.rpm

1.6 生產環境配置chrony 時間同步服務器

全部機器 yum install chrony*
1. node01:

vim /etc/chrony.conf 增長
----
註釋掉相關默認的server
server ntp1.aliyun.com iburst

allow 172.17.100.0.0/16

service chronyd start
chkconfig chronyd on 

2. node02---node05 同步node01

vim /etc/chrony.conf
----
註釋掉默認的server
server 172.17.100.131 iburst
----

service chronyd start 
chkconfig chronyd on

1.7 安裝mysql5.7數據庫主節點上面安裝

卸載：mariadb-libs 的包：
----
 ＃rpm -e mariadb-libs mariadb-devel --nodeps
---
安裝mysql5.7 所須要依賴包

 # yum install -y perl-JSON libaio 

解壓 mysql 5.7.21 集成安裝包

tar -xvf mysql-5.7.21-1.el7.x86_64.rpm-bundle.tar

刪掉一個包mysql-community-server-minimal-5.7.21-1.el7.x86_64.rpm 這個minimal包否則裝不了mysql5.7 

# rm -rf mysql-community-server-minimal-5.7.21-1.el7.x86_64.rpm
# rpm -ivh mysql-community-* 
# mysqld --initialize-insecure --user=mysql

注：上面的安裝步驟是用忽略安裝mysql包安裝順序，系統將自動識別安裝順序

啓動mysql5.7 

#service mysqld start
# chkconfig mysqld on

image_1c7e35kbh1q4rnhbu3n1bboulj9.png-472.2kB

image_1c7e362vhmlm1sc09321c4ct4om.png-83.8kB

image_1c7e36rqphuevg7963g3m6h13.png-233.7kB

注：初次使用mysql的時候,須要更改mysql 的密碼，不然查詢不了mysql數據庫當中的數據。 
#mysql -uroot -p
mysql > set password = password('Bigdata_1234');
mysql> flush privileges;

1.8 修改系統參數

vim /etc/sysctl.conf

##內核默認參數 
kernel.sysrq = 0 
kernel.core_uses_pid = 1 
kernel.msgmnb = 65536 
kernel.msgmax = 65536 
kernel.shmmax = 68719476736 
kernel.shmall = 4294967296 
##打開文件數參數(20*1024*1024) 
fs.file-max= 20971520 
##WEB Server參數 
net.ipv4.tcp_tw_reuse=1 
net.ipv4.tcp_tw_recycle=1 
net.ipv4.tcp_fin_timeout=30 
net.ipv4.tcp_keepalive_time=1200 
net.ipv4.ip_local_port_range = 1024 65535 
net.ipv4.tcp_rmem=4096 87380 8388608 
net.ipv4.tcp_wmem=4096 87380 8388608 
net.ipv4.tcp_max_syn_backlog=8192 
net.ipv4.tcp_max_tw_buckets = 5000 

##TCP補充參數 
net.ipv4.ip_forward = 0 
net.ipv4.conf.default.rp_filter = 1 
net.ipv4.conf.default.accept_source_route = 0 
net.ipv4.tcp_syncookies = 1 
net.ipv4.tcp_sack = 1 
net.ipv4.tcp_window_scaling = 1 
net.core.wmem_default = 8388608 
net.core.rmem_default = 8388608 
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216 
net.core.netdev_max_backlog = 262144 
net.ipv4.tcp_max_orphans = 3276800 
net.ipv4.tcp_timestamps = 0 
net.ipv4.tcp_synack_retries = 1 
net.ipv4.tcp_syn_retries = 1 
net.ipv4.tcp_mem = 94500000 915000000 927000000 
##禁用ipv6 
net.ipv6.conf.all.disable_ipv6 =1 
net.ipv6.conf.default.disable_ipv6 =1 
##swap使用率優化 
vm.swappiness=0 
---
執行如下命令使參數生效: 
# sysctl -p 
同步到全部機器

二： CDH5.12.2 安裝與配置

2.1 安裝CM 所依賴的包：(所有節點都安裝)

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python

2.2 安裝cloudera-server 與更改配置 （節點所有安裝）

rpm -ivh cloudera-manager-* 
rpm -ivh enterprise-debuginfo-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm

image_1c7e3ni54o0410941digmf71vb91g.png-272.2kB

在mysql 當中建立CM 的庫

# mysql -uroot -pBigdata_1234
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to 'cmf'@'%' identified by 'cmf_1234' with grant option;
mysql> grant all privileges on cmf.* to 'cmf'@'node01.yangyang.com' identified by 'cmf_1234' with grant option;
修改cm數據庫
#cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.propertiesbak 
# vim /etc/cloudera-scm-server/db.properties

image_1c7j93pkt1lq21tve7fa6m114sm9.png-235.8kB

2.3 安裝 離線cdh parcel-repo

#cd /root/software
#cp -ap CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel* /opt/cloudera/parcel-repo/
#cp -p manifest.json /opt/cloudera/parcel-repo/
#cd /opt/cloudera/parcel-repo
mv CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha1  (去掉 這個 1) CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha   
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/ （全部主機都要執行這條命令）

image_1c7e4qmep1a8p1jj11a501cvkvt947.png-690.5kB

2.4 jdbc 配置

拷貝jdbc 到 java的目錄
  tar -zxvf mysql-connector-java-5.1.44.tar.gz
  cd mysql-connector-java-5.1.44
  cp mysql-connector-java-5.1.44-bin.jar  /usr/share/java/
  cd /usr/share/java/
  mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar

image_1c7e4vhpbnvv781rgo97tl1n4k.png-757.2kB

2.5 修改config.ini 文件

#cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.inibak 
#vim /etc/cloudera-scm-agent/config.ini
修改server_host 爲主節點的IP 地址
---

server_host=172.17.100.131

---
同步到全部主機節點 
scp /etc/cloudera-scm-agent/config.ini node02:/etc/cloudera-scm-agent/
scp /etc/cloudera-scm-agent/config.ini node03:/etc/cloudera-scm-agent/

image_1c7e55cio9eh1o5j1jtu1ev31lh51.png-79kB
image_1c7e59rdcklsf5q2lf10f61vhh5e.png-142.6kB

2.6 啓動CM-server與agent-server

node01.yangyang.com 主機

cd /etc/init.d
./cloudera-scm-server start

image_1c7e5g8k61lcr10491qt11bnnfr35r.png-207kB

查看啓動日誌

查看啓動結果：
        須要經過查看server的啓動日誌文件
        /var/log/cloudera-scm-server
    tail -f cloudera-scm-server.log

    看到jetty 表示啓動成功 會有 7180 端口啓動：

image_1c7e5hvv41up5oegb3o1jp1qhp68.png-265.5kB

2.7打開web瀏覽器：

http://172.17.100.131:7180

image_1c7e5kettqm0o4erj1l5c12256l.png-217.4kB

用戶名： admin
密碼： admin

image_1c7e5m89t1j3eagc14b5qb43r672.png-348kB

啓動 全部的agent 節點
cd /etc/init.d/
./cloudera-scm-agent start

image_1c7e5pbee1a81rkcf8l6rgund7f.png-197kB

image_1c7e64o6f15mr1sv51mij2phs8v7s.png-355kB

image_1c7e65poo71p1n7o1iii1njj11c189.png-313.3kB

image_1c7e66oie1r1eudsfhg1ugt1ffv8m.png-384.8kB

image_1c7e68c5p6l0fmisce1tlp1vp8a3.png-166.9kB 

一直到分發結束 -- 而後繼續

image_1c7e76eea1d1o1sld32o184kuntag.png-94.1kB

image_1c7e77ki2jr11p674u61og410clat.png-381.9kB

全部節點執行：
#echo never > /sys/kernel/mm/transparent_hugepage/defrag
#echo never > /sys/kernel/mm/transparent_hugepage/enabled

vim /etc/rc.local 

在最下面增長：

---
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
---
同步全部機器

image_1c7e7bjm71lu710gd1sb0obl85eba.png-295.8kB

從新運行界面：

image_1c7e7dapi1rdd1f731gqq1oh1p8dbn.png-285.7kB

image_1c7e7eg2ms0h9ts16cs1uj369qc4.png-370.7kB

配置角色：

image_1c7e7hm2811eqo3m1ms6m4biguch.png-351.4kB

image_1c7e7ij031cth91g1ngf1l6j1a1fcu.png-265.4kB

數據庫受權：
mysql -uroot -pBigdata_1234
---
建立hive 所須要的庫並受權：

mysql >create database cdh_hive default character set latin1; 
字符必須爲拉丁(latin1),若是uft8可能會報錯Specified key was too long; max key length is 767 bytes 
mysql >grant all privileges on cdh_hive.* to 'hive'@'%' identified by 'Hive_123' with grant option;
mysql >grant all privileges on cdh_hive.* to 'hive'@'node01.yangyang.com' identified by 'Hive_123' with grant option;
mysql >flush privileges;

image_1c7e8t4c4dvtgg51isc5lve0hev.png-222kB

建立oozie 所須要的庫
mysql >create database cdh_oozie default character set utf8; 
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'%' identified by 'Oozie_123' with grant option;
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'node01.yangyang.com' identified by 'Oozie_123' with grant option;
mysql >flush privileges;

image_1c7e8ufqvspdl8i1nup1oojdvfc.png-241.3kB

建立hue所須要的庫：
mysql >create database cdh_hue default character set utf8; 
mysql >grant all privileges on cdh_hue.* to ‘hue’@'%' identified by 'Hue_123' with grant option;
mysql >grant all privileges on cdh_hue.* to 'hue'@'node01.yangyang.com' identified by 'Hue_123' with grant option;
mysql >flush privileges;

image_1c7e8vurmv6g1t0v1i0lg4t1q41fp.png-228.5kB

建立CM 的 監控端Activity Monitor 的庫
Activity Monitor:
mysql >create database cdh_am default character set utf8; 
mysql >grant all privileges on cdh_am.* to ‘am’@'%' identified by 'Am_123' with grant option;
mysql >grant all privileges on cdh_am.* to 'am'@'node01.yangyang.com' identified by 'Am_123' with grant option;
mysql >flush privileges;

image_1c7e95pvdlvn15h713b21uuo1d13gj.png-280.9kB

image_1c7e96lu71vq7sge71vv3i1speh0.png-345.5kB

image_1c7e98c5m4mj1rg7v4s1enh18i3hd.png-328.4kB

image_1c7e9io911qqrkkr8aq1r66o1hq.png-365.2kB

image_1c7e9jj2t1ca2eva1qqf36r1a6ni7.png-203.6kB

image_1c7e9kar33up9lg1gvra7ckqjik.png-399.4kB

image_1c7e9nto5tk644l1vl11ehb1k7j1.png-409.8kB

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程