大數據平臺CDH5.14.2 的安裝配置

時間  2019-11-10
標籤 數據 平臺 cdh5.14.2 cdh 安裝 配置 简体版
原文   原文鏈接

大數據平臺CDH5.14.2 的安裝配置

標籤（空格分隔）： 大數據平臺構建php

  • 一：系統環境初始化
  • 二：安裝CDH5.14.2 平臺
  • 三：分配主機與分配角色

一： 系統環境初始化

1.1: 系統環境介紹

系統： CentOS7.5X64

cat /etc/hosts 
---
172.17.100.11   node-01.flyfish
172.17.100.12   node-02.flyfish
172.17.100.13   node-03.flyfish
172.17.100.14   node-04.flyfish
172.17.100.15   kbs-01.flyfish
172.17.100.16   kbs-02.flyfish
-----

本次 安裝只配置其中三臺機器： 172.17.100.11到 172.17.100.13

image_1cfk2nlhohkr1gij12s3g1djh6m.png-52.1kB

1.2: 關閉 firewalld，iptables 與 selinux

yum install -y firewall* iptable* 
  (1) 關閉firewalld
 systemctl stop firewalld.service
 systemctl disable firewalld.service
 firewall-cmd --state
 (2) 關閉iptables 
 iptables -F
 systemctl stop iptables.service 
 service iptables save
 systemctl disable iptables.service 
 (3) 禁用 selinux 
  sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config  
  setenforce 0
  getenforce 0
  sestatus
  重啓機器
   reboot

image_1cfk2mh0a1ts11uju185810s71o539.png-98.8kB

1.3:無密鑰登陸配置

作root用戶無密鑰認證
 ssh-keygen ---一直敲回車到最後
 cat id_rsa.pub >> authorized_keys 
 將全部的公鑰導入authorized_keys  分發到 全部的 機器的.ssh/ 下面
 而後測試

image_1cfk2qgsn1a8v13u11vv2b8m1aep13.png-99.1kB

1.4 配置jdk1.7環境 （全部機器都配置）

1): 卸載原來的jdk
rpm -e java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64 java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64 copy-jdk-configs-2.2-3.el7.noarch java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-devel-1.7.0.141-2.6.10.5.el7.x86_64 java-1.8.0-openjdk-devel-1.8.0.131-11.b12.el7.x86_64 java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el7_3.x86_64 tzdata-java-2017b-1.el7.noarch python-javapackages-3.4.1-11.el7.noarch javapackages-tools-3.4.1-11.el7.noarch --nodeps

2) 安裝 jdk:

rpm -ivh oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

vim /etc/profile 

最後增長：

### JDK ####

export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
---

java -version

image_1cfk4ks6a108j1o8n9b419n5nle1t.png-81.9kB

image_1cfk4lk4k7du1ok713kt1ehm1rjf2a.png-43.7kB

image_1cfk4g7n49di1qgm1qhvpv61ia61g.png-54kB

1.5：chrony 時間同步服務器

全部機器 yum install chrony*
1. node-01.flyfish:
vim /etc/chrony.conf 增長
----
註釋掉相關默認的server
server ntp1.aliyun.com iburst
allow 172.17.100.0.0/16
service chronyd start
chkconfig chronyd on 
2. node-02.flyfish---node-03.flyfish 同步node-01.flyfish
vim /etc/chrony.conf
----
註釋掉默認的server
server 172.17.100.11 iburst
----
service chronyd start 
chkconfig chronyd on

image_1cfk4tp0m17m033m1ij0vordq72q.png-155.6kB

image_1cfk5143p15qqbm01g13t2ka7k37.png-102.3kB

image_1cfk52aq0b48nevn0i15cftm544.png-176.2kB

image_1cfk54h8n1qc9hgsjlm1abunea4h.png-148.4kB

1.6: 安裝maridb-server 數據庫

1）安裝mariadb-server
yum install -y mariadb*

2) 啓動 mariadb-server

service mariadb start 
chkcofig mariadb on 

3) 配置數據庫密碼：

mysql_secure_installation

4） 登陸mariadb-server 數據庫

   mysql -uroot -pflyfish225.com

image_1cfk5ddalrb810tn1p9rer60i5b.png-263.9kB

image_1cfk5clm611li14dr1pb7am72kv4u.png-98.3kB

image_1cfk5ev5v1q84g57h718tmqnd5o.png-92.8kB

image_1cfk5hsnk8r01834j5j4oa1vr565.png-146.2kB

image_1cfk5ij9ip0vcb1hq61h4019eh6i.png-152kB

image_1cfk5j82u1njs1u4v1kfkudb12da6v.png-79.3kB

image_1cfk5kg5h16q62ia1kk313sbc0i7c.png-128.2kB

1.8 關於系統參數配置

vim /etc/sysctl.conf
##內核默認參數 
kernel.sysrq = 0 
kernel.core_uses_pid = 1 
kernel.msgmnb = 65536 
kernel.msgmax = 65536 
kernel.shmmax = 68719476736 
kernel.shmall = 4294967296 
##打開文件數參數(20*1024*1024) 
fs.file-max= 20971520 
##WEB Server參數 
net.ipv4.tcp_tw_reuse=1 
net.ipv4.tcp_tw_recycle=1 
net.ipv4.tcp_fin_timeout=30 
net.ipv4.tcp_keepalive_time=1200 
net.ipv4.ip_local_port_range = 1024 65535 
net.ipv4.tcp_rmem=4096 87380 8388608 
net.ipv4.tcp_wmem=4096 87380 8388608 
net.ipv4.tcp_max_syn_backlog=8192 
net.ipv4.tcp_max_tw_buckets = 5000 
##TCP補充參數 
net.ipv4.ip_forward = 0 
net.ipv4.conf.default.rp_filter = 1 
net.ipv4.conf.default.accept_source_route = 0 
net.ipv4.tcp_syncookies = 1 
net.ipv4.tcp_sack = 1 
net.ipv4.tcp_window_scaling = 1 
net.core.wmem_default = 8388608 
net.core.rmem_default = 8388608 
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216 
net.core.netdev_max_backlog = 262144 
net.ipv4.tcp_max_orphans = 3276800 
net.ipv4.tcp_timestamps = 0 
net.ipv4.tcp_synack_retries = 1 
net.ipv4.tcp_syn_retries = 1 
net.ipv4.tcp_mem = 94500000 915000000 927000000 
##禁用ipv6 
net.ipv6.conf.all.disable_ipv6 =1 
net.ipv6.conf.default.disable_ipv6 =1 
##swap使用率優化 
vm.swappiness=0 
---
執行如下命令使參數生效: 
# sysctl -p 
同步到全部機器

image_1cfk5ti4gimj1na614om1em1suj7p.png-84.8kB

1.7 修改文件句柄數

vim /etc/security/limits.conf
---

*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited

---

image_1cfk61cq8efemoi7dc1o7l1tes86.png-44kB

二：安裝CDH5.14.2 平臺

2.1: 安裝配置依賴包(所有節點都安裝)

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python

2.2：安裝cloudera-server 與更改配置 （節點所有安裝）

rpm -ivh cloudera-manager*
rpm -ivh enterprise-debuginfo-5.14.3-1.cm5143.p0.4.el7.x86_64.rpm

image_1cfk6tui1s9i15gq5urngiqk8j.png-172.7kB

2.3:在mariadb-server 當中建立CM 的庫

mysql -uroot -pflyfish225.com
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to 'cmf'@'%' identified by 'cmf_1234' with grant option;
mysql> grant all privileges on cmf.* to 'cmf'@'node-01.flyfish' identified by 'cmf_1234' with grant option;
mysql> flush privileges;

image_1cfk7aqu2n4mbp4uaubea2hg9g.png-216.7kB

2.4:安裝離線cdh parcel-repo

#cd /root/software
#cp -p CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel* /opt/cloudera/parcel-repo/
#cp -p manifest.json /opt/cloudera/parcel-repo/
#cd /opt/cloudera/parcel-repo
mv CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha1   (去掉最後的1) CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha   
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/ （全部主機都要執行這條命令）

image_1cfk82kl814h51uvvs1k1ctbmm6an.png-217.2kB

2.5 配置jdbc的驅動

rpm -ivh mysql-connector-java-8.0.11-1.el7.noarch.rpm
cd /usr/share/java
cp -p mysql-connector-java-8.0.11.jar mysql-connector-java.jar
chmod 777 mysql-connector-java.jar

image_1cfk8g6hrthveu382intcspqb4.png-80.7kB

image_1cfk8gn5s9badhqcpn6kd15dbh.png-51.8kB

image_1cfk8h4071lvai1h19jk16121m36bu.png-37.6kB

image_1cfke9mmc1sglc723tm14d6h0k93.png-50.4kB

2.6 配置數據的註冊生成表

1）備份db生成表的文件
cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.propertiesbak 

2）生成表

/usr/share/cmf/schema/scm_prepare_database.sh mysql cmf cmf cmf_1234

image_1cfk9ui3a4m41jeut7q7jtpv49.png-111.6kB

2.7:修改config.ini 文件

vim /etc/cloudera-scm-agent/config.ini
---
server_host=172.17.100.11
---

scp /etc/cloudera-scm-agent/config.ini node-02.flyfish:/etc/cloudera-scm-agent/

scp /etc/cloudera-scm-agent/config.ini node-03.flyfish:/etc/cloudera-scm-agent/

image_1cfk8ktkjutqvd610qti7luraco.png-123.5kB

image_1cfk8nlcrh7bnv5k81nq2hnsd5.png-89.5kB

2.8: 啓動CM-SERVER

cd /etc/init.d/
./cloudera-scm-server start

image_1cfka3cs3197625g27uks75n9m.png-79.1kB

查看日誌：
cd /var/log/cloudera-scm-server

tail -100f cloudera-scm-server.log

查看有7180 端口啓動表示CM管理server啓動

netstat -nultp |grep 7180

image_1cfkacouq1l8vsgb10u91v1b11p81g.png-332.5kB

image_1cfkaa9bj1t577o3q4p1k261fmq13.png-62.2kB

使用瀏覽器web打開：

http://node-01.flyfish:7180

image_1cfkah1derna1grs1ebd82l3k1t.png-187.5kB

image_1cfkahsd3u1i173nb51nhqp9j2a.png-372.1kB

image_1cfkaigmh80ca00e6h16mebkj2n.png-365.6kB

image_1cfkaj1ofs0418ffkvh90r1v5h34.png-345.9kB

啓動agent 端：

cd /etc/init.d/
./cloudera-scm-agent start

image_1cfkaltndl9sauc1ts81nn92554h.png-93.4kB

image_1cfkan2pn13fva9v6h31m324tq4u.png-84.9kB

image_1cfkaned3foamrobn81c6p19ki5b.png-74.3kB

三： 分配主機與分配角色

image_1cfkaq29p1gsmsfh4atqf2lp5o.png-354.8kB

image_1cfkar2741uf210ud1t6k15e7jqb65.png-303.6kB

image_1cfkas06v9679n15vi1h3i19n6i.png-347.4kB

image_1cfkasikho1iipl1q9s1un51snt6v.png-263kB

image_1cfkb1baqdjlu7lj5f1rjo18ol7c.png-204.7kB

image_1cfkbc5g3kk3154o1hjgkrfqia7p.png-236.9kB

image_1cfkbluv4132k1fl7mb1qdd1vc786.png-422.9kB

全部節點執行：
#echo never > /sys/kernel/mm/transparent_hugepage/defrag
#echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local 
在最下面增長：
---
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
---
同步全部機器

image_1cfkbu5mfbqb1681ini19tp9e116.png-222.6kB

image_1cfkbvvpt1rq16r61av11ker10uh1j.png-366.5kB

image_1cfkc2ohl199vk0ll3siqgkst20.png-638.1kB

image_1cfkca9vp99hsa685t1niv1li02d.png-392.1kB

image_1cfkcaqmt1r16dip1s9lu17138t2q.png-370kB

image_1cfkcepb7fjm15471j9h1efa5nv3n.png-419kB

mysql 受權的配置

數據庫受權：
mysql -uroot -pflyfish225.com
---
建立hive 所須要的庫並受權：
mysql >create database cdh_hive default character set latin1; 
字符必須爲拉丁(latin1),若是uft8可能會報錯Specified key was too long; max key length is 767 bytes 
mysql >grant all privileges on cdh_hive.* to 'hive'@'%' identified by 'Hive_123' with grant option;
mysql >grant all privileges on cdh_hive.* to 'hive'@'node-01.flyfish' identified by 'Hive_123' with grant option;
mysql >flush privileges; 

建立oozie 所須要的庫
mysql >create database cdh_oozie default character set utf8; 
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'%' identified by 'Oozie_123' with grant option;
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'node-01.flyfish' identified by 'Oozie_123' with grant option;
mysql >flush privileges; 

建立hue所須要的庫：
mysql >create database cdh_hue default character set utf8; 
mysql >grant all privileges on cdh_hue.* to ‘hue’@'%' identified by 'Hue_123' with grant option;
mysql >grant all privileges on cdh_hue.* to 'hue'@'node-01.flyfish' identified by 'Hue_123' with grant option;
mysql >flush privileges; 

建立CM 的 監控端Activity Monitor 的庫
Activity Monitor:
mysql >create database cdh_am default character set utf8; 
mysql >grant all privileges on cdh_am.* to ‘am’@'%' identified by 'Am_123' with grant option;
mysql >grant all privileges on cdh_am.* to 'am'@'node-01.flyfish' identified by 'Am_123' with grant option;
mysql >flush privileges;

建立RM 的 監控端 Reports Manager 的庫
Reports Manager
mysql >create database cdh_rm default character set utf8; 
mysql >grant all privileges on cdh_rm.* to 'rm’@'%' identified by 'Rm_123' with grant option;
mysql >grant all privileges on cdh_rm.* to 'rm'@'node-01.flyfish' identified by 'Rm_123' with grant option;
mysql >flush privileges;

image_1cfkcvc7s1j1i1ei71sos15tk1s5744.png-200.1kB

image_1cfkd01d1s2h14gh1jl1feai0i4h.png-180.7kB

image_1cfkd0f61dju4v2pgp54eke4u.png-128.2kB

image_1cfkd4vl01vmh9f317t41gai1a965b.png-462.9kB

image_1cfkd6de6h4sdmb80p1thu4f65o.png-380.6kB

image_1cfkd6uaqm1uo7m1frjk71ad765.png-439.7kB

image_1cfkd7njnf1t1tod34m1c3bmeu6i.png-372.4kB

image_1cfkdjtup2kes509t71klt134c6v.png-462.5kB

image_1cfkdl1er1gar1q01klhdjq1vj27s.png-207.3kB

image_1cfkdlqe41dben6umg1v011tl489.png-524.8kB

image_1cfke4i101op81o711asq5gaos28m.png-526.1kB

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程