【原創】大數據基礎之CM5（Cloudera Manager）+CDH5離線安裝

CM/CDH 5.16.1

CM官方：https://www.cloudera.com/products/product-components/cloudera-manager.html

CDH官方：https://www.cloudera.com/products/open-source/apache-hadoop/key-cdh-components.html

CDH5.16.1 軟件列表：https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_516.html

 

一 簡介

1 CDH

CDH is Cloudera’s 100% open source platform distribution, including Apache Hadoop and built specifically to meet enterprise demands. CDH delivers everything you need for enterprise use right out of the box. By integrating Hadoop with more than a dozen other critical open source projects, Cloudera has created a functionally advanced system that helps you perform end-to-end Big Data workflows.

2 Cloudera Manager

Cloudera Manager is the industry’s trusted tool for managing Hadoop in production.

 

二 安裝 

1 準備jdk

1）卸載可能的openjdk

# rpm -qa|grep openjdk
java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64
java-1.8.0-openjdk-devel-1.8.0.191.b12-1.el7_6.x86_64
java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64
# rpm -e --nodeps java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 java-1.8.0-openjdk-devel-1.8.0.191.b12-1.el7_6.x86_64 java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64

2）安裝oracle jdk

# wget https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
# rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
# ln -s /usr/java/jdk1.8.0_141-cloudera/bin/java /usr/bin/java

2 建立用戶

# useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3 下載

# wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
# mkdir /opt/cloudera-manager
# tar xvzf cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz -C /opt/cloudera-manager

# mkdir -p /opt/cloudera/parcels
# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

4 準備server

1）確保文件存在：/usr/share/java/mysql-connector-java.jar

# cp mysql-connector-java.jar /usr/share/java/mysql-connector-java.jar

or

# yum install mysql-connector-java

2）初始化mysql

mysql> create database scm;
mysql> grant all privileges on scm.* to scm@'%' identified by 'scm';

# /opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh -h 192.168.0.1 mysql scm scm scm

3）其餘

# mkdir /var/lib/cloudera-scm-server
# chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server

4）下載cdh

# wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
# wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
# wget http://archive.cloudera.com/cdh5/parcels/latest/manifest.json
# mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha

# mkdir -p /opt/cloudera/parcel-repo
# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
# mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo

5 啓動server

# /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start

日誌位於 /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-server/

6 逐個啓動agent

# vi /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini
server_host=$server_ip

# /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start

日誌位於 /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-agent/

7 開機啓動

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent

修改

CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
->
CMF_DEFAULTS=${CMF_DEFAULTS:-/opt/cloudera-manager/cm-5.16.1/etc/default}

不然會報錯

File not found: /usr/sbin/cmf-agent

修改後同步到全部節點，而後配置開機啓動

# ln -s /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
# chkconfig --add cloudera-scm-agent
# chkconfig cloudera-scm-agent on
# chkconfig --list cloudera-scm-agent

# ln -s /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
# chkconfig --add cloudera-scm-server
# chkconfig cloudera-scm-server on
# chkconfig --list cloudera-scm-server

8 訪問

http://$server_ip:7180 用戶名密碼爲 admin/admin 進入嚮導頁面，一步步配置完成；

 

 

當使用一些功能時須要安裝新的庫，具體以下：

好比Activity Monitor須要

mysql> create database amon;
mysql> grant all privileges on amon.* to amon@'%' identified by 'amon';

 

可能遇到的問題

1）若是看不到下載版本的cdh，檢查CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1是否修改成CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha

2）正在安裝選定 Parcel
報錯：主機運行情況不良。

查看agent日誌：

# tail -f /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-agent/cloudera-scm-agent.log
[15/Feb/2019 14:44:28 +0000] 7241 MainThread agent ERROR Error, CM server guid updated, expected b6f92bed-e8fc-45e0-b627-fead8cd288e6, received 1db60b99-1f0c-4226-bd2f-a9baa568cf96

解決方法

# /bin/rm /opt/cloudera-manager/cm-5.16.1/lib/cloudera-scm-agent/cm_guid
# /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent restart

安裝parcel正常

3）若是安裝過程當中出現任何問題，將server和agent停掉，而後重複‘初始化mysql’過程，而後重啓server和agent便可；

 

參考：https://www.cloudera.com/documentation/enterprise/5-16-x/topics/installation.html