Cloudera Manager 安裝 CDH5.x 心得

Cloudera Manager 安裝 CDH5.x 心得

廢話很少說，先展現下這幾天搗鼓的成果

• Cloudera Manager 管理配置界面

• Hbase 管理界面及Hbase Web UI

[站外圖片上傳中...(image-1c6abb-1513138023093)]

• HDFS 管理界面 及 NameNode Web UI

[站外圖片上傳中...(image-82c66f-1513138023093)]

• Hive 管理界面 及 HiveServer2 Web UI

• Hue 管理界面 及 Web UI

首次登陸會提示讓建立管理員帳號和密碼，筆者 admin admin

• Oozie 管理界面 及 Oozie Web UI

# Oozie  Web UI 報錯，後期修復

Oozie web console is disabled.
To enable Oozie web console install the Ext JS library.

參考：http://cdh01.aniu.so:11000/oozie/docs/DG_QuickStart.html

• YARN (MR2 Included) 管理界面 及 Web UI

• ResourceManager Web UI

• HistoryServer Web UI

[站外圖片上傳中...(image-47c126-1513138023093)]

• Zookeeper 管理界面

筆者這裏zookeeper安裝的時候選擇的默認，所以只安裝了一個zookeeper，但我的感受後期應該須要增長zookeeper的界面數量

下面開始說安裝的注事事項

• 一、配置環境要符合要求，要純淨的系統環境

# 筆者環境
# CM env
192.168.1.137 cdh01.aniu.so CentOS6.9 16G Memory 100G LVM卷 （Manger 節點）
192.168.1.148 cdh02.aniu.so CentOS6.9 4G Memory 70G LVM卷
192.168.1.149 cdh03.aniu.so CentOS6.9 4G Memory 70G LVM卷
192.168.1.150 cdh04.aniu.so CentOS6.9 4G Memory 70G LVM卷

建議小白參考筆者的環境配置，主機名能夠自定義

#對四個節點的系統進行更新，安裝開發工具包
yum update -y && yum -y groupinstall "Development Tools"

• 二、關閉防火牆、禁用Selinux

# 關閉防火牆
/etc/init.d/iptables stop && /etc/init.d/ip6tables stop
chkconfig iptables off && chkconfig ip6tables off

# 建議採用修改內核參數的方式關閉ip6tables 
vim /etc/modprobe.d/dist.conf # 編輯此文件，在最後加入：

# Disable ipv6
alias net-pf-10 off
alias ipv6 off

# 禁用selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0  # 不重啓臨時生效

• 三、內核參數調整

# 內存參數調整
sysctl -w vm.swappiness=10 或者 編輯vim /etc/sysctl.conf，在最後加入：

vm.swappiness = 10

編輯啓動項vim /etc/rc.local,最後加入：
echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo never > /sys/kernel/mm/transparent_hugepage/enabled

注：上面全部操做在全部節點都須要執行

• 四、全部節點間配置免密認證

# CM節點執行
ssh-keygen -t rsa -b 2048 # 有確認提示，一直按回車便可
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
# 筆者 hosts.conf
# CM node
192.168.1.137 cdh01.aniu.so
192.168.1.148 cdh02.aniu.so
192.168.1.149 cdh03.aniu.so
192.168.1.150 cdh04.aniu.so

# 同步密鑰
for ip in $(awk '{print $1}' hosts.conf );do scp ~/.ssh/authorized_keys root@$ip:/root/.ssh ;done
或者使用
ssh-copy-id root@cdh01.aniu.so
ssh-copy-id root@cdh02.aniu.so
ssh-copy-id root@cdh03.aniu.so
ssh-copy-id root@cdh04.aniu.so
# 上面操做也須要在全部節點執行

• 五、使用cloudera-manger repo安裝CM

# 在CM節點執行
wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d
wget https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/cloudera-cdh5.repo -P /etc/yum.repos.d
# 
yum clean all && yum makecache # 建議執行不強制
yum install oracle-j2sdk1.7 -y
yum install cloudera-manager-daemons cloudera-manager-server -y

# 在其餘節點執行
wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d
yum install oracle-j2sdk1.7 -y

# 配置JAVA_HOME
編輯vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
保存退出執行：
source /etc/profile 使更改的環境變量生效

# 在全部節點執行配置JAVA_HOME的操做

• 六、CM節點安裝數據庫，或使用已有的數據

# 筆者使用mysql57-community.repo,安裝的mysql
[mysql57-community]
name=MySQL 5.7 Community Server
baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/6/$basearch/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
#
yum install mysql-community-embedded  mysql-community-server  mysql-community-devel mysql-community-client -y

# 筆者my.cnf
[root@cdh01 yum.repos.d]# cat /etc/my.cnf
[client]
port        = 3306
socket      = /var/lib/mysql/mysql.sock

[mysqld]
datadir     = /opt/mysql
socket      = /var/lib/mysql/mysql.sock
#skip-grant-tables 
skip-ssl
disable-partition-engine-check
port        = 3306
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 1M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
thread_cache_size = 8
query_cache_size = 8M
tmp_table_size = 16M
performance_schema_max_table_instances = 500

explicit_defaults_for_timestamp = true
max_connections = 500
max_connect_errors = 100
open_files_limit = 8192

log-bin=mysql-bin
binlog_format=mixed
server-id   = 1
expire_logs_days = 10
early-plugin-load = ""

default_storage_engine          = InnoDB
innodb_file_per_table           = 1
innodb_data_home_dir            = /opt/mysql
innodb_data_file_path           = ibdata1:1024M;ibdata2:10M:autoextend
innodb_log_group_home_dir       = /opt/mysql
innodb_buffer_pool_size         = 16M
innodb_log_file_size            = 5M
innodb_log_buffer_size          = 8M
innodb_flush_log_at_trx_commit  = 1
innodb_lock_wait_timeout        = 50
innodb_log_files_in_group       = 3 
innodb_buffer_pool_size         = 12G
innodb_log_file_size            = 512M
innodb_log_buffer_size          = 256M
innodb_flush_log_at_trx_commit  = 2
innodb_lock_wait_timeout        = 150
innodb_open_files               = 600
innodb_max_dirty_pages_pct      = 50
innodb_file_per_table           = 1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
xinteractive-timeout

symbolic-links=0
slow_query_log
long_query_time = 5
slow_query_log_file = /var/log/mysql-slow.log
log-error = /var/log/mysqld.log
pid-file = /var/run/mysqld/mysqld.pid

# 初始化mysql，並設置啓動數據庫設置root密碼
/usr/sbin/mysqld --initialize --user=mysql --socket=/var/lib/mysql/mysql.sock # 先執行
mysql_secure_installation # 再執行

# 建立CM啓動用到的數據庫
mysql -u root -pAniuops123. -e "create database cmf DEFAULT CHARACTER SET utf8;"
mysql -u root -pAniuops123. -e "GRANT ALL PRIVILEGES ON `cmf`.* TO 'cmf'@'localhost' IDENTIFIED BY 'Aniunas123.'";"

• 啓動cloudera-scm-server，並配置parcel

# 生成db配置文件
/usr/share/cmf/schema/scm_prepare_database.sh mysql cmf cmf Aniucmf123.

# 啓動cloudera-scm-server
/etc/init.d/cloudera-scm-server start # 查看啓動日誌

# 配置parcel離線
cd /opt/cloudera/parcel-repo/ # 而後下載

wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel
wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha1
wget http://archive.cloudera.com/cdh5/parcels/latest/manifest.json

# 注：讀者根據cloudera當前CDH最新版本更改下載用到的URL
mv CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha1 CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha # 強制執行、默認使用本地的parcels包，不更改sha1,cloudera-scm-server啓動安裝時會去cloudera官網找匹配的parcel安裝包

重啓cloudera-scm-server，查看實時日誌
/etc/init.d/cloudera-scm-server restart
tailf /var/log/cloudera-scm-server/cloudera-scm-server.log

• 經過CM管理界面安裝CDH，注意事項

# CM server啓動成功便可經過http://192.168.1.137:7180訪問，默認帳戶密碼：admin admin

# **重點內容** 下面的話很重要：


不要勾選：單用戶模式 ，筆者在此模式下安裝屢次都沒成功，有心人能夠測試

• 能一次性安裝成功的最好，安裝不成功建議多試幾回，對初始化完成的虛擬機進行快照操做，便於恢復