017-Centos7.6+CDH 6.2 安裝和使用

時間 2019-12-12

標籤 centos7.6+cdh centos cdh 6.2 安裝使用欄目 CentOS 简体版

原文原文鏈接

這是堅持技術寫做計劃（含翻譯）的第17篇，定個小目標999，每週最少2篇。html

本文主要介紹hadoop發行版CDH最新版(6.2)的安裝。java

準備

硬件配置

IP	HostName	OS	Cores	Mem	Disk	Rules
192.168.20.24	cdh01.anjia.com	centos7.6 mini	4	16G	100G	CM6,CDH agent
192.168.20.25	cdh02.anjia.com	centos7.6 mini	4	16G	100G	CDH agent
192.168.20.28	cdh03.anjia.com	centos7.6 mini	4	16G	100G	CDH agent

軟件包

本地搭建鏡像源，加速構建。參考 018-CDH6.2構建本地源加速CDH安裝mysql

安裝前準備

CM存儲空間規劃

參考 Storage Space Planning for Cloudera Manager sql

配置網絡名

# cdh01.anjia.com 192.168.20.24
sudo hostnamectl set-hostname cdh01.anjia.com
cat << EOF | sudo tee -a /etc/hosts
192.168.20.24  cdh01.anjia.com  cdh01
192.168.20.25  cdh02.anjia.com  cdh02
192.168.20.28  cdh03.anjia.com  cdh03
EOF
echo "HOSTNAME=cdh01.anjia.com" >> /etc/sysconfig/network

# cdh02.anjia.com 192.168.20.25
sudo hostnamectl set-hostname cdh02.anjia.com
cat << EOF | sudo tee -a /etc/hosts
192.168.20.24  cdh01.anjia.com  cdh01
192.168.20.25  cdh02.anjia.com  cdh02
192.168.20.28  cdh03.anjia.com  cdh03
EOF
echo "HOSTNAME=cdh02.anjia.com" >> /etc/sysconfig/network

# cdh03.anjia.com 192.168.20.28
sudo hostnamectl set-hostname cdh03.anjia.com
cat << EOF | sudo tee -a /etc/hosts
192.168.20.24  cdh01.anjia.com  cdh01
192.168.20.25  cdh02.anjia.com  cdh02
192.168.20.28  cdh03.anjia.com  cdh03
EOF
echo "HOSTNAME=cdh03.anjia.com" >> /etc/sysconfig/network
複製代碼

參考 Configure Network Names數據庫

禁用防火牆

每臺都執行centos

sudo iptables-save > ~/firewall.rules
sudo systemctl disable firewalld
sudo systemctl stop firewalld
複製代碼

參考 Disabling the Firewallbash

開啓ntp同步時鐘

每臺都執行網絡

yum install -y ntp
sed -i "/^server/ d" /etc/ntp.conf
cat << EOF | sudo tee -a /etc/ntp.conf
server ntp1.aliyun.com
server ntp2.aliyun.com
server ntp3.aliyun.com
server ntp4.aliyun.com
EOF
sudo systemctl start ntpd
sudo systemctl enable ntpd
ntpdate -u ntp1.aliyun.com
hwclock --systohc
複製代碼

修改repo

每臺都執行oracle

sudo wget https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
sed -i "s/https\:\/\/archive.cloudera.com/http\:\/\/192.168.20.24\/cloudera-repos/g"  /etc/yum.repos.d/cloudera-manager.repo
sudo rpm --import https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera
複製代碼

優化虛擬內存需求率

每臺都執行app

echo 'vm.swappiness = 10' >> /etc/sysctl.conf
sysctl -p
複製代碼

解決透明大頁面問題

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
複製代碼

安裝jdk

每臺都執行

sudo yum install -y oracle-j2sdk1.8

cat << EOF | sudo tee -a /etc/profile
#設置java環境
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/
export CLASSPATH=.:\$JAVA_HOME/lib:\$JAVA_HOME/jre/lib:\$CLASSPATH
export PATH=\$JAVA_HOME/bin:\$JAVA_HOME/jre/bin:\$PATH
EOF
source /etc/profile
複製代碼

安裝CM和CDH

安裝CM

在cdh01.anjia.com執行

sudo yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
複製代碼

啓用Auto-TLS,在CM6.2.0人工不啓用auto-tls會致使登錄不成功

sudo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services
複製代碼

安裝數據庫

本文選擇MariaDB
在cdh01.anjia.com執行

sudo yum install -y mariadb-server
sudo systemctl stop mariadb
cat << EOF | sudo tee -a /etc/my.cnf.d/cdh-db.cnf
[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MariaDB, if you enable the binary log and do not set
#a server_id, MariaDB will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
EOF


sudo systemctl enable mariadb
sudo systemctl start mariadb

sudo /usr/bin/mysql_secure_installation

[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
[...]
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
[...]
All done!  If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB! 複製代碼

安裝數據庫驅動

每臺都執行

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
複製代碼

建立數據庫

在cdh01.anjia.com執行

mysql -u root -p<password>

CREATE DATABASE <database> DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';
複製代碼

這一步scm是必須的，其他的數據庫能夠等後邊真實使用時再建立。

Service	Database	User
Cloudera Manager Server	scm	scm
Activity Monitor	amon	amon
Reports Manager	rman	rman
Hue	hue	hue
Hive Metastore Server	metastore	hive
Sentry Server	sentry	sentry
Cloudera Navigator Audit Server	nav	nav
Cloudera Navigator Metadata Server	navms	navms
Oozie	oozie	oozie

配置CM數據庫

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh [options] <databaseType> <databaseName> <databaseUser> <password>
# 輸出
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!
複製代碼

詳細參數，參考 Syntax for scm_prepare_database.sh

安裝CDH和其餘軟件

sudo systemctl start cloudera-scm-server
# 啓動後，監聽server日誌，大約1-3分鐘左右
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
# 直到日誌出現，以下內容，即啓動成功。
2019-05-08 13:12:26,523 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
複製代碼

參考資料