kudu_CM安裝準備工做

 

Cloudera Manager簡介：

hadoop: https://yq.aliyun.com/articles/60759

----------------------------------------------下載----------------------------------------------

--下載安裝包：
1.Cloudera Manager倉庫鏡像包下載地址：
https://archive.cloudera.com/cdh5/parcels/5.12/

CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel.sha1
manifest.json

2.CDH parcel安裝包地址：
http://archive.cloudera.com/cm5/repo-as-tarball/5.12.2/

cm5.12.2-centos7.tar.gz

3.kudu的parcel安裝包地址：
http://archive.cloudera.com/kudu/parcels/latest/

KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel.sha1

4.kudu 的csd安裝包地址：
http://archive.cloudera.com/kudu/csd/
注意： 經過 Cloudera Manager 安裝parcel時sha1格式的文件須要提早修改成sha。

----------------------------------------------準備工做----------------------------------------------

--配置

修改主機名：
hostnamectl set-hostname Node1

--1.修改hosts文件 全部節點

[root@node1 .ssh]# cat /etc/hosts
127.0.0.1 localhost.Node1
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.171 Node1
192.168.2.172 Node2
192.168.2.173 Node3　

--2.配置公鑰認證 免密登陸
2.1 配置公鑰認證
ssh-keygen -t rsa
一路回車，生成無密碼的密鑰對

2.2 將公鑰添加到認證文件中：

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

2.3 scp文件到全部受管節點（ Node2 和 Node3）的~/.ssh目錄：

scp ~/.ssh/authorized_keys root@Node2:~/.ssh/
scp ~/.ssh/authorized_keys root@Node3:~/.ssh/

2.4若是沒有.ssh目錄，手動建立並賦權限

[root@node1 ~]# mkdir .ssh
[root@node1 ~]# chmod 700 .ssh/

--3.關閉防火牆

[root@node1 .ssh]# systemctl stop firewalld.service
[root@node1 .ssh]# firewall-cmd --state
not running

　　

--4.關閉SELINUX
sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
setenforce 0

--5.配置NTP服務 同步時間

--6.優化虛擬內存需求率(全部節點)
1)檢查虛擬內存需求率
　　cat /proc/sys/vm/swappiness

2)臨時下降虛擬內存需求率
　　sysctl vm.swappiness=0

永久下降虛擬內存需求率
使用命令 vi /etc/sysctl.conf 增長
　　vm.swappiness = 0
並運行以下命令使生效
　　sysctl -p

--7.解決透明大頁面問題(全部節點)

1)檢查透明大頁面問題

執行代碼：

echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo never > /sys/kernel/mm/transparent_hugepage/enabled　　

以禁用此設置，而後將同一命令添加到 /etc/rc.local 等初始化腳本中，以便在系統重啓時予以設置。

cat /sys/kernel/mm/transparent_hugepage/defrag

　　若是顯示爲：

　　[always] madvise never

 

2)臨時關閉透明大頁面問題
echo never > /sys/kernel/mm/transparent_hugepage/defrag
確認配置生效：
cat /sys/kernel/mm/transparent_hugepage/defrag
應該顯示爲：
always madvise [never]
3)配置開機自動生效
使用命令 vim /etc/rc.local,加入以下內容

echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo never > /sys/kernel/mm/transparent_hugepage/enabled

--8.安裝JDK

卸載自帶的OpenJdk，
使用 rpm -qa | grep java 查詢java相關的包，
使用 rpm -e --nodeps 包名 卸載。

 

--------------------------------------- -----------------安裝配置MariaDB--------------------------------------- 

a. 確保/var/lib/mysql目錄有足夠大的磁盤空間
b. 卸載自帶的mysql。
　　yum remove mysql
1.添加yum源

vi /etc/yum.repos.d/MariaDB.repo

[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.1/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1　　

2.安裝MariaDB
　　yum install MariaDB-server MariaDB-client MariaDB-devel --enablerepo=MariaDB.repo

 

3.配置my.conf
vim /etc/my.cnf
用如下內容替換

[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

3.啓動MariaDB
　　service mysql start

啓動失敗：
失敗1：
    /usr/bin/mysqld_safe_helper: Can't create/write to file '/var/log/mysqld.log' (Errcode: 13 "Permission denied")
受權：
    chown -R mysql /var/log/
失敗2：
    2018-02-05 11:22:22 139647932491520 [ERROR] mysqld: Can't create/write to file '/var/run/mysqld/mysqld.pid' (Errcode: 13 "Permission denied")
    2018-02-05 11:22:22 139647932491520 [ERROR] Can't start server: can't create PID file: Permission denied
解決：
[root@Node1 log]# cat /var/log/mysqld.log
[root@Node1 run]# mkdir mysqld
[root@Node1 log]# chown -R mysql /var/run/mysqld
[root@Node1 log]# chgrp -R mysql /var/run/mysqld

 

 

4.查看MariaDB版本
mysql --version

5.設置開機啓動
chkconfig mysql on

6.初始化數據庫

sudo /usr/bin/mysql_secure_installation

Enter current password for root (enter for none):  Enter

Set root password? [Y/n] y
New password: 123456
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!

Remove anonymous users? [Y/n] y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

7.使用mysql -uroot -p進入mysql命令行，建立數據庫和用戶：

 

[root@Node1 log]# mysql -uroot -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 12
Server version: 10.1.30-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database hive DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database hue DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database oozie DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';   
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> exit
Bye
 

 

-------------------------------------------安裝mysql JDBC 驅動（管理節點）-----------------------------------------

下載mysql JDBC 驅動放到目錄 /usr/share/java/ 並修更名爲mysql-connector-java.jar
下載地址：https://dev.mysql.com/downloads/connector/j/

[root@Node1 mysql-connector-java-5.1.45]# cp mysql-connector-java-5.1.45-bin.jar /usr/share/java/mysql-connector-java.jar

 

----------------------------------------- 安裝配置 Cloudera Manager（管理節點）---------------------------------------
創建Cloudera Manager安裝文件自定義存儲庫

a.安裝httpd服務器

查詢一下是否已經安裝了apache
rpm -qa httpd 
若是尚未則進行安裝
yum -y install httpd
啓動apache
service httpd start
開機自啓動
chkconfig httpd on

1.將Cloudera Manager倉庫鏡像包cm5.12.2-centos7.tar.gz
解壓到/var/www/html目錄，文件目錄結構以下
[root@Node1 cm]# tar zxvf /root/softwares/kudu/cm5.12.2-centos7.tar.gz -C /var/www/html

2.經過rpm安裝包本地安裝 Cloudera Manager
到目錄 /var/www/html/cm/5/RPMS/x86_64

--本地安裝rpm包，不驗證簽名

yum --nogpgcheck localinstall cloudera-manager-daemons-5.12.*.rpm cloudera-manager-server-5.12.*.rpm enterprise-debuginfo-5.12.*.rpm

3.Parcel和csd格式文件上傳
a. 上傳下列文件到Parcel包的存放路徑： /root/cloudera/parcel-repo/
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel.sha1
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel.sha1
manifest.json

 

 

4.上傳最新的kudu csd文件到csd包存放路徑： /root/cloudera/csd/
KUDU-5.10.2.jar

 

5.配置 Cloudera Manager Server 數據庫

使用命令scm_prepare_database.sh建立Cloudera Manager Server數據庫配置文件
命令格式以下
/usr/share/cmf/schema/scm_prepare_database.sh database-type [options] database-name username password
如：

 

/usr/share/cmf/schema/scm_prepare_database.sh mysql -hNode1 -uroot -p123456 --scm-host Node1 scm scm scm

成功：

[root@node1 cloudera-scm-server]# /usr/share/cmf/schema/scm_prepare_database.sh mysql -hNode1 -uroot -p123456 --scm-host Node1 scm scm scm
JAVA_HOME=/usr/java/jdk1.7.0_79
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.7.0_79/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
[root@node1 cloudera-scm-server]#　

執行完成後生成數據庫配置文件/etc/cloudera-scm-server/db.properties
# Auto-generated by scm_prepare_database.sh on 2018年 01月 10日 星期三 14:42:32 CST
#
# For information describing how to configure the Cloudera Manager Server
# to connect to databases, see the "Cloudera Manager Installation Guide."
#
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.setupType=EXTERNAL
com.cloudera.cmf.db.password=scm

報錯：
mysql>use mysql;
mysql>desc user;
mysql>GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY "123456";　　//爲root添加遠程鏈接的能力。
mysql>update user set Password = password('123456') where User='root';
mysql>select Host,User,Password  from user where User='root'; 
mysql>flush privileges;
mysql>exit

grant all on scm.* to 'root'@'%' identified by '123456' with grant option;

cat /etc/cloudera-scm-server/db.properties

6.啓動Cloudera Manager Server

[root@node1 ~]# service cloudera-scm-server start
Starting cloudera-scm-server (via systemctl): Warning: cloudera-scm-server.service changed on disk. Run 'systemctl daemon-reload' to reload units.
[ 肯定 ]
[root@node1 ~]# systemctl daemon-reload
[root@node1 ~]# service cloudera-scm-server start
Starting cloudera-scm-server (via systemctl): [ 肯定 ]

啓動失敗報錯沒有JDK
一個是OS的環境變量，一個是腳本運行時候的設置
[root@Node1 init.d]# pwd
/etc/init.d
[root@Node1 init.d]# vim cloudera-scm-server
添加：
export JAVA_HOME=/root/jdk1.7.0_79

再次啓動：
[root@Node1 init.d]# service cloudera-scm-server start
查看日誌：
[root@Node1 init.d]# cat /var/log/cloudera-scm-server/cloudera-scm-server.out 
JAVA_HOME=/root/jdk1.7.0_79
/usr/share/cmf/bin/cmf-server:行129: /root/jdk1.7.0_79/bin/java: 權限不夠
/usr/share/cmf/bin/cmf-server: 第 129 行:exec: /root/jdk1.7.0_79/bin/java: 沒法執行: 權限不夠

修改jdk路徑
[root@Node1 ~]# cd /usr/
[root@Node1 usr]# mkdir java/
[root@Node1 init.d]# echo $JAVA_HOME
/usr/java/jdk1.7.0_79

再次啓動：
[root@Node1 init.d]# service cloudera-scm-server start

--CM

等待大概兩分鐘，訪問 http://192.168.2.171:7180/ 進入管理端 (登錄名:admin 密碼:admin)

--kuduhttp://192.168.2.171:8051/