CDH安裝筆記

系統環境準備

一、網絡配置

vi /etc/sysconfig/network

vi /etc/hosts

 

二、ssh免密鑰登陸

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

scp /etc/hosts node3:/etc/

cat /tmp/id_dsa.pub >> authorized_keys

 

三、防火牆關閉

service iptables stop

chkconfig iptables off

 

四、SELINUX關閉

setenforce 0

vi /etc/selinux/config (SELINUX=disabled)

 

五、安裝JDK配置環境變量

vi /etc/profile

export JAVA_HOME=/opt/jdk

export PATH=$PATH:$JAVA_HOME/bin:

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

 

六、安裝NTP

設置開機啓動 chkconfig ntpd on

設置時間同步

ntpdate -u s2m.time.edu.cn或ntpdate -u time.nist.gov

 

七、安裝配置mysql（在server機器上安裝mysql）

yum install mysql-server

service mysqld start

chkconfig mysqld on

chkconfig --list mysqld

use mysql;

select host, user from user;

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123' WITH GRANT OPTION;

delete from user where host != '%';

flush privileges;

 

八、下載第三方依賴包

yum install chkconfig python bind-util psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb -y

 

node1	server agent	mysql	依賴包
node2	agent		依賴包
node3	agent		依賴包

 

CM安裝

一、COPY文件

mkdir /opt/cloudera-manager

tar zxvf cloudera-manager*.tar.gz -C /opt/cloudera-manager

 

二、建立用戶cloudera-scm

useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

 

下載cm網址

http://archive.cloudera.com/cm5/cm/5/

 

下載cdh5

http://archive.cloudera.com/cdh5/parcels/

 

三、配置CM Agent

修改文件

vi /opt/cloudera-manager/cm-5.4.7/etc/cloudera-scm-agent/config.ini

 

修改節點名稱server_host

 

四、配置cm server數據庫

mkdir /usr/share/java

上傳mysql的jar包，名稱必須修改爲mysql-connector-java.jar

grant all on *.* to 'temp'@'%' identified by 'temp' with grant option;

cd /opt/cloudera-manager/cm-5.4.7/share/cmf/schema/

./scm_prepare_database.sh mysql temp -h node1 -utemp -ptemp --scm-host node1 scm scm scm

 

提示All done, your SCM database is configured correctly!

則表示成功

 

五、建立Parcel目錄

server節點

mkdir -p /opt/cloudera/parcel-repo

chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

agent節點

mkdir -p /opt/cloudera/parcels

chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

 

六、製做CDH本地源

將

manifest.json

CDH-5.9.0-1.cdh5.9.0.p0.23-el5.parcel.sha1

CDH-5.9.0-1.cdh5.9.0.p0.23-el5.parcel

文件移動到/opt/cloudera/parcel-repo目錄下

 

七、啓動CM Server、Agent

cd /opt/cloudera-manager/cm-5.4.7/etc/init.d/

./cloudera-scm-server start

./cloudera-scm-agent start

 

查看日誌

tail -f /opt/cloudera-manager/cm-5.4.7/log/cloudera-scm-server/cloudera-scm-server.log

 

登陸

http://ip:7180

echo 10 > /proc/sys/vm/swappiness

echo never > /sys/kernel/mm/transparent_hugepage/defrag

 

建立hive數據庫

create database hive DEFAULT CHARACTER SET utf8;

grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';

 

建立oozie數據庫

create database oozie DEFAULT CHARACTER SET utf8;

grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

 

建立sqoop數據庫

create database sqoop DEFAULT CHARACTER SET utf8;

grant all on sqoop.* TO 'sqoop'@'%' IDENTIFIED BY 'sqoop';

以上內容參考北京尚學堂視頻課程