CDH的hadoop與Spark套件組安裝

1.解壓JDBC並複製到cm目錄中

tar -zxvf mysql-connector-java-5.1.43.tar.gz 

cp /opt/mysql-connector-java-5.1.43/mysql-connector-java-5.1.43-bin.jar /opt/cm-5.10.0/share/cmf/lib/

2.初始化CM5數據庫

/opt/cm-5.10.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123456 --scm-host localhost scm scm scm

3.配置server_host並複製

vim /opt/cm-5.10.0/etc/cloudera-scm-agent/config.ini

注意：裏面的server_host必定要配成主機的名稱，如server_host=spark001

scp -r /opt/cm-5.10.0 root@spark002:/opt/

4.添加新用戶

注意是每一個節點都要添加

useradd --system --home=/opt/cm-5.10.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

5.將CDH5的壓縮包移動到/opt/cloudera/parcel-repo/

#移動文件命令
mv CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel /opt/cloudera/parcel-repo/
mv CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha1 /opt/cloudera/parcel-repo/
mv manifest.json /opt/cloudera/parcel-repo/
#重命名成sha
mv CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha1 CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha

6.啓動主節點和從節點

1.主節點啓動

#啓動服務器
/opt/cm-5.10.0/etc/init.d/cloudera-scm-server start
#啓動Agent服務
/opt/cm-5.10.0/etc/init.d/cloudera-scm-agent start

2.啓動其餘節點

#啓動Agent服務
/opt/cm-5.10.0/etc/init.d/cloudera-scm-agent start

7.解決交換設置及透明大頁面壓縮問題

在每一個節點上運行

echo 0 > /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag