A record--Offline deployment of Big Data Platform CDH Cluster

時間 2019-11-13

標籤 record offline deployment big data platform cdh cluster 简体版

原文原文鏈接

目錄html

A record--Offline deployment of Big Data Platform CDH Cluster

A record--Offline deployment of Big Data Platform CDH Cluster

Tags： Cloudera-Manager CDH Hadoop Deploy Clusterjava

Abstract: Deployment and Management of Hadoop clusters need tools, such as Cloudera Manager. In this article, I compare the tools briefly, and then record the step of deploying CDH cluster offline in detail. Finally, I expound the theory of 'handle delicately'.node

Preface

The emergence of Big Data technology led by Apache Hadoop, makes small and medium-sized enterprises also have the ability to handle the storage and processing of big data. At present, there'r lots of Hadoop distributions, such as HUAWEI Distribution, Intel Distribution, Cloudera’s Distribution Including Apache Hadoop (CDH free), and Hortonworks Data Platform (HDP free), etc. All of these are based on the Hadoop Apache Community Edition.python

The deployment and management of a Hadoop cluster which has tens or more nodes needs advanced tools. Apache Ambari from Hortonworks is this kind of tools, it provided an easy-to-use RESTfull web site to manage Hadoop. Cloudera also provided a similar tool, Cloudera Manager(CM) to configure, monitor and manage CDH clusters.mysql

The main content of this paper is a record of building a CDH cluster. Special attention is required to choose Cloudera Manager version, which depends on Operating system, el7 isn't supported by Cloudera Manager at this moment. You should follow [the official document][1], otherwise the installation will run into a stone wall.linux

This paper is based on CentOS 6.5, 64-bit；Cloudera Manager 5.3.6；JDK 1.7.web

Deploy CDH

Configure network (All nodes)

[root@cdh-server ~]# vi /etc/sysconfig/network #修改hostname:
NETWORKING=yes
HOSTNAME=cdh-server

[root@cdh-server ~]# vi /etc/hosts #修改ip與主機名的對應關係:
192.168.180.173 cdh-server
192.168.180.175 node175
[root@cdh-server ~]# service network restart #重啓網絡服務生效

Install JDK (All nodes)

#卸載OpenJDK
[root@cdh-server user1]# rpm -qa | grep java
[root@cdh-server user1]# rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
[root@cdh-server user1]# rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
[root@cdh-server user1]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
#安裝JDK
[root@cdh-server user1]# chmod a+x jdk-7u79-linux-x64.rpm
[root@cdh-server user1]# rpm -ivh jdk-7u79-linux-x64.rpm
[root@cdh-server user1]# echo "JAVA_HOME=/usr/java/jdk1.7.0_79/" >>

Install MySQL (Master)

[user1@cdh-server]$ cd /home/user1
[user1@cdh-server]$ tar -zxvf mysql-5.6.26-linux-glibc2.5-x86_64.tar.gz
[user1@cdh-server]$ mv mysql-5.6.26-linux-glibc2.5-x86_64 mysql-5.6.26
[user1@cdh-server]$ cd mysql-5.6.26/
[user1@cdh-server]$ vi support-files/my.cnf #新建文件

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[mysqld]
character-set-server=utf8
default-storage-engine=INNODB

# Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = /home/user1/mysql-5.6.26/data
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /home/user1/mysql-5.6.26/data
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 16M
innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Initialize MySQL (Master)

[user1@cdh-server]$ ./scripts/mysql_install_db --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data --user=user1
[user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &
[user1@cdh-server]$ ./bin/mysqladmin -u root password '123456'

[user1@cdh-server mysql-5.6.26]$ ./bin/mysql -uroot -p'123456'
#hive
mysql> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

#Activity Monitor使用
mysql> create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.01 sec)

#Navigator Audit Server使用
mysql> create database audit DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.01 sec)

#Navigator Metadata Server
mysql> create database metadata DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.01 sec)

mysql> grant all privileges on *.* to 'root'@'localhost' identified by '123456' with grant option;
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on *.* to 'root'@'cdh-server' identified by '123456' with grant option;
Query OK, 0 rows affected (0.00 sec)

#this user scm is for  cloudera manager
mysql> grant all privileges on *.* to 'scm'@'localhost' identified by 'scm' with grant option;
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on *.* to 'scm'@'cdh-server' identified by 'scm' with grant option;
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

Deploy & Start CM-Server (Master)

[user1@cdh-server ~]$ tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
[user1@cdh-server ~]$ cp mysql-connector-java-5.1.33-bin.jar ./cm-5.3.6/share/cmf/lib/

[user1@cdh-server ~]$ su - root
[root@cdh-server ~]# cd /home/user1/

[root@cdh-server user1]# cp -rf cloudera /opt
[root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
[root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
[root@cdh-server user1]# mv manifest.json /opt/cloudera/parcel-repo/manifest.json
[root@cdh-server user1]# ./cm-5.3.6/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost:3306 -uroot -p123456 --scm-host localhost scm scm scm
[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]
[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log

Stop iptables (All nodes)

#中止iptables
[root@cdh-server user1]# service iptables stop 
#經過瀏覽器訪問驗證
http://192.168.180.173:7180/

Deploy & Start CM-Agent (Slaves)

[root@cdh-server user1]# tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
[root@cdh-server user1]# vi cm-5.3.6/etc/cloudera-scm-agent/config.ini

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# Hostname of the CM server.
#server_host=localhost
server_host=cdh-server
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@cdh-server user1]# useradd -G sys --home=/home/user1/cm-5.3.6/run/cloudera-scm-server --no-create-home  --comment "Cloudera SCM User" cloudera-scm
[root@cdh-server user1]# useradd --comment "Cloudera SCM User" cloudera-scm #若上一步執行正確，則此步省略

[root@cdh-server user1]# echo 0 > /proc/sys/vm/swappiness

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]
[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log

Configure CDH

Load Cloudera Manager http://192.168.180.173:7180/, then create a new Cluster names Cluster_user1，Start and configure Services.sql

#安裝配置hive出錯時，在hiveServer上:
[root@hive-server user1]# cp mysql-connector-java-5.1.33-bin.jar /opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/hive/lib/

#同理：use this jar for Navigator Audit Server and Navigator Metadata Server or Activity Server
[root@cdh-server user1]# cp mysql-connector-java-5.1.33-bin.jar /usr/share/java/mysql-connector-java.jar

Others

Stop CDH

Stop Cloudera Management Service & Cluster_user1
Stop Agent (Slaves)

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent stop

Stop Server (Master)

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server stop

Start CDH

Start MySQL (Master)

[user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &

Start Agent (Slaves)

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]
[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log

Start Server (Master)

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]
[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log

The theory of handle delicately

Handle delicately is a kind of feeling, but also a skill. When in seller's market, a company can make a profit while may needn't to handle delicately. However, if it wishes to pursue more, it will handle more delicately, such as pay more attention to detail, User Experience or others. When in buyer's market, the relation between supply and demand makes companies handle delicately to survive.json

Handle delicately is not only the driving force pushing social to continually advance, but also the result of this advance. Today, lots of internet firms are in buyer's market, who has more users who will win the fight, delicately handling makes them at the forefront of social evolution and technological innovation.瀏覽器

Handle delicately doesn't only appear in companies, but also individuals, regions and countries, etc. The contry who handles more delicately, who is more developed. The company who pays more attention to details, who is more competitive. But, the one who handles more delicately doesn't mean it will make more profits, for various reasons.

Writer: @Angel Wang

aitanjupt@hotmail.com
2015 - 10 - 18
[1]: http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/pcm_os.html