最近本身寫了一個Hadoop自動化部署腳本，包括Hadoop集羣自動化部署腳本和Hadoop增長單節點自動化部署腳本。須要快速部署Hadoop集羣的童鞋能夠使用該腳本。這些腳本我在用5臺虛擬機進行了測試，若是在使用中還有bug，歡迎指出。本文主要介紹Hadoop集羣自動化部署腳本，安裝的Hadoop版本爲2.6.0。html
安裝Hadoop2.6.0集羣須要依賴JDK和Zookeeper。本文安裝的JDK版本爲jdk-7u60-linux-x64，Zookeeper版本爲zookeeper-3.4.6。java
該部署腳本由兩部分構成：root用戶下執行的腳本和Hadoop啓動用戶下執行的腳本。這些腳本都只須要在一臺服務器上執行便可，執行腳本的服務器做爲Hadoop的Master服務器。下面分別進行說明。node
root腳本的目錄結構以下：linux
該目錄下的init.conf文件爲root執行腳本使用的配置文件，在執行腳本以前須要對該配置文件進行修改。文件內容以下：shell
#jdk file and version JDK_FILE_TAR=jdk-7u60-linux-x64.tar.gz #jdk unpack name JDK_FILE=jdk1.7.0_60 #java home JAVAHOME=/usr/java #Whether install the package for dependence,0 means no,1 means yes IF_INSTALL_PACKAGE=1 #host conf ALLHOST="hadoop1master hadoop1masterha hadoop1slave1 hadoop1slave2 hadoop1slave3" ALLIP="192.168.0.180 192.168.0.184 192.168.0.181 192.168.0.182 192.168.0.183" #zookeeper conf ZOOKEEPER_TAR=zookeeper-3.4.6.tar.gz ZOOKEEPERHOME=/usr/local/zookeeper-3.4.6 SLAVELIST="hadoop1slave1 hadoop1slave2 hadoop1slave3" #hadoop conf HADOOP_TAR=hadoop-2.6.0.tar.gz HADOOPHOME=/usr/local/hadoop-2.6.0 HADOOP_USER=hadoop2 HADOOP_PASSWORD=hadoop2 #root conf: $MASTER_HA $SLAVE1 $SLAVE2 $SLAVE3 ROOT_PASSWORD="hadoop hadoop hadoop hadoop"
下面是個別參數的解釋及注意事項：bootstrap
該目錄下包含password.expect、scp.expect、otherInstall.expect三個文件。password.expect用來設置hadoop啓動用戶的密碼；scp.expect用來遠程傳輸文件；otherInstall.expect用來遠程執行其餘服務器上的installRoot.sh。這三個文件都在installRoot.sh中被調用。
password.expect文件內容以下：bash
#!/usr/bin/expect -f set user [lindex $argv 0] set password [lindex $argv 1] spawn passwd $user expect "New password:" send "$password\r" expect "Retype new password:" send "$password\r" expect eof
其中argv 0和argv 1都是在installRoot.sh腳本中進行傳值的。其餘兩個文件argv *也是這樣傳值的。
scp.expect文件內容以下：服務器
#!/usr/bin/expect -f # set dir, host, user, password set dir [lindex $argv 0] set host [lindex $argv 1] set user [lindex $argv 2] set password [lindex $argv 3] set timeout -1 spawn scp -r $dir $user@$host:/root/ expect { "(yes/no)?" { send "yes\n" expect "*assword:" { send "$password\n"} } "*assword:" { send "$password\n" } } expect eof
otherInstall.expect文件內容以下：ssh
#!/usr/bin/expect -f # set dir, host, user, password set dir [lindex $argv 0] set name [lindex $argv 1] set host [lindex $argv 2] set user [lindex $argv 3] set password [lindex $argv 4] set timeout -1 spawn ssh -q $user@$host "$dir/$name" expect { "(yes/no)?" { send "yes\n" expect "*assword:" { send "$password\n"} } "*assword:" { send "$password\n" } } expect eof
這裏就是安裝Hadoop集羣及其依賴所需的安裝包。oop
該腳本是在root用戶下須要執行的腳本，文件內容以下：
#!/bin/bash if [ $USER != "root" ]; then echo "[ERROR]:Must run as root"; exit 1 fi # Get absolute path and name of this shell readonly PROGDIR=$(readlink -m $(dirname $0)) readonly PROGNAME=$(basename $0) hostname=`hostname` source /etc/profile # import init.conf source $PROGDIR/conf/init.conf echo "install start..." # install package for dependence if [ $IF_INSTALL_PACKAGE -eq 1 ]; then yum -y install expect >/dev/null 2>&1 echo "expect install successful." # yum install openssh-clients #scp fi #stop iptables or open ports, now stop iptables service iptables stop chkconfig iptables off FF_INFO=`service iptables status` if [ -n "`echo $FF_INFO | grep "Firewall is not running"`" ]; then echo "Firewall is already stop." else echo "[ERROR]:Failed to shut down the firewall.Exit shell." exit 1 fi #stop selinux setenforce 0 SL_INFO=`getenforce` if [ $SL_INFO == "Permissive" -o $SL_INFO == "disabled" ]; then echo "selinux is already stop." else