hadoop集羣部署

首先說明下，整理的比較亂，遇到問題，解決問題便可

1. 須要確認部署的服務器ip

0 1 2 3 表明四個ip

另外須要一臺服務器，作遠程操控用

2. 在操控機上 執行 ssh-keygen，生成本機祕鑰文件（若是已經有，跳過本步驟），好比用戶 test，祕鑰文件路徑爲 /home/test/.ssh/

操控機上須要安裝ansible 

配置ansible安裝源

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-6.repo

 安裝ansible

1 yum -y install ansible 

 準備 ansible host清單文件： hosts

內容以下：

【hadoop_host】
0ip ansible_ssh_user=test
1ip ansible_ssh_user=test
2ip ansible_ssh_user=test 
3ip ansible_ssh_user=test

確保操控機到 hadoop集羣服務 的網路ok，至少ssh沒問題

前提工做準備完成後，準備進行初始化工做，初始化工做包括（sudo無密碼，遠程容許sudo執行，ulimit 系統調優）

1.首先設置test用戶執行sudo無密碼操做
前提是test用戶在wheel組
ansible -i ./hosts hadoop_host -m shell -a " sed 's/^# %wheel.*NOPASSWD: ALL/%wheel ALL=(ALL) NOPASSWD: ALL/' -i /etc/sudoers" -s --ask-sudo-pass-k 是要求輸入密碼選項
輸入密碼後，此命令的做用是test用在在遠程服務器上能夠sudo無密碼
2. 操做 test用戶能夠遠程執行sudo命令權限
ansible -i ./hosts hadoop_host -m shell -a " sed -i '/Defaults.*requiretty/a Defaults: test \!requiretty' /etc/sudoers" -s --ask-sudo-pass3. ulimit參數調整

ansible -i ./hosts hadoop_host -m shell -a " sed -i  '$ a fs.file-max = 65535'  /etc/sysctl.conf && sudo sed -i 's/1024/65535/' /etc/security/limits.d/90-nproc.conf && sudo sed -i '$ a * soft nofile 65535\\n* hard nofile 65535' /etc/security/limits.conf    " -s --ask-sudo-passk

接下來須要準備ssh無祕鑰命令（從操控機到hadoop集羣各服務器）

參考本隨筆

 http://www.cnblogs.com/jackchen001/p/6381270.html

安裝jdk，並在hadoop集羣配置java環境變量

 前提是，ssh無祕鑰通道已打通
 1 生成jdk環境變量文件
 echo '
 export JAVA_HOME=/usr/java/latest/
 export PATH=$JAVA_HOME/bin:$PATH ' >> java.sh
 2 安裝jdk
 ansible -i ./hosts hadoop_host -m yum -a "name=jdk state=present" -s
 3 傳送jdk環境變量文件
 ansible -i ./hosts hadoop_host -m copy -a "src=java.sh dest=/etc/profile.d/" -s
 4 更改java安裝目錄屬組權限
  ansible -i ./hosts hadoop_host -m shell -a "chown -R hadoop.hadoop /usr/java/" -s

查閱好的ansible模塊介紹文章

http://breezey.blog.51cto.com/2400275/1555530/

hadoop集羣hosts映射文件
生產hosts
echo '0 master
1 slave
2 slave
3 slave' >> /tmp/hosts
發送至hadoop集羣服務器上
ansible -i ./hosts hadoop_host -m copy -a "src=/tmp/hosts dest=/etc/hosts" -s
更改hostname
ansible -i ./hosts hadoop_host -m shell -a "sed -i 's/.localdomain//g' /etc/sysconfig/network && service network restart " -s

 

下載hadoop，並配置

下載hadoop安裝包
 ansible -i ./hosts hadoop_host -m  ger_url -a "url=http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.0.0-alpha2/hadoop-3.0.0-alpha2.tar.gz dest=/opt/" -s
這個下載命令執行的話，hadoop集羣都會去下載hadoop安裝包，形成網絡資源的浪費
最好是在操控機上下載，並配置好，而後發送到hadoop集羣服務器上
發送命令以下
ansible -i ./hosts hadoop_host -m copy -a "src=/opt/hadoop dest=/opt/ owner=hadoop user=hadoop mode=0755" -s

hadoop環境變量配置

生成hadoop環境變量文件
echo '
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/opt/hadoop/lib/native/"
export HADOOP_COMMON_LIB_NATIVE_DIR="/opt/hadoop/lib/native/"
' >> hadoop.sh

將hadoop環境文件發送至集羣
ansible -i ./hosts hadoop_host -m copy -a "src=hadoo.sh dest=/etc/profiled./" -s

最重要的一件事，hadoop用須要在集羣服務器之間能夠無祕鑰ssh，並執行命令

操控機上
建立hadoop用戶，並設置hadoo用戶密碼
http://www.cnblogs.com/jackchen001/p/6381270.html
操控機上
ansible -i ./hosts hadoop_host -m shell -a "ssh-keygen -q" -s
等hadoop用戶建立完，並設置密碼，ssh無祕鑰操做都作完後
須要在每臺集羣服務器上，執行 rsync_key playbook
以確保集羣服務器之間hadoop有權限能夠自由執行命令並ssh
讓hadoop用戶能夠sudo
ansible -i ./hosts hadoop_host -m  shell -a "sed -i '$ a %hadoop ALL=(ALL) NOPASSWD: ALL ' /etc/sudoers" -s

作好這件事，而後就上傳hadoop配置文件

http://hadoop.apache.org/docs/current/ hadoop官網

配置好的hadoop配置文件以下：

core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/opt/hadoop/tmp</value>
</property>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://master:9000</value>
</property>
<property> 
  <name>dfs.name.dir</name>           
  <value>/opt/hadoop/name</value> 
</property>

</configuration>

hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
    <name>dfs.replication</name>  
    <value>3</value>
</property>
<property>
    <name>dfs.namenode.name.dir</name>  
    <value>file:/opt/hadoop/name1,/opt/hadoop/name2</value>
</property>

<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/opt/hadoop/data1,/opt/hadoop/data2</value>
</property>

<property>
    <name>dfs.namnode.secondary.http-address</name>
    <value>slave1:9001</value>
</property>


</configuration>

mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>

    <property>  
        <name>mapreduce.framework.name</name>  
        <value>yarn</value>  
    </property>

<property>
    <name>mapred.job.tracker</name>  
    <value>master:9001</value>
</property>
<property>
    <name>mapred.system.dir</name>  
    <value>/opt/hadoop/mapred_system</value>
</property>
<property>
    <name>mapred.local.dir</name>  
    <value>/opt/hadoop/mapred_local</value>
</property>

<property>
        <name>mapreduce.application.classpath</name>
        <value>
                /opt/hadoop/etc/hadoop,
                /opt/hadoop/lib/native/*,
                /opt/hadoop/share/hadoop/common/*,
                /opt/hadoop/share/hadoop/common/lib/*,
                /opt/hadoop/share/hadoop/hdfs/*,
                /opt/hadoop/share/hadoop/hdfs/lib/*,
                /opt/hadoop/share/hadoop/mapreduce/*,
                /opt/hadoop/share/hadoop/mapreduce/lib/*,
                /opt/hadoop/share/hadoop/yarn/*,
                /opt/hadoop/share/hadoop/yarn/lib/*
        </value>
</property>

</configuration>

yarn-site.xml
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->

    <property>  
        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value>  
    </property>  
    <property>  
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>  
        <value>org.apache.hadoop.mapred.ShuffleHandle</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.resource-tracker.address</name>  
        <value>master:8025</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.scheduler.address</name>  
        <value>master:8030</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.address</name>  
        <value>master:8040</value>  
    </property>
    <property>  
        <name>yarn.resourcemanager.admin.address</name>  
        <value>master:8033</value>  
    </property>
    <property>  
        <name>yarn.resourcemanager.webapp.address</name>  
        <value>master:8034</value>  
    </property>

</configuration>

master文件內容
master
slaves文件內容
slave1
slave2
slave3
workers文件內容
slave1
slave2
slave3

hadoop-env.sh 文件中增長
export JAVA_HOME=/usr/java/latest

將以上配置文件在操控機上改好，而後傳送至集羣服務器上

總結了一半，本身懵了，我好尷尬啊！！！！！