Yarn Scheduler Load Simulator YARN調度負載模擬器

    項目起源是由於有人但願有個模擬環境來模擬公平調度器和容量調度器，以便合理配置調度器，下降生產環境出問題的風險，詳見https://issues.apache.org/jira/browse/YARN-1021。以後在hadoop2.3.0就增長了這個工具。

    首先設定環境變量：

     export HADOOP_HOME=/usr/hadoop-2.3.0

     export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop #此目錄放置sls-runner.xml文件

     sls-runner.xml文件內容以下：

<configuration>

  <!-- SLSRunner configuration -->
  <property>
    <name>yarn.sls.runner.pool.size</name>
    <value>100</value>
  </property>
  
  <!-- Nodes configuration -->
  <property>
    <name>yarn.sls.nm.memory.mb</name>
    <value>10240</value>
  </property>
  <property>
    <name>yarn.sls.nm.vcores</name>
    <value>10</value>
  </property>
  <property>
    <name>yarn.sls.nm.heartbeat.interval.ms</name>
    <value>1000</value>
  </property>
  
  <!-- Apps configuration -->
  <property>
    <name>yarn.sls.am.heartbeat.interval.ms</name>
    <value>1000</value>
  </property>
  <property>
    <name>yarn.sls.am.type.mapreduce</name>
    <value>org.apache.hadoop.yarn.sls.appmaster.MRAMSimulator</value>
  </property>

  <!-- Containers configuration -->
  <property>
    <name>yarn.sls.container.memory.mb</name>
    <value>1024</value>
  </property>
  <property>
    <name>yarn.sls.container.vcores</name>
    <value>1</value>
  </property>

  <!--  metrics  -->
  <property>
    <name>yarn.sls.metrics.switch</name>
    <value>ON</value>
  </property>
  <property>
    <name>yarn.sls.metrics.web.address.port</name>
    <value>10001</value>
  </property>
  <property>
    <name>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler</name>
    <value>org.apache.hadoop.yarn.sls.scheduler.FifoSchedulerMetrics</value>
  </property>
  <property>
    <name>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</name>
    <value>org.apache.hadoop.yarn.sls.scheduler.FairSchedulerMetrics</value>
  </property>
  <property>
    <name>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</name>
    <value>org.apache.hadoop.yarn.sls.scheduler.CapacitySchedulerMetrics</value>
  </property>
  
</configuration>

• yarn.sls.runner.pool.size

  The simulator uses a thread pool to simulate the NM and AM running , and this parameter specifies the number of threads in the pool.

• yarn.sls.nm.memory.mb

  The total memory for each NMSimulator.

• yarn.sls.nm.vcores

  The total vCores for each NMSimulator.

• yarn.sls.nm.heartbeat.interval.ms

  The heartbeat interval for each NMSimulator.

• yarn.sls.am.heartbeat.interval.ms

  The heartbeat interval for each AMSimulator.

• yarn.sls.am.type.mapreduce

  The AMSimulator implementation for MapReduce-like applications. Users can specify implementations for other type of applications.

• yarn.sls.container.memory.mb

  The memory required for each container simulator.

• yarn.sls.container.vcores

  The vCores required for each container simulator.

• yarn.sls.runner.metrics.switch

  The simulator introduces Metrics to measure the behaviors of critical components and operations. This field specifies whether we open (ON) or close (OFF) the Metrics running.

• yarn.sls.metrics.web.address.port

  The port used by simulator to provide real-time tracking. The default value is 10001.

• org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler

  The implementation of scheduler metrics of Fifo Scheduler.

• org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

  The implementation of scheduler metrics of Fair Scheduler.

• org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

  The implementation of scheduler metrics of Capacity Scheduler.

而後使用apache rumen對jobhistory文件進行解析，生成json格式文件以便sls解析：

hadoop jar hadoop-rumen-2.3.0.jar org.apache.hadoop.tools.rumen.TraceBuilder -write-job-trace file:///home/user/job-trace.json file:///home/user/topology.output file:///home/user/logs/history/done

file:///home/user/logs/history/done 用戶集羣存放運行完成jobhistory的目錄，通常在hdfs裏，能夠經過hadoop fs -get取到本地目錄

file:///home/user/job-trace.json file:///home/user/topology.output 生成的sls要讀取的文件

運行模擬器

slsrun.sh --input-rumen=/home/user/  --output-dir=/usr/sls/sample-result

--input-rumen 本例就是file:///home/user/job-trace.json file:///home/user/topology.output 對應的路徑

若是運行報錯：

java.lang.NullPointerException
at org.apache.hadoop.yarn.sls.web.SLSWebApp.<init>(SLSWebApp.java:82)
at 

在https://issues.apache.org/jira/browse/YARN-1021中查看comment，解決方法以下：

bin/slsrun.sh --input-sls=sls-file/sls-jobs.json --output-dir=output_sls --nodes=sls-file/sls-nodes.json