[Yarn-Tunning]

1. overview
   • A yarn cluster is composed of host machines. Hosts provide memory and CPU resources. A vcore, or virtual core, is a usage share of a host CPU.
   • Tuning YARN consists primarily of optimally defining containers on your work hosts. You can think of a container as a rectangular graph consisting of memory and vcores. Containers perform task.
   • Some tasks use a great deal of memory, with minimal processing on a large volume of data.
   • Other task requires a great deal of processing power, but use less memory.
   • The Yarn ResourceManager allocates memory and vcores to use all available resources in the most efficient way possible. Ideally, few or no resources are left idle.
   • An application is a YARN client program consisting of one or more tasks. Typically , a task uses all of the available resource in container. A task cannot consume more than its designated allocation, ensuring that it cannot use all the host CPU cycles or exceed its memory allotment
2. Cluster Configuration
   • Worker Host Configuration
     • to define the configuration for a single worker host computer in your cluster

     • Host Componets	Quantity	Description
       RAM	256	Gigabytes
       CPU	48	8CPUs 6cores
       HDD(Hard Disk Driver)	36	12 * 3T SATA
       Ethernet	2	1 Gigabit Ethernet

   • Worker Host Planning

     • to allocate resources on each worker machine.

     • Service	Category	CPU(cores)	Memory(MB)
       OS		1	8192
       CM agent		1	1024
       Others		0	0
       HDFS DataNode		1	1024
       Impala
       HBase
       Solr
       Yarn NodeManager		1	1024
       Available Resources
       Yarn available cores		176
       Yarn available memory		250880

     • Avaiable = Total - OtherUsed

   • Cluster Size

     • Having defined the specifications for each host in your cluster, enter the number of worker needed to support your business case. To see the benefits of parallel computing, set the number of hosts to a minimum of 10

3. Yarn Configuration

   • • yarn.nodemanager.resource.cpu-vcores = 176 

     • yarn.nodemanager.resource.memory-mb = 250880

   • go to http://<ResourceManagerIP>:8088/ to verify the "Memory Total" and "Vcores Total"

     • Vcores Total = 176 * 10

     • Memory Total = 250880 * 10

4. Container Setting

   • Yarn Container Configuration Property(Vcores)	Value	Description
     yarn.scheduler.minimum-allocation-vcores	1	Minimum vcores reservation for a container
     yarn.scheduler.maximum-allocation-vcores	32	Maximum vcores reservation for a container
     yarn.scheduler.increment-allocation-vcores	1	Vcore allocation must be a multiple this value
     Yarn Container Configuration Property(Memory)
     yarn.scheduler.minimum-allocation-mb	1024	Minimum memory reservation for a container
     yarn.scheduler.maximum-allocation-mb	8192	Maximum memory reservation for a container
     yarn.scheduler.increment-allocation-mb	1024	memory allocation must be a multiple this value