YARN (Yet Another Resource Navigator) was introduced in the second version of Hadoop and this is a technology to manage clusters. However, at the time of launch, Apache Software Foundation described it as a redesigned resource manager, but now it is known as a large-scale distributed operating system, which is used for Big data applications.
YARN provided new capabilities to Apache Hadoop by decoupling resource management and scheduling capabilities. Now, with the help of YARN, interactive queries can be run on Hadoop and data streaming can also be done at the same time. This article describes YARN in detail and it includes YARN architecture, its features, and other functionalities.
An Introduction to YARN
YARN is a pre-requisite for Hadoop and provides security, data governance tools, resource management functionality across Hadoop clusters. YARN also extends the power of Hadoop by including new cost-effective processing, and linear-scale storage of beneficial technologies. A consistent framework is provided to developers and ISVs to write data, access applications which can run in Hadoop. 
Through following listed features YARN enhances the Hadoop capabilities:
- Scalability: Processing power of data center keep on increasing rapidly. Resource manager of YARN keeps on focusing exclusively on scheduling and can manage many clusters and thousands of nodes as a result of which they can manage petabytes of data.
- Compatibility: Without any disruption to the Hadoop1 processes all existing Map-Reduce applications can run YARN.
- Cluster Utilization: Dynamic cluster allocation, used in YARN improves cluster utilization. It is far better than static Map Reduce technologies used in earlier Hadoop versions
- Multi-tenancy: This feature improves ROI of the organization by allowing multiple engine access.
Components of YARN
YARN can split the job responsibilities of Job or Task Tracker into separate entities which are listed below:
- Resource Manager
- Application Manager
- Node Manager
- A dedicated container for every Node Manager
Here among above-listed components resource manager works as the master node of YARN and is responsible to take resource inventory and can run Scheduler like important and critical services. Resource managers can allocate required resources to the running applications.
Read: Difference Between Apache Hadoop and Spark Framework
As it does not track and monitor the application status so it is a pure scheduler. So,the resource manager is basically used to manage clusters of distributed applications of Hadoop YARN.
Resource Manager works with an Application manager and node managers present on every node in the following way:
- Resource manager instructions are followed by Node managers and they then manage the resources of a single node.
- Application Managers can negotiate the resources of the resource manager and can coordinate with node managers by starting containers.
Here in this architecture of Resource manager again has four components with the help of which it executes its responsibilities perfectly. These components are:
- Resource Scheduler: Resource scheduler basically schedule all running activities of the applications and decide that which activity needs to run, when it must run, and for how long it will run?
- Application Master Liveness Monitor: Application master is responsible to launch, monitor and requesting containers for any specific application
- Node Manager Liveness Monitor: It tracks the available resources for data processing on its slave nodes and send on the basis of this information, they can send the reports to Resource Manager about all available resources.
- Several Event Handler: It handles and manages all events that happen during any application execution.
Resource management is a great feature which is present in YARN and it was launched to fulfill following tasks:
- To guarantee a fixed time to complete even the critical tasks
- Proper and reasonable cluster scheduling by allocating resources in a fair manner to each user group
- To prevent other users to access the clusters
A Walkthrough of Application Execution in YARN
By far it has been clear that YARN is responsible to run and schedule the applications so that they can complete their task in a particular time frame. Here it is obvious that any application runs or executes in following steps:
Read: How to Install Apache Pig on Linux?
- Submission of Application
- Application Master Instance Bootstrapping
- Execution of Application through application master instance
In the following diagram, a brief introduction of application execution is shown: Above shown steps are listed below for application execution:
- To launch the specific application, master an application, and its specifications are submitted by the client.
- Resource manager starts the application master to launch any specific application.
- Application Master registers with Resource manager through which the client program can query resource manager and it can directly communicate with its application master.
- During operation execution appropriate resource containers are negotiated by application master by following resource-request protocol.
- After allocating appropriate container, application master launches it and provide its specification to the Node Manager. The launch specifications include all necessary information through which Application master can communicate properly with containers.
- The code which executes within the container provides all information of the application like its status, progress status to application master through the application-specific.
- During application execution direct communication between application master and program executes.
- After completion of application execution, it deregisters itself with the resource manager and shuts down the process.
The complete process of Application Startup can also be represented in the following manner: As per above diagram, there are three below listed actors of YARN:
- Job Submitter
- Resource Manager
- Node Manager
The complete process can be summarized as:
- An application is submitted by the client
- A container is allocated by Resource Manager
- Resource Manager contacts related Node Manager
- Containers are launched by Node Manager
- Application Master is executed by Containers
A single application is responsible for the execution of Application Master. It asks for containers from resource scheduler and executes specific programs on the containers. So, we can say that Resource Manager is the core component of YARN and it occupies the role of Job tracker of MR version1. It is the central controlling authority for managing resources and allocating them to the proper and appropriate application. Through two main components named: Scheduler and ApplicationManager and it allocates resources to the applications.
RM and Components Interfacing
Following services are used for the interaction of Resource Manager and other components:
Read: Top 20 Apache Solr Interview Questions & Answers for Freshers and Experienced
- ClientService: All RPC interfaces are handled by this component like application submission, termination, clustering and other.
- AdminService: This service ensures that all admin requests will be satisfied
- Resorce Tracker Service: All nodes information are forwarded to YARN scheduler
- NodesListManager: All valid and excluded nodes are managed and are responsible to read host configuration files and track the application progress.
Final Words:
YARN has provided an exclusive feature to Hadoop system. In Hadoop version1, it was not able to manage the resources properly, and the user often finds it difficult to allocate resources properly. Through YARN now the scheduling and allocating the resource has become easier and the complete processing speed has been enhanced.
Through its various components, it can dynamically allocate various resources and schedule the application processing. For large volume data processing, it is quite necessary to manage the available resources properly so that every application can leverage them.
HDFS separation from Map Reduce has made the Hadoop environment more efficient and quicker. To know more about YARN and its capabilities you should join Hadoop training and certification program at JanBask right away.
FaceBook
Twitter
LinkedIn
Pinterest
Email
Hadoop Course
Upcoming Batches
Trending Courses
Cyber Security
- Introduction to cybersecurity
- Cryptography and Secure Communication
- Cloud Computing Architectural Framework
- Security Architectures and Models
Upcoming Class
6 days 28 Feb 2025
QA
- Introduction and Software Testing
- Software Test Life Cycle
- Automation Testing and API Testing
- Selenium framework development using Testing
Upcoming Class
-1 day 21 Feb 2025
Salesforce
- Salesforce Configuration Introduction
- Security & Automation Process
- Sales & Service Cloud
- Apex Programming, SOQL & SOSL
Upcoming Class
4 days 26 Feb 2025
Business Analyst
- BA & Stakeholders Overview
- BPMN, Requirement Elicitation
- BA Tools & Design Documents
- Enterprise Analysis, Agile & Scrum
Upcoming Class
-1 day 21 Feb 2025
MS SQL Server
- Introduction & Database Query
- Programming, Indexes & System Functions
- SSIS Package Development Procedures
- SSRS Report Design
Upcoming Class
-1 day 21 Feb 2025
Data Science
- Data Science Introduction
- Hadoop and Spark Overview
- Python & Intro to R Programming
- Machine Learning
Upcoming Class
6 days 28 Feb 2025
DevOps
- Intro to DevOps
- GIT and Maven
- Jenkins & Ansible
- Docker and Cloud Computing
Upcoming Class
0 day 22 Feb 2025
Hadoop
- Architecture, HDFS & MapReduce
- Unix Shell & Apache Pig Installation
- HIVE Installation & User-Defined Functions
- SQOOP & Hbase Installation
Upcoming Class
6 days 28 Feb 2025
Python
- Features of Python
- Python Editors and IDEs
- Data types and Variables
- Python File Operation
Upcoming Class
0 day 22 Feb 2025
Artificial Intelligence
- Components of AI
- Categories of Machine Learning
- Recurrent Neural Networks
- Recurrent Neural Networks
Upcoming Class
14 days 08 Mar 2025
Machine Learning
- Introduction to Machine Learning & Python
- Machine Learning: Supervised Learning
- Machine Learning: Unsupervised Learning
Upcoming Class
27 days 21 Mar 2025
Tableau
- Introduction to Tableau Desktop
- Data Transformation Methods
- Configuring tableau server
- Integration with R & Hadoop
Upcoming Class
6 days 28 Feb 2025
Hadoop Course
Upcoming Batches
Receive Latest Materials and Offers on Hadoop Course
Mar 20, 2018 
420.1k
Through following listed features YARN enhances the Hadoop capabilities:
Here in this architecture of Resource manager again has four components with the help of which it executes its responsibilities perfectly. These components are:
Above shown steps are listed below for application execution:
As per above diagram, there are three below listed actors of YARN:
So, we can say that Resource Manager is the core component of YARN and it occupies the role of Job tracker of MR version1. It is the central controlling authority for managing resources and allocating them to the proper and appropriate application. Through two main components named: Scheduler and ApplicationManager and it allocates resources to the applications.
FaceBook
Twitter
LinkedIn
Pinterest
Email
Oct 13, 2017
615k
615k