YARN- Empowering The Hadoop Functionalities

YARN (Yet Another Resource Navigator) was introduced in the second version of Hadoop and this is a technology to manage clusters. However, at the time of launch, Apache Software Foundation described it as a redesigned resource manager, but now it is known as a large-scale distributed operating system, which is used for Big data applications.

YARN provided new capabilities to Apache Hadoop by decoupling resource management and scheduling capabilities. Now, with the help of YARN, interactive queries can be run on Hadoop and data streaming can also be done at the same time. This article describes YARN in detail and it includes YARN architecture, its features, and other functionalities.

An Introduction to YARN                                                                         

YARN is a pre-requisite for Hadoop and provides security, data governance tools, resource management functionality across Hadoop clusters. YARN also extends the power of Hadoop by including new cost-effective processing, and linear-scale storage of beneficial technologies. A consistent framework is provided to developers and ISVs to write data, access applications which can run in Hadoop. Through following listed features YARN enhances the Hadoop capabilities:

1. Scalability: Processing power of data center keep on increasing rapidly. Resource manager of YARN keeps on focusing exclusively on scheduling and can manage many clusters and thousands of nodes as a result of which they can manage petabytes of data.
2. Compatibility: Without any disruption to the Hadoop1 processes all existing Map-Reduce applications can run YARN.
3. Cluster Utilization: Dynamic cluster allocation, used in YARN improves cluster utilization. It is far better than static Map Reduce technologies used in earlier Hadoop versions
4. Multi-tenancy: This feature improves ROI of the organization by allowing multiple engine access.

Components of YARN

YARN can split the job responsibilities of Job or Task Tracker into separate entities which are listed below:

• Resource Manager
• Application Manager
• Node Manager
• A dedicated container for every Node Manager

Here among above-listed components resource manager works as the master node of YARN and is responsible to take resource inventory and can run Scheduler like important and critical services. Resource managers can allocate required resources to the running applications.

Read: CCA Spark & Hadoop Developer Certification Exam Practice Tests

As it does not track and monitor the application status so it is a pure scheduler. So,the resource manager is basically used to manage clusters of distributed applications of Hadoop YARN.

Resource Manager works with an Application manager and node managers present on every node in the following way:

1. Resource manager instructions are followed by Node managers and they then manage the resources of a single node.
2. Application Managers can negotiate the resources of the resource manager and can coordinate with node managers by starting containers.

Here in this architecture of Resource manager again has four components with the help of which it executes its responsibilities perfectly. These components are:

• Resource Scheduler: Resource scheduler basically schedule all running activities of the applications and decide that which activity needs to run, when it must run, and for how long it will run?
• Application Master Liveness Monitor: Application master is responsible to launch, monitor and requesting containers for any specific application
• Node Manager Liveness Monitor: It tracks the available resources for data processing on its slave nodes and send on the basis of this information, they can send the reports to Resource Manager about all available resources.
• Several Event Handler: It handles and manages all events that happen during any application execution.

Resource management is a great feature which is present in YARN and it was launched to fulfill following tasks:

• To guarantee a fixed time to complete even the critical tasks
• Proper and reasonable cluster scheduling by allocating resources in a fair manner to each user group
• To prevent other users to access the clusters

A Walkthrough of Application Execution in YARN

By far it has been clear that YARN is responsible to run and schedule the applications so that they can complete their task in a particular time frame. Here it is obvious that any application runs or executes in following steps:

Read: What is Spark? Apache Spark Tutorials Guide for Beginner

• Submission of Application
• Application Master Instance Bootstrapping
• Execution of Application through application master instance

In the following diagram, a brief introduction of application execution is shown: Above shown steps are listed below for application execution:

1. To launch the specific application, master an application, and its specifications are submitted by the client.
2. Resource manager starts the application master to launch any specific application.
3. Application Master registers with Resource manager through which the client program can query resource manager and it can directly communicate with its application master.
4. During operation execution appropriate resource containers are negotiated by application master by following resource-request protocol.
5. After allocating appropriate container, application master launches it and provide its specification to the Node Manager. The launch specifications include all necessary information through which Application master can communicate properly with containers.
6. The code which executes within the container provides all information of the application like its status, progress status to application master through the application-specific.
7. During application execution direct communication between application master and program executes.
8. After completion of application execution, it deregisters itself with the resource manager and shuts down the process.

The complete process of Application Startup can also be represented in the following manner: As per above diagram, there are three below listed actors of YARN:

1. Job Submitter
2. Resource Manager
3. Node Manager

The complete process can be summarized as:

1. An application is submitted by the client
2. A container is allocated by Resource Manager
3. Resource Manager contacts related Node Manager
4. Containers are launched by Node Manager
5. Application Master is executed by Containers

A single application is responsible for the execution of Application Master. It asks for containers from resource scheduler and executes specific programs on the containers. So, we can say that Resource Manager is the core component of YARN and it occupies the role of Job tracker of MR version1. It is the central controlling authority for managing resources and allocating them to the proper and appropriate application. Through two main components named: Scheduler and ApplicationManager and it allocates resources to the applications.

RM and Components Interfacing

Following services are used for the interaction of Resource Manager and other components:

Read: A Complete List of Sqoop Commands Cheat Sheet with Example

1. ClientService: All RPC interfaces are handled by this component like application submission, termination, clustering and other.
2. AdminService: This service ensures that all admin requests will be satisfied

• Resorce Tracker Service: All nodes information are forwarded to YARN scheduler

1. NodesListManager: All valid and excluded nodes are managed and are responsible to read host configuration files and track the application progress.

Final Words:

YARN has provided an exclusive feature to Hadoop system. In Hadoop version1, it was not able to manage the resources properly, and the user often finds it difficult to allocate resources properly. Through YARN now the scheduling and allocating the resource has become easier and the complete processing speed has been enhanced.

Through its various components, it can dynamically allocate various resources and schedule the application processing. For large volume data processing, it is quite necessary to manage the available resources properly so that every application can leverage them.

HDFS separation from Map Reduce has made the Hadoop environment more efficient and quicker. To know more about YARN and its capabilities you should join Hadoop training and certification program at JanBask right away.

FaceBook

Twitter

LinkedIn

Pinterest

Email

Trending Courses

Cyber Security

• Introduction to cybersecurity
• Cryptography and Secure Communication 
• Cloud Computing Architectural Framework
• Security Architectures and Models

Upcoming Class

-1 day 21 Dec 2024

View Details

QA

• Introduction and Software Testing
• Software Test Life Cycle
• Automation Testing and API Testing
• Selenium framework development using Testing

Upcoming Class

6 days 28 Dec 2024

View Details

Salesforce

• Salesforce Configuration Introduction
• Security & Automation Process
• Sales & Service Cloud
• Apex Programming, SOQL & SOSL

Upcoming Class

8 days 30 Dec 2024

View Details

Business Analyst

• BA & Stakeholders Overview
• BPMN, Requirement Elicitation
• BA Tools & Design Documents
• Enterprise Analysis, Agile & Scrum

Upcoming Class

5 days 27 Dec 2024

View Details

MS SQL Server

• Introduction & Database Query
• Programming, Indexes & System Functions
• SSIS Package Development Procedures
• SSRS Report Design

Upcoming Class

5 days 27 Dec 2024

View Details

Data Science

• Data Science Introduction
• Hadoop and Spark Overview
• Python & Intro to R Programming
• Machine Learning

Upcoming Class

12 days 03 Jan 2025

View Details

DevOps

• Intro to DevOps
• GIT and Maven
• Jenkins & Ansible
• Docker and Cloud Computing

Upcoming Class

-1 day 21 Dec 2024

View Details

Hadoop

• Architecture, HDFS & MapReduce
• Unix Shell & Apache Pig Installation
• HIVE Installation & User-Defined Functions
• SQOOP & Hbase Installation

Upcoming Class

6 days 28 Dec 2024

View Details

Python

• Features of Python
• Python Editors and IDEs
• Data types and Variables
• Python File Operation

Upcoming Class

-1 day 21 Dec 2024

View Details

Artificial Intelligence

• Components of AI
• Categories of Machine Learning
• Recurrent Neural Networks
• Recurrent Neural Networks

Upcoming Class

13 days 04 Jan 2025

View Details

Machine Learning

• Introduction to Machine Learning & Python
• Machine Learning: Supervised Learning
• Machine Learning: Unsupervised Learning

Upcoming Class

5 days 27 Dec 2024

View Details

Tableau

• Introduction to Tableau Desktop
• Data Transformation Methods
• Configuring tableau server
• Integration with R & Hadoop

Upcoming Class

6 days 28 Dec 2024

View Details

Related Posts

A Complete List of Sqoop Commands Cheat Sheet with Example

 Jul 23, 2024  786.3k

Read More

Big Data Hadoop Developer Career Path & Future Scope

 Nov 08, 2021  242k

Read More

CCA Spark & Hadoop Developer Certification Exam Practice Tests

 Jan 30, 2019  316.4k

Read More