Hadoop “Unable to load native-hadoop library for your platform” warning

4.1K Asked by BellaBlake in Big Data Hadoop , Asked on Apr 19, 2021

I installed Hadoop on the server running CentOs. As I run start-dfs.shor stop-dfs.sh I got the following error :

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

I am using the Hadoop 2.2.0 Version. In hadoop-env.sh I have also added these two environment variables.

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/usr/local/hadoop/lib/"
export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"

Getting the error “unable to load native hadoop library for your platform”.

Any solution what I have to do?

Answered by Bella Blake

This error arises that “ unable to load native hadoop library for your platform” as the native library of Hadoop is built in for a 32 bit system, your system maybe 64 bit.

export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"

So, it is giving a warning but not much of Hadoop functionalities will be affected.You can ignore the warning or if you want to remove it then :

Either replace the 32 bit library by downloading Hadoop source code and recompiling it.

Or in hadoop-env.sh where you have added environment variables,

add the word NATIVE like this :

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native"

Note:

There may be two possibilities: You have installed the wrong Java JDK8 package. Please ensure to download the 64-bit JDK8 and remove your current 32-bit JDK8. It's a warning due to the Hadoop libraries being compiled for 32bits.

Your Answer

Answer (1)

Ranjana

The warning "Unable to load native-hadoop library for your platform" is a common message when running Hadoop. This message indicates that Hadoop is unable to find or load the native library specific to your operating system. The native library is used to provide better performance for certain Hadoop operations, but it is not strictly necessary for Hadoop to function. Here's how you can address this warning:

Steps to Resolve the Warning

Install Hadoop Native Libraries:

Ensure you have the Hadoop native libraries installed on your system. These libraries are usually included in the Hadoop distribution.

  Set the HADOOP_HOME Environment Variable:

Make sure the HADOOP_HOME environment variable is set correctly to point to your Hadoop installation directory.

  export HADOOP_HOME=/path/to/hadoopexport PATH=$PATH:$HADOOP_HOME/binSet the JAVA_LIBRARY_PATH:

Add the native library directory to the JAVA_LIBRARY_PATH environment variable.

  export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/nativeVerify Permissions:

Ensure that the user running Hadoop has read and execute permissions on the native library files.

Check Compatibility:

Make sure the native libraries are compatible with your platform. The libraries are platform-specific, so you need to have the correct version for your operating system and architecture.

Download Precompiled Native Libraries:

If the native libraries are not included in your Hadoop distribution, you can download precompiled native libraries for your platform. For example, for Hadoop 3.3.1, you can find the binaries on the Apache Hadoop Releases page.

Build Native Libraries from Source:

If precompiled binaries are not available for your platform, you may need to build the native libraries from the source.

  cd $HADOOP_HOMEmvn package -Pdist,native -DskipTests -Dtar

Check Hadoop Configuration Files:

Ensure your Hadoop configuration files (core-site.xml, hdfs-site.xml, etc.) are correctly set up and do not contain incorrect paths or settings that might prevent the native library from being found.

Ignoring the Warning

If you cannot resolve the issue or it is not critical for your use case, you can choose to ignore the warning. Hadoop will still function correctly without the native libraries, but you might not get the best performance.

Example Configuration

Here's an example of how your .bashrc or .bash_profile might look:

  # Set Hadoop-related environment variablesexport HADOOP_HOME=/usr/local/hadoopexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop# Set JAVA_HOME (we assume JDK 8 is installed)export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64# Add Hadoop bin/ directory to PATHexport PATH=$PATH:$HADOOP_HOME/bin

# Native library path

export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native

By following these steps and ensuring your environment is set up correctly, you should be able to address the "Unable to load native-hadoop library for your platform" warning effectively.

4 Months