How to filter in NaN (pandas)?

1.9K Asked by AndrewJenkins in Tableau , Asked on Jul 2, 2021

I have a pandas dataframe (df), and I want to do something like:

newdf = df[(df.var1 == 'a') & (df.var2 == NaN)]

I've tried replacing NaN with np.NaN, or 'NaN' or 'nan' etc, but nothing evaluates to True. There's no pd.NaN.

I can use df.fillna(np.nan) before evaluating the above expression but that feels hackish and I wonder if it will interfere with other pandas operations that rely on being able to identify pandas-format NaN's later.

How pandas filter nan? Any advice is appreciated. Thank you.

Answered by Bella Blake

When NaN is being compared to itself it returns false. You can use pd.isnull(df.var2) instead of NaN.

Note: To filter out the rows of pandas dataframe that has missing values in Last_Namecolumn, we will first find the index of the column with non null values with pandas notnull() function. It will return a boolean series, where True for not null and False for null values or missing

Your Answer

Answer (1)

Ranjana

In Pandas, you can filter out NaN (Not a Number) values from a DataFrame using the dropna() method or boolean indexing.

Here's how you can do it:

Using dropna() method:import pandas as pd

# Example DataFrame

df = pd.DataFrame({'A': [1, 2, None, 4],

                   'B': [5, None, 7, 8]})

# Drop rows containing NaN values

filtered_df = df.dropna()

print(filtered_df)

This will drop any row that contains at least one NaN value.

Using boolean indexing:

import pandas as pd

import numpy as np

# Example DataFrame

df = pd.DataFrame({'A': [1, 2, np.nan, 4],

                   'B': [5, np.nan, 7, 8]})

# Filter rows where NaN values are present in any column

filtered_df = df[~df.isnull().any(axis=1)]

print(filtered_df)

In this example, df.isnull().any(axis=1) returns a boolean Series indicating whether there are any NaN values in each row. Then, ~ is used to negate this Series, and it is used to filter out rows that contain any NaN values.

Both methods will result in a DataFrame with rows that do not contain any NaN values. Choose the method that best fits your workflow and preferences.

6 Months