[Python] Improve code that finds nan values with a condition and removes them

I have a dataframe where each column starts and finished with certain number of nan values. Somewhere in the middle of a column there is a continuous list of values. It can happen that a nan value "interrupts" the data. I want to iterate over each column, find such values and then remove the whole row.

For example, I want to find the np.nan between 9 and 13 and remove it:

[np.nan, np.nan, np.nan, 1, 4, 6, 6, 9, np.nan, 13, np.nan, np.nan]


Conditions for removal:

1. if value has at least one data point before
2. if value has at least one data point after
3. if value is nan

I wrote code that does this already, but it's slow and kind of wordy.

import pandas as pd
import numpy as np


data = {'A': [np.nan, np.nan, np.nan, 1, 4, 6, 6, 9, np.nan, 13, np.nan, np.nan], 'B': [np.nan, np.nan, np.nan, 11, 3, 16, 13, np.nan, np.nan, 12, np.nan, np.nan]}
df = pd.DataFrame(data)

def get_nans(column):
output = []
for index_to_check, value in column.items():
has_value_before = not column[:index_to_check].isnull().all()
has_value_after = not column[index_to_check + 1:].isnull().all()
is_nan = np.isnan(value)
output.append(not( has_value_before and has_value_after and is_nan))
return output

for column in df.columns:
df = df[get_nans(df[column])]

print(df)



How can I improve my code, vectorize it etc?

Continue reading...