[Python] Pandera validation behavior for NaN failure cases

Suppose we use a minimal example for a panderas dataframe-wide validation (cf. this Stackoverflow post):

import numpy as np
import pandas as pd
import pandera as pa

dataframe = pd.DataFrame({'column_A': ['ABC company', 'BBB company', 'ABC company', 'CCC company'],
'column_B': ['1000', np.NaN, '2000', np.NaN]
})

# define your dataframe level test
check_AB = pa.Check(
lambda df: (df['column_A'].str.contains('ABC')) & (~df['column_B'].isna()),
name='check_AB'
)

schema = pa.DataFrameSchema(
columns={
'column_A': pa.Column(pa.String),
'column_B': pa.Column(pa.String, nullable=True)
},
checks=check_AB # <- dataframe wide check
)

schema.validate(dataframe, lazy=True)


The NaN values do not show up among the failure cases, only the non-NaN values:

Now we adjust the example:

dataframe = pd.DataFrame({'column_A': ['ABC company', np.nan, 'ABC company', np.nan],
'column_B': [1000, np.NaN, 2000, 4]
})

# define your dataframe level test
check_AB = pa.Check(
lambda df: (df['column_A'].str.contains('ABC')) & (df['column_B'] >= 7),
name='check_AB'
)


Again, only the non-NaN values are reported among the failure cases:

We can see this in the error object itself as well (see schema_context DataFrameSchema):

try: schema.validate(dataframe, lazy=True)
except pa.errors.SchemaErrors as e: e2 = e; print(e2.failure_cases)




Finally, if all failure cases are NaNs, pandera does not give a validation error:

dataframe = pd.DataFrame({'column_A': ['ABC company', np.nan, 'ABC company', np.nan],
'column_B': [1000, np.NaN, 2000, np.nan]
})

# define your dataframe level test
check_AB = pa.Check(
lambda df: (df['column_A'].str.contains('ABC')) & (df['column_B'] >= 7),
name='check_AB'
)

schema = pa.DataFrameSchema(
columns={
'column_A': pa.Column(pa.String, nullable=True),
'column_B': pa.Column(pa.Float, nullable=True)
},
checks=check_AB # <- dataframe wide check
)




Questions: Why does Panderas not report NaNs as failure cases? And why does an all-NaN-failure-case-situation prevent the correct triggering of the validation errors?

Continue reading...