[Python] How do I capture missing nan values from Pandas 2.3.0 using Pydantic 2.11.7

Prerequisites:

• Python 3.11.7
• Pandas 2.3.0
• Numpy 2.1.3
• Pydantic 2.11.7

In the pandas documentation, it states that missing values for numeric data types are filled in with numpy.nan:

https://pandas.pydata.org/docs/user_guide/missing_data.html#values-considered-missing

In [1]:
import pandas as pd
import numpy as np

In [2]:
s = pd.Series([1, 2], dtype=np.int64).reindex([0, 1, 2])

In [3]:
s
Out[3]:
0 1.0
1 2.0
2 NaN
dtype: float64


In pydantic models, it should be possible to capture numpy.nan with Literal[numpy.nan]. By chaining this functionality with typing.Union, this allows one to create pydantic models with complex functional behaviours, e.g. positive integers or numpy.nan (e.g. to create a model which will accept any positive integer or a missing value):

In [4]:
import numpy as np
import pydantic

from typing import Union, Literal

class Col(pydantic.BaseModel):
precision: Union[
pydantic.conint(ge=1),
Literal[np.nan]
]


However, taking the above pandas example, pydantic validation fails once it hits that missing value:

In [5]:
for i in s.values:
print(i)
Col(precision=i)
Out[5]:
1.0
2.0
nan

---------------------------------------------------------------------------
ValidationError Traceback (most recent call last)
Cell In[5], line 3
1 for i in s.values:
2 print(i)
----> 3 Col(precision=i)

File ~/........./python3.11/site-packages/pydantic/main.py:253, in BaseModel.__init__(self, **data)
251 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks
252 __tracebackhide__ = True
--> 253 validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
254 if self is not validated_self:
255 warnings.warn(
256 'A custom validator is returning a value other than `self`.\n'
257 "Returning anything other than `self` from a top level model validator isn't supported when validating via `__init__`.\n"
258 'See the `model_validator` docs (https://docs.pydantic.dev/latest/concepts/validators/#model-validators) for more details.',
259 stacklevel=2,
260 )

ValidationError: 2 validation errors for Col
precision.constrained-int
Input should be a finite number [type=finite_number, input_value=nan, input_type=float64]
For further information visit https://errors.pydantic.dev/2.11/v/finite_number
precision.literal[nan]
Input should be nan [type=literal_error, input_value=nan, input_type=float64]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error


If I call the pydantic class with a regular numpy.nan value, it works just fine:

In [6]:
Col(precision=np.nan)
Out[6]:
Col(precision=nan)


So it seems as if Pandas is using something other than numpy.nan to fill in missing values. Does anyone know what that might be, or how I might catch it with Pydantic & typing?


As an aside, I have a somewhat hacky workaround by using pydantic.confloat(allow_inf_nan=True) instead of Literal[np.nan], but since confloat permits all float data types I then have to add a secondary model validator to throw an exception if the given value is not actually nan. I dislike this solution though it works, as I feel there should be something much more elegant that I could do:

In [7]:
import math

class Col2(pydantic.BaseModel):
precision: Union[
pydantic.conint(ge=1),
pydantic.confloat(allow_inf_nan)=True
]

@pydantic.model_validator(mode="after")
def validate_nans(self):
if self.precision is not None and isinstance(self.precision, float):
if self.precision.is_integer():
assert self.precision >= 0
else:
assert math.isnan(self.precision)


This works as expected for all types of nan (that I know about for now anyway):

In [8]:
import pytest

bad_nan = s.iloc[-1]

test_values = [
(0, True), (-1, False), (-1.234, False), (1.234, False), (2, True),
(3.0, True), (-3.0, False)
(bad_nan, True), (np.nan, True), (float('nan'), True), (np.float64('nan'), True)
(False, True), (True, True) # Bool False is 0, True is 1. This is fine.
]

for val, should_pass in test_values:
if should_pass:
Col2(precision=val)
else:
with pytest.raises(pydantic.ValidationError):
Col2(precision=val)
Out[8]:
============================== 1 passed in 0.05s ===============================


Other solutions I have tried but that don't work for one reason or another:

• 
  Annotated[pydantic.conint(ge=0), pydantic.AllowInfNan()]
  
  Breaks because you can't mix int and nan floats together in one annotated type.
  
  
• 
  pydantic.confloat(ge=0, multiple_of=1, allow_inf_nan=True)
  
  Breaks because you can't spec the ge argument with allow_inf_nan simultaneously (nan values are not greater than or less than 0, so that fails the validation). Interestingly though there's no issues with spec'ing multiple_of=1 and allow_inf_nan=True together.
  

Continue reading...