[Python] pandas.to_parquet pyarrow.lib.ArrowInvalid: Could not convert Timedelta

I have a huge multiindex dataframe in long format. There's only one "value" column. Some entries in "value" are of type pd.Timedelta.

I got an error when trying to save that dataframe as parquet file using pd.to_parquet:

pyarrow.lib.ArrowInvalid: ("Could not convert Timedelta('2903 days 00:00:00') with type Timedelta: tried to convert to double", 'Conversion failed for column value with type object')

If I convert the particular value from the error message to numpy, I get the following:

array([Timedelta('2903 days 00:00:00')], dtype=object)

I set up a toy example to check if it is possible at all to convert pd.Timedelta to parquet. The following code works just fine:

import pandas as pd

idx = pd.MultiIndex.from_tuples(
[("A", "x"), ("A", "y"), ("B", "x"), ("B", "y")],
names=["idx1", "idx2"])
data = {"value": [
pd.Timedelta(days=10),
2.5,
pd.Timedelta(days=20),
5
]}

df = pd.DataFrame(data=data, index=idx)
df.to_parquet("test.parquet")

x = pd.read_parquet("test.parquet")


A simple df.iloc[0, :].to_numpy() delivers the exact same type as in my real dataframe: array([Timedelta('10 days 00:00:00')], dtype=object).

I am wondering what might be the difference of my original dataframe compared to my toy example?

Continue reading...