[Python] Handle empty columns in polars when concatenating dataframes

I want to be able to concatenate dataframes in polars where the dataframes have the same columns, but some of the dataframes have no data for a subset of the columns.

More precisely, I am looking for the polars equivalent of this pandas minimal working example:

from io import StringIO
import polars as pl
import pandas as pd

TESTDATA1 = StringIO("""
col1,col2,col3
1,1,"a"
2,1,"b"
"""
)

TESTDATA2 = StringIO("""
col1,col2,col3
1,,"a"
2,,"b"
"""
)

df = pd.concat(
[
pd.read_csv(TESTDATA1),
pd.read_csv(TESTDATA2),
],
)
print(df)


This prints

col1 col2 col3
0 1 1.0 a
1 2 1.0 b
0 1 NaN a
1 2 NaN b


I tried the following polars implementation which does not work for me:


TESTDATA1 = StringIO("""
col1,col2,col3
1,1,"a"
2,1,"b"
""")

TESTDATA2 = StringIO("""
col1,col2,col3
1,,"a"
2,,"b"
""")

df = pl.concat(
[
pl.read_csv(TESTDATA1),
pl.read_csv(TESTDATA2),
],
how ="diagonal"
)



I get the error message:

SchemaError: cannot vstack: because column datatypes (dtypes) in the two DataFrames do not match for left.name='col2' with left.dtype=i64 != right.dtype=str with right.name='col2'


It seems that the empty column is treated as str in polars and can't be merged with the other dataframe where it is of type i64.

I understand that this is a solution for my problem:

df = pl.concat(
[
pl.read_csv(TESTDATA1),
pl.read_csv(TESTDATA2).with_columns(pl.col("col2").cast(pl.Int64)),
],
how ="diagonal"
)


But in reality, I have some twenty columns that are potentially null and I don't want to cast all of them.

What works in pandas and polars is the situation where the empty column is removed from the dataframe, i.e.


TESTDATA1 = StringIO("""
col1,col2,col3
1,1,"a"
2,1,"b"
""")

TESTDATA2 = StringIO("""
col1,col3
1,"a"
2,"b"
""")

pl.concat(
[
pl.read_csv(TESTDATA1),
pl.read_csv(TESTDATA2),
],
how ="diagonal"
)


In pandas, I could also drop the empty column by calling .dropna(how="all",axis=1) but I don't know the equivalent for this in polars.

So, to summarize:

• How can I concatenate dataframes in polars if some of them have columns with no data (null)?
• How can I achieve the equivalent of .dropna(how="all",axis=1) in polars?

Thanks!

Continue reading...