[Python] how to preserve (unknown) datatypes when creating a df from existing dfs with...

I want to create a summary file of a few dozen original files, of which I only extract the last row (df.tail(1)) of each original file with 1000s of columns, which might be int, float, str, bool.

My problem is, that sometimes column names differ some (which is fine, I just have to retain the information), the majority of column names are the same. When using pd.concat on these dfs, nans are created for all columns which do not exist in all files, which leads to a change in datatype for ints.

df1 = pd.DataFrame([(2, 3, True, "Test")], columns=["x", "y", "z", "a"])
df2 = pd.DataFrame([(4, 6)], columns=["x", "b"])

df3 = pd.concat([df1, df2], ignore_index = True)


--> df3:

resulting df3

But datatype preservation is necessary for further file handling (ints have to stay ints)!

I tried adding the information of the subsequent dfs by using dicts

df3 = df1.copy(deep=True)

res_dict = {"x": 4, "y": 6}
df3.loc[len(df3.index)] = res_dict


but this doesn't preserve datatype either.

The only solution I found so far, is to add an all empty row when adding values of a new original file, accessing cells explicitely, and if a new column is to be added, to also add the column all empty before adding information to the specific cell

for loop_no, key in enumerate(res_dict.keys()):
if loop_no == 0:
row_no = len(df3)
df3.loc[row_no] = [""] * len(df3.columns)

if key in df3.columns:
df3.loc[row_no, key] = res_dict[key]
else:
df3[key] = ""
df3.loc[row_no, key] = res_dict[key]


But this is really tedious and with the amount of data I have to assemble not exactly efficient.

Continue reading...