[Python] Pandas : suppression de colonnes ne fonctionne pas après requête SQL dans un pipeline...

I'm currently working on an ETL pipeline that extracts data from a PostgreSQL database, then transforms and stores it in an Excel file using pandas.

One of the transformation steps consists in deleting some columns from the SQL query and adding a new custom column.

Creating the new column works correctly, but deleting columns doesn't seem to work, and I don't understand why.

Here's the relevant code compartment:

Translated with DeepL.com (free version)

for op_type, df_list in dataframes_by_operation_type.items():
if df_list:
df_consolidated = pd.concat(df_list, ignore_index=True)

if not df_consolidated.empty:
possible_id_column_names = [f"ID {w['id_display_name']}" for w in WALLETS if 'id_display_name' in w]
def find_operator_id(row):
for id_col_name in possible_id_column_names:
if id_col_name in row.index and pd.notna(row[id_col_name]):
return row[id_col_name]
return None

df_consolidated['ID Opérateur'] = df_consolidated.apply(find_operator_id, axis=1)

cols_to_drop = [col for col in possible_id_column_names if col in df_consolidated.columns]
if cols_to_drop:
df_consolidated = df_consolidated.drop(columns=cols_to_drop)

result_consolidated = storage_service.save_to_excel(
df=df_consolidated,
operator_name="all",
wallet_name="all",
operation_type=op_type,
period_type="daily",
company_id=company['id'],
company_name=company['name']
)
overall_results.append(result_consolidated)


So I need your help.

Continue reading...