[Python] How to pass a dynamic list of CSV files from Snakemake input to a Pandas DataFrame...

I'm working on a Snakemake workflow where I need to combine multiple CSV files into a single Pandas DataFrame. The list of CSV files is dynamic—it depends on upstream rules and wildcard patterns.

Here's a simplified version of what I have in my Snakefile:

rule combine_tables:
input:
expand("results/{sample}/data.csv", sample=SAMPLES)
output:
"results/combined/all_data.csv"
run:
import pandas as pd
dfs = [pd.read_csv(f) for f in input]
combined = pd.concat(dfs)
combined.to_csv(output[0], index=False)



This works when the files exist, but I’d like to know:

What's the best practice for handling missing or corrupt files in this context?

Is there a more "Snakemake-idiomatic" way to dynamically list and read input files for Pandas operations?

How do I ensure proper file ordering or handle metadata like sample names if not all CSVs are structured identically?

Continue reading...