[Python] Best way to manually interfere in `snakemake` workflow and automatically backup the...

Sometimes in a long workflow, manual interference in form of modifying an intermediate result is required as automatization is just not perfect. In my case, I segment raw input images with deep learning and afterwards perform analysis of these segmentations. As the segmentation is not always perfect, I manually correct the segmentation sometimes. As the manual correction takes some time, I also want to backup these manual corrections to a separate location in case something happens to my local computer.

For this, I use the following simplified snakemake workflow. The folder data/backup is a symbolic link to a mounted network drive.

import os

rule segment:
input:
img="data/raw/{file}.tif",
output:
segmentation="data/segmentations/{file}.tif",
script: "src/segment.py"

rule backup:
input:
img="data/segmentations/{file}_manual.tif",
output:
img="data/backup/{file}_manual.tif",
shell: "cp {input.img} {output.img}"

def analyze_input_func(wildcards):
path = "data/segmentations"
auto_path = os.path.join(path, f"{wildcards.file}.tif")
manual_path = auto_path.replace(".tif", "_manual.tif")
if os.path.isfile(manual_path):
backup_path = f"data/backup/{wildcards.file}_manual.tif"
return {
"segmentation": manual_path,
"backup": backup_path,
}
else:
return {"segmentation": auto_path}

rule analyse:
input:
unpack(analyze_input_func),
output:
csv="data/properties/{file}.csv",
script: "src/analyse.py"


Do you have suggestions on how to improve this workflow? I feel that this workflow could be written more elegantly. Especially, as the path to data/segmentations/{file}.tif has to be defined independently in two places: in the segment rule and in the analyze_input_func. This requires more maintenance when changing this path and is prone to errors.

Disclaimer: I have asked, to include a manual input function into snakemake as new feature.

Continue reading...