I’m currently working with a large dataset and need help merging multiple .fasta files. Although I’m not an expert, I’ve attempted to automate this process using a Python script. However, the merging operation is extremely slow and takes an unreasonable amount of time to complete. To give you more context, my dataset is structured as follows: main_folder/ ├── folder_1/ │ ├── file_1.fasta │ ├── file_2.fasta │ └── ... ├── folder_2/ │ └── ... ├── ... └── folder_N/ ├── file_1.fasta └── file_N.fasta I’m working specifically with the GPD database (Virus - Bioinformatics), and I'm looking for a more efficient way to merge all .fasta files from these subdirectories into a single file. Any suggestions or improvements to speed up the process would be greatly appreciated. def merge_all_fasta_from_subfolders(db_root: Path, output_fasta: Path, batch_size: int = 5000): """ Merge all .fasta files from all subfolders in db_root into a single output .fasta using batch writing. """ start_time = time.time() fasta_files = list(db_root.rglob("*.fasta")) if not fasta_files: print(f"⚠️ No FASTA files found in: {db_root}") return with open(output_fasta, "w") as out_handle: batch: list[SeqRecord] = [] total_written = 0 for file in fasta_files: try: for record in SeqIO.parse(file, "fasta"): batch.append(record) if len(batch) >= batch_size: SeqIO.write(batch, out_handle, "fasta") total_written += len(batch) batch.clear() except Exception as e: print(f"❌ Error reading {file.name}: {e}") if batch: SeqIO.write(batch, out_handle, "fasta") total_written += len(batch) elapsed = time.time() - start_time print(f"✅ Total sequences written: {total_written} in {elapsed:.2f} seconds.") Continue reading...
