[Python] Memory leak when reading a large tar archive

I have a large 30GB tar archive with 32KB files in it. My task is to get a list of files and sort them from the smallest to the largest.

After that, create a sorted list and save it to disk to transfer it to another program for processing. Technically, I can't put the entire list of files in RAM, so I decided to use an external sorting algorithm. However, I constantly get memory leaks, when running the code, after creating the entire 15th chunk, RAM consumption is about 6GB, which is generally not normal. I can't figure out where the memory is leaking and what I'm doing wrong.

example of archive content file names:

1740296766220.dump 1740296766320.dump 1740296766331.dump 1740296766339.dump 1740296766380.dump 1740296766391.dump 1740296766410.dump
​

import tarfile
import os
import gc
import heapq
import tempfile


def parse_filename(filename):
""" Extracts the numeric part of the file name for sorting. """
try:
return int(filename.split('.')[0])
except ValueError:
return float('inf')


def filename_generator(tar_path):
""" A generator that lazily reads file names from a tar archive. """
with tarfile.open(tar_path, 'r') as tar:
for member in tar:
yield member.name


def external_sort_tar(tar_path, output_file, chunk_size=1_000_000):
temp_files = []
temp_dir = tempfile.mkdtemp() # Temporary directory for chunks
print(temp_dir)

def save_chunk(filenames, chunk_index):
""" Sorts and saves chunks to a temporary file. """
temp_file = os.path.join(temp_dir, f"chunk_{chunk_index}.txt")
filenames.sort(key=parse_filename)
with open(temp_file, "w") as tf:
tf.writelines("\n".join(filenames) + "\n")
return temp_file

filenames = []
chunk_index = 0
for name in filename_generator(tar_path):
filenames.append(name)
if len(filenames) >= chunk_size: # We limit the volume of the chunk
temp_files.append(save_chunk(filenames, chunk_index))
filenames = []
gc.collect()
chunk_index += 1

if filenames:
temp_files.append(save_chunk(filenames, chunk_index))

# Multipath merging of sorted chunks
with open(output_file, "w") as out:
file_handles = [open(f, "r") for f in temp_files]
out.writelines(heapq.merge(*file_handles, key=parse_filename))

# Closing and deleting temporary files
for f in file_handles:
f.close()
for f in temp_files:
os.remove(f)
os.rmdir(temp_dir)


external_sort_tar('example.tar', "tar.map", chunk_size=1000000)


I also tried to set chunk_size to 1000, however, it didn't help, it's not related to the size of the chunk.

What are your ideas?

Script for generating a test archive - to reproduce the problem

#!/bin/bash

# Ask the user for the number of files to generate (default: 10000)
read -p "How many files to generate? (default: 10000): " FILE_COUNT
FILE_COUNT=${FILE_COUNT:-10000}

# Get the current Unix timestamp in milliseconds
START_TIMESTAMP=$(date +%s%3N)

# Create a temporary directory for the files
TMP_DIR=$(mktemp -d)
echo "Temporary directory created: $TMP_DIR"

# Generate files with random data
for ((i=0; i<FILE_COUNT; i++)); do
FILENAME="$TMP_DIR/$((START_TIMESTAMP + i)).dump"
head -c 100 /dev/urandom > "$FILENAME"
done

echo "$FILE_COUNT files created. Archiving..."

# Create a tar archive without compression (files are placed directly in the root of the archive)
ARCHIVE_NAME="dumps_$(date +%s).tar"
tar -cf "$ARCHIVE_NAME" -C "$TMP_DIR" $(ls "$TMP_DIR")

echo "Files added to archive: $ARCHIVE_NAME"

# Remove the temporary directory
rm -rf "$TMP_DIR"
echo "Temporary files removed. Done!"

Continue reading...