[Python] Memory Not Released After Each Request Despite Cleanup Attempts

We're running a FastAPI service that fetches data from Trino, processes it using PyArrow and Polars, and uploads the result to AWS S3 in Parquet format. However, we're facing a persistent issue where memory is not released after each request, even after explicitly attempting cleanup.

Architecture overview:

• Framework: FastAPI
• Data Source: Trino
• Processing: PyArrow, Polars
• Storage: AWS S3 (Parquet format)

Flow:

1. API receives a request
2. Fetch data from Trino
3. Convert to PyArrow Table
4. Upload to S3 using write_to_dataset
5. Attempt memory cleanup

We've tried:

1. 
   Using del to delete large objects
   
   Deleted PyArrow tables and Polars DataFrames after upload.
   
   import pyarrow as pa
   import pyarrow.parquet as pq
   import pyarrow.fs as pafs
   # Fetch and process data
   data = fetch_data_from_trino()
   arrow_table = pa.table(list(zip(*data)))
   # Initialize S3 filesystem with optimal configuration for high-performance writes
   s3_filesystem = pafs.S3FileSystem()
   # Upload to S3
   pq.write_to_dataset(
   arrow_table,
   root_path=f"{RESULT_STORAGE_BUCKET}/{s3_storage_path}",
   partition_cols=['organisation'],
   filesystem=s3_filesystem
   )
   # Attempt to free memory
   del data
   del arrow_table
   
   
   Observation: Memory usage remained high even after deletion.
   
   "resource_stats": {
   "memory_mb": {
   "start": 140.3004568,
   "peak": 589.02921865,
   "end": 587.01258
   }
   }
2. 
   Forcing garbage collection with gc.collect()
   
   Called after del.
   
   import os
   import psutil
   
   
   def force_garbage_collection_and_cleanup() -> Dict[str, int]:
   try:
   # Get memory usage before cleanup
   memory_before = psutil.Process(os.getpid()).memory_info().rss / (1024 ** 2)
   
   # Force garbage collection - run multiple times for thorough cleanup
   objects_collected = 0
   for _ in range(3): # Run GC multiple times for better cleanup
   collected = gc.collect()
   objects_collected += collected
   
   # Get memory usage after cleanup
   memory_after = psutil.Process(os.getpid()).memory_info().rss / (1024 ** 2)
   
   except Exception as cleanup_error:
   logger.warning(f"Failed to perform garbage collection: {str(cleanup_error)}")
   
   
   Observation: no significant drop in memory usage.
   
   "resource_stats": {
   "memory_mb": {
   "start": 190.56,
   "peak": 579.6,
   "end": 579.7
   }
   }
3. 
   Manual memory release with malloc_trim and deep GC
   
   Custom force_memory_release() function:
   
   • Runs gc.collect() across all generations
   • Calls malloc_trim(0) (Linux)
   
   import gc
   import sys
   import psutil
   import platform
   import ctypes
   
   
   def force_memory_release():
   """
   Force Python to release memory back to the operating system.
   
   This method:
   1. Runs garbage collection multiple times to ensure all unreferenced objects are cleaned
   2. On Linux (ECS), calls malloc_trim() to force glibc to release memory to OS
   3. Tracks and reports memory freed
   
   Returns:
   dict: Memory statistics before and after cleanup
   """
   # Get memory usage before cleanup
   process = psutil.Process()
   memory_before_mb = process.memory_info().rss / 1024 / 1024
   
   # Force garbage collection multiple times
   # First pass: collect unreferenced objects
   collected_gen0 = gc.collect(0) # Young generation
   collected_gen1 = gc.collect(1) # Middle generation
   collected_gen2 = gc.collect(2) # Old generation
   
   # Second pass: ensure all cyclic references are broken
   collected_final = gc.collect()
   
   total_objects_collected = collected_gen0 + collected_gen1 + collected_gen2 + collected_final
   
   # Force memory release to OS on Linux systems (ECS containers)
   malloc_trim_success = False
   if platform.system() == 'Linux':
   try:
   # Load glibc and call malloc_trim to release memory to OS
   libc = ctypes.CDLL("libc.so.6")
   malloc_trim = libc.malloc_trim
   malloc_trim.argtypes = [ctypes.c_size_t]
   malloc_trim.restype = ctypes.c_int
   
   # malloc_trim(0) releases all possible memory to OS
   result = malloc_trim(0)
   malloc_trim_success = bool(result)
   
   if malloc_trim_success:
   print("Successfully called malloc_trim() to release memory to OS")
   else:
   print("malloc_trim() was called but returned 0 (no memory released)")
   
   except Exception as e:
   print(f"Could not call malloc_trim(): {e}")
   malloc_trim_success = False
   
   # Get memory usage after cleanup
   memory_after_mb = process.memory_info().rss / 1024 / 1024
   memory_freed_mb = max(0, memory_before_mb - memory_after_mb)
   
   cleanup_stats = {
   'objects_collected': total_objects_collected,
   'memory_before_mb': round(memory_before_mb, 2),
   'memory_after_mb': round(memory_after_mb, 2),
   'memory_freed_mb': round(memory_freed_mb, 2),
   'malloc_trim_success': malloc_trim_success,
   'platform': platform.system()
   }
   
   print(
   f"Memory cleanup completed: "
   f"Objects collected: {total_objects_collected}, "
   f"Memory freed: {memory_freed_mb:.2f}MB "
   f"({memory_before_mb:.2f}MB -> {memory_after_mb:.2f}MB), "
   f"malloc_trim: {'success' if malloc_trim_success else 'not available/failed'}"
   )
   
   return cleanup_stats
   
   
   force_memory_release()
   
   
   
   Observation: this method showed partial success in reducing memory usage. However, memory was not fully returned to baseline, indicating potential native memory fragmentation or memory held by external libraries.
   
   "resource_stats": {
   "memory_mb": {
   "start": 216.47,
   "peak": 460,
   "end": 460
   }
   }
4. 
   Periodic memory cleanup script
   
   Background script runs every 10 seconds:
   
   • Triggers GC
   • Attempts to release PyArrow memory
   • Calls malloc_trim
   
   #!/bin/bash
   
   # Run a Python script that performs memory cleanup
   /usr/local/bin/python3 <<EOF
   import gc
   import pyarrow as pa
   import psutil
   import os
   import ctypes
   import time
   
   def log(msg):
   print(f"[{time.strftime('%Y-%m-%d %H:%M:%S')}] {msg}")
   
   process = psutil.Process(os.getpid())
   rss_before = process.memory_info().rss / (1024 ** 2)
   arrow_before = pa.total_allocated_bytes() / (1024 ** 2)
   
   log(f"Memory before cleanup: {rss_before:.2f} MB (RSS), {arrow_before:.2f} MB (PyArrow)")
   
   # Perform garbage collection
   gc.collect()
   
   # Release unused memory from PyArrow
   pa.default_memory_pool().release_unused()
   
   # Try to return memory to OS (glibc only)
   try:
   ctypes.CDLL("libc.so.6").malloc_trim(0)
   except Exception as e:
   log(f"malloc_trim failed: {e}")
   
   rss_after = process.memory_info().rss / (1024 ** 2)
   arrow_after = pa.total_allocated_bytes() / (1024 ** 2)
   
   log(f"Memory after cleanup: {rss_after:.2f} MB (RSS), {arrow_after:.2f} MB (PyArrow)")
   EOF
   
   
   
   Observation: while PyArrow memory usage dropped significantly, RSS memory did not fully return to baseline. This suggests native memory fragmentation or memory still held by other libraries.
   
   "resource_stats": {
   "memory_mb": {
   "start": 141.30078125,
   "peak": 634.94921875,
   "end": 635.0
   }
   }

Questions:

• What could be causing memory to remain high after each request, even after aggressive cleanup attempts?
• Are there known memory retention issues with PyArrow, Polars, or Trino clients in long-running FastAPI services?
• Any best practices for managing native memory in such a pipeline?

Continue reading...