[Python] Python xml iterparse consumes memory

I want to process large XML file (50GB) that is archived in zip file (5GB). My idea was to process it in stream mode because of the file size.

I implemented code:

import xml.etree.ElementTree as ET
import zipfile
import gc

def process_big_xml(path_to_zip, path_to_xml):
with zipfile.ZipFile(path_to_zip, "r") as z:
with z.open(path_to_xml) as file_streamed:
context = ET.iterparse(file_streamed, events=("start", "end"))
processed_lines = 0
for event, elem in context:
if event == "end" and elem.tag == "record":
...
(processing data in XML)
...
processed_lines += 1

# Clearing memory
elem.clear()
del elem

if processed_lines % 25000 == 0:
gc.collect() # Using gc to reduce memory usage



The memory usage is increasing linearly despite using elem.clear(), del elem and gc.collect(). Where is the mistake? What can be improved? I tried also with lxml - memory usage was OK, but function was x times slower.

Continue reading...