I'm running an experiment on copy-on-write mechanism on Python Multiprocessing. I created a large file of 10GB and load the file into large_object in main. file_path = 'dummy_large_file.bin' try: large_object = load_large_object(file_path) print("File loaded successfully") except ValueError as e: print(e) except Exception as e: print(f"An error occurred: {e}") # Create a dummy file of 10GB for testing dummy_file_path = 'dummy_large_file.bin' with open(dummy_file_path, 'wb') as dummy_file: dummy_file.seek(10 * 1024 * 1024 * 1024 - 1) dummy_file.write(b"A") print(f"Dummy file created at {dummy_file_path}") I intentionally pass the large_object to the multiprocessing workers to see the workers copy the large object. And to make sure multiprocessing copy, I modify the content of the object. # Multi-processing num_workers = 4 chunk_size = 10 processes = [] for i in range(num_workers): start = i * chunk_size p = multiprocessing.Process(target=process_chunk, args=(large_object, dummy_file_path, start, chunk_size)) processes.append(p) p.start() for p in processes: p.join() # Multi-processing def process_chunk(object, file_path, start, size): random_chunk = bytes([random.randint(0, 255) for _ in range(size)]) object[start:start + size] = random_chunk # Modify the chunk to a random binary number print(f"large object [{start}:{start + size}]: {object[start:start + size]}") # Process the chunk The expectation here is the program should hold large_object of 10GB, and it copies the large_object to 4 other workers - which sums to 40GB. So the total memory should be 50GB. However, I'm only seeing the total memory of 20GB. Why there's only 20GB memory consumption? Does python implement some other lazy loading or granular copy for large objects? Continue reading...
