This is a script I am using to test how much page-locked memory torch can allocate on my machine (htop shows 94.3 GB RAM). import torch pinned_GB_counter = 0 # keep a reference to avoid garbage collection gc_blocker = [] GB_bytes = 1024*1024*1024 while True: try: input(f"Press Enter to increase the pinned GB counter from {pinned_GB_counter} to {pinned_GB_counter+1}") pinned_GB_counter += 1 gc_blocker.append(torch.empty(GB_bytes, dtype=torch.uint8, pin_memory=True)) print(f"pinned GB_counter: {pinned_GB_counter}") except Exception as e: print(f"Ran out of pinned memory after {pinned_GB_counter} GB") print(e) break Every time I run it, I hit a larger number before crashing. The error message is always: CUDA error: invalid argument CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. The numbers I've reached (in order) are: 16, 21, 25, 29, 32, 35, 45, 56, 62. Is torch.empty() lazy by any chance? Is the OS "organizing" its page tables behind the scenes that allows me to page-lock more memory? My expectation would be that the amount of pinnable memory by torch would be constant. Continue reading...
