When using Cupy, I get tons of messages about ignored OutOfMemoryErrors, e.g. as below: Exception ignored in: 'cupy.cuda.thrust.cupy_malloc' Traceback (most recent call last): File "cupy/cuda/memory.pyx", line 738, in cupy.cuda.memory.alloc File "cupy/cuda/memory.pyx", line 1424, in cupy.cuda.memory.MemoryPool.malloc File "cupy/cuda/memory.pyx", line 1445, in cupy.cuda.memory.MemoryPool.malloc File "cupy/cuda/memory.pyx", line 1116, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc File "cupy/cuda/memory.pyx", line 1137, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc File "cupy/cuda/memory.pyx", line 1382, in cupy.cuda.memory.SingleDeviceMemoryPool._try_malloc File "cupy/cuda/memory.pyx", line 1385, in cupy.cuda.memory.SingleDeviceMemoryPool._try_malloc cupy.cuda.memory.OutOfMemoryError: Out of memory allocating 5,607,143,424 bytes (allocated so far: 7,307,858,432 bytes). In my understanding, these messages are generated by Cupy trying to allocate some memory but failing because the GPU memory is full of allocated but unused (i.e. cache areas) blocks. After failing with the allocation, Cupy frees some allocated but unused blocks and re-tries the allocation, which goes well. Since the generated error message is related to Cupy's internal memory management, and since the application is running simply fine, I find these messages very annoying to be displayed on the console. Is there any way to suppress such messages? Those messages make the console completely messy and unusable. Continue reading...
