[Python] dealing with reading/writing large files inside a k8s pod with a memory limit

I have several applications that run inside a k8s pod, where the pod has a specific memory limit.

I noticed that if i run a python script that deal with a large file, e.g: unzip a large file example.zip, the pod memory usage really spikes. As i understand, even if i use an unzip function that uses streaming, the whole data of the large zip file will be written into the os system cache, and the memory usage of the pod includes the page cache, therefore can spike and cause OOM errors.

When i could have control over how i read the file, using python, i could use libc.posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED) to drop the file from the cache and control the process so my cache wouldnt explode. But if i use unzip, i have no such control.

Is there a better, more general way to deal with large files inside a pod? or maybe only way is to try to clear the cache with every application i use? (posix_fadvise)?

edit: i was wrong and even if i read the zip file chunk by chunk, i am able to call posix_fadvise on the zip output and input file descriptor, so this solution works.

on the other hand, i wonder if there isnt a more "global" system oriented solution and not an applicative one?

Continue reading...