[Python] How can I open multiple files (number of files unknown beforehand) using "with open"...

I need to use a with open statement for opening the files, because I need to open a few hundred files together and merge them using K-way merge.

So at this point, I need very fast I/O that does not store the whole/huge portion of file in memory (because there are hundreds of files, each of ~10MB). I just need to read one line at a time for K-way merge. Reducing memory usage is my primary focus right now.

I learned that with open is the most efficient technique, but I cannot understand how to open all the files together in a single with open statement.

Continue reading...