From the Python documentation: By default, the pickle data format uses a relatively compact binary representation. If you need optimal size characteristics, you can efficiently compress pickled data. I'm going to be serializing several gigabytes of data at the end of a process that runs for several hours, and I'd like the result to be as small as possible on disk. However, Python offers several different ways to compress data. Is there one of these that's particularly efficient for pickled files? The data I'm pickling mostly consists of nested dictionaries and strings, so if there's a more efficient way to compress e.g. JSON, that would work too. The time for compression and decompression isn't important, but the time this process takes to generate the data makes trial and error inconvenient. Continue reading...
