I implemented a custom logging.StreamHandler that stores logs on S3. It works fine in a single process, but I cannot handle scenarios where processes are forked. I'm on Linux and only need to support that platform. The general approach is this: class S3Handler(logging.StreamHandler): def __init__(self): super().__init__(io.StringIO()) # create unique target location and counter # counter for rotating file names def emit(self, record): # emit record # if buffer is large enough: store data to file def close(self): # store buffer to file This works like a charm for a single process. I configure the logger in a config file and use logging.fileConfig to set everything up. In my code I use LOG = logging.getLogger(__name__) everywhere on the module level. But my project uses Luigi as a library, which forks subprocesses and things become a bit strange: The different processes use the same "copy" of the handler, so they use the same unique target location and will overwrite each others data. This makes sense, but while figuring it out I also saw unexpected closing of handlers, which I could not really pin down. There is very little information about logging in the context of forking, so I dig into the source code of the logging library. I found _at_fork_reinit() which I overwrote. My implementation of it creates a new unique target location and a fresh io.StringIO buffer. This approach works fine in the sense that data is not overwritten anymore. I get output for the individual subprocess in individual files. But log messages are now "multiplied": I see logs from the main process showing up in logs from the subprocesses. I have not idea how this is possible, because no file handles are involved. Everything happens in memory and that should be isolated between processes. Any hint how Python logging is supposed to work and to be used in the context of a fork would be highly appreciated. Continue reading...
