[Python] Redirect standard output of program run using subprocess to rolling log file

I have a code (Fortran), called from a Python wrapper using subprocess, that can write a lot of information to standard output. Sometimes, if the output is too large, it causes the code to crash. Usually, we don't need all the output, just the last few lines that tell us whether the code ran successfully or ran into a problem. So, I want to redirect the standard output to a log file of finite size, say a few lines. I found that the logging package allows this with RotatingFileHandler. However, I cannot manage to tell subprocess to write to the log file handled by the RotatingFileHandler.

Currently, my code is as follow

import subprocess as subp
import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger( 'My rolling log' )
handler = RotatingFileHandler( 'test.log', maxBytes=1000,
backupCount=1, mode='w+', encoding='latin-1' )
logger.addHandler( handler )
# define a fileno method because subprocess needs one:
logger.fileno = logger.handlers[0].stream.fileno
stdout = logger

s = subp.Popen( 'mycode.x', stdin=subp.PIPE, stdout=stdout, stderr=subp.PIPE )

sortie, err = s.communicate( input=cmd.encode() )


This does save the output to a the log file, but it is not rolling as I want. Could you point out what I am doing wrong ?

Continue reading...