[Python] Python logging with json config file

I played around with the logging module and found some things that are not quite clear to me.

First of all, because it is a big program I wanted the code to be as simple as possible, therefore I used a json config file.

{
"version": 1,
"disable_existing_loggers": false,
"formatters": {
"simple": {
"format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
},
"extra": {
"format": "%(asctime)-16s %(name)-8s %(filename)-12s %(lineno)-6s %(funcName)-30s %(levelname)-8s %(message)s",
"datefmt": "%m-%d %H:%M:%S"
}
},

"handlers": {
"console": {
"class": "logging.StreamHandler",
"level": "DEBUG",
"formatter": "simple",
"stream": "ext://sys.stdout"
},

"info_file_handler": {
"class": "logging.handlers.RotatingFileHandler",
"level": "INFO",
"formatter": "simple",
"filename": "info.log",
"maxBytes": 10485760,
"backupCount": 40,
"encoding": "utf8"
},

"error_file_handler": {
"class": "logging.handlers.RotatingFileHandler",
"level": "ERROR",
"formatter": "simple",
"filename": "errors.log",
"maxBytes": 10485760,
"backupCount": 40,
"encoding": "utf8"
}
},

"loggers": {
"my_module": {
"level": "ERROR",
"handlers": ["console"],
"propagate": "no"
}
},

"root": {
"level": "INFO",
"handlers": ["console", "info_file_handler", "error_file_handler"]
}
}


This json file is imported and used. It seems to work fine. Unfortunately there are some problems. After finishing the code (still working on it) the logger does not close the file. And due to this in the next run it is appending the new run to the old log file. No possibility to delete it or truncate() without problems. Thus I wanted to remove the handlers writing the files. Unfortunately it seems like logger.handlers is empty, and thus I can not find them anywhere. Probably got an idea?

Also the log files have a major problem. The logger is just appending everything new without starting a new line. Which ends up in an endless one-line log.

main test program:

import sys
import os
import json
import logging.config

import my_module

def setup_logging(
default_path='logging.json',
default_level=logging.INFO,
env_key='LOG_CFG'
):
"""
Setup logging configuration
"""
path = default_path
value = os.getenv(env_key, None)
if value:
path = value
if os.path.exists(path):
with open(path, 'rt') as f:
config = json.load(f)
logging.config.dictConfig(config)
else:
logging.basicConfig(level=default_level)


def main():

setup_logging()
logger = logging.getLogger(__name__)
logger.info('Startlogging:')

my_module.foo()
b = my_module.Bar()
b.bar()

print logger.handlers

if __name__ == '__main__':
main()

Continue reading...