Develop Reference

How to set level for logging in python from configuration file

I am trying to set loggers for my python code, I want to set the level of the log from the configuration file. But unable to do by me. Here the code is given below, If you noticed that in the given below code can see logger.setLevel(logging.INFO). I don't want to directly mention as a hardcoded value logging.INFO. Need to get this from the config file, is it possible?
import logging
from logging.config import fileConfig
from datetime import date
class Log:
#staticmethod
def trace():
today = date.today()
# dd/mm/YY
d1 = today.strftime("%d_%m_%Y")
# Gets or creates a logger
logger = logging.getLogger(__name__)
# set log level
logger.setLevel(logging.INFO)
# define file handler and set formatter
file_handler = logging.FileHandler('log/'+d1+'_logfile.log')
formatter = logging.Formatter('%(asctime)s : %(levelname)s : %(name)s : %(message)s')
file_handler.setFormatter(formatter)
# add file handler to logger
logger.addHandler(file_handler)
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)
return logger

You can always use Python built-in Configuration file parser
Have the log levels in a config file and read that value. Since that value will be in string, you can define the dictionary mapping in your code. See below for an example.
import configparser
config= configparser.ConfigParser()
config.read('configfile')
log_level_info = {'logging.DEBUG': logging.DEBUG,
'logging.INFO': logging.INFO,
'logging.WARNING': logging.WARNING,
'logging.ERROR': logging.ERROR,
}
print(config['DEFAULT']['LOG_LEVEL'])
my_log_level_from_config = config['DEFAULT']['LOG_LEVEL']
my_log_level = log_level_info.get(my_log_level_from_config, logging.ERROR)
logger.setLevel(my_log_level)
Your config file would be like below:
user#Inspiron:~/code/advanced_python$ cat configfile
[DEFAULT]
LOG_LEVEL = logging.INFO
user#Inspiron:~/code/advanced_python$

If I understood correctly, you need a way to set your logging level at runtime instead of a hard-coded value. I would say you have two options.
The first solution would be to parse your configuration file, and set the level of logging accordingly. If you don't want to parse it everytime the Log class is invoked, in your main you can set a variable that you pass to the Log class.
The second one, that I also suggest, would be to set handlers with python logging class https://docs.python.org/3/library/logging.config.html

logging level (logging.INFO) is an integer value. can you pass numbers from your config file to set log level
print(logging.INFO)
print(logging.WARN)
print(logging.DEBUG)
print(logging.ERROR)
20
30
10
40

Python logging - new log file each loop iteration

I would like to generate a new log file on each iteration of a loop in Python using the logging module. I am analysing data in a for loop, where each iteration of the loop contains information on a new object. I would like to generate a log file per object.
I looked at the docs for the logging module and there is capability to change log file on time intervals or when the log file fills up, but I cannot see how to iteratively generate a new log file with a new name. I know ahead of time how many objects are in the loop.
My imagined pseudo code would be:
import logging
for target in targets:
logfile_name = f"{target}.log"
logging.basicConfig(format='%(asctime)s - %(levelname)s : %(message)s',
datefmt='%Y-%m/%dT%H:%M:%S',
filename=logfile_name,
level=logging.DEBUG)
# analyse target infomation
logging.info('log target info...')
However, the logging information is always appended to the fist log file for target 1.
Is there a way to force a new log file at the beginning of each loop?

Rather than using logging directly, you need to use logger objects. Go thorough the docs here.
Create a new logger object as a first statement in the loop. The below is a working solution.
import logging
import sys
def my_custom_logger(logger_name, level=logging.DEBUG):
"""
Method to return a custom logger with the given name and level
"""
logger = logging.getLogger(logger_name)
logger.setLevel(level)
format_string = ("%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:"
"%(lineno)d — %(message)s")
log_format = logging.Formatter(format_string)
# Creating and adding the console handler
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setFormatter(log_format)
logger.addHandler(console_handler)
# Creating and adding the file handler
file_handler = logging.FileHandler(logger_name, mode='a')
file_handler.setFormatter(log_format)
logger.addHandler(file_handler)
return logger
if __name__ == "__main__":
for item in range(10):
logger = my_custom_logger(f"Logger{item}")
logger.debug(item)
This writes to a different log file for each iteration.

This might not be the best solution, but it will create new log file for each iteration. What this is doing is, adding a new file handler in each iteration.
import logging
targets = ["a", "b", "c"]
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
for target in targets:
log_file = "{}.log".format(target)
log_format = "|%(levelname)s| : [%(filename)s]--[%(funcName)s] : %(message)s"
formatter = logging.Formatter(log_format)
# create file handler and set the formatter
file_handler = logging.FileHandler(log_file)
file_handler.setFormatter(formatter)
# add handler to the logger
logger.addHandler(file_handler)
# sample message
logger.info("Log file: {}".format(target))

This is not necessarily the best answer but worked for my case, and just wanted to put it here for future references. I created a function that looks as follows:
def logger(filename, level=None, format=None):
"""A wrapper to the logging python module
This module is useful for cases where we need to log in a for loop
different files. It also will allow more flexibility later on how the
logging format could evolve.
Parameters
----------
filename : str
Name of logfile.
level : str, optional
Level of logging messages, by default 'info'. Supported are: 'info'
and 'debug'.
format : str, optional
Format of logging messages, by default '%(message)s'.
Returns
-------
logger
A logger object.
"""
levels = {"info": logging.INFO, "debug": logging.DEBUG}
if level is None:
level = levels["info"]
else:
level = levels[level.lower()]
if format is None:
format = "%(message)s"
# https://stackoverflow.com/a/12158233/1995261
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
logger = logging.basicConfig(filename=filename, level=level, format=format)
return logger
As you can see (you might need to scroll down the code above to see the return logger line), I am using logging.basicConfig(). All modules I have in my package that log stuff, have the following at the beginning of the files:
import logging
import other stuff
logger = logging.getLogger()
class SomeClass(object):
def some_method(self):
logger.info("Whatever")
.... stuff
When doing a loop, I have call things this way:
if __name__ == "__main__":
for i in range(1, 11, 1):
directory = "_{}".format(i)
if not os.path.exists(directory):
os.makedirs(directory)
filename = directory + "/training.log"
logger(filename=filename)
I hope this is helpful.

I'd like to slightly modify #0Nicholas's method. The direction is right, but the first FileHandler will continue log information into the first log file as long as the function is running. Therefore, we would want to pop the handler out of the logger's handlers list:
import logging
targets = ["a", "b", "c"]
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
log_format = "|%(levelname)s| : [%(filename)s]--[%(funcName)s] : %(message)s"
formatter = logging.Formatter(log_format)
for target in targets:
log_file = f"{target}.log"
# create file handler and set the formatter
file_handler = logging.FileHandler(log_file)
file_handler.setFormatter(formatter)
# add handler to the logger
logger.addHandler(file_handler)
# sample message
logger.info(f"Log file: {target}")
# close the log file
file_handler.close()
# remove the handler from the logger. The default behavior is to pop out
# the last added one, which is the file_handler we just added in the
# beginning of this iteration.
logger.handlers.pop()

Here is a working version for this problem. I was only able to get it to work if the targets already have .log before going into the loop so you may want to add one more for before going into targets and override all targets with .log extension
import logging
targets = ["a.log","b.log","c.log"]
for target in targets:
log = logging.getLogger(target)
formatter = logging.Formatter('%(asctime)s - %(levelname)s : %(message)s', datefmt='%Y-%m/%dT%H:%M:%S')
fileHandler = logging.FileHandler(target, mode='a')
fileHandler.setFormatter(formatter)
streamHandler = logging.StreamHandler()
streamHandler.setFormatter(formatter)
log.addHandler(fileHandler)
log.addHandler(streamHandler)
log.info('log target info...')

two filehandlers of two different loggers get the same content

I wrote a script that collects the event logs from cloudwatch, writes them to two files then sends the content of each file by mail. As I want to receive two emails, each loggers has a different type of logs, I created two loggers:
logLevel = logging.INFO
doWait = object()
if os.path.exists ('.debug'):
logLevel = logging.DEBUG
logger1 = logging.getLogger()
logger2 = logging.getLogger()
logger1.setLevel(logLevel)
logger2.setLevel(logLevel)
formatter = logging.Formatter('%(asctime)s :: %(levelname)s :: %(message)s')
fileHandler1 = os.path.join('/tmp/', 'error-admin-' + datetime.datetime.utcnow().strftime('%Y-%m-%d') + '.log')
fileHandler2 = os.path.join('/tmp/', 'error-user-' + datetime.datetime.utcnow().strftime('%Y-%m-%d') + '.log')
fh1 = logging.handlers.RotatingFileHandler(fileHandler)
fh2 = logging.handlers.RotatingFileHandler(fileHandler2)
fh1.setLevel(logLevel)
fh2.setLevel(logLevel)
fh1.setFormatter(formatter)
fh2.setFormatter(formatter)
logger1.addHandler(fh1)
logger2.addHandler(fh2)
in the main, I call functions like that:
function1(arg1,arg2,logger1)
function2(arg1,arg2,logger2)
Now, my problem is, that when I execute my script, I get the same content in the both filehandlers, even that I separated the loggers. Why does that happen?

This happens because logger1 and logger2 both point at the root Logger underneath.
From the docs: Multiple calls to getLogger() with the same name will always return a reference to the same Logger object.
You need two separate Logger objects:
logger1 = logging.getLogger('error-admin')
logger2 = logging.getLogger('error-user')

How do I set a different name to an individual log handler?

I have two log handlers in my code: a StreamHandler to write INFO level logs from the same module to stdout, and a FileHandler to write more verbose, DEBUG logs to a file. This is my code:
import sys
import logging
log = logging.getLogger('mymodule')
log.setLevel(logging.DEBUG)
logf = logging.FileHandler('file.log')
logf.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
log.addHandler(logf)
logs = logging.StreamHandler(sys.stdout)
logs.setLevel(logging.INFO)
logs.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
log.addHandler(logs)
However, I want the FileHandler to also write DEBUG info from other modules. I can achieve this if I remove the name from logging.getLogger(), but this will also affect my StreamHandler, which I only want to print output from my own module.
So is there a way to have either of the handlers use a different name?

There's a name attribute available on the base Handler class. I took a look at the Python source and doesn't look like it's used internally so you could manually set each handler to a different name.

Python logger not respecting setLevel?

I've spent a bit of time looking through the site at Python logger questions hoping my would be resolved there. I've set up a logger with two stream handlers that have both different formats and levels of logging, here's a functional snippet from my codebase:
import os
import time
import logging
LOG_LEVELS = [logging.ERROR, logging.WARNING, logging.INFO, logging.DEBUG]
TEST_RESULT_LEVELV_NUM = 51
# http://stackoverflow.com/a/11784984/196832
def status(self, message, *args, **kws):
self._log(TEST_RESULT_LEVELV_NUM, message, args, **kws)
logging.addLevelName(TEST_RESULT_LEVELV_NUM, "RESULT")
logging.Logger.result = status
def setup_logging(level=0, quiet=False, logdir=None):
logger = logging.getLogger('juju-test')
ffmt = logging.Formatter('%(asctime)s %(name)s %(levelname)-8s: %(message)s')
cfmt = logging.Formatter('%(name)s %(levelname)s: %(message)s')
#logger.setLevel(0)
if level >= len(LOG_LEVELS):
level = len(LOG_LEVELS) - 1
if logdir:
if not os.path.exists(logdir):
os.makedirs(logdir)
logfile = os.path.join(logdir, 'juju-test.%s.log' % int(time.time()))
fh = logging.FileHandler(logfile)
# Always at least log to INFO for file, unless DEBUG is requested
fh.setLevel(LOG_LEVELS[level if level >= 2 else 2])
fh.setFormatter(ffmt)
logger.addHandler(fh)
if not quiet:
ch = logging.StreamHandler()
ch.setLevel(LOG_LEVELS[level])
ch.setFormatter(cfmt)
logger.addHandler(ch)
return logger
I've been using an argparse to feed this, but for testing purposes if you feed the following to function:
logger = setup_logging(level=1, logdir="/tmp/oofrab/")
logger.info('Informative!')
logger.warn('Whoa buddy!')
logger.error('Look what you did.')
logger.result("They told me not to make a custom logging level, I'll show them!")
logger.debug('Lots of bugs, man')
I'd expect to see status, error, and warn in the console. Then status, error, warn and info in the log. However, I only see down to warn in both console and log file, despite selecting logging.INFO (key 2 in the LOG_LEVELS list) for the file handler. Is this expected?
I'm not using basicConfig or anything else when building the logger, why can't I have these two custom levels?

Apparently, logging.NOTSET does not mean "All levels", but rather defaults. So setting the parent logger to level 0 does only reverts it to it's default accepted levels. That being said, if I set logger.setLevel to logging.DEBUG that essentially sets logging to accept ALL levels then passes filtering on to the various handlers to filter further.
To get around this (and potential custom log levels) I've set the initial logger level to 1