Develop Reference

Python Loggin - Only log to file

Basically, I want to initialise a logger for each purpose, such that each logger will log to a file, but I notice that the content of log has been printed to console as well. I don't want that to happen, but I don't know how to adjust my code.
def getLogger(loggerName:str, fileDir:str,
level = logging.DEBUG,
format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'):
"""return a logger for logging
somehow the logDecorator function does
Args:
loggerName (str): name given to the logger
fileDir (str): directory
level ([type], optional): [description]. Defaults to logging.DEBUG.
format (str, optional): [description]. Defaults to '%(asctime)s - %(name)s - %(levelname)s - %(message)s'.
Returns:
logger: [description]
"""
myLogger = logging.getLogger(loggerName)
myLogger.setLevel(level)
myFormat = logging.Formatter(format)
myHandler = logging.FileHandler(fileDir, mode='a')
myHandler.setFormatter(myFormat)
myLogger.addHandler(myHandler)
# console only prints WARNING & ABOVE
# ideally, what I thought is that if I don't not
# add a SteamHandler, the log won't even print to console at all
# but adding a SteamHandler still prints EVERYTHING to the console
# despite the level set as WARNING
console = logging.StreamHandler()
console.setLevel(logging.WARNING)
console.setFormatter(myFormat)
myLogger.addHandler(console)
return myLogger
I have tried this:
mainLogger = logging.getLogger()
mainLogger.setLevel(logging.WARNING) # only log WARNING and above
but It won't work as well, still printing to console.
So I am lost in my understanding of LOGGING module, I am not sure where have I done wrong.
Example of using the code:
aRandomLogger = getLogger(loggerName = 'ProgressLogger', fileDir = '../logs/ProgressLogger.log')
aRandomLogger.info('This is an information, I want it to be log to file, but not on console')
aRandomLogger.warning('this is a warning, I want it be be log to file, and shown on console')
Turns out, both logs are saved to file and to console.

It has been awhile, but I have managed to figure out the solution to this.
myLogger = logging.getLogger(loggerName)
myLogger.propagate = False
Setting propagate to False will prevent the logger from producing unwanted logs everywhere.

Python multiple loggers not working. How to configure multiple loggers with different levels?

I am trying to configure two loggers, one logger is for INFO level, and the other logger is for DEBUG level. I would like DEBUG content to only go to my log file, and I would like the INFO content to go both a log file and to the console. Please see the below code. Nothing is being written into my files and nothing is being displayed in the console.
logFileDir = os.path.join(os.getcwd(), '.logs')
if not os.path.exists(logFileDir):
os.mkdir(logFileDir)
infoLogFileDir = os.path.join(logFileDir, 'INFO')
if not os.path.exists(infoLogFileDir):
os.mkdir(infoLogFileDir)
debugLogFileDir = os.path.join(logFileDir, 'DEBUG')
if not os.path.exists(debugLogFileDir):
os.mkdir(debugLogFileDir)
LOG_FORMAT = ("%(asctime)s [%(levelname)s]: %(message)s in %(pathname)s:%(lineno)d")
#DEBUG LOGGER
debugLogFileName = os.path.join(debugLogFileDir, 'EFDebugLog.log')
debugLogger = logging.getLogger("debugLogger")
debugLogger.setLevel(logging.DEBUG)
debugHandler = logging.handlers.RotatingFileHandler(filename=debugLogFileName,maxBytes=5000000, backupCount=100)
debugHandler.setLevel(logging.DEBUG)
debugHandler.setFormatter(Formatter(LOG_FORMAT))
debugLogger.addHandler(debugHandler)
#INFO LOGGER
infoLogFileName = os.path.join(infoLogFileDir, 'EFInfoLog.log')
infoLogger = logging.getLogger("infoLogger")
infoLogger.setLevel(logging.INFO)
infoHandler = logging.handlers.RotatingFileHandler(filename=infoLogFileName,maxBytes=5000000, backupCount=100)
infoHandler.setLevel(logging.INFO)
infoHandler.setFormatter(Formatter(LOG_FORMAT))
infoLogger.addHandler(infoHandler)
infoLogger.addHandler(logging.StreamHandler())

The logging.* functions that you are calling log to the root logger. That's why you don't see any output; you haven't configured any handlers for the root logger. You have only configured handlers for your own loggers, which you are not using.
If you want to use the logging.* functions, you need to first configure the root logger, which you can get by calling getLogger without any arguments. So the code might look like:
import logging
import logging.handlers
root_logger = logging.getLogger()
info_handler = logging.handlers.RotatingFileHandler(filename='infolog.txt')
info_handler.setLevel(logging.INFO)
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
debug_handler = logging.handlers.RotatingFileHandler(filename='debuglog.txt')
debug_handler.setLevel(logging.DEBUG)
root_logger.addHandler(stream_handler)
root_logger.addHandler(debug_handler)
root_logger.addHandler(info_handler)
# this is needed, since the severity is WARNING by default,
# i.e. it would not log any debug messages
root_logger.setLevel(logging.DEBUG)
root_logger.debug('this is a debug message')
root_logger.info('this is an info message')

Python logging - new log file each loop iteration

I would like to generate a new log file on each iteration of a loop in Python using the logging module. I am analysing data in a for loop, where each iteration of the loop contains information on a new object. I would like to generate a log file per object.
I looked at the docs for the logging module and there is capability to change log file on time intervals or when the log file fills up, but I cannot see how to iteratively generate a new log file with a new name. I know ahead of time how many objects are in the loop.
My imagined pseudo code would be:
import logging
for target in targets:
logfile_name = f"{target}.log"
logging.basicConfig(format='%(asctime)s - %(levelname)s : %(message)s',
datefmt='%Y-%m/%dT%H:%M:%S',
filename=logfile_name,
level=logging.DEBUG)
# analyse target infomation
logging.info('log target info...')
However, the logging information is always appended to the fist log file for target 1.
Is there a way to force a new log file at the beginning of each loop?

Rather than using logging directly, you need to use logger objects. Go thorough the docs here.
Create a new logger object as a first statement in the loop. The below is a working solution.
import logging
import sys
def my_custom_logger(logger_name, level=logging.DEBUG):
"""
Method to return a custom logger with the given name and level
"""
logger = logging.getLogger(logger_name)
logger.setLevel(level)
format_string = ("%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:"
"%(lineno)d — %(message)s")
log_format = logging.Formatter(format_string)
# Creating and adding the console handler
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setFormatter(log_format)
logger.addHandler(console_handler)
# Creating and adding the file handler
file_handler = logging.FileHandler(logger_name, mode='a')
file_handler.setFormatter(log_format)
logger.addHandler(file_handler)
return logger
if __name__ == "__main__":
for item in range(10):
logger = my_custom_logger(f"Logger{item}")
logger.debug(item)
This writes to a different log file for each iteration.

This might not be the best solution, but it will create new log file for each iteration. What this is doing is, adding a new file handler in each iteration.
import logging
targets = ["a", "b", "c"]
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
for target in targets:
log_file = "{}.log".format(target)
log_format = "|%(levelname)s| : [%(filename)s]--[%(funcName)s] : %(message)s"
formatter = logging.Formatter(log_format)
# create file handler and set the formatter
file_handler = logging.FileHandler(log_file)
file_handler.setFormatter(formatter)
# add handler to the logger
logger.addHandler(file_handler)
# sample message
logger.info("Log file: {}".format(target))

This is not necessarily the best answer but worked for my case, and just wanted to put it here for future references. I created a function that looks as follows:
def logger(filename, level=None, format=None):
"""A wrapper to the logging python module
This module is useful for cases where we need to log in a for loop
different files. It also will allow more flexibility later on how the
logging format could evolve.
Parameters
----------
filename : str
Name of logfile.
level : str, optional
Level of logging messages, by default 'info'. Supported are: 'info'
and 'debug'.
format : str, optional
Format of logging messages, by default '%(message)s'.
Returns
-------
logger
A logger object.
"""
levels = {"info": logging.INFO, "debug": logging.DEBUG}
if level is None:
level = levels["info"]
else:
level = levels[level.lower()]
if format is None:
format = "%(message)s"
# https://stackoverflow.com/a/12158233/1995261
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
logger = logging.basicConfig(filename=filename, level=level, format=format)
return logger
As you can see (you might need to scroll down the code above to see the return logger line), I am using logging.basicConfig(). All modules I have in my package that log stuff, have the following at the beginning of the files:
import logging
import other stuff
logger = logging.getLogger()
class SomeClass(object):
def some_method(self):
logger.info("Whatever")
.... stuff
When doing a loop, I have call things this way:
if __name__ == "__main__":
for i in range(1, 11, 1):
directory = "_{}".format(i)
if not os.path.exists(directory):
os.makedirs(directory)
filename = directory + "/training.log"
logger(filename=filename)
I hope this is helpful.

I'd like to slightly modify #0Nicholas's method. The direction is right, but the first FileHandler will continue log information into the first log file as long as the function is running. Therefore, we would want to pop the handler out of the logger's handlers list:
import logging
targets = ["a", "b", "c"]
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
log_format = "|%(levelname)s| : [%(filename)s]--[%(funcName)s] : %(message)s"
formatter = logging.Formatter(log_format)
for target in targets:
log_file = f"{target}.log"
# create file handler and set the formatter
file_handler = logging.FileHandler(log_file)
file_handler.setFormatter(formatter)
# add handler to the logger
logger.addHandler(file_handler)
# sample message
logger.info(f"Log file: {target}")
# close the log file
file_handler.close()
# remove the handler from the logger. The default behavior is to pop out
# the last added one, which is the file_handler we just added in the
# beginning of this iteration.
logger.handlers.pop()

Here is a working version for this problem. I was only able to get it to work if the targets already have .log before going into the loop so you may want to add one more for before going into targets and override all targets with .log extension
import logging
targets = ["a.log","b.log","c.log"]
for target in targets:
log = logging.getLogger(target)
formatter = logging.Formatter('%(asctime)s - %(levelname)s : %(message)s', datefmt='%Y-%m/%dT%H:%M:%S')
fileHandler = logging.FileHandler(target, mode='a')
fileHandler.setFormatter(formatter)
streamHandler = logging.StreamHandler()
streamHandler.setFormatter(formatter)
log.addHandler(fileHandler)
log.addHandler(streamHandler)
log.info('log target info...')

Confused when using Python logging module

I want to log to 2 different files, but different things.
I was trying this:
runid = str(uuid.uuid1())
logger = logging.getLogger('logger1')
other_logger = logging.getLogger('logger2')
logger.setLevel(logging.DEBUG)
debug_handler = logging.FileHandler('log/debug.log')
info_handler = logging.FileHandler('log/info.log')
other_handler = logging.FileHandler('log/other_info.log')
debug_handler.setLevel(logging.DEBUG)
info_handler.setLevel(logging.INFO)
other_handler.setLevel(logging.INFO)
formatter = logging.Formatter(runid + ' - %(asctime)s - %(levelname)s - %(funcName)s - %(message)s')
debug_handler.setFormatter(formatter)
info_handler.setFormatter(formatter)
other_handler.setFormatter(formatter)
logger.addHandler(debug_handler)
logger.addHandler(info_handler)
other_logger.addHandler(other_handler)
logger.info('message1')
other_logger.info('message2')
But logger and other_logger are working as one and I get both messages in all files, no matter if I call in logger or other_logger.
According doc:
"Loggers have the following attributes and methods. Note that Loggers
are never instantiated directly, but always through the module-level
function logging.getLogger(name). Multiple calls to getLogger() with
the same name will always return a reference to the same Logger
object."
But the parent object is always the same, as in this small test:
import logging
log1 = logging.getLogger('hey')
log2 = logging.getLogger('you')
print log1.parent, log2.parent
enrique#enrique-mbp:$ python /tmp/test.py
<logging.RootLogger object at 0x26f0810> <logging.RootLogger object at 0x26f0810>
How can I solve this?

The problem is here:
logger.setLevel(logging.DEBUG)
You need to set the level for other_logger as well:
logger.setLevel(logging.DEBUG)
other_logger.setLevel(logging.DEBUG) # or INFO because that is the lowest level being used by a handler
Without this, other_logger remains at the default logging level, logging.WARNING, which prevents the other_logger.info('message2') line from logging anything.

Different logging levels for filehandler and display in Python

I am using the logging module in Python to write debug and error messages.
I want to write to file all messages of logging.DEBUG or greater.
However, I only want to print to the screen messages of logging.WARNING or greater.
Is this possible using just one Logger and one FileHandler?

As it has been mentioned, handlers are so easy to create and add that you're probably better off just using two handlers. If, however, for some reason you want to stick to one, the Python logging cookbook has a section describing more or less what you want to do: logging to both console and file, but at different levels (it even shows you how to do different formatting). It does it with a single StreamHandler rather than a FileHandler, though:
import logging
# set up logging to file - see previous section for more details
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M',
filename='/temp/myapp.log',
filemode='w')
# define a Handler which writes INFO messages or higher to the sys.stderr
console = logging.StreamHandler()
console.setLevel(logging.INFO)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
# Now, we can log to the root logger, or any other logger. First the root...
logging.info('Jackdaws love my big sphinx of quartz.')
# Now, define a couple of other loggers which might represent areas in your
# application:
logger1 = logging.getLogger('myapp.area1')
logger2 = logging.getLogger('myapp.area2')
logger1.debug('Quick zephyrs blow, vexing daft Jim.')
logger1.info('How quickly daft jumping zebras vex.')
logger2.warning('Jail zesty vixen who grabbed pay from quack.')
logger2.error('The five boxing wizards jump quickly.')
Edit: As discussed in the comments this code still generates two handlers, but "hides" one construction through the use of basicConfig(). I would strongly encourage you to create both explicitly.

You can append lots of handlers to the same logger with different loglevel, but root loglevel must be as verbose as the most verbose handler. This example log messages to file with log.debug() and log.info() to console:
log = logging.getLogger("syslog2elastic")
log.setLevel(logging.DEBUG) # this must be DEBUG to allow debug messages through
console = logging.StreamHandler()
console.setLevel(logging.INFO)
formatter = logging.Formatter("%(asctime)s.%(msecs)03d - %(name)s:%(lineno)d - %(levelname)s - %(message)s", "%Y%m%d%H%M%S")
console.setFormatter(formatter)
log.addHandler(console)
fh = RotatingFileHandler(args.logfile, maxBytes=104857600, backupCount=5)
fh.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s.%(msecs)03d - %(message)s", "%Y%m%d%H%M%S")
fh.setFormatter(formatter)
log.addHandler(fh)

No. File and Screen output means you need two handlers.

You can do this in the following way, making two loggers, one for file logging, another one for console logging. Make sure to set the root logger to the most verbose of the two.
import logging
logging.getLogger().setLevel(logging.DEBUG) # This must be as verbose as the most verbose handler
formatter = logging.Formatter(
'%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s [%(lineno)s]: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S',
)
console_logging = logging.StreamHandler()
console_logging.setLevel(logging.WARNING)
console_logging.setFormatter(formatter)
logging.getLogger().addHandler(console_logging)
file_logging = logging.FileHandler('file.log')
file_logging.setLevel(logging.DEBUG)
file_logging.setFormatter(formatter)
logging.getLogger().addHandler(file_logging)