Develop Reference

How can I set the level on a custom python logger?

I have a custom python logger
# logger.py
import logging
#logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
c_handler = logging.StreamHandler()
c_handler.setLevel(logging.DEBUG)
c_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
c_handler.setFormatter(c_format)
logger.addHandler(c_handler)
I have set the level to DEBUG, but only WARNINGS (and above) are shown
from ..logger import logger
...
logger.debug('this is a debug log message')
logger.warning('too hot to handle')
...
my_module.logger:too hot to handle
if I uncomment the line
logging.basicConfig(level=logging.DEBUG)
then I get the DEBUG level, but two copies of the message
my_module.logger - DEBUG - this is a debug log message
DEBUG:my_module.logger:this is a debug log message
my_module.logger - WARNING - too hot to handle
WARNING:my_module.logger:too hot to handle
I am not importing logging at any other point in the package
How should I configure the logger?

TL;DR Use logger.setLevel(logging.DEBUG)
According to Python documentation, a handler processes messages with a level equal to or higher than the handler is set to (via .setLevel()).
But also note, emphasis mine:
When a logger is created, the level is set to NOTSET (which causes all messages to be processed when the logger is the root logger, or delegation to the parent when the logger is a non-root logger). Note that the root logger is created with level WARNING.
So without logging.basicConfig, there's no "root logger" at program startup, and your first getLogger() creates a stub root logger with default level WARNING, and your logger with level NOTSET (which fallbacks to that of the root logger). As a result your logger.debug message is thrown away before it gets handled.
With logging.basicConfig, you explicitly create a root logger with the given level and a StreamHandler with default Formatter. Your new getLogger() is attached to the root logger and any log record is propagated to the root logger - thus printing twice with a different formatter (the default one indeed).
The stub root logger created by the first call to getLogger() has no handler attached so any propagated record is not printed out.
If you want to have full control over your logging facility, it's better to give your logger an explicit level than relying on basicConfig, which creates a root logger that you may not want:
logger.setLevel(logging.DEBUG)

Having read the docs again I realise that propagate is the attribute that I need to use to turn off the ancestor logging output. So my logger becomes
# logger.py
import logging
logging.basicConfig(level=logging.DEBUG)
logger.propagate = False
logger = logging.getLogger(__name__)
c_handler = logging.StreamHandler()
c_handler.setLevel(logging.DEBUG)
c_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
c_handler.setFormatter(c_format)
logger.addHandler(c_handler)
And I get just one log message and the debug level is used

Unexpected behavior of the Python logging module with info and warning messages

I have a question regarding logging module in Python.
If I instantiate a logger and set its level to INFO without adding any handlers, then I expect that info messages will be printed on the screen.
However, in reality, warning messages are printed but info messages are not.
After adding explicitly a handler, both levels are printed.
Precisely, the following small script:
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.info("This is an info message")
logger.warning("This is a warning message")
print("*** Add a stream handler explicitly")
handler = logging.StreamHandler()
logger.addHandler(handler)
logger.info("This is an info message")
logger.warning("This is a warning message")
gives output
This is a warning message
*** Add a stream handler explicitly
This is an info message
This is a warning message
(checked in Python 3.7.6 and 3.8.2).
I would expect that either no messages are printed without a handler, or both levels are printed after setting level to INFO.

I suggest you use loguru. It's a really simple to use tool and for me it's very intuitive.
You can add a default logger just using this line:
from loguru import logger
If you want to change the logging level, just setup the logger like this:
logger.remove() # This disables the default logger
logger.add(sys.stderr, level="INFO") # This adds a default logger (format and colors included) but at the logging level you need
And that's it. All the scripts that use the loguru logger in your code will be set up like this. By default it outputs to stderr but you can change it however you need.

Python: How to create and use a custom logger in python use logging module?

I am trying to create a custom logger as in the code below. However, no matter what level I pass to the function, logger only prints warning messages. For example even if I set the argument level = logging.DEBUG by default my code fails to log the debug or info messages. Can someone point out the problem here.
import boto3
import logging
def get_logger(name=__name__, level=logging.DEBUG):
# Create log handler
logHandler = logging.StreamHandler()
logHandler.setLevel(level)
# Set handler format
logFormat = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s", datefmt="%d-%b-%y")
logHandler.setFormatter(logFormat)
# Create logger
logger = logging.getLogger(name)
# Add handler to logger
logger.addHandler(logHandler)
# Stop propagating the log messages to root logger
# logger.propagate = False
return logger
def listBuckets():
logThis = get_logger(level=logging.DEBUG)
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
logThis.debug(msg='This message is from logger')
print(bucket.name)
listBuckets()

You are missing the fact that a) every logger's ultimate ancestor is the root logger (which has level WARNING by default) and b) that both, loggers and handlers have levels.
The docs state:
When a logger is created, the level is set to NOTSET (which causes all
messages to be processed when the logger is the root logger, or
delegation to the parent when the logger is a non-root logger).
So, you create a logger and a StreamHandler with their default level NOTSET. Your logger is an implicit descendant of the root logger. You set the handler to level DEBUG, but not the logger using that handler.
Since the level on your logger still is NOTSET, when a log event occurs, its chain of ancestors is traversed ...
... until either an ancestor with a level other than NOTSET is found, or
the root is reached.
[...]
If the root is reached, and it has a level of NOTSET, then all
messages will be processed. Otherwise, the root’s level will be used
as the effective level.
Which means, you immediately end up at the root logger to determine the effective log level; it is set to WARNING as per the root logger's default.
You can check this with the parent and level properties and the getEffectiveLevel method on the logger object:
logThis = get_logger()
print(logThis.parent) # <RootLogger root (WARNING)>
print(logThis.level) # 0 (= NOTSET)
print(logThis.getEffectiveLevel()) # 30 (= WARNING) from root logger
To have your logger handle the messages on and above the desired level itself, simply set it on the logger via logger.setLevel(level) in your get_logger function.

A couple of points to know (Read these logging docs for details)
A parent's / ancestor's log level takes precedence while evaluating the effective log level
A root logger, for example that created with getLogger, has WARNING as it default log level
You have set the log level of the handler (logHandler) but not the root (logger). At this point, no handler can have a log level less than the root's, ie, WARNING
logHandler.setLevel(level)
logger.addHandler(logHandler)
logThis.debug(msg='This message is from logger') # Does not log
logThis.warn(msg='This message is from logger') # Logs
So, set the root level to something reasonable and you should be good to go
logHandler.setLevel('WARNING') # or NOTSET
logThis.debug(msg='This message is from logger') # Logs!

Logging setLevel is being ignored

The below code is copied from the documentation. I am supposed to be able to see all the info logs. But I don't. I am only able to see the warn and above even though I've set setLevel to INFO.
Why is this happening? foo.py:
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
Output:
workingDirectory$ python foo.py
warn message
error message
critical message
Where did the info and debug messages go??

Replace the line
logger.setLevel(logging.DEBUG)
with
logging.basicConfig(level=logging.DEBUG, format='%(message)s')
and it should work as expected. If you don't configure logging with any handlers (as in your post - you only configure a level for your logger, but no handlers anywhere), you'll get an internal handler "of last resort" which is set to output just the message (with no other formatting) at the WARNING level.

Try running logging.basicConfig() in there. Of note, I see you mention INFO, but use DEBUG. As written, it should show all five messages. Swap out DEBUG with INFO, and you should see four messages.
import logging
logging.basicConfig()
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
edit: Do you have logging set up elsewhere in your code already? Can't reproduce the exact behavior you note with the specific code provided.

As pointed by some users, using:
logging.basicConfig(level=logging.DEBUG, format='%(message)s')
like written in the accepted answer is not a good option because it sets the log level for the root logger, so it may lead to unexpected behaviours (eg. third party libraries may start to log debug messages if you set loglevel=logging.DEBUG)
In my opinion the best solution is to set log level just for your logger, like this:
import logging
logger = logging.getLogger('MyLogger')
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
Not really intuitive solution, but is necessary if you want to set log level only for 'MyLogger' and leave the root logger untouched.
So, why is logging.basicConfig(level=logging.DEBUG, format='%(message)s') setting the log level globally?
Well, actually it doesn't. As said, it's just changing the configuration of the root logger and, as described in the python documentation:
Loggers should NEVER be instantiated directly, but always through the
module-level function logging.getLogger(name). Multiple calls to
getLogger() with the same name will always return a reference to the
same Logger object.
So, logging.basicConfig is creating a StreamHandler with a default Formatter and adding it to the root logger.
The point is that if any other library is using the "root logger", you're going to set that log level for that library too so it can happen that you start to see debug logs from third party libraries.
This is why I think it's better to create your own logger and set your own formatters and handlers, so you can leave the root logger untouched.

This is technically also an "answer", because it can "solve" the problem. BUT I definitely DO NOT like it. It is not intuitive, and I lost 2+ hours over it.
Before:
import logging
logger = logging.getLogger('foo')
logger.setLevel(logging.INFO)
logger.info('You can not see me')
# Or you can just use the following one-liner in command line.
# $ python -c "import logging; logger = logging.getLogger('foo'); logger.setLevel(logging.INFO); logger.info('You can not see me')"
After:
import logging
logging.debug('invisible magic') # <-- magic
logger = logging.getLogger('foo')
logger.setLevel(logging.INFO)
logger.info('But now you can see me')
# Or you can just use the following one-liner in command line.
$ python -c "import logging; logging.debug('invisible magic'); logger = logging.getLogger('foo'); logger.setLevel(logging.INFO); logger.info('But now you see me')"
PS: Comparing it to the current chosen answer, and #Vinay-Sajip's explanation, I can kind of understand why. But still, I wish it was not working that way.

If you want this to work WITHOUT basicConfig, you have to first set up the lowest possible level you'll log onto the logger. Since the logger sets a minimum threshold, handlers which have a lower threshold but belong to the same logger won't get those lower threshold messages since they're ignored by the logger in the first place. Intuitive, but not obvious.
We start by doing this:
lgr = logging.getLogger(name)
lgr.setLevel(logging.DEBUG)
Then, set up the handlers with the different levels you need, in my case I want DEBUG logging on stdout and INFO logging to a rotating file, so I do the following:
rot_hndlr = RotatingFileHandler('filename.log',
maxBytes=log_size,
backupCount=3)
rot_hndlr.setFormatter(formatter)
rot_hndlr.setLevel(logging.INFO)
lgr.addHandler(rot_hndlr)
stream_hndlr = logging.StreamHandler()
stream_hndlr.setFormatter(stream_formatter)
lgr.addHandler(stream_hndlr)
Then, to test, I do this:
lgr.debug("Hello")
lgr.info("There")
My stdout (console) will look like this:
Hello
There
and my filename.log file will look like this:
There

In short, change the level in logging.basicConfig will influence the global settings.
You should better set level for each logger and the specific handler in the logger.
The following is an example that displays all levels on the console and only records messages >= errors in log_file.log. Notice the level for each handler is different.
import logging
# Define logger
logger = logging.getLogger('test')
# Set level for logger
logger.setLevel(logging.DEBUG)
# Define the handler and formatter for console logging
consoleHandler = logging.StreamHandler() # Define StreamHandler
consoleHandler.setLevel(logging.DEBUG) # Set level
concolsFormatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s') # Define formatter
consoleHandler.setFormatter(concolsFormatter) # Set formatter
logger.addHandler(consoleHandler) # Add handler to logger
# Define the handler and formatter for file logging
log_file = 'log_file'
fileHandler = logging.FileHandler(f'{log_file}.log') # Define FileHandler
fileHandler.setLevel(logging.ERROR) # Set level
fileFormatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') # Define formatter
fileHandler.setFormatter(fileFormatter) # Set formatter
logger.addHandler(fileHandler) # Add handler to logger
# Test
logger.debug('This is a debug')
logger.info('This is an info')
logger.warning('This is a warning')
logger.error('This is an error')
logger.critical('This is a critical')
Console output:
# Test
test - DEBUG - This is a debug
test - INFO - This is an info
test - WARNING - This is a warning
test - ERROR - This is an error
test - CRITICAL - This is a critical
File log_file.log content:
2021-09-22 12:50:50,938 - test - ERROR - This is an error
2021-09-22 12:50:50,938 - test - CRITICAL - This is a critical
To review your logger's level:
logger.level
The result should be one of the following:
10 # DEBUG
20 # INFO
30 # WARNING
40 # ERROR
50 # CRITICAL
To review your handlers's levels:
logger.handlers
[<StreamHandler stderr (DEBUG)>,
<FileHandler ***/log_file.log (ERROR)>]

The accepted answer does not work for me on Win10, Python 3.7.2.
My solution:
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
It's order sensitive.

You have to set the basicConfig of the root logger to DEBUG, then you can set the level of your individual loggers to more restrictive levels.
This is not what I expected. Here is what I had to do:
#!/usr/bin/env python3
import logging
# by default this is WARNING. Leaving it as WARNING here overrides
# whatever setLevel-ing you do later so it seems they are ignored.
logging.basicConfig(level=logging.DEBUG)
l = logging.getLogger(__name__)
l.setLevel(level=logging.INFO)
# if I hadn't called basicConfig with DEBUG level earlier,
# info messages would STILL not be shown despite calling
# setLevel above. However now debug messages will not be shown
# for l because setLevel set it to INFO
l.warning('A warning message will be displayed')
l.info('A friendly info message will be displayed')
l.debug('A friendly debug message will not be displayed')

Most of the answers that I've found for this issue uses the basicConfig of the root logger.
It's not helpful for those who intend to use multiple independent loggers that were not initialised with basicConfig. The use of basicConfig implies that the loglevels of ALL loggers will be changed. It also had the unfortunate side effect of generating duplicate logs.
So I tried over several days experimenting with different ways to manipulate the loglevels and came up with one that finally worked.
The trick was to not only change the log levels of all the handlers but also the all the handlers of the parent of the logger.
def setLevel(self, infoLevel):
# To dynamically reset the loglevel, you need to also change the parent levels as well as all handlers!
self.logger.parent.setLevel(infoLevel)
for handler in self.logger.parent.handlers:
handler.setLevel(infoLevel)
self.logger.setLevel(infoLevel)
for handler in self.logger.handlers:
handler.setLevel(infoLevel)
The inspiration came from the fact that the basicConfig changes the root logger settings, so I was trying to do the same without using basicConfig.
For those that are interested, I did a little Python project on Github that illustrates the different issues with setting loglevel of the logger (it works partially), proves the SLogger (Sample Logger) implementation works, and also illustrates the duplicate log issue with basicConfig when using multiple loggers not initialised with it.
https://github.com/FrancisChung/python-logging-playground
TLDR: If you're only interested in a working sample code for the logger, the implentation is listed below
import logging
CRITICAL = 50
FATAL = CRITICAL
ERROR = 40
WARNING = 30
WARN = WARNING
INFO = 20
DEBUG = 10
NOTSET = 0
class SLogger():
"""
SLogger : Sample Logger class using the standard Python logging Library
Parameters:
name : Name of the Logger
infoLevel : logging level of the Logger (e.g. logging.DEBUG/INFO/WARNING/ERROR)
"""
def __init__(self, name: str, infoLevel=logging.INFO):
try:
if name is None:
raise ValueError("Name argument not specified")
logformat = '%(asctime)s %(levelname)s [%(name)s %(funcName)s] %(message)s'
self.logformat = logformat
self.name = name.upper()
self.logger = logging.getLogger(self.name)
self.logger.setLevel(infoLevel)
self.add_consolehandler(infoLevel, logformat)
except Exception as e:
if self.logger:
self.logger.error(str(e))
def error(self, message):
self.logger.error(message)
def info(self, message):
self.logger.info(message)
def warning(self, message):
self.logger.warning(message)
def debug(self, message):
self.logger.debug(message)
def critical(self, message):
self.logger.critical(message)
def setLevel(self, infoLevel):
# To dynamically reset the loglevel, you need to also change the parent levels as well as all handlers!
self.logger.parent.setLevel(infoLevel)
for handler in self.logger.parent.handlers:
handler.setLevel(infoLevel)
self.logger.setLevel(infoLevel)
for handler in self.logger.handlers:
handler.setLevel(infoLevel)
return self.logger.level
def add_consolehandler(self, infoLevel=logging.INFO,
logformat='%(asctime)s %(levelname)s [%(name)s %(funcName)s] %(message)s'):
sh = logging.StreamHandler()
sh.setLevel(infoLevel)
formatter = logging.Formatter(logformat)
sh.setFormatter(formatter)
self.logger.addHandler(sh)

Create object the right way, e.g. inspired by Google:
import logging
formatter = logging.Formatter('%(asctime)s %(threadName)s: %(message)s')
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
handler.setFormatter(formatter)
log.addHandler(handler)
log.debug('debug message')
log.info('info message')
log.warn('warn message')
log.error('error message')
log.critical('critical message')
2022-11-22 23:17:59,342 MainThread: debug message
2022-11-22 23:17:59,342 MainThread: info message
2022-11-22 23:17:59,342 MainThread: warn message
2022-11-22 23:17:59,342 MainThread: error message
2022-11-22 23:17:59,342 MainThread: critical message

As pointed out by #ManuelFedele, logging.basicConfig is not a good solution.
#VinaySajip explained that the setLevel is ignored because the logger is using the internal handler "of last resort", whose level is set to WARNING.
This explanation was also helpful:
The Handler.setLevel() method, just as in logger objects, specifies the lowest severity that will be dispatched to the appropriate destination. Why are there two setLevel() methods? The level set in the logger determines which severity of messages it will pass to its handlers. The level set in each handler determines which messages that handler will send on.
So a good solution is to add a handler to the logger, with the appropriate level:
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG) # or whatever level should be displayed on the console
logger.addHandler(ch)
Output
>>> logger.debug('debug message')
debug message
>>> logger.info('info message')
info message
>>> logger.warn('warn message')
<stdin>:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
warn message
>>> logger.error('error message')
error message
>>> logger.critical('critical message')
critical message

logging.info doesn't show up on console but warn and error do

When I log an event with logging.info, it doesn't appear in the Python terminal.
import logging
logging.info('I am info') # no output
In contrast, events logged with logging.warn do appear in the terminal.
import logging
logging.warn('I am warning') # outputs "I am warning"
Is there a environment level change I can to make logging.info print to the console? I want to avoid making changes in each Python file.

The root logger always defaults to WARNING level. Try calling
logging.getLogger().setLevel(logging.INFO)
and you should be fine.

Like #ztyx said that default logger level is WARNING. You have to set it to a lower level
You can do it by using logging.basicConfig and setting logger level:
logging.basicConfig(level=logging.DEBUG)

The above solutions didn't work for me, but the code here did:
# set up logging to file
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M',
filename='/temp/myapp.log',
filemode='w')
# define a Handler which writes INFO messages or higher to the sys.stderr
console = logging.StreamHandler()
console.setLevel(logging.INFO)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
(I omitted parts of the code for the sake of readability)

This will work
import logging
logging.basicConfig()
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.info('its working')

In more recent versions of Python 3 (tested with Python 3.8), console logging requires creating a console handler to correctly show info messages.
The following example is modified from the Configuring Logging example in the Python documentation:
import logging
# create logger
logger = logging.getLogger('__name__')
level = logging.INFO
logger.setLevel(level)
# ----> console info messages require these lines <----
# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(level)
# add ch to logger
logger.addHandler(ch)
# -----------------------------------------------------
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')
Running the above code generates the following output:
info message
warn message
error message
critical message
Here is this same code without the console handler.
import logging
# create logger
logger = logging.getLogger('__name__')
level = logging.INFO
logger.setLevel(level)
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')
Without the console handler, the output does not include the info message:
warn message
error message
critical message
I do not understand why this is the case as it seems unnecessary.

What's the minimum required code for a working module-level logger? I did an experiment (with python version 3.8.6).
The take-away is:
logging.basicConfig() is needed (however, specifying level=... is NOT needed)
it's necessary to configure the root logger: logging.getLogger().setLevel(...)
So, a minimum working example is:
The library/module code does NOT need to configure the logger:
# library/module code: lib.py
import logging
LOGGER = logging.getLogger('x.y.z')
def some_function():
LOGGER.info("hi")
The application code need to configure the logger with 2 lines at minimum:
# Application Code
import logging, lib
logging.basicConfig()
logging.getLogger().setLevel(logging.INFO) # configure root logger
main() # code that will trigger lib
Here's the experiment:
In [1]: import logging
In [2]: lg = logging.getLogger('x.y.z')
In [3]: lg.info(1)
In [4]: logging.basicConfig()
In [5]: lg.info(1)
In [6]: logging.basicConfig(level=logging.INFO)
In [7]: lg.info(1)
In [8]: logging.basicConfig()
In [9]: logging.getLogger().setLevel(logging.INFO)
In [10]: lg.info(1)
INFO:x.y.z:1

For those using absl.logging, the equivalent command is
from absl import logging
logging.set_verbosity(logging.INFO)

If you are using Django to power your server, you just simply need to change the log level in your settings.py file as such:
"handlers": {
"console": {
-- "level": "WARNING",
++ "level": "INFO",
"class": "logging.StreamHandler",
"formatter": "stackdriver",
}
},
More examples in the documentation here:
https://docs.djangoproject.com/en/4.0/topics/logging/#configuring-logging-1

logging.info() will use the root logger for logging.
According to official doc,
if you do not set an explicit handler for the logger, a special handler called lastResort will be used.
See the code here. By default the logging level of lastResort (it is stream handler) is 30.
we can change its level to output info message.
# setting both the logger and handler's level will work as expected.
logger.setLevel(logging.DEBUG)
logging.lastResort.setLevel(logging.DEBUG)
However, this is like a hack and never a encouraged action.
Using logging.basicConfig()
If we want to do logging real quick, we can use the method logging.basicConfig.
logging.basicConfig(level=logging.DEBUG)
This will create the logger. The logger.level will be the level we set here.
A stream handler will be also created, with level NOTSET.
Without a level param, the default level for root logger is WARNING.
reference
logger logging without handler: logger logging without a handler