Develop Reference

python logger prints everything twice

I try using this openshift-restclient-python library. My custom Logger prints everything twice after I run into this bug.
modules/logging/Logging.py
import logging
class CustomLogger:
logger=None
def __init__(self):
if (CustomLogger.logger==None):
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
stdout_handler = logging.StreamHandler()
stdout_handler.setLevel(logging.DEBUG)
logger.addHandler(stdout_handler)
CustomLogger.logger=logger
def getLogger(self):
return CustomLogger.logger
logger = CustomLogger().getLogger()
This ist my main.py:
#!/usr/bin/env python3
import sys
from modules.logging.Logging import logger
from kubernetes import client, config
from openshift.dynamic import DynamicClient
from openshift.helper.userpassauth import OCPLoginConfiguration
import warnings
warnings.filterwarnings("ignore")
apihost = 'myhost'
username = 'myuser'
password = 'insecure'
ca_cert = '/path/to/cert'
kubeConfig = OCPLoginConfiguration(ocp_username=username, ocp_password=password)
kubeConfig.host = apihost
kubeConfig.verify_ssl = False
kubeConfig.ssl_ca_cert = ca_cert
kubeConfig.get_token()
k8s_client = client.ApiClient(kubeConfig)
logger.warning("this is printed once")
dyn_client = DynamicClient(k8s_client)
logger.warning("this is printed twice")
v1_projects = dyn_client.resources.get(api_version='project.openshift.io/v1', kind='Project')
project_list = v1_projects.get()
sys.exit(0)
executing the main.py I get the following output
this is printed once
ERROR:root:load cache error: ResourceList.__init__() got an unexpected keyword argument 'base_resource_lookup'
this is printed twice
WARNING:modules.logging.Logging:this is printed twice
If I do not use my custom logger but a simple configuration as below in main.py then everything is printed once.
import logging
logging.basicConfig(level=logging.DEBUG)
I have found this answer so I also tried removing any handler but the only handler is the one that contains my customization, so I end up with a basic logger.
What am I doing wrong?
Thanks
EDIT:
There is an easier way reproducing the issue.
I still have my custom logger as posted before but my main.py now:
#!/usr/bin/env python3
import sys
from modules.logging.Logging import logger
import logging
print(logger.handlers)
print("number handlers: " +str(len(logger.handlers)))
logger.warning("this is printed once")
logging.basicConfig(level=logging.DEBUG)
logger.warning("this is printed twice")
print("number handlers: " +str(len(logger.handlers)))
for h in logger.handlers:
logger.removeHandler(h)
print("number handlers: " +str(len(logger.handlers)))
logger.warning("still get printed")
sys.exit(0)
the output:
[<StreamHandler <stderr> (DEBUG)>]
number handlers: 1
this is printed once
this is printed twice
WARNING:modules.logging.Logging:this is printed twice
number handlers: 1
number handlers: 0
WARNING:modules.logging.Logging:still get printed
The code logging.basicConfig(level=logging.DEBUG) doesn't add another handler but cause everything to be logged. I actually only want the customized logs printed by the streamingHandler. How can I revert what is done by logging.basicConfig(level=logging.DEBUG)?

Please try remove this peace of code from class CustomLogger
stdout_handler = logging.StreamHandler()
stdout_handler.setLevel(logging.DEBUG)
logger.addHandler(stdout_handler)
btw i was struggling with the same. Found answers using search on this website.
https://stackoverflow.com/a/26730545/15637940
https://stackoverflow.com/a/70876179/15637940
and a lot more answered questions...

I solved it that way:
class CustomLogger:
logger=None
def __init__(self):
if (CustomLogger.logger==None):
logging.basicConfig(filename='/dev/null', filemode='w', format='%(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
fmt = '%(asctime)s | %(message)s'
stdout_handler = logging.StreamHandler()
stdout_handler.setLevel(logging.DEBUG)
stdout_handler.setFormatter(CustomFormatter(fmt))
logger.addHandler(stdout_handler)
CustomLogger.logger=logger
def getLogger(self):
return CustomLogger.logger
logger = CustomLogger().getLogger()
It seems that the library I am using at some place logs to the RootLogger. According to this answer logging.basicConfig() is a constructor of a streamHandler that connects to the RootLogger.
If I use logger = logging.getLogger('root') instead of logger = logging.getLogger(__name__) then everything is printed once. However, in that case everything the library logs on DEBUG-Level is printed to the terminal.
The line logging.basicConfig(filename='/dev/null', filemode='w', format='%(name)s - %(levelname)s - %(message)s') causes that everything logged by the root logger is printed to /dev/null.

Python logging in several modules outputs twice

I'm not very familiar with python logging and I am trying to get it to log output to the console. I've gotten it to work but it seems that the output is seen twice in the console and I'm not sure why. I've looked at the other questions asked here about similar situations but I could not find anything that helped me.
I have three modules, lets call them ma, mb, mc and a main. The main imports these 3 modules and calls their functions.
In each module I set up a logger like so:
ma.py
import logging
logger = logging.getLogger('test.ma') #test.mb for mb.py/test.mc for mc.py
logger.setLevel(logging.DEBUG)
console_log = logging.StreamHandler()
console_log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
console_log.setFormatter(formatter)
logger.addHandler(console_log)
...
...
#right before the end of each module, after I'm finished logging what I need.
logger.removeHandler(console_log)
mb.py
import logging
logger = logging.getLogger('test.mb')
logger.setLevel(logging.DEBUG)
console_log = logging.StreamHandler()
console_log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
console_log.setFormatter(formatter)
logger.addHandler(console_log)
...
...
#end of file
logger.removeHandler(console_log)
mc.py
import logging
logger = logging.getLogger('test.mc')
logger.setLevel(logging.DEBUG)
console_log = logging.StreamHandler()
console_log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
console_log.setFormatter(formatter)
logger.addHandler(console_log)
...
...
#end of file
logger.removeHandler(console_log)
The main problem I'm having is that output is printing twice and for some parts of the program it is not formatted. Any help would appreciated, thanks!

A better approach to using a global logger is to use a function that wraps the call to logging.getLogger:
import logging
def get_logger(name):
logger = logging.getLogger(name)
if not logger.handlers:
# Prevent logging from propagating to the root logger
logger.propagate = 0
console = logging.StreamHandler()
logger.addHandler(console)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
console.setFormatter(formatter)
return logger
def main():
logger = get_logger(__name__)
logger.setLevel(logging.DEBUG)
logger.info("This is an info message")
logger.error("This is an error message")
logger.debug("This is a debug message")
if __name__ == '__main__':
main()
Output
2017-03-09 12:02:41,083 - __main__ - This is an info message
2017-03-09 12:02:41,083 - __main__ - This is an error message
2017-03-09 12:02:41,083 - __main__ - This is a debug message
Note: Using a function also allows us to set logger.propagate to False to prevent log messages from being sent to the root logger. This is almost certainly the cause of the duplicated output you are seeing.
Update: After calling get_logger, the desired level must be set by calling logger.setLevel. If this is not done, then only error messages will be seen.

To log in multiple modules, to one global file, create a logging instance and then get that instance in whichever modules you like. So for example in main, i would have:
import logging
logging.basicConfig(filename='logfile.log', level=logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
logger = logging.getLogger('Global Log')
Then, in every module that I want to be able to append to this log, I have:
import logging
logger = logging.getLogger('Global Log')
Now everytime you access the logger in your module, you are accessing the same logging instance. This way, you should only have to set up and format the logger once.
More about logging instance in the docs:
https://docs.python.org/2/library/logging.html#logging.getLogger

In my case I was seeing logging twice due to the way /dev/log redirects records to the rsyslog daemon I have locally running.
Using a
handler = logging.handlers.SysLogHandler(address='/run/systemd/journal/syslog')
solved my problem of double logging.

Logger to not show INFO from external libaries

I want logger to print INFO messages from all my code but not from 3rd party libraries. This is discussed in multiple places, but the suggested solution does not work for me. Here is my simulation of external library, extlib.py:
#!/usr/bin/env python3
from logging import info
def f():
info("i am extlib f()")
My module:
#!/usr/bin/env python3
import logging
from logging import info
import extlib
logging.basicConfig(level=logging.INFO)
info("I am mymodule")
extlib.f()
Output:
INFO:root:I am mymodule
INFO:root:i am extlib f()
My attempt to only enable INFO for local module:
#!/usr/bin/env python3
import logging
from logging import info
import extlib
logging.getLogger(__name__).setLevel(logging.INFO)
info("I am mymodule")
extlib.f()
Output: nothing
Desired output:
INFO:root:I am mymodule
What am I doing wrong?

The problem is that they aren't using the logger class in the external library. If they were then you could filter it out. I'm not certain you can stop them from logging information since they are using the info function call. Here is a workaround though.
Suppressing output of module calling outside library
Update:
here's how you do loggers
import logging
# create logger
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# add formatter to ch
ch.setFormatter(formatter)
# add ch to logger
logger.addHandler(ch)
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')

Why does logging seem to require a handler in each module in order to print to console?

In my programs, I generally want to log to both a file and to the screen. If I import a module, I want the calls to logger within that module's functions/classes to also result in a log to a file and the screen. Test_module_A does this, and test_module_B logs to the file but not the screen. Is this because the loggers in the modules propagate up to the root logger which through basicConfig is only setup to log to the file? I just want to make sure I'm using logging properly, as I am going to rewrite all my code to use this type of logging rather than "print." Going forward, I am only going to use logging and not print, as I figure logging is more flexible. Is this a good idea?
main.py
import logging
logger = logging.getLogger(__name__)
task_file='temp.txt'
format_str = '%(asctime)s %(module)s %(levelname)s: %(message)s'
datefmt_str = '%m/%d/%Y %I:%M:%S%p'
logging.basicConfig(filename=task_file, format=format_str, datefmt=datefmt_str,
level=logging.INFO)
console = logging.StreamHandler()
#console.setLevel(logging.INFO)
formatter = logging.Formatter(format_str)
formatter.datefmt = datefmt_str
console.setFormatter(formatter)
logger.addHandler(console)
logger.info("from main.py")
import test_module_A
import test_module_B
test_module_A.py
import logging
logger = logging.getLogger(__name__)
format_str = '%(asctime)s %(module)s %(levelname)s: %(message)s'
datefmt_str = '%m/%d/%Y %I:%M:%S%p'
console = logging.StreamHandler()
formatter = logging.Formatter(format_str)
formatter.datefmt = datefmt_str
console.setFormatter(formatter)
logger.addHandler(console)
logger.info("upon import of test_module_A")
test_module_B.py
import logging
logger = logging.getLogger(__name__)
logger.info("upon import of test_module_B")
Upon running main.py:
#screen output
05/06/2014 12:36:33AM main INFO: from main.py
05/06/2014 12:36:33AM test_module_A INFO: upon import of test_module_A
# test.txt
05/06/2014 12:36:33AM main INFO: from main.py
05/06/2014 12:36:33AM test_module_A INFO: upon import of test_module_A
05/06/2014 12:36:33AM test_module_B INFO: upon import of test_module_B

It's not good practice to configure logging (add handlers, etc.) in each module: the way to do it is to just have
logger = logging.getLogger(__name__)
and actual logging calls in each module as required, and only have the configuration done in one place (called from the main script early on)
if __name__ == '__main__':
logging.basicConfig(level=..., format=..., ...) #or other configuration code
handler = logging.FileHandler(...)
# set formatters, filters etc.
logging.getLogger().addHandler(handler) # add to root logger
main()
The basicConfig() call adds to the root logger a handler which writes to sys.stderr, and together with the FileHandler added explicitly, the logs from all modules should be written to both screen and file.

Suspend the formatting of the logger, then go back to it

I have a logging configuration where I log to a file and to the console:
logging.basicConfig(filename=logfile, filemode='w',
level=numlevel,
format='%(asctime)s - %(levelname)s - %(name)s:%(funcName)s - %(message)s')
# add console messages
console = logging.StreamHandler()
console.setLevel(logging.INFO)
consoleformatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
console.setFormatter(consoleformatter)
logging.getLogger('').addHandler(console)
At some point in my script, i need to interact with the user by printing a summary and asking for confirmation. The summary is currently produced by prints in a loop. I would like to suspend the current format of the console logs, so that I can printout one big block of text with a question at the end and wait for user input. But i still want all of this to be logged to file!
The function that does this is in a module, where I tried the following :
logger = logging.getLogger(__name__)
def summaryfunc:
logger.info('normal logging business')
clearformatter = logging.Formatter('%(message)s')
logger.setFormatter(clearformatter)
logger.info('\n##########################################')
logger.info('Summary starts here')
Which yields the error: AttributeError: 'Logger' object has no attribute 'setFormatter'
I understand that a logger is a logger, not a handler, but i'm not sure on how to get things to work...
EDIT:
Following the answers, my problem turned into : how can i suspend logging to the console when interacting with the user, while still being able to log to file. IE: suspend only the streamHandler. Since this is happening in a module, the specifics of the handlers are defined elsewhere, so here is how i did it :
logger.debug('Normal logging to file and console')
root_logger = logging.getLogger()
stream_handler = root_logger.handlers[1]
root_logger.removeHandler(stream_handler)
print('User interaction')
logger.info('Logging to file only')
root_logger.addHandler(stream_handler)
logger.info('Back to logging to both file and console')
This relies on the streamHandler always being the second in the list returned by handlers but i believe this is the case because it's in the order I added the handlers to the root logger...

I agree with Vinay that you should use print for normal program output and only use logging for logging purpose. However, if you still want to switch formatting in the middle, then switch back, here is how to do it:
import logging
def summarize():
console_handler.setFormatter(logging.Formatter('%(message)s'))
logger.info('Here is my report')
console_handler.setFormatter(console_formatter)
numlevel = logging.DEBUG
logfile = 's2.log'
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
console_handler = logging.StreamHandler()
console_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
console_handler.setFormatter(console_formatter)
logger.addHandler(console_handler)
file_handler = logging.FileHandler(filename=logfile, mode='w')
file_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s:%(funcName)s - %(message)s')
file_handler.setFormatter(file_formatter)
logger.addHandler(file_handler)
logger.info('Before summary')
summarize()
logger.info('After summary')
Discussion
The script creates a logger object and assigned to it two handlers: one for the console and one for file.
In the function summarize(), I switched in a new formatter for the console handler, do some logging, then switched back.
Again, let me remind that you should not use logging to display normal program output.
Update
If you want to suppress console logging, then turn it back on. Here is a suggestion:
def interact():
# Remove the console handler
for handler in logger.handlers:
if not isinstance(handler, logging.FileHandler):
saved_handler = handler
logger.removeHandler(handler)
break
# Interact
logger.info('to file only')
# Add the console handler back
logger.addHandler(saved_handler)
Note that I did not test the handlers against logging.StreamHandler since a logging.FileHandler is derived from logging.StreamHandler. Therefore, I removed those handlers that are not FileHandler. Before removing, I saved that handler for later restoration.
Update 2: .handlers = []
In the main script, if you have:
logger = logging.getLogger(__name__) # __name__ == '__main__'
Then in a module, you do:
logger = logging.getLogger(__name__) # __name__ == module's name, not '__main__'
The problem is, in the script, __name__ == '__main__' and in the module, __name__ == <the module's name> and not '__main__'. In order to achieve consistency, you will need to make up some name and use them in both places:
logger = logging.getLogger('MyScript')

Logging shouldn't be used to provide the actual output of your program - a program is supposed to run the same way if logging is turned off completely. So I would suggest that it's better to do what you were doing before, i.e. prints in a loop.