Develop Reference

How to log different classes into different files?

Let's say I have different classes defined like this
class PS1:
def __init__(self):
pass
def do_sth(self):
logging.info('PS1 do sth')
class PS2:
def __init__(self):
pass
def do_sth(self):
logging.info('PS2 do sth')
How can I make PS1 log into PS1.log and PS2 log into PS2.log? I know I can set up loggers and file handlers and do something likes ps1_logger.info instead of logging.info. However, my current code base already has hundreds of logging.info, do I have to change them all?

When you call logging methods on the module level you are creating logs directly at the root logger, so you no longer can separate them by logger. The only chance left is to separate your logs by Handler with a filter. Since you don't want to update your logging calls to add any information about where the call happens you have to inspect the stack to find the calling class. The code below does this, but I highly recommend not using this solution in production and refactor your code to not send everything to the root logger. Look into logging adapters and/or the extra keyword argument for an alternative solution that modifies the logging calls.
import logging
import inspect
root = logging.getLogger()
ps1_handler = logging.FileHandler('ps1.log')
ps2_handler = logging.FileHandler('ps2.log')
def make_class_filter(class_name):
def filter(record):
for fi in inspect.stack():
if 'self' in fi[0].f_locals and class_name in str(fi[0].f_locals['self']):
return True
return False
return filter
ps1_handler.addFilter(make_class_filter('PS1'))
ps2_handler.addFilter(make_class_filter('PS2'))
root.addHandler(ps1_handler)
root.addHandler(ps2_handler)
class PS1:
def do_sth(self):
logging.warning('PS1 do sth')
class PS2:
def do_sth(self):
logging.warning('PS2 do sth')
PS1().do_sth()
PS2().do_sth()

Change log-level via mocking

I want to change the log-level temporarily.
My current strategy is to use mocking.
with mock.patch(...):
my_method_which_does_log()
All logging.info() calls inside the method should get ignored and not logged to the console.
How to implement the ... to make logs of level INFO get ignored?
The code is single-process and single-thread and executed during testing only.

I want to change the log-level temporarily.
A way to do this without mocking is logging.disable
class TestSomething(unittest.TestCase):
def setUp(self):
logging.disable(logging.WARNING)
def tearDown(self):
logging.disable(logging.NOTSET)
This example would only show messages of level WARNING and above for each test in the TestSomething class. (You call disable at the start and end of each test as needed. This seems a bit cleaner.)
To unset this temporary throttling, call logging.disable(logging.NOTSET):
If logging.disable(logging.NOTSET) is called, it effectively removes this overriding level, so that logging output again depends on the effective levels of individual loggers.

I don't think mocking is going to do what you want. The loggers are presumably already instantiated in this scenario, and level is an instance variable for each of the loggers (and also any of the handlers that each logger has).
You can create a custom context manager. That would look something like this:
Context Manager
import logging
class override_logging_level():
"A context manager for temporarily setting the logging level"
def __init__(self, level, process_handlers=True):
self.saved_level = {}
self.level = level
self.process_handlers = process_handlers
def __enter__(self):
# Save the root logger
self.save_logger('', logging.getLogger())
# Iterate over the other loggers
for name, logger in logging.Logger.manager.loggerDict.items():
self.save_logger(name, logger)
def __exit__(self, exception_type, exception_value, traceback):
# Restore the root logger
self.restore_logger('', logging.getLogger())
# Iterate over the loggers
for name, logger in logging.Logger.manager.loggerDict.items():
self.restore_logger(name, logger)
def save_logger(self, name, logger):
# Save off the level
self.saved_level[name] = logger.level
# Override the level
logger.setLevel(self.level)
if not self.process_handlers:
return
# Iterate over the handlers for this logger
for handler in logger.handlers:
# No reliable name. Just use the id of the object
self.saved_level[id(handler)] = handler.level
def restore_logger(self, name, logger):
# It's possible that some intervening code added one or more loggers...
if name not in self.saved_level:
return
# Restore the level for the logger
logger.setLevel(self.saved_level[name])
if not self.process_handlers:
return
# Iterate over the handlers for this logger
for handler in logger.handlers:
# Reconstruct the key for this handler
key = id(handler)
# Again, we could have possibly added more handlers
if key not in self.saved_level:
continue
# Restore the level for the handler
handler.setLevel(self.saved_level[key])
Test Code
# Setup for basic logging
logging.basicConfig(level=logging.ERROR)
# Create some loggers - the root logger and a couple others
lr = logging.getLogger()
l1 = logging.getLogger('L1')
l2 = logging.getLogger('L2')
# Won't see this message due to the level
lr.info("lr - msg 1")
l1.info("l1 - msg 1")
l2.info("l2 - msg 1")
# Temporarily override the level
with override_logging_level(logging.INFO):
# Will see
lr.info("lr - msg 2")
l1.info("l1 - msg 2")
l2.info("l2 - msg 2")
# Won't see, again...
lr.info("lr - msg 3")
l1.info("l1 - msg 3")
l2.info("l2 - msg 3")
Results
$ python ./main.py
INFO:root:lr - msg 2
INFO:L1:l1 - msg 2
INFO:L2:l2 - msg 2
Notes
The code would need to be enhanced to support multithreading; for example, logging.Logger.manager.loggerDict is a shared variable that's guarded by locks in the logging code.

Using #cryptoplex's approach of using Context Managers, here's the official version from the logging cookbook:
import logging
import sys
class LoggingContext(object):
def __init__(self, logger, level=None, handler=None, close=True):
self.logger = logger
self.level = level
self.handler = handler
self.close = close
def __enter__(self):
if self.level is not None:
self.old_level = self.logger.level
self.logger.setLevel(self.level)
if self.handler:
self.logger.addHandler(self.handler)
def __exit__(self, et, ev, tb):
if self.level is not None:
self.logger.setLevel(self.old_level)
if self.handler:
self.logger.removeHandler(self.handler)
if self.handler and self.close:
self.handler.close()
# implicit return of None => don't swallow exceptions

You could use dependency injection to pass the logger instance to the method you are testing. It is a bit more invasive though since you are changing your method a little, however it gives you more flexibility.
Add the logger parameter to your method signature, something along the lines of:
def my_method( your_other_params, logger):
pass
In your unit test file:
if __name__ == "__main__":
# define the logger you want to use:
logging.basicConfig( stream=sys.stderr )
logging.getLogger( "MyTests.test_my_method" ).setLevel( logging.DEBUG )
...
def test_my_method(self):
test_logger = logging.getLogger( "MyTests.test_my_method" )
# pass your logger to your method
my_method(your_normal_parameters, test_logger)
python logger docs: https://docs.python.org/3/library/logging.html

I use this pattern to write all logs to a list. It ignores logs of level INFO and smaller.
logs=[]
import logging
def my_log(logger_self, level, *args, **kwargs):
if level>logging.INFO:
logs.append((args, kwargs))
with mock.patch('logging.Logger._log', my_log):
my_method_which_does_log()

Python logging handler to append to list

Somewhat similar to this (unanswered) question here:
https://stackoverflow.com/questions/33136398/python-logging-handler-that-appends-messages-to-a-list
I am looking to use a handler to simply append anything (passing a filter) to a python list. I would envisage some kind of code like this:
import logging
import sys
mylog = logging.getLogger()
mylog.setLevel(logging.DEBUG)
log_list = []
lh = logging.SomeHandler(log_list)
lh.setLevel(logging.DEBUG)
mylog.addHandler(lh)
mylog.warning('argh')
print log_list[0]
The question is therefore - how can I implement this? What SomeHandler can I use?

Here is a naive, non thread-safe implementation:
import logging
class ListHandler(logging.Handler): # Inherit from logging.Handler
def __init__(self, log_list):
# run the regular Handler __init__
logging.Handler.__init__(self)
# Our custom argument
self.log_list = log_list
def emit(self, record):
# record.message is the log message
self.log_list.append(record.msg)

#imriqwe's answer is correct for a non thread-safe implementation, but if you need to be thread-safe, one solution is to use a queue.Queue() instead of a list. Here is some code I am using in an in-process project to generate a tkinter log window.
import logging
import queue
class QueuingHandler(logging.Handler):
"""A thread safe logging.Handler that writes messages into a queue object.
Designed to work with LoggingWidget so log messages from multiple
threads can be shown together in a single ttk.Frame.
The standard logging.QueueHandler/logging.QueueListener can not be used
for this because the QueueListener runs in a private thread, not the
main thread.
Warning: If multiple threads are writing into this Handler, all threads
must be joined before calling logging.shutdown() or any other log
destinations will be corrupted.
"""
def __init__(self, *args, message_queue, **kwargs):
"""Initialize by copying the queue and sending everything else to superclass."""
logging.Handler.__init__(self, *args, **kwargs)
self.message_queue = message_queue
def emit(self, record):
"""Add the formatted log message (sans newlines) to the queue."""
self.message_queue.put(self.format(record).rstrip('\n'))
To use, create a queue, create the handler using the queue, then add it to the logger (this example also creates a log file in the current directory):
LOG_FORMAT = '%(asctime)s: %(name)8s: %(levelname)8s: %(message)s'
# Setup root logger to write to a log file.
logging.basicConfig(filename='gui-test.log',
filemode='w',
format=LOG_FORMAT,
level=logging.DEBUG
)
# Get a child logger
logger = logging.getLogger(name='gui')
# Build our QueuingHandler
message_queue = queue.Queue()
handler = QueuingHandler(message_queue=message_queue, level=logging.DEBUG)
# Change the date/time format for the GUI to drop the date
formatter = logging.Formatter(LOG_FORMAT)
formatter.default_time_format = '%H:%M:%S'
handler.setFormatter(formatter)
# Add our QueuingHandler into the logging heirarchy at the lower level
logger.addHandler(handler)
Now all you have to do is read your messages from the queue.

Create a Log Box with tkinter text Widget

I want to create a box were the user is informed of what the application is actually doing.
I created a Text Widget were to show the print statements that I wrote in key points of the applications, so that it could serve as a log box.
To do this, I redirected the stdout to a subclass of the widget itself "upgraded" with a write method as I saw here in another post.
This does indeed work, but I noticed a problem that makes the box almost useless.
If you run the code, you can see that the sentences appear all at once. More puzzling for me is that not only the sentences of the "wait2" functions
appear togheter, but even the print statement of the calling function, "wait1", is shown at the end of the process.
Why this behaviour? what can I do to see the statement shown in the box as they are executed?
from Tkinter import *
import sys
import time
root = Tk()
class addwritemethod(object):
def __init__(self, widget):
self.widget = widget
def write(self, string):
self.widget.configure(state="normal")
self.widget.insert("end", string)
self.widget.see("end")
self.widget.configure(state="disabled")
def wait1():
print "Can you see me?"
wait2()
def wait2():
for i in range(10):
time.sleep(5)
print "Long time no see!"
t_go = Button(root, text= "Start!", width =12, command = wait1)
t_go.pack(side = LEFT, padx = 20, pady = 20)
tlog = Text(root,wrap = "word")
tlog.pack(side="top", fill="both", expand=True)
tlog.configure(state="disabled")
sys.stdout = addwritemethod(tlog)
mainloop()
EDIT: I want to say thanks to the people who answered me and an apology: I did not give all the required information.
I put time.sleep() in the test code only to show you the behaviour. In the real application, I trasfer a file via ssh with Paramiko and I don't use sleep().
Maybe I choose the wrong example, but the result is the same, all the print stament are shown at the same moment.

You could also use the built-in logging module to achieve your goal of
creating a box where the user is informed of what the application is actually doing ... a log box.
I had this same need and converged on the recommendations provided here and here.
I have an example below that I created to illustrate the concept of logging to a GUI control using Tkinter. The example below logs to a text control as you ask, but you can send log messages to other GUI components by replacing/copying the class MyHandlerText with other handler classes like MyHandlerLabel, MyHandlerListbox, etc. (choose your own names for the handler classes). Then you'd have a handler for a variety of GUI controls of interest. The big "a-ha" moment for me was the module-level getLogger concept encouraged by python.org.
import Tkinter
import logging
import datetime
# this item "module_logger" is visible only in this module,
# (but you can create references to the same logger object from other modules
# by calling getLogger with an argument equal to the name of this module)
# this way, you can share or isolate loggers as desired across modules and across threads
# ...so it is module-level logging and it takes the name of this module (by using __name__)
# recommended per https://docs.python.org/2/library/logging.html
module_logger = logging.getLogger(__name__)
class simpleapp_tk(Tkinter.Tk):
def __init__(self,parent):
Tkinter.Tk.__init__(self,parent)
self.parent = parent
self.grid()
self.mybutton = Tkinter.Button(self, text="ClickMe")
self.mybutton.grid(column=0,row=0,sticky='EW')
self.mybutton.bind("<ButtonRelease-1>", self.button_callback)
self.mytext = Tkinter.Text(self, state="disabled")
self.mytext.grid(column=0, row=1)
def button_callback(self, event):
now = datetime.datetime.now()
module_logger.info(now)
class MyHandlerText(logging.StreamHandler):
def __init__(self, textctrl):
logging.StreamHandler.__init__(self) # initialize parent
self.textctrl = textctrl
def emit(self, record):
msg = self.format(record)
self.textctrl.config(state="normal")
self.textctrl.insert("end", msg + "\n")
self.flush()
self.textctrl.config(state="disabled")
if __name__ == "__main__":
# create Tk object instance
app = simpleapp_tk(None)
app.title('my application')
# setup logging handlers using the Tk instance created above
# the pattern below can be used in other threads...
# ...to allow other thread to send msgs to the gui
# in this example, we set up two handlers just for demonstration (you could add a fileHandler, etc)
stderrHandler = logging.StreamHandler() # no arguments => stderr
module_logger.addHandler(stderrHandler)
guiHandler = MyHandlerText(app.mytext)
module_logger.addHandler(guiHandler)
module_logger.setLevel(logging.INFO)
module_logger.info("from main")
# start Tk
app.mainloop()

When you call sleep, the application does exactly that: it sleeps. When it's sleeping it can't update the display. As a general rule you should never call sleep in a GUI.
That being said, a quick fix is to make sure you call update after printing something to the log, so that Tkinter has a chance to update the screen. Add self.widget.update_idletasks() at the end of write (redrawing the screen is considered an "idle task").
This isn't a proper fix but it's good enough to illustrate why the data isn't appearing. A proper fix involves not calling sleep. There are many examples on stackoverflow related to this, and almost all of them involve using the after method.

Python logging in multiple modules

I want to write a logger which I can use in multiple modules. I must be able to enable and disable it from one place. And it must be reusable.
Following is the scenario.
switch_module.py
class Brocade(object):
def __init__(self, ip, username, password):
...
def connect(self):
...
def disconnect(self):
...
def switch_show(self):
...
switch_module_library.py
import switch_module
class Keyword_Mapper(object):
def __init__(self, keyword_to_execute):
self._brocade_object = switch_module.Brocade(ip, username, password)
...
def map_keyword_to_command(self)
...
application_gui.py
class GUI:
# I can open a file containing keyword for brocade switch
# in this GUI in a tab and tree widget(it uses PyQt which I don't know)
# Each tab is a QThread and there could be multiple tabs
# Each tab is accompanied by an execute button.
# On pressing exeute it will read the string/keywords from the file
# and create an object of keyword_Mapper class and call
# map_key_word_to_command method, execute the command on brocade
# switch and log the results. Current I am logging the result
# only from the Keyword_Mapper class.
Problem I have is how to write a logger which could be enabled and disabled at will
and it must log to one file as well as console from all three modules.
I tried writing global logger in int.py and then importing it in all three modules
and had to give a common name so that they log to the same file, but then
ran into trouble since there is multi-threading and later created logger to
log to a file which has thread-id in its name so that I can have each log
per thread.
What if I am required to log to different file rather than the same file?
I have gone through python logging documentation but am unable to get a clear picture
about writing a proper logging system which could be reused.
I have gone through this link
Is it better to use root logger or named logger in Python
but due to the GUI created by someone other than me using PyQt and multi-threading I am unable to get my head around logging here.

I my project I only use a root logger (I dont have the time to create named loggers, even if it would be nice). So if you don't want to use a named logger, here is a quick solution:
I created a function to set up logger quickly:
import logging
def initLogger(level=logging.DEBUG):
if level == logging.DEBUG:
# Display more stuff when in a debug mode
logging.basicConfig(
format='%(levelname)s-%(module)s:%(lineno)d-%(funcName)s: %(message)s',
level=level)
else:
# Display less stuff for info mode
logging.basicConfig(format='%(levelname)s: %(message)s', level=level)
I created a package for it so that I can import it anywhere.
Then, in my top level I have:
import LoggingTools
if __name__ == '__main__':
# Configure logger
LoggingTools.initLogger(logging.DEBUG)
#LoggingTools.initLogger(logging.INFO)
Depending if I am debugging or not, I using the corresponding statement.
Then in each other files, I just use the logging:
import logging
class MyClass():
def __init__(self):
logging.debug("Debug message")
logging.info("Info message")