Right now I have a central module in a framework that spawns multiple processes using the Python 2.6 multiprocessing module. Because it uses multiprocessing, there is module-level multiprocessing-aware log, LOG = multiprocessing.get_logger(). Per the docs, this logger (EDIT) does not have process-shared locks so that you don't garble things up in sys.stderr (or whatever filehandle) by having multiple processes writing to it simultaneously.
The issue I have now is that the other modules in the framework are not multiprocessing-aware. The way I see it, I need to make all dependencies on this central module use multiprocessing-aware logging. That's annoying within the framework, let alone for all clients of the framework. Are there alternatives I'm not thinking of?
I just now wrote a log handler of my own that just feeds everything to the parent process via a pipe. I've only been testing it for ten minutes but it seems to work pretty well.
(Note: This is hardcoded to RotatingFileHandler, which is my own use case.)
Update: #javier now maintains this approach as a package available on Pypi - see multiprocessing-logging on Pypi, github at https://github.com/jruere/multiprocessing-logging
Update: Implementation!
This now uses a queue for correct handling of concurrency, and also recovers from errors correctly. I've now been using this in production for several months, and the current version below works without issue.
from logging.handlers import RotatingFileHandler
import multiprocessing, threading, logging, sys, traceback
class MultiProcessingLog(logging.Handler):
def __init__(self, name, mode, maxsize, rotate):
logging.Handler.__init__(self)
self._handler = RotatingFileHandler(name, mode, maxsize, rotate)
self.queue = multiprocessing.Queue(-1)
t = threading.Thread(target=self.receive)
t.daemon = True
t.start()
def setFormatter(self, fmt):
logging.Handler.setFormatter(self, fmt)
self._handler.setFormatter(fmt)
def receive(self):
while True:
try:
record = self.queue.get()
self._handler.emit(record)
except (KeyboardInterrupt, SystemExit):
raise
except EOFError:
break
except:
traceback.print_exc(file=sys.stderr)
def send(self, s):
self.queue.put_nowait(s)
def _format_record(self, record):
# ensure that exc_info and args
# have been stringified. Removes any chance of
# unpickleable things inside and possibly reduces
# message size sent over the pipe
if record.args:
record.msg = record.msg % record.args
record.args = None
if record.exc_info:
dummy = self.format(record)
record.exc_info = None
return record
def emit(self, record):
try:
s = self._format_record(record)
self.send(s)
except (KeyboardInterrupt, SystemExit):
raise
except:
self.handleError(record)
def close(self):
self._handler.close()
logging.Handler.close(self)
The only way to deal with this non-intrusively is to:
Spawn each worker process such that its log goes to a different file descriptor (to disk or to pipe.) Ideally, all log entries should be timestamped.
Your controller process can then do one of the following:
If using disk files: Coalesce the log files at the end of the run, sorted by timestamp
If using pipes (recommended): Coalesce log entries on-the-fly from all pipes, into a central log file. (E.g., Periodically select from the pipes' file descriptors, perform merge-sort on the available log entries, and flush to centralized log. Repeat.)
QueueHandler is native in Python 3.2+, and does exactly this. It is easily replicated in previous versions.
Python docs have two complete examples: Logging to a single file from multiple processes
For those using Python < 3.2, just copy QueueHandler into your own code from: https://gist.github.com/vsajip/591589 or alternatively import logutils.
Each process (including the parent process) puts its logging on the Queue, and then a listener thread or process (one example is provided for each) picks those up and writes them all to a file - no risk of corruption or garbling.
Below is another solution with a focus on simplicity for anyone else (like me) who get here from Google. Logging should be easy! Only for 3.2 or higher.
import multiprocessing
import logging
from logging.handlers import QueueHandler, QueueListener
import time
import random
def f(i):
time.sleep(random.uniform(.01, .05))
logging.info('function called with {} in worker thread.'.format(i))
time.sleep(random.uniform(.01, .05))
return i
def worker_init(q):
# all records from worker processes go to qh and then into q
qh = QueueHandler(q)
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logger.addHandler(qh)
def logger_init():
q = multiprocessing.Queue()
# this is the handler for all log records
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(levelname)s: %(asctime)s - %(process)s - %(message)s"))
# ql gets records from the queue and sends them to the handler
ql = QueueListener(q, handler)
ql.start()
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# add the handler to the logger so records from this process are handled
logger.addHandler(handler)
return ql, q
def main():
q_listener, q = logger_init()
logging.info('hello from main thread')
pool = multiprocessing.Pool(4, worker_init, [q])
for result in pool.map(f, range(10)):
pass
pool.close()
pool.join()
q_listener.stop()
if __name__ == '__main__':
main()
Yet another alternative might be the various non-file-based logging handlers in the logging package:
SocketHandler
DatagramHandler
SyslogHandler
(and others)
This way, you could easily have a logging daemon somewhere that you could write to safely and would handle the results correctly. (E.g., a simple socket server that just unpickles the message and emits it to its own rotating file handler.)
The SyslogHandler would take care of this for you, too. Of course, you could use your own instance of syslog, not the system one.
As of 2020 it seems there is a simpler way of logging with multiprocessing.
This function will create the logger. You can set the format here and where you want your output to go (file, stdout):
def create_logger():
import multiprocessing, logging
logger = multiprocessing.get_logger()
logger.setLevel(logging.INFO)
formatter = logging.Formatter(\
'[%(asctime)s| %(levelname)s| %(processName)s] %(message)s')
handler = logging.FileHandler('logs/your_file_name.log')
handler.setFormatter(formatter)
# this bit will make sure you won't have
# duplicated messages in the output
if not len(logger.handlers):
logger.addHandler(handler)
return logger
In the init you instantiate the logger:
if __name__ == '__main__':
from multiprocessing import Pool
logger = create_logger()
logger.info('Starting pooling')
p = Pool()
# rest of the code
Now, you only need to add this reference in each function where you need logging:
logger = create_logger()
And output messages:
logger.info(f'My message from {something}')
Hope this helps.
A variant of the others that keeps the logging and queue thread separate.
"""sample code for logging in subprocesses using multiprocessing
* Little handler magic - The main process uses loggers and handlers as normal.
* Only a simple handler is needed in the subprocess that feeds the queue.
* Original logger name from subprocess is preserved when logged in main
process.
* As in the other implementations, a thread reads the queue and calls the
handlers. Except in this implementation, the thread is defined outside of a
handler, which makes the logger definitions simpler.
* Works with multiple handlers. If the logger in the main process defines
multiple handlers, they will all be fed records generated by the
subprocesses loggers.
tested with Python 2.5 and 2.6 on Linux and Windows
"""
import os
import sys
import time
import traceback
import multiprocessing, threading, logging, sys
DEFAULT_LEVEL = logging.DEBUG
formatter = logging.Formatter("%(levelname)s: %(asctime)s - %(name)s - %(process)s - %(message)s")
class SubProcessLogHandler(logging.Handler):
"""handler used by subprocesses
It simply puts items on a Queue for the main process to log.
"""
def __init__(self, queue):
logging.Handler.__init__(self)
self.queue = queue
def emit(self, record):
self.queue.put(record)
class LogQueueReader(threading.Thread):
"""thread to write subprocesses log records to main process log
This thread reads the records written by subprocesses and writes them to
the handlers defined in the main process's handlers.
"""
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue
self.daemon = True
def run(self):
"""read from the queue and write to the log handlers
The logging documentation says logging is thread safe, so there
shouldn't be contention between normal logging (from the main
process) and this thread.
Note that we're using the name of the original logger.
"""
# Thanks Mike for the error checking code.
while True:
try:
record = self.queue.get()
# get the logger for this record
logger = logging.getLogger(record.name)
logger.callHandlers(record)
except (KeyboardInterrupt, SystemExit):
raise
except EOFError:
break
except:
traceback.print_exc(file=sys.stderr)
class LoggingProcess(multiprocessing.Process):
def __init__(self, queue):
multiprocessing.Process.__init__(self)
self.queue = queue
def _setupLogger(self):
# create the logger to use.
logger = logging.getLogger('test.subprocess')
# The only handler desired is the SubProcessLogHandler. If any others
# exist, remove them. In this case, on Unix and Linux the StreamHandler
# will be inherited.
for handler in logger.handlers:
# just a check for my sanity
assert not isinstance(handler, SubProcessLogHandler)
logger.removeHandler(handler)
# add the handler
handler = SubProcessLogHandler(self.queue)
handler.setFormatter(formatter)
logger.addHandler(handler)
# On Windows, the level will not be inherited. Also, we could just
# set the level to log everything here and filter it in the main
# process handlers. For now, just set it from the global default.
logger.setLevel(DEFAULT_LEVEL)
self.logger = logger
def run(self):
self._setupLogger()
logger = self.logger
# and here goes the logging
p = multiprocessing.current_process()
logger.info('hello from process %s with pid %s' % (p.name, p.pid))
if __name__ == '__main__':
# queue used by the subprocess loggers
queue = multiprocessing.Queue()
# Just a normal logger
logger = logging.getLogger('test')
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(DEFAULT_LEVEL)
logger.info('hello from the main process')
# This thread will read from the subprocesses and write to the main log's
# handlers.
log_queue_reader = LogQueueReader(queue)
log_queue_reader.start()
# create the processes.
for i in range(10):
p = LoggingProcess(queue)
p.start()
# The way I read the multiprocessing warning about Queue, joining a
# process before it has finished feeding the Queue can cause a deadlock.
# Also, Queue.empty() is not realiable, so just make sure all processes
# are finished.
# active_children joins subprocesses when they're finished.
while multiprocessing.active_children():
time.sleep(.1)
All current solutions are too coupled to the logging configuration by using a handler. My solution has the following architecture and features:
You can use any logging configuration you want
Logging is done in a daemon thread
Safe shutdown of the daemon by using a context manager
Communication to the logging thread is done by multiprocessing.Queue
In subprocesses, logging.Logger (and already defined instances) are patched to send all records to the queue
New: format traceback and message before sending to queue to prevent pickling errors
Code with usage example and output can be found at the following Gist: https://gist.github.com/schlamar/7003737
Since we can represent multiprocess logging as many publishers and one subscriber (listener), using ZeroMQ to implement PUB-SUB messaging is indeed an option.
Moreover, PyZMQ module, the Python bindings for ZMQ, implements PUBHandler, which is object for publishing logging messages over a zmq.PUB socket.
There's a solution on the web, for centralized logging from distributed application using PyZMQ and PUBHandler, which can be easily adopted for working locally with multiple publishing processes.
formatters = {
logging.DEBUG: logging.Formatter("[%(name)s] %(message)s"),
logging.INFO: logging.Formatter("[%(name)s] %(message)s"),
logging.WARN: logging.Formatter("[%(name)s] %(message)s"),
logging.ERROR: logging.Formatter("[%(name)s] %(message)s"),
logging.CRITICAL: logging.Formatter("[%(name)s] %(message)s")
}
# This one will be used by publishing processes
class PUBLogger:
def __init__(self, host, port=config.PUBSUB_LOGGER_PORT):
self._logger = logging.getLogger(__name__)
self._logger.setLevel(logging.DEBUG)
self.ctx = zmq.Context()
self.pub = self.ctx.socket(zmq.PUB)
self.pub.connect('tcp://{0}:{1}'.format(socket.gethostbyname(host), port))
self._handler = PUBHandler(self.pub)
self._handler.formatters = formatters
self._logger.addHandler(self._handler)
#property
def logger(self):
return self._logger
# This one will be used by listener process
class SUBLogger:
def __init__(self, ip, output_dir="", port=config.PUBSUB_LOGGER_PORT):
self.output_dir = output_dir
self._logger = logging.getLogger()
self._logger.setLevel(logging.DEBUG)
self.ctx = zmq.Context()
self._sub = self.ctx.socket(zmq.SUB)
self._sub.bind('tcp://*:{1}'.format(ip, port))
self._sub.setsockopt(zmq.SUBSCRIBE, "")
handler = handlers.RotatingFileHandler(os.path.join(output_dir, "client_debug.log"), "w", 100 * 1024 * 1024, 10)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s;%(levelname)s - %(message)s")
handler.setFormatter(formatter)
self._logger.addHandler(handler)
#property
def sub(self):
return self._sub
#property
def logger(self):
return self._logger
# And that's the way we actually run things:
# Listener process will forever listen on SUB socket for incoming messages
def run_sub_logger(ip, event):
sub_logger = SUBLogger(ip)
while not event.is_set():
try:
topic, message = sub_logger.sub.recv_multipart(flags=zmq.NOBLOCK)
log_msg = getattr(logging, topic.lower())
log_msg(message)
except zmq.ZMQError as zmq_error:
if zmq_error.errno == zmq.EAGAIN:
pass
# Publisher processes loggers should be initialized as follows:
class Publisher:
def __init__(self, stop_event, proc_id):
self.stop_event = stop_event
self.proc_id = proc_id
self._logger = pub_logger.PUBLogger('127.0.0.1').logger
def run(self):
self._logger.info("{0} - Sending message".format(proc_id))
def run_worker(event, proc_id):
worker = Publisher(event, proc_id)
worker.run()
# Starting subscriber process so we won't loose publisher's messages
sub_logger_process = Process(target=run_sub_logger,
args=('127.0.0.1'), stop_event,))
sub_logger_process.start()
#Starting publisher processes
for i in range(MAX_WORKERS_PER_CLIENT):
processes.append(Process(target=run_worker,
args=(stop_event, i,)))
for p in processes:
p.start()
I also like zzzeek's answer but Andre is correct that a queue is required to prevent garbling. I had some luck with the pipe, but did see garbling which is somewhat expected. Implementing it turned out to be harder than I thought, particularly due to running on Windows, where there are some additional restrictions about global variables and stuff (see: How's Python Multiprocessing Implemented on Windows?)
But, I finally got it working. This example probably isn't perfect, so comments and suggestions are welcome. It also does not support setting the formatter or anything other than the root logger. Basically, you have to reinit the logger in each of the pool processes with the queue and set up the other attributes on the logger.
Again, any suggestions on how to make the code better are welcome. I certainly don't know all the Python tricks yet :-)
import multiprocessing, logging, sys, re, os, StringIO, threading, time, Queue
class MultiProcessingLogHandler(logging.Handler):
def __init__(self, handler, queue, child=False):
logging.Handler.__init__(self)
self._handler = handler
self.queue = queue
# we only want one of the loggers to be pulling from the queue.
# If there is a way to do this without needing to be passed this
# information, that would be great!
if child == False:
self.shutdown = False
self.polltime = 1
t = threading.Thread(target=self.receive)
t.daemon = True
t.start()
def setFormatter(self, fmt):
logging.Handler.setFormatter(self, fmt)
self._handler.setFormatter(fmt)
def receive(self):
#print "receive on"
while (self.shutdown == False) or (self.queue.empty() == False):
# so we block for a short period of time so that we can
# check for the shutdown cases.
try:
record = self.queue.get(True, self.polltime)
self._handler.emit(record)
except Queue.Empty, e:
pass
def send(self, s):
# send just puts it in the queue for the server to retrieve
self.queue.put(s)
def _format_record(self, record):
ei = record.exc_info
if ei:
dummy = self.format(record) # just to get traceback text into record.exc_text
record.exc_info = None # to avoid Unpickleable error
return record
def emit(self, record):
try:
s = self._format_record(record)
self.send(s)
except (KeyboardInterrupt, SystemExit):
raise
except:
self.handleError(record)
def close(self):
time.sleep(self.polltime+1) # give some time for messages to enter the queue.
self.shutdown = True
time.sleep(self.polltime+1) # give some time for the server to time out and see the shutdown
def __del__(self):
self.close() # hopefully this aids in orderly shutdown when things are going poorly.
def f(x):
# just a logging command...
logging.critical('function number: ' + str(x))
# to make some calls take longer than others, so the output is "jumbled" as real MP programs are.
time.sleep(x % 3)
def initPool(queue, level):
"""
This causes the logging module to be initialized with the necessary info
in pool threads to work correctly.
"""
logging.getLogger('').addHandler(MultiProcessingLogHandler(logging.StreamHandler(), queue, child=True))
logging.getLogger('').setLevel(level)
if __name__ == '__main__':
stream = StringIO.StringIO()
logQueue = multiprocessing.Queue(100)
handler= MultiProcessingLogHandler(logging.StreamHandler(stream), logQueue)
logging.getLogger('').addHandler(handler)
logging.getLogger('').setLevel(logging.DEBUG)
logging.debug('starting main')
# when bulding the pool on a Windows machine we also have to init the logger in all the instances with the queue and the level of logging.
pool = multiprocessing.Pool(processes=10, initializer=initPool, initargs=[logQueue, logging.getLogger('').getEffectiveLevel()] ) # start worker processes
pool.map(f, range(0,50))
pool.close()
logging.debug('done')
logging.shutdown()
print "stream output is:"
print stream.getvalue()
I'd like to suggest to use the logger_tt library: https://github.com/Dragon2fly/logger_tt
The multiporcessing_logging library is not working on my macOSX, while logger_tt does.
just publish somewhere your instance of the logger. that way, the other modules and clients can use your API to get the logger without having to import multiprocessing.
I liked zzzeek's answer. I would just substitute the Pipe for a Queue since if multiple threads/processes use the same pipe end to generate log messages they will get garbled.
The concurrent-log-handler seems to do the job perfectly. Tested on Windows. Supports also POSIX systems.
Main idea
Create a separate file with a function that returns a logger. The logger must have fresh instance of ConcurrentRotatingFileHandler for each process. Example function get_logger() given below.
Creating loggers is done at the initialization of the process. For a multiprocessing.Process subclass it would mean the beginning of the run() method.
Detailed instructions
I this example, I will use the following file structure
.
│-- child.py <-- For a child process
│-- logs.py <-- For setting up the logs for the app
│-- main.py <-- For a main process
│-- myapp.py <-- For starting the app
│-- somemodule.py <-- For an example, a "3rd party module using standard logging"
Code
Child process
# child.py
import multiprocessing as mp
import time
from somemodule import do_something
class ChildProcess(mp.Process):
def __init__(self):
self.logger = None
super().__init__()
def run(self):
from logs import get_logger
self.logger = get_logger()
while True:
time.sleep(1)
self.logger.info("Child process")
do_something()
Simple child process that inherits multiprocessing.Process and simply logs to file text "Child process"
Important: The get_logger() is called inside the run(), or elsewhere inside the child process (not module level or in __init__().) This is required as get_logger() creates ConcurrentRotatingFileHandler instance, and new instance is needed for each process.
The do_something is used just to demonstrate that this works with 3rd party library code which does not have any clue that you are using concurrent-log-handler.
Main Process
# main.py
import logging
import multiprocessing as mp
import time
from child import ChildProcess
from somemodule import do_something
class MainProcess(mp.Process):
def __init__(self):
self.logger = logging.getLogger()
super().__init__()
def run(self):
from logs import get_logger
self.logger = get_logger()
self.child = ChildProcess()
self.child.daemon = True
self.child.start()
while True:
time.sleep(0.5)
self.logger.critical("Main process")
do_something()
The main process that logs into file two times a second "Main process". Also inheriting from multiprocessing.Process.
Same comments for get_logger() and do_something() apply as for the child process.
Logger setup
# logs.py
import logging
import os
from concurrent_log_handler import ConcurrentRotatingFileHandler
LOGLEVEL = logging.DEBUG
def get_logger():
logger = logging.getLogger()
if logger.handlers:
return logger
# Use an absolute path to prevent file rotation trouble.
logfile = os.path.abspath("mylog.log")
logger.setLevel(LOGLEVEL)
# Rotate log after reaching 512K, keep 5 old copies.
filehandler = ConcurrentRotatingFileHandler(
logfile, mode="a", maxBytes=512 * 1024, backupCount=5, encoding="utf-8"
)
filehandler.setLevel(LOGLEVEL)
# create also handler for displaying output in the stdout
ch = logging.StreamHandler()
ch.setLevel(LOGLEVEL)
formatter = logging.Formatter(
"%(asctime)s - %(module)s - %(levelname)s - %(message)s [Process: %(process)d, %(filename)s:%(funcName)s(%(lineno)d)]"
)
# add formatter to ch
ch.setFormatter(formatter)
filehandler.setFormatter(formatter)
logger.addHandler(ch)
logger.addHandler(filehandler)
return logger
This uses the ConcurrentRotatingFileHandler from the concurrent-log-handler package. Each process needs a fresh ConcurrentRotatingFileHandler instance.
Note that all the arguments for the ConcurrentRotatingFileHandler should be the same in every process.
Example app
# myapp.py
if __name__ == "__main__":
from main import MainProcess
p = MainProcess()
p.start()
Just a simple example on how to start the multiprocess application
Example of 3rd party module using standard logging
# somemodule.py
import logging
logger = logging.getLogger("somemodule")
def do_something():
logging.info("doing something")
Just a simple example to test if loggers from 3rd party code will work normally.
Example output
2021-04-19 19:02:29,425 - main - CRITICAL - Main process [Process: 103348, main.py:run(23)]
2021-04-19 19:02:29,427 - somemodule - INFO - doing something [Process: 103348, somemodule.py:do_something(7)]
2021-04-19 19:02:29,929 - main - CRITICAL - Main process [Process: 103348, main.py:run(23)]
2021-04-19 19:02:29,931 - somemodule - INFO - doing something [Process: 103348, somemodule.py:do_something(7)]
2021-04-19 19:02:30,133 - child - INFO - Child process [Process: 76700, child.py:run(18)]
2021-04-19 19:02:30,137 - somemodule - INFO - doing something [Process: 76700, somemodule.py:do_something(7)]
2021-04-19 19:02:30,436 - main - CRITICAL - Main process [Process: 103348, main.py:run(23)]
2021-04-19 19:02:30,439 - somemodule - INFO - doing something [Process: 103348, somemodule.py:do_something(7)]
2021-04-19 19:02:30,944 - main - CRITICAL - Main process [Process: 103348, main.py:run(23)]
2021-04-19 19:02:30,946 - somemodule - INFO - doing something [Process: 103348, somemodule.py:do_something(7)]
2021-04-19 19:02:31,142 - child - INFO - Child process [Process: 76700, child.py:run(18)]
2021-04-19 19:02:31,145 - somemodule - INFO - doing something [Process: 76700, somemodule.py:do_something(7)]
2021-04-19 19:02:31,449 - main - CRITICAL - Main process [Process: 103348, main.py:run(23)]
2021-04-19 19:02:31,451 - somemodule - INFO - doing something [Process: 103348, somemodule.py:do_something(7)]
How about delegating all the logging to another process that reads all log entries from a Queue?
LOG_QUEUE = multiprocessing.JoinableQueue()
class CentralLogger(multiprocessing.Process):
def __init__(self, queue):
multiprocessing.Process.__init__(self)
self.queue = queue
self.log = logger.getLogger('some_config')
self.log.info("Started Central Logging process")
def run(self):
while True:
log_level, message = self.queue.get()
if log_level is None:
self.log.info("Shutting down Central Logging process")
break
else:
self.log.log(log_level, message)
central_logger_process = CentralLogger(LOG_QUEUE)
central_logger_process.start()
Simply share LOG_QUEUE via any of the multiprocess mechanisms or even inheritance and it all works out fine!
Below is a class that can be used in Windows environment, requires ActivePython.
You can also inherit for other logging handlers (StreamHandler etc.)
class SyncronizedFileHandler(logging.FileHandler):
MUTEX_NAME = 'logging_mutex'
def __init__(self , *args , **kwargs):
self.mutex = win32event.CreateMutex(None , False , self.MUTEX_NAME)
return super(SyncronizedFileHandler , self ).__init__(*args , **kwargs)
def emit(self, *args , **kwargs):
try:
win32event.WaitForSingleObject(self.mutex , win32event.INFINITE)
ret = super(SyncronizedFileHandler , self ).emit(*args , **kwargs)
finally:
win32event.ReleaseMutex(self.mutex)
return ret
And here is an example that demonstrates usage:
import logging
import random , time , os , sys , datetime
from string import letters
import win32api , win32event
from multiprocessing import Pool
def f(i):
time.sleep(random.randint(0,10) * 0.1)
ch = random.choice(letters)
logging.info( ch * 30)
def init_logging():
'''
initilize the loggers
'''
formatter = logging.Formatter("%(levelname)s - %(process)d - %(asctime)s - %(filename)s - %(lineno)d - %(message)s")
logger = logging.getLogger()
logger.setLevel(logging.INFO)
file_handler = SyncronizedFileHandler(sys.argv[1])
file_handler.setLevel(logging.INFO)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
#must be called in the parent and in every worker process
init_logging()
if __name__ == '__main__':
#multiprocessing stuff
pool = Pool(processes=10)
imap_result = pool.imap(f , range(30))
for i , _ in enumerate(imap_result):
pass
I have a solution that's similar to ironhacker's except that I use logging.exception in some of my code and found that I needed to format the exception before passing it back over the Queue since tracebacks aren't pickle'able:
class QueueHandler(logging.Handler):
def __init__(self, queue):
logging.Handler.__init__(self)
self.queue = queue
def emit(self, record):
if record.exc_info:
# can't pass exc_info across processes so just format now
record.exc_text = self.formatException(record.exc_info)
record.exc_info = None
self.queue.put(record)
def formatException(self, ei):
sio = cStringIO.StringIO()
traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
s = sio.getvalue()
sio.close()
if s[-1] == "\n":
s = s[:-1]
return s
If you have deadlocks occurring in a combination of locks, threads and forks in the logging module, that is reported in bug report 6721 (see also related SO question).
There is a small fixup solution posted here.
However, that will just fix any potential deadlocks in logging. That will not fix that things are maybe garbled up. See the other answers presented here.
Here's my simple hack/workaround... not the most comprehensive, but easily modifiable and simpler to read and understand I think than any other answers I found before writing this:
import logging
import multiprocessing
class FakeLogger(object):
def __init__(self, q):
self.q = q
def info(self, item):
self.q.put('INFO - {}'.format(item))
def debug(self, item):
self.q.put('DEBUG - {}'.format(item))
def critical(self, item):
self.q.put('CRITICAL - {}'.format(item))
def warning(self, item):
self.q.put('WARNING - {}'.format(item))
def some_other_func_that_gets_logger_and_logs(num):
# notice the name get's discarded
# of course you can easily add this to your FakeLogger class
local_logger = logging.getLogger('local')
local_logger.info('Hey I am logging this: {} and working on it to make this {}!'.format(num, num*2))
local_logger.debug('hmm, something may need debugging here')
return num*2
def func_to_parallelize(data_chunk):
# unpack our args
the_num, logger_q = data_chunk
# since we're now in a new process, let's monkeypatch the logging module
logging.getLogger = lambda name=None: FakeLogger(logger_q)
# now do the actual work that happens to log stuff too
new_num = some_other_func_that_gets_logger_and_logs(the_num)
return (the_num, new_num)
if __name__ == '__main__':
multiprocessing.freeze_support()
m = multiprocessing.Manager()
logger_q = m.Queue()
# we have to pass our data to be parallel-processed
# we also need to pass the Queue object so we can retrieve the logs
parallelable_data = [(1, logger_q), (2, logger_q)]
# set up a pool of processes so we can take advantage of multiple CPU cores
pool_size = multiprocessing.cpu_count() * 2
pool = multiprocessing.Pool(processes=pool_size, maxtasksperchild=4)
worker_output = pool.map(func_to_parallelize, parallelable_data)
pool.close() # no more tasks
pool.join() # wrap up current tasks
# get the contents of our FakeLogger object
while not logger_q.empty():
print logger_q.get()
print 'worker output contained: {}'.format(worker_output)
There is this great package
Package:
https://pypi.python.org/pypi/multiprocessing-logging/
code:
https://github.com/jruere/multiprocessing-logging
Install:
pip install multiprocessing-logging
Then add:
import multiprocessing_logging
# This enables logs inside process
multiprocessing_logging.install_mp_handler()
For whoever might need this, I wrote a decorator for multiprocessing_logging package that adds the current process name to logs, so it becomes clear who logs what.
It also runs install_mp_handler() so it becomes unuseful to run it before creating a pool.
This allows me to see which worker creates which logs messages.
Here's the blueprint with an example:
import sys
import logging
from functools import wraps
import multiprocessing
import multiprocessing_logging
# Setup basic console logger as 'logger'
logger = logging.getLogger()
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setFormatter(logging.Formatter(u'%(asctime)s :: %(levelname)s :: %(message)s'))
logger.setLevel(logging.DEBUG)
logger.addHandler(console_handler)
# Create a decorator for functions that are called via multiprocessing pools
def logs_mp_process_names(fn):
class MultiProcessLogFilter(logging.Filter):
def filter(self, record):
try:
process_name = multiprocessing.current_process().name
except BaseException:
process_name = __name__
record.msg = f'{process_name} :: {record.msg}'
return True
multiprocessing_logging.install_mp_handler()
f = MultiProcessLogFilter()
# Wraps is needed here so apply / apply_async know the function name
#wraps(fn)
def wrapper(*args, **kwargs):
logger.removeFilter(f)
logger.addFilter(f)
return fn(*args, **kwargs)
return wrapper
# Create a test function and decorate it
#logs_mp_process_names
def test(argument):
logger.info(f'test function called via: {argument}')
# You can also redefine undecored functions
def undecorated_function():
logger.info('I am not decorated')
#logs_mp_process_names
def redecorated(*args, **kwargs):
return undecorated_function(*args, **kwargs)
# Enjoy
if __name__ == '__main__':
with multiprocessing.Pool() as mp_pool:
# Also works with apply_async
mp_pool.apply(test, ('mp pool',))
mp_pool.apply(redecorated)
logger.info('some main logs')
test('main program')
One of the alternatives is to write the mutliprocessing logging to a known file and register an atexit handler to join on those processes read it back on stderr; however, you won't get a real-time flow to the output messages on stderr that way.
Simplest idea as mentioned:
Grab the filename and the process id of the current process.
Set up a [WatchedFileHandler][1]. The reasons for this handler are discussed in detail here, but in short there are certain worse race conditions with the other logging handlers. This one has the shortest window for the race condition.
Choose a path to save the logs to such as /var/log/...
Related
I have a multiprocessing setup, with a worker function and a parent function. Here is a simplified version:
from functools import partial
from typing import List
import logging
from logging.handlers import QueueHandler, QueueListener
from multiprocessing import Manager, Pool, Queue
def parent_func(log_file: str, task_list: List[str]):
# Setup the root logger
logger = getLogger()
logger.addHandler(FileHandler(log_file))
# Setup the logging listener, which should log to the log file and to stdout
manager = Manager()
queue = manager.Queue()
# Copy all handlers from the root logger
listener = QueueListener(queue, *logger.handlers)
listener.start()
# Fork into workers
with Pool() as pool:
func = partial(
worker_func,
logging_queue=queue,
)
pool.map(func, task_list)
listener.stop()
def worker_func(
data: str,
logging_queue: Queue,
):
# Note: this logger is totally disconnected from the parent logger since it shares no handlers
logger = getLogger(data)
logger.handlers = []
logger.addHandler(QueueHandler(logging_queue))
logger.info("Processing {}".format(data))
The problems are:
Using this code, each logger.info call in the worker function logs everything to stdout twice, and
If I remove the listener variable entirely, I somehow still receive logs from the worker functions (even though they're running on a different process!)
This implies that I'm doing something very wrong, because the logging queue doesn't even seem to be necessary. What am I doing wrong here?
Somewhat similar to this (unanswered) question here:
https://stackoverflow.com/questions/33136398/python-logging-handler-that-appends-messages-to-a-list
I am looking to use a handler to simply append anything (passing a filter) to a python list. I would envisage some kind of code like this:
import logging
import sys
mylog = logging.getLogger()
mylog.setLevel(logging.DEBUG)
log_list = []
lh = logging.SomeHandler(log_list)
lh.setLevel(logging.DEBUG)
mylog.addHandler(lh)
mylog.warning('argh')
print log_list[0]
The question is therefore - how can I implement this? What SomeHandler can I use?
Here is a naive, non thread-safe implementation:
import logging
class ListHandler(logging.Handler): # Inherit from logging.Handler
def __init__(self, log_list):
# run the regular Handler __init__
logging.Handler.__init__(self)
# Our custom argument
self.log_list = log_list
def emit(self, record):
# record.message is the log message
self.log_list.append(record.msg)
#imriqwe's answer is correct for a non thread-safe implementation, but if you need to be thread-safe, one solution is to use a queue.Queue() instead of a list. Here is some code I am using in an in-process project to generate a tkinter log window.
import logging
import queue
class QueuingHandler(logging.Handler):
"""A thread safe logging.Handler that writes messages into a queue object.
Designed to work with LoggingWidget so log messages from multiple
threads can be shown together in a single ttk.Frame.
The standard logging.QueueHandler/logging.QueueListener can not be used
for this because the QueueListener runs in a private thread, not the
main thread.
Warning: If multiple threads are writing into this Handler, all threads
must be joined before calling logging.shutdown() or any other log
destinations will be corrupted.
"""
def __init__(self, *args, message_queue, **kwargs):
"""Initialize by copying the queue and sending everything else to superclass."""
logging.Handler.__init__(self, *args, **kwargs)
self.message_queue = message_queue
def emit(self, record):
"""Add the formatted log message (sans newlines) to the queue."""
self.message_queue.put(self.format(record).rstrip('\n'))
To use, create a queue, create the handler using the queue, then add it to the logger (this example also creates a log file in the current directory):
LOG_FORMAT = '%(asctime)s: %(name)8s: %(levelname)8s: %(message)s'
# Setup root logger to write to a log file.
logging.basicConfig(filename='gui-test.log',
filemode='w',
format=LOG_FORMAT,
level=logging.DEBUG
)
# Get a child logger
logger = logging.getLogger(name='gui')
# Build our QueuingHandler
message_queue = queue.Queue()
handler = QueuingHandler(message_queue=message_queue, level=logging.DEBUG)
# Change the date/time format for the GUI to drop the date
formatter = logging.Formatter(LOG_FORMAT)
formatter.default_time_format = '%H:%M:%S'
handler.setFormatter(formatter)
# Add our QueuingHandler into the logging heirarchy at the lower level
logger.addHandler(handler)
Now all you have to do is read your messages from the queue.
Logging is needed for a multiprocess python app. Using queues appears to be the best solution. The logutils library provides this.
Is it possible to set the log levels for two handler independently? For example, in the test below I would like STREAM 1 to have WARNING messages and STREAM 2 to have INFO messages. In the test log at the end of the code an INFO message is created that should not output to the console from STREAM 1 handler (WARNING). However, it outputs to both handlers.
For reference, I have been using this page http://plumberjack.blogspot.co.uk/2010/09/improved-queuehandler-queuelistener.html by Vinay Sajip who is the author of the library.
# System imports
import logging
import logging.handlers
try:
import Queue as queue
except ImportError:
import queue
# Custom imports
from logutils.queue import QueueHandler, QueueListener
# Get queue
q = queue.Queue(-1)
# Setup stream handler 1 to output WARNING to console
h1 = logging.StreamHandler()
f1 = logging.Formatter('STREAM 1 WARNING: %(threadName)s: %(message)s')
h1.setFormatter(f1)
h1.setLevel(logging.WARNING) # NOT WORKING. This should log >= WARNING
# Setup stream handler 2 to output INFO to console
h2 = logging.StreamHandler()
f2 = logging.Formatter('STREAM 2 INFO: %(threadName)s: %(message)s')
h2.setFormatter(f2)
h2.setLevel(logging.INFO) # NOT WORKING. This should log >= WARNING
# Start queue listener using the stream handler above
ql = QueueListener(q, h1, h2)
ql.start()
# Create log and set handler to queue handle
root = logging.getLogger()
root.setLevel(logging.DEBUG) # Log level = DEBUG
qh = QueueHandler(q)
root.addHandler(qh)
root.info('Look out!') # Create INFO message
ql.stop()
You can use the respect_handler_level argument that's been added to the QueueListener initializer from Python version 3.5, so that the listener will respect handlers individual level.
From QueueListener's doc:
If respect_handler_level is True, a handler’s level is respected (compared with the level for the message) when deciding whether to pass messages to that handler; otherwise, the behaviour is as in previous Python versions - to always pass each message to each handler.
In your case you should replace the initialization of the QueueListener to:
ql = QueueListener(q, h1, h2, respect_handler_level=True)
This is a limitation in the implementation of the QueueListener.handle() method. This is currently:
def handle(self, record):
record = self.prepare(record)
for handler in self.handlers:
handler.handle(record)
To do what you want, it should be
def handle(self, record):
record = self.prepare(record)
for handler in self.handlers:
# CHANGED HERE TO ADD A CONDITION TO CHECK THE HANDLER LEVEL
if record.levelno >= handler.level:
handler.handle(record)
I will fix this at some point because I think this is better, but for now you can subclass QueueListener and override the handle method in the subclass.
I used this subclass to override the queue listener class. The two other methods addHandler and removeHandler allow for addition and removal of handlers. The CustomQueueListener should be used just like QueueListener. Follow the other logging examples for how to use addHandler() and removeHandler().
class CustomQueueListener(QueueListener):
def __init__(self, queue, *handlers):
super(CustomQueueListener, self).__init__(queue, *handlers)
"""
Initialise an instance with the specified queue and
handlers.
"""
# Changing this to a list from tuple in the parent class
self.handlers = list(handlers)
def handle(self, record):
"""
Override handle a record.
This just loops through the handlers offering them the record
to handle.
:param record: The record to handle.
"""
record = self.prepare(record)
for handler in self.handlers:
if record.levelno >= handler.level: # This check is not in the parent class
handler.handle(record)
def addHandler(self, hdlr):
"""
Add the specified handler to this logger.
"""
if not (hdlr in self.handlers):
self.handlers.append(hdlr)
def removeHandler(self, hdlr):
"""
Remove the specified handler from this logger.
"""
if hdlr in self.handlers:
hdlr.close()
self.handlers.remove(hdlr)
I've seen a few questions regarding putting logs from different processes together when using the multiprocessing module in Python. I would like to do the opposite, produce separate log files for different processes, and they should log everything that happens when calling other modules without being mangled. In the example below I have a main program (main.py) and two modules (module1.py and module2.py), and I want the main logger (mainlog) to write to stdout, which it does fine. I also want a separate file for each process including logging from module1 and module2.
main.py:
import logging
import multiprocessing as mpr
import module1
import sys
mainlog = logging.getLogger("main")
h = logging.StreamHandler(sys.stdout)
mainlog.addHandler(h)
logging.root.setLevel(logging.DEBUG)
for i in xrange(0,3):
mainlog.info("Starting process ... %s", i)
log = logging.getLogger("module1")
h = logging.FileHandler("process_{0}.log".format(i))
fmt = logging.Formatter(fmt="%(levelname)-10s:%(filename)-20s:%(message)s")
h.setFormatter(fmt)
log.addHandler(h)
log.setLevel(logging.DEBUG)
p = mpr.Process(target=module1.do_something, args=(i,))
p.start()
A module1.py:
import logging
import module2
log = logging.getLogger("module1")
def do_something(i):
for j in xrange(0,100):
log.debug("do something. process %2s. iteration %2s", i,j)
module2.multiply(j,2)
And a module2.py:
import logging
log = logging.getLogger("module2")
def multiply(x,y):
log.debug("... multiplying %s x %s = %s", x,y, x*y)
return x*y
Instead I get the following output:
Starting process ... 0
Starting process ... 1
No handlers could be found for logger "module2"
Starting process ... 2
No handlers could be found for logger "module2"
No handlers could be found for logger "module2"
And 3 individual logging files (process_0.log, ...) that contain the messages from all processes together, instead of only one. Nothing from module2.py is logged. What am I doing wrong?
You need to configure logging in the child processes. They start off with a clean slate, and logging isn't configured in them. Are you on Windows, by any chance? Almost nothing is inherited from the parent process to the child process in Windows, whereas on POSIX, fork() semantics can allow some things to be inherited.
I ended up creating a subclass of logging.Logger to manage switch between logging to main and logging to disk. Now I can switch when necessary inside a Process:
import logging
import sys
class CGLogger(logging.Logger):
def __init__(self,name):
logging.Logger.__init__(self,name)
self.mainhandler = logging.StreamHandler(sys.stdout)
self.addHandler(self.mainhandler)
def stop_main_logging(self):
self.removeHandler(self.mainhandler)
def log_to_file(self, fn):
self.filehandler = logging.FileHandler(fn)
self.addHandler(self.filehandler)
def stop_logging_to_file(self):
self.removeHandler(self.filehandler)
def restart_main_logging(self):
self.addHandler(self.mainhandler)
def switch_to_file_logging(self, fn):
self.stop_main_logging()
self.log_to_file(fn)
def switch_to_main_logging(self):
self.stop_logging_to_file()
self.restart_main_logging(fn)
logging.setLoggerClass(CGLogger)
I've recently been given the challenge of working multiprocessing into our software. I want a main process to spawn subprocesses, and I need some way of sending logging information back to the main process. This is mainly because a module we use writes warning and error messages to a logging object, and we want these messages to appear in the gui, which runs in the main process.
The obvious approach was to write a small class with a write() method that puts() onto a queue, and then use this class in a logging stream handler. The main process would then get() from this queue to send the text to the gui. But this didn't seem to work, and I don't know why
I wrote some sample code to demonstrate the problem. It uses a logging object to write a queue in a subprocess, and then the main process tries to read from the queue, but fails. Can someone help me figure out what is wrong with this?
import time, multiprocessing, queue, logging
class FileLikeQueue:
"""A file-like object that writes to a queue"""
def __init__(self, q):
self.q = q
def write(self, t):
self.q.put(t)
def flush(self):
pass
def func(q):
"""This function just writes the time every second for five
seconds and then returns. The time is sent to the queue and
to a logging object"""
stream = FileLikeQueue(q)
log = logging.getLogger()
infohandler = logging.StreamHandler(stream)
infohandler.setLevel(logging.INFO)
infoformatter = logging.Formatter("%(message)s")
infohandler.setFormatter(infoformatter)
log.addHandler(infohandler)
t1 = time.time()
while time.time() - t1 < 5: #run for five seconds
log.info('Logging: ' + str(time.time()))
q.put('Put: %s' % str(time.time()))
time.sleep(1)
def main():
q = multiprocessing.Queue()
p = multiprocessing.Process(target=func, args=(q,))
p.start()
#read the queue until it is empty
while True:
try:
t = q.get()
except queue.Empty:
break
print(t)
if __name__ == '__main__':
main()
I expect the output to be:
Logging: 1333629221.01
Put: 1333629221.01
Logging: 1333629222.02
Put: 1333629222.02
Logging: 1333629223.02
Put: 1333629223.02
Logging: 1333629224.02
Put: 1333629224.02
Logging: 1333629225.02
Put: 1333629225.02
But what I get is:
Put: 1333629221.01
Put: 1333629222.02
Put: 1333629223.02
Put: 1333629224.02
Put: 1333629225.02
So the put() operation in func() works, but the logging doesn't. Why?
Thank you.
Your problem is with the configuration of the logging module:
You need to call log.setLevel(logging.INFO). The default log level is WARNING, so your logs have no effect.
You did call setLevel on the handler object, but the logged messages never reach the handler because they are filtered by the logger. There is no need to call setLevel on the handler itself, because it processes all messages by default.