Related
I have a logger that has a RotatingFileHandler.
I want to redirect all Stdout and Stderr to the logger.
How to do so?
Not enough rep to comment, but I wanted to add the version of this that worked for me in case others are in a similar situation.
class LoggerWriter:
def __init__(self, level):
# self.level is really like using log.debug(message)
# at least in my case
self.level = level
def write(self, message):
# if statement reduces the amount of newlines that are
# printed to the logger
if message != '\n':
self.level(message)
def flush(self):
# create a flush method so things can be flushed when
# the system wants to. Not sure if simply 'printing'
# sys.stderr is the correct way to do it, but it seemed
# to work properly for me.
self.level(sys.stderr)
and this would look something like:
log = logging.getLogger('foobar')
sys.stdout = LoggerWriter(log.debug)
sys.stderr = LoggerWriter(log.warning)
UPDATE for Python 3:
Including a dummy flush function which prevents an error where the function is expected (Python 2 was fine with just linebuf='').
Note that your output (and log level) appears different if it is logged from an interpreter session vs being run from a file. Running from a file produces the expected behavior (and output featured below).
We still eliminate extra newlines which other solutions do not.
class StreamToLogger(object):
"""
Fake file-like stream object that redirects writes to a logger instance.
"""
def __init__(self, logger, level):
self.logger = logger
self.level = level
self.linebuf = ''
def write(self, buf):
for line in buf.rstrip().splitlines():
self.logger.log(self.level, line.rstrip())
def flush(self):
pass
Then test with something like:
import StreamToLogger
import sys
import logging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename='out.log',
filemode='a'
)
log = logging.getLogger('foobar')
sys.stdout = StreamToLogger(log,logging.INFO)
sys.stderr = StreamToLogger(log,logging.ERROR)
print('Test to standard out')
raise Exception('Test to standard error')
See below for old Python 2.x answer and the example output:
All of the prior answers seem to have problems adding extra newlines where they aren't needed. The solution that works best for me is from http://www.electricmonk.nl/log/2011/08/14/redirect-stdout-and-stderr-to-a-logger-in-python/, where he demonstrates how send both stdout and stderr to the logger:
import logging
import sys
class StreamToLogger(object):
"""
Fake file-like stream object that redirects writes to a logger instance.
"""
def __init__(self, logger, log_level=logging.INFO):
self.logger = logger
self.log_level = log_level
self.linebuf = ''
def write(self, buf):
for line in buf.rstrip().splitlines():
self.logger.log(self.log_level, line.rstrip())
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename="out.log",
filemode='a'
)
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl
stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl
print "Test to standard out"
raise Exception('Test to standard error')
The output looks like:
2011-08-14 14:46:20,573:INFO:STDOUT:Test to standard out
2011-08-14 14:46:20,573:ERROR:STDERR:Traceback (most recent call last):
2011-08-14 14:46:20,574:ERROR:STDERR: File "redirect.py", line 33, in
2011-08-14 14:46:20,574:ERROR:STDERR:raise Exception('Test to standard error')
2011-08-14 14:46:20,574:ERROR:STDERR:Exception
2011-08-14 14:46:20,574:ERROR:STDERR::
2011-08-14 14:46:20,574:ERROR:STDERR:Test to standard error
Note that self.linebuf = '' is where the flush is being handled, rather than implementing a flush function.
If it's an all-Python system (i.e. no C libraries writing to fds directly, as Ignacio Vazquez-Abrams asked about) then you might be able to use an approach as suggested here:
class LoggerWriter:
def __init__(self, logger, level):
self.logger = logger
self.level = level
def write(self, message):
if message != '\n':
self.logger.log(self.level, message)
and then set sys.stdout and sys.stderr to LoggerWriter instances.
You can use redirect_stdout context manager:
import logging
from contextlib import redirect_stdout
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.write = lambda msg: logging.info(msg) if msg != '\n' else None
with redirect_stdout(logging):
print('Test')
or like this
import logging
from contextlib import redirect_stdout
logger = logging.getLogger('Meow')
logger.setLevel(logging.INFO)
formatter = logging.Formatter(
fmt='[{name}] {asctime} {levelname}: {message}',
datefmt='%m/%d/%Y %H:%M:%S',
style='{'
)
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.write = lambda msg: logger.info(msg) if msg != '\n' else None
with redirect_stdout(logger):
print('Test')
Output Redirection Done Right!
The Problem
logger.log and the other functions (.info/.error/etc.) output each call as a separate line, i.e. implicitly add (formatting and) a newline to it.
sys.stderr.write on the other hand just writes its literal input to stream, including partial lines. For example: The output "ZeroDivisionError: division by zero" is actually 4(!) separate calls to sys.stderr.write:
sys.stderr.write('ZeroDivisionError')
sys.stderr.write(': ')
sys.stderr.write('division by zero')
sys.stderr.write('\n')
The 4 most upvoted approaches (1, 2, 3, 4) thus result in extra newlines -- simply put "1/0" into your program and you will get the following:
2021-02-17 13:10:40,814 - ERROR - ZeroDivisionError
2021-02-17 13:10:40,814 - ERROR - :
2021-02-17 13:10:40,814 - ERROR - division by zero
The Solution
Store the intermediate writes in a buffer. The reason I am using a list as buffer rather than a string is to avoid the Shlemiel the painter’s algorithm. TLDR: It is O(n) instead of potentially O(n^2)
class LoggerWriter:
def __init__(self, logfct):
self.logfct = logfct
self.buf = []
def write(self, msg):
if msg.endswith('\n'):
self.buf.append(msg.removesuffix('\n'))
self.logfct(''.join(self.buf))
self.buf = []
else:
self.buf.append(msg)
def flush(self):
pass
# To access the original stdout/stderr, use sys.__stdout__/sys.__stderr__
sys.stdout = LoggerWriter(logger.info)
sys.stderr = LoggerWriter(logger.error)
2021-02-17 13:15:22,956 - ERROR - ZeroDivisionError: division by zero
For versions below Python 3.9, you could replace replace msg.removesuffix('\n') with either msg.rstrip('\n') or msg[:-1].
As an evolution to Cameron Gagnon's response, I've improved the LoggerWriterclass to:
class LoggerWriter(object):
def __init__(self, writer):
self._writer = writer
self._msg = ''
def write(self, message):
self._msg = self._msg + message
while '\n' in self._msg:
pos = self._msg.find('\n')
self._writer(self._msg[:pos])
self._msg = self._msg[pos+1:]
def flush(self):
if self._msg != '':
self._writer(self._msg)
self._msg = ''
now uncontrolled exceptions look nicer:
2018-07-31 13:20:37,482 - ERROR - Traceback (most recent call last):
2018-07-31 13:20:37,483 - ERROR - File "mf32.py", line 317, in <module>
2018-07-31 13:20:37,485 - ERROR - main()
2018-07-31 13:20:37,486 - ERROR - File "mf32.py", line 289, in main
2018-07-31 13:20:37,488 - ERROR - int('')
2018-07-31 13:20:37,489 - ERROR - ValueError: invalid literal for int() with base 10: ''
With flush added to Vinay Sajip's answer:
class LoggerWriter:
def __init__(self, logger, level):
self.logger = logger
self.level = level
def write(self, message):
if message != '\n':
self.logger.log(self.level, message)
def flush(self):
pass
Quick but Fragile One-Liner
sys.stdout.write = logger.info
sys.stderr.write = logger.error
What this does is simply assign the logger functions to the stdout/stderr .write call which means any write call will instead invoke the logger functions.
The downside of this approach is that both calls to .write and the logger functions typically add a newline so you will end up with extra lines in your log file, which may or may not be a problem depending on your use case.
Another pitfall is that if your logger writes to stderr itself we get infinite recursion (a stack overflow error). So only output to a file.
Solving problem where StreamHandler causes infinite Recurison
My logger was causing an infinite recursion, because the Streamhandler was trying to write to stdout, which itself is a logger -> leading to infinite recursion.
Solution
Reinstate the original sys.__stdout__ for the StreamHandler ONLY, so that you can still see the logs showing in the terminal.
class DefaultStreamHandler(logging.StreamHandler):
def __init__(self, stream=sys.__stdout__):
# Use the original sys.__stdout__ to write to stdout
# for this handler, as sys.stdout will write out to logger.
super().__init__(stream)
class LoggerWriter(io.IOBase):
"""Class to replace the stderr/stdout calls to a logger"""
def __init__(self, logger_name: str, log_level: int):
""":param logger_name: Name to give the logger (e.g. 'stderr')
:param log_level: The log level, e.g. logging.DEBUG / logging.INFO that
the MESSAGES should be logged at.
"""
self.std_logger = logging.getLogger(logger_name)
# Get the "root" logger from by its name (i.e. from a config dict or at the bottom of this file)
# We will use this to create a copy of all its settings, except the name
app_logger = logging.getLogger("myAppsLogger")
[self.std_logger.addHandler(handler) for handler in app_logger.handlers]
self.std_logger.setLevel(app_logger.level) # the minimum lvl msgs will show at
self.level = log_level # the level msgs will be logged at
self.buffer = []
def write(self, msg: str):
"""Stdout/stderr logs one line at a time, rather than 1 message at a time.
Use this function to aggregate multi-line messages into 1 log call."""
msg = msg.decode() if issubclass(type(msg), bytes) else msg
if not msg.endswith("\n"):
return self.buffer.append(msg)
self.buffer.append(msg.rstrip("\n"))
message = "".join(self.buffer)
self.std_logger.log(self.level, message)
self.buffer = []
def replace_stderr_and_stdout_with_logger():
"""Replaces calls to sys.stderr -> logger.info & sys.stdout -> logger.error"""
# To access the original stdout/stderr, use sys.__stdout__/sys.__stderr__
sys.stdout = LoggerWriter("stdout", logging.INFO)
sys.stderr = LoggerWriter("stderr", logging.ERROR)
if __name__ == __main__():
# Load the logger & handlers
logger = logging.getLogger("myAppsLogger")
logger.setLevel(logging.DEBUG)
# HANDLER = logging.StreamHandler()
HANDLER = DefaultStreamHandler() # <--- replace the normal streamhandler with this
logger.addHandler(HANDLER)
logFormatter = logging.Formatter("[%(asctime)s] - %(name)s - %(levelname)s - %(message)s")
HANDLER.setFormatter(logFormatter)
# Run this AFTER you load the logger
replace_stderr_and_stdout_with_logger()
And then finally call the replace_stderr_and_stdout_with_logger() after you've initialised your logger (the last bit of the code)
If you want to logging info and error messages into separates stream (info into stdout, errors into stderr) you can use this trick:
class ErrorStreamHandler(log.StreamHandler):
"""Print input log-message into stderr, print only error/warning messages"""
def __init__(self, stream=sys.stderr):
log.Handler.__init__(self, log.WARNING)
self.stream = stream
def emit(self, record):
try:
if record.levelno in (log.INFO, log.DEBUG, log.NOTSET):
return
msg = self.format(record)
stream = self.stream
# issue 35046: merged two stream.writes into one.
stream.write(msg + self.terminator)
self.flush()
except RecursionError: # See issue 36272
raise
except Exception:
self.handleError(record)
class OutStreamHandler(log.StreamHandler):
"""Print input log-message into stdout, print only info/debug messages"""
def __init__(self, loglevel, stream=sys.stdout):
log.Handler.__init__(self, loglevel)
self.stream = stream
def emit(self, record):
try:
if record.levelno not in (log.INFO, log.DEBUG, log.NOTSET):
return
msg = self.format(record)
stream = self.stream
# issue 35046: merged two stream.writes into one.
stream.write(msg + self.terminator)
self.flush()
except RecursionError: # See issue 36272
raise
except Exception:
self.handleError(record)
Usage:
log.basicConfig(level=settings.get_loglevel(),
format="[%(asctime)s] %(levelname)s: %(message)s",
datefmt='%Y/%m/%d %H:%M:%S', handlers=[ErrorStreamHandler(), OutStreamHandler(settings.get_loglevel())])
I need to execute code inside while loop every x seconds without stoping loop work
I have trying threading and lock combinations but it is still not working. I am working on python 3.7.4, pycharm 2019.2
#!/usr/bin/env python3
import configparser
import logging
import threading
import time
import ts3
__all__ = ["notify_bot"]
logging.basicConfig(filename='ts3bot.log',
level=logging.INFO,
format="%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s] %(message)s",
)
logging.getLogger().addHandler(logging.StreamHandler())
def notify_bot(ts3conn, config, lock):
logging.info("Start Notify Bot ...")
lock.acquire()
ts3conn.exec_("servernotifyregister", event="server")
lock.release()
while True:
event = ts3conn.wait_for_event()
try:
reasonid_ = event[0]["reasonid"]
except KeyError:
continue
if reasonid_ == "0":
logging.info("User joined Lobby:")
logging.info(event[0])
servergroups = event[0]['client_servergroups']
guestname = event[0]['client_nickname']
lock.acquire()
if not set(servergroups):
print(f"s1 {guestname}")
else:
print(f"s2{guestname}")
lock.release()
return None
def keep_alive(ts3conn, lock):
while True:
logging.info("Send keep alive!")
lock.acquire()
ts3conn.send_keepalive()
lock.release()
time.sleep(5)
if __name__ == "__main__":
logging.info("Start TS Bot ...")
config = configparser.ConfigParser()
config.sections()
config.read("settings_test.ini")
logging.info("Config loaded!")
HOST = config['server']['url']
PORT = config['server']['query_port']
USER = config['server']['query_user']
PASS = config['server']['query_pw']
SID = config['server']['sid']
NAME = config['bot']['name']
logging.info("Connecting to query interface ...")
URI = f"telnet://{USER}:{PASS}#{HOST}:{PORT}"
try:
with ts3.query.TS3ServerConnection(URI) as ts3conn:
ts3conn.exec_("use", sid=SID)
ts3conn.query("clientupdate", client_nickname="x123d")
logging.info("Connected!")
lock = threading.Lock()
notify_thread = threading.Thread(target=notify_bot, args=(ts3conn, config, lock), daemon=True,
name="notify")
keep_alive_thread = threading.Thread(target=keep_alive, args=(ts3conn, lock), daemon=True,
name="keep_alive")
notify_thread.start()
keep_alive_thread.start()
keep_alive_thread.join()
notify_thread.join()
except KeyboardInterrupt:
logging.INFO(60 * "=")
logging.info("TS Bot terminated by user!")
logging.INFO(60 * "=")
After run work for 1 person who join server and do nothing, dont send keep alive and dont work at all
you can use Bibio TIME
You can check it from official python website (https://docs.python.org/3/library/time.html)
Personally, for simple things, I find the _thread library easier. Here's a function that you can run in a thread, and an example of starting that thread:
import _thread
def mythread(arg1):
while True:
time.sleep(arg1)
do.whatever()
_thread.start_new_thread(mythread, (5,))
The important thing to note is the second argument I passed to the _thread.start_new_thread function. It must be a tuple, which is why there is a comma after the 5. Even if your function doesn't require any arguments, you have to pass a tuple.
I am using time module and threading,
I'v made some changes and it seems to work
#!/usr/bin/env python3
import configparser
import logging
import threading
import time
import ts3
logging.basicConfig(filename='ts3bot.log',
level=logging.INFO,
format="%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s] %(message)s",
)
logging.getLogger().addHandler(logging.StreamHandler())
def notify_bot(ts3conn):
logging.info("Start Notify Bot ...")
ts3conn.exec_("servernotifyregister", event="server")
while True:
event = ts3conn.wait_for_event()
try:
reasonid_ = event[0]["reasonid"]
except KeyError:
continue
if reasonid_ == "0":
logging.info("User joined Lobby:")
logging.info(event[0])
servergroups = event[0]['client_servergroups']
guestname = event[0]['client_nickname']
if not set(servergroups):
print(f"s1 {guestname}")
else:
print(f"s2{guestname}")
return None
def keep_alive(ts3conn, time):
while True:
logging.info("Send keep alive!")
ts3conn.send_keepalive()
time.sleep(20)
if __name__ == "__main__":
logging.info("Start TS Bot ...")
config = configparser.ConfigParser()
config.sections()
config.read("settings_test.ini")
logging.info("Config loaded!")
HOST = config['server']['url']
PORT = config['server']['query_port']
USER = config['server']['query_user']
PASS = config['server']['query_pw']
SID = config['server']['sid']
NAME = config['bot']['name']
logging.info("Connecting to query interface ...")
URI = f"telnet://{USER}:{PASS}#{HOST}:{PORT}"
try:
with ts3.query.TS3ServerConnection(URI) as ts3conn:
ts3conn.exec_("use", sid=SID)
ts3conn.query("clientupdate", client_nickname="x123d")
logging.info("Connected!")
notify_thread = threading.Thread(target=notify_bot, args=(ts3conn,), daemon=True,
name="notify")
keep_alive_thread = threading.Thread(target=keep_alive, args=(ts3conn, time), daemon=True,
name="keep_alive")
notify_thread.start()
keep_alive_thread.start()
keep_alive_thread.join()
notify_thread.join()
except KeyboardInterrupt:
logging.INFO(60 * "=")
logging.info("TS Bot terminated by user!")
logging.INFO(60 * "=")
It looks like ts3conn.send_keepalive() making error, when I delete it, code work fine, when I'v add it, code stop working after send ts3conn.send_keepalive() once
I'm using the recipe directly from the cookbook, with the exception of a single line {ctrl-f for "print(rd,"} printing whatever pumps out of the socket:
import pickle
import logging
import logging.handlers
import socketserver
import struct
class LogRecordStreamHandler(socketserver.StreamRequestHandler):
"""Handler for a streaming logging request.
This basically logs the record using whatever logging policy is
configured locally.
"""
def handle(self):
"""
Handle multiple requests - each expected to be a 4-byte length,
followed by the LogRecord in pickle format. Logs the record
according to whatever policy is configured locally.
"""
while True:
chunk = self.connection.recv(4)
if len(chunk) < 4:
break
slen = struct.unpack('>L', chunk)[0]
chunk = self.connection.recv(slen)
while len(chunk) < slen:
chunk = chunk + self.connection.recv(slen - len(chunk))
obj = self.unPickle(chunk)
record = logging.makeLogRecord(obj)
self.handleLogRecord(record)
def unPickle(self, data):
return pickle.loads(data)
def handleLogRecord(self, record):
# if a name is specified, we use the named logger rather than the one
# implied by the record.
if self.server.logname is not None:
name = self.server.logname
else:
name = record.name
logger = logging.getLogger(name)
# N.B. EVERY record gets logged. This is because Logger.handle
# is normally called AFTER logger-level filtering. If you want
# to do filtering, do it at the client end to save wasting
# cycles and network bandwidth!
logger.handle(record)
class LogRecordSocketReceiver(socketserver.ThreadingTCPServer):
"""
Simple TCP socket-based logging receiver suitable for testing.
"""
allow_reuse_address = True
def __init__(self, host='localhost',
port=logging.handlers.DEFAULT_TCP_LOGGING_PORT,
handler=LogRecordStreamHandler):
socketserver.ThreadingTCPServer.__init__(self, (host, port), handler)
self.abort = 0
self.timeout = 1
self.logname = None
def serve_until_stopped(self):
import select
abort = 0
while not abort:
rd, wr, ex = select.select([self.socket.fileno()],
[], [],
self.timeout)
print(rd, wr, ex)
if rd:
self.handle_request()
abort = self.abort
def main():
logging.basicConfig(
format='%(relativeCreated)5d %(name)-15s %(levelname)-8s %(message)s')
tcpserver = LogRecordSocketReceiver()
print('About to start TCP server...')
tcpserver.serve_until_stopped()
if __name__ == '__main__':
main()
To test, I use the recipe directly from the cookbook:
import logging, logging.handlers
rootLogger = logging.getLogger('')
rootLogger.setLevel(logging.DEBUG)
socketHandler = logging.handlers.SocketHandler('localhost',
logging.handlers.DEFAULT_TCP_LOGGING_PORT)
# don't bother with a formatter, since a socket handler sends the event as
# an unformatted pickle
rootLogger.addHandler(socketHandler)
# Now, we can log to the root logger, or any other logger. First the root...
logging.info('Jackdaws love my big sphinx of quartz.')
But the thing just pumps out:
About to start TCP server...
[] [] []
[] [] []
[] [] []
[] [] []
....
Ad infinitum.
However, if I start up a basic socket client in another terminal:
import socket
import logging.handlers
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("localhost",logging.handlers.DEFAULT_TCP_LOGGING_PORT))
s.send('blah'.encode())
I get:
....
[] [] []
[] [] []
[] [] []
[3] [] []
[] [] []
....
So the error is in the logging.handlers.SocketHandler, but it doesn't give any errors or anything, it just doesn't work.
Any thoughts?
UPDATE:
This seemed like it might be helpful:
>>> socketHandler.__dict__
{'filters': [], '_name': None, 'level': 0, 'formatter': None, 'lock': <unlocked _thread.RLock object owner=0 count=0 at 0x7f6094d20330>, 'host': 'localhost', 'port': 9020, 'address': ('localhost', 9020), 'sock': None, 'closeOnError': False, 'retryTime': 1531926559.6880224, 'retryStart': 1.0, 'retryMax': 30.0, 'retryFactor': 2.0, 'retryPeriod': 8.0}
UPDATE:
It looks like it might be an IPV6 vs IPV4 problem. Specifically, SocketHandler.makeSocket() calls socket.create_connection, which in turn calls socket.getaddrinfo. On my machine, socket.getaddrinfo only returns IPV6 stuff, and coincidentally, socket.create_connection goes on to throw a ConnectionRefused error when it gets to the .connect call.
Still only 85% sure this is an ipv6 vs ipv4 problem, but circumventing the call to socket.create_connection solves the problem for me. Here's a clean way to do it:
from logging import *
import logging.handlers
import socket
class SocketHandler(logging.handlers.SocketHandler):
def makeSocket(self, timeout=1):
result = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
result.connect(self.address)
except OSError:
result.close()
raise
return result
rootLogger = getLogger('')
rootLogger.setLevel(DEBUG)
socketHandler = SocketHandler('localhost',
logging.handlers.DEFAULT_TCP_LOGGING_PORT)
rootLogger.addHandler(socketHandler)
I have a logger that has a RotatingFileHandler.
I want to redirect all Stdout and Stderr to the logger.
How to do so?
Not enough rep to comment, but I wanted to add the version of this that worked for me in case others are in a similar situation.
class LoggerWriter:
def __init__(self, level):
# self.level is really like using log.debug(message)
# at least in my case
self.level = level
def write(self, message):
# if statement reduces the amount of newlines that are
# printed to the logger
if message != '\n':
self.level(message)
def flush(self):
# create a flush method so things can be flushed when
# the system wants to. Not sure if simply 'printing'
# sys.stderr is the correct way to do it, but it seemed
# to work properly for me.
self.level(sys.stderr)
and this would look something like:
log = logging.getLogger('foobar')
sys.stdout = LoggerWriter(log.debug)
sys.stderr = LoggerWriter(log.warning)
UPDATE for Python 3:
Including a dummy flush function which prevents an error where the function is expected (Python 2 was fine with just linebuf='').
Note that your output (and log level) appears different if it is logged from an interpreter session vs being run from a file. Running from a file produces the expected behavior (and output featured below).
We still eliminate extra newlines which other solutions do not.
class StreamToLogger(object):
"""
Fake file-like stream object that redirects writes to a logger instance.
"""
def __init__(self, logger, level):
self.logger = logger
self.level = level
self.linebuf = ''
def write(self, buf):
for line in buf.rstrip().splitlines():
self.logger.log(self.level, line.rstrip())
def flush(self):
pass
Then test with something like:
import StreamToLogger
import sys
import logging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename='out.log',
filemode='a'
)
log = logging.getLogger('foobar')
sys.stdout = StreamToLogger(log,logging.INFO)
sys.stderr = StreamToLogger(log,logging.ERROR)
print('Test to standard out')
raise Exception('Test to standard error')
See below for old Python 2.x answer and the example output:
All of the prior answers seem to have problems adding extra newlines where they aren't needed. The solution that works best for me is from http://www.electricmonk.nl/log/2011/08/14/redirect-stdout-and-stderr-to-a-logger-in-python/, where he demonstrates how send both stdout and stderr to the logger:
import logging
import sys
class StreamToLogger(object):
"""
Fake file-like stream object that redirects writes to a logger instance.
"""
def __init__(self, logger, log_level=logging.INFO):
self.logger = logger
self.log_level = log_level
self.linebuf = ''
def write(self, buf):
for line in buf.rstrip().splitlines():
self.logger.log(self.log_level, line.rstrip())
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename="out.log",
filemode='a'
)
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl
stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl
print "Test to standard out"
raise Exception('Test to standard error')
The output looks like:
2011-08-14 14:46:20,573:INFO:STDOUT:Test to standard out
2011-08-14 14:46:20,573:ERROR:STDERR:Traceback (most recent call last):
2011-08-14 14:46:20,574:ERROR:STDERR: File "redirect.py", line 33, in
2011-08-14 14:46:20,574:ERROR:STDERR:raise Exception('Test to standard error')
2011-08-14 14:46:20,574:ERROR:STDERR:Exception
2011-08-14 14:46:20,574:ERROR:STDERR::
2011-08-14 14:46:20,574:ERROR:STDERR:Test to standard error
Note that self.linebuf = '' is where the flush is being handled, rather than implementing a flush function.
If it's an all-Python system (i.e. no C libraries writing to fds directly, as Ignacio Vazquez-Abrams asked about) then you might be able to use an approach as suggested here:
class LoggerWriter:
def __init__(self, logger, level):
self.logger = logger
self.level = level
def write(self, message):
if message != '\n':
self.logger.log(self.level, message)
and then set sys.stdout and sys.stderr to LoggerWriter instances.
You can use redirect_stdout context manager:
import logging
from contextlib import redirect_stdout
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.write = lambda msg: logging.info(msg) if msg != '\n' else None
with redirect_stdout(logging):
print('Test')
or like this
import logging
from contextlib import redirect_stdout
logger = logging.getLogger('Meow')
logger.setLevel(logging.INFO)
formatter = logging.Formatter(
fmt='[{name}] {asctime} {levelname}: {message}',
datefmt='%m/%d/%Y %H:%M:%S',
style='{'
)
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.write = lambda msg: logger.info(msg) if msg != '\n' else None
with redirect_stdout(logger):
print('Test')
Output Redirection Done Right!
The Problem
logger.log and the other functions (.info/.error/etc.) output each call as a separate line, i.e. implicitly add (formatting and) a newline to it.
sys.stderr.write on the other hand just writes its literal input to stream, including partial lines. For example: The output "ZeroDivisionError: division by zero" is actually 4(!) separate calls to sys.stderr.write:
sys.stderr.write('ZeroDivisionError')
sys.stderr.write(': ')
sys.stderr.write('division by zero')
sys.stderr.write('\n')
The 4 most upvoted approaches (1, 2, 3, 4) thus result in extra newlines -- simply put "1/0" into your program and you will get the following:
2021-02-17 13:10:40,814 - ERROR - ZeroDivisionError
2021-02-17 13:10:40,814 - ERROR - :
2021-02-17 13:10:40,814 - ERROR - division by zero
The Solution
Store the intermediate writes in a buffer. The reason I am using a list as buffer rather than a string is to avoid the Shlemiel the painter’s algorithm. TLDR: It is O(n) instead of potentially O(n^2)
class LoggerWriter:
def __init__(self, logfct):
self.logfct = logfct
self.buf = []
def write(self, msg):
if msg.endswith('\n'):
self.buf.append(msg.removesuffix('\n'))
self.logfct(''.join(self.buf))
self.buf = []
else:
self.buf.append(msg)
def flush(self):
pass
# To access the original stdout/stderr, use sys.__stdout__/sys.__stderr__
sys.stdout = LoggerWriter(logger.info)
sys.stderr = LoggerWriter(logger.error)
2021-02-17 13:15:22,956 - ERROR - ZeroDivisionError: division by zero
For versions below Python 3.9, you could replace replace msg.removesuffix('\n') with either msg.rstrip('\n') or msg[:-1].
As an evolution to Cameron Gagnon's response, I've improved the LoggerWriterclass to:
class LoggerWriter(object):
def __init__(self, writer):
self._writer = writer
self._msg = ''
def write(self, message):
self._msg = self._msg + message
while '\n' in self._msg:
pos = self._msg.find('\n')
self._writer(self._msg[:pos])
self._msg = self._msg[pos+1:]
def flush(self):
if self._msg != '':
self._writer(self._msg)
self._msg = ''
now uncontrolled exceptions look nicer:
2018-07-31 13:20:37,482 - ERROR - Traceback (most recent call last):
2018-07-31 13:20:37,483 - ERROR - File "mf32.py", line 317, in <module>
2018-07-31 13:20:37,485 - ERROR - main()
2018-07-31 13:20:37,486 - ERROR - File "mf32.py", line 289, in main
2018-07-31 13:20:37,488 - ERROR - int('')
2018-07-31 13:20:37,489 - ERROR - ValueError: invalid literal for int() with base 10: ''
With flush added to Vinay Sajip's answer:
class LoggerWriter:
def __init__(self, logger, level):
self.logger = logger
self.level = level
def write(self, message):
if message != '\n':
self.logger.log(self.level, message)
def flush(self):
pass
Quick but Fragile One-Liner
sys.stdout.write = logger.info
sys.stderr.write = logger.error
What this does is simply assign the logger functions to the stdout/stderr .write call which means any write call will instead invoke the logger functions.
The downside of this approach is that both calls to .write and the logger functions typically add a newline so you will end up with extra lines in your log file, which may or may not be a problem depending on your use case.
Another pitfall is that if your logger writes to stderr itself we get infinite recursion (a stack overflow error). So only output to a file.
Solving problem where StreamHandler causes infinite Recurison
My logger was causing an infinite recursion, because the Streamhandler was trying to write to stdout, which itself is a logger -> leading to infinite recursion.
Solution
Reinstate the original sys.__stdout__ for the StreamHandler ONLY, so that you can still see the logs showing in the terminal.
class DefaultStreamHandler(logging.StreamHandler):
def __init__(self, stream=sys.__stdout__):
# Use the original sys.__stdout__ to write to stdout
# for this handler, as sys.stdout will write out to logger.
super().__init__(stream)
class LoggerWriter(io.IOBase):
"""Class to replace the stderr/stdout calls to a logger"""
def __init__(self, logger_name: str, log_level: int):
""":param logger_name: Name to give the logger (e.g. 'stderr')
:param log_level: The log level, e.g. logging.DEBUG / logging.INFO that
the MESSAGES should be logged at.
"""
self.std_logger = logging.getLogger(logger_name)
# Get the "root" logger from by its name (i.e. from a config dict or at the bottom of this file)
# We will use this to create a copy of all its settings, except the name
app_logger = logging.getLogger("myAppsLogger")
[self.std_logger.addHandler(handler) for handler in app_logger.handlers]
self.std_logger.setLevel(app_logger.level) # the minimum lvl msgs will show at
self.level = log_level # the level msgs will be logged at
self.buffer = []
def write(self, msg: str):
"""Stdout/stderr logs one line at a time, rather than 1 message at a time.
Use this function to aggregate multi-line messages into 1 log call."""
msg = msg.decode() if issubclass(type(msg), bytes) else msg
if not msg.endswith("\n"):
return self.buffer.append(msg)
self.buffer.append(msg.rstrip("\n"))
message = "".join(self.buffer)
self.std_logger.log(self.level, message)
self.buffer = []
def replace_stderr_and_stdout_with_logger():
"""Replaces calls to sys.stderr -> logger.info & sys.stdout -> logger.error"""
# To access the original stdout/stderr, use sys.__stdout__/sys.__stderr__
sys.stdout = LoggerWriter("stdout", logging.INFO)
sys.stderr = LoggerWriter("stderr", logging.ERROR)
if __name__ == __main__():
# Load the logger & handlers
logger = logging.getLogger("myAppsLogger")
logger.setLevel(logging.DEBUG)
# HANDLER = logging.StreamHandler()
HANDLER = DefaultStreamHandler() # <--- replace the normal streamhandler with this
logger.addHandler(HANDLER)
logFormatter = logging.Formatter("[%(asctime)s] - %(name)s - %(levelname)s - %(message)s")
HANDLER.setFormatter(logFormatter)
# Run this AFTER you load the logger
replace_stderr_and_stdout_with_logger()
And then finally call the replace_stderr_and_stdout_with_logger() after you've initialised your logger (the last bit of the code)
If you want to logging info and error messages into separates stream (info into stdout, errors into stderr) you can use this trick:
class ErrorStreamHandler(log.StreamHandler):
"""Print input log-message into stderr, print only error/warning messages"""
def __init__(self, stream=sys.stderr):
log.Handler.__init__(self, log.WARNING)
self.stream = stream
def emit(self, record):
try:
if record.levelno in (log.INFO, log.DEBUG, log.NOTSET):
return
msg = self.format(record)
stream = self.stream
# issue 35046: merged two stream.writes into one.
stream.write(msg + self.terminator)
self.flush()
except RecursionError: # See issue 36272
raise
except Exception:
self.handleError(record)
class OutStreamHandler(log.StreamHandler):
"""Print input log-message into stdout, print only info/debug messages"""
def __init__(self, loglevel, stream=sys.stdout):
log.Handler.__init__(self, loglevel)
self.stream = stream
def emit(self, record):
try:
if record.levelno not in (log.INFO, log.DEBUG, log.NOTSET):
return
msg = self.format(record)
stream = self.stream
# issue 35046: merged two stream.writes into one.
stream.write(msg + self.terminator)
self.flush()
except RecursionError: # See issue 36272
raise
except Exception:
self.handleError(record)
Usage:
log.basicConfig(level=settings.get_loglevel(),
format="[%(asctime)s] %(levelname)s: %(message)s",
datefmt='%Y/%m/%d %H:%M:%S', handlers=[ErrorStreamHandler(), OutStreamHandler(settings.get_loglevel())])
I have been working with pyinotify and I am having issues with it where after multiple changes to a folder it simply stops receiving notifications. I have a feeling it is something to do with the the fact that two threads are running; namely the notifier thread and the wxpython thread.
The purpose of the app is to essentially load a picture to the screen when it detects an ip connection, monitor a folder for the file 'Checklist' and based on that file do some processing i.e move files around.
It works intermittently but being a python newbie Im not exactly sure what the issue might be as I have basically taken the threaded example and worked around it. Sometimes, it only gets one notification and stops receiving file change notifications.
Additionally, if I restart the linux box and try again, it works for a good number of file changes and then stops receiving notifications again which makes me think that perhaps its not releasing the watches properly?
Any help would be greatly appreciated as well as optimizations and improvements are very welcome. I'm sure I could learn a lot from the feedback. The code is below
import pyinotify
import os.path
import shutil
import errno
import subprocess
import logging
import wx
import time
import signal
import sys
#update CHECKLIST name
CHECKLIST = 'Checklist' #this must exist in the update archive
#static values
DIR_UPDATE = 'd'
FILE_UPDATE = 'f'
PING_IP = ' localhost' # change in production
#paths
WATCH_PATH = '/home/test'
LOG_PATH = '/home/test/update.log'
CONNECTED_IMG = 'update.jpg'
UPDATING_IMG = 'updating.png'
#msgs
UPDATEFOUND_MSG = ' Update Found '
UPDATEUNZIPPEDSTART_MSG = ' Update unzipping '
UPDATEUNZIPPED_MSG = ' Update unzipped '
UPDATEFILE_MSG = ' Update file '
UPDATEFILEMSG_CONT = ' moved into path '
REMOVEFILEMSG_CONT = ' removed from update folder '
UPDATECOMPLETE_CONT = ' Update complete'
ROADANGELRESTART_MSG = ' Update restarting app '
DIRCREATED_MSG = ' Directory created at path '
#errors
UPDATEFAILED_MSG = ' Update process failed on '
BADLYFORMED_MSG = ' Badly formed src/dest combination '
UPDATESRCUNAVAILABLE = ' Invalid update file specified '
UPDATEDESTUNAVAILABLE = ' Invalid update destination specified '
INVALIDUPDATEFORMAT = ' Invalid format string '
#on startup create the watchfolder if it doesnt exist
WM = pyinotify.WatchManager() # Watch Manager
WM_MASK = pyinotify.IN_CLOSE_WRITE # watched events
#setup logger
LOGGER = logging.getLogger('Updater')
LOG_HANDLE = logging.FileHandler(LOG_PATH)
FORMATTER = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
LOG_HANDLE.setFormatter(FORMATTER)
LOGGER.addHandler(LOG_HANDLE)
LOGGER.setLevel(logging.INFO)
#Global values used primarily in the main function loop
HANDLER = None
NOTIFIER = None
WDD = None
UPDATE_UI = None
WINDOW = None
CURR_IMG = None
LAST_CURRIMG = None
class EventHandler(pyinotify.ProcessEvent):
VERBOSE = False
""" Main class to monitor file events and process accordingly"""
def process_IN_CLOSE_WRITE(self, event):
""" Only executes when a Checklist has finished being written to"""
path = event.pathname
print 'evt'
#look for the update_ready file before processing
if (os.path.basename(path) == 'Checklist'):
EventHandler.parse_updates(WATCH_PATH)
global CURR_IMG
CURR_IMG = os.path.join(WATCH_PATH, UPDATING_IMG)
show_window()
print 'update completed'
time.sleep(1000)
#classmethod
def parse_updates(cls, path):
""" parses update files """
#handle errors for opening the file
file_path = os.path.join(path, CHECKLIST)
print file_path
files = open(file_path)
#handle errors for malformed tuples-done
#handle errors for unavailable files-done
#handle errors for unavailable dests-done #created automatically
#handle permission errors
for line in files:
#remove linebreaks etc and ensure its not empty
if line.strip():
array = line.split('=')
length = len(array)
if length == 3:
EventHandler.process_line(path, array)
else:
if length > 0:
EventHandler.print_bad_msg(array[0])
else:
EventHandler.print_bad_msg()
print 'removing ', file_path
os.remove(file_path) #remove the checklist file
#classmethod
def mkdir(cls, path):
""" makes a directory from a path"""
try:
os.mkdir(path)
print DIRCREATED_MSG, path
except OSError, err:
print err
if err.errno != errno.EEXIST: #thrown when the dir already exists
return False
#classmethod
def move_file(cls, src, dest):
""" moves a file from src to dest and remove after
expects that the dest already exists at this point
otherwise ignores the move"""
#print 'moving from', src, 'to ', dest
if os.path.isfile(dest):
shutil.copy2(src, dest)
else:
print UPDATEDESTUNAVAILABLE
#remove the src file when done
os.remove(src)
#classmethod
def process_line(cls, path, array):
""" process a line from the checklist"""
#remove newlines etc
update_file = array[0].strip()
update_src = os.path.join(path, update_file)
update_dest = array[1].strip()
update_type = array[2].strip()
#ensure we have valid values in all three fields
if update_file and update_dest and update_type:
#ensure the src file exists
if os.path.isfile(update_src):
#check if destination is directory and
#copy the file into the directory
if update_type == DIR_UPDATE:
EventHandler.mkdir(update_dest)
dest = os.path.join(update_dest, update_file)
EventHandler.move_file(update_src, dest)
else:
EventHandler.move_file(update_src, update_dest)
else:
print UPDATESRCUNAVAILABLE
else:
print INVALIDUPDATEFORMAT
#classmethod
def print_bad_msg(cls, msg = ''):
""" print a badly formed message with optional value"""
if msg:
print BADLYFORMED_MSG, msg
else:
print BADLYFORMED_MSG
class UpdateFrame(wx.Frame):
""" Displays update images to screen"""
def __init__(self, path):
wx.Frame.__init__(self, None, wx.ID_ANY)
image_file = path
image = wx.Bitmap(image_file)
image_size = image.GetSize()
# set the frame size to fit the screen size
self.SetClientSize(wx.DisplaySize())
# bitmap's upper left corner is in frame position (x, y)
# by default pos=(0, 0)
wx.StaticBitmap(self, wx.ID_ANY, image, size = image_size)
# the parent is the frame
self.SetTitle('Update Mode')
def ping_ip():
""" ping once to establish connection """
ret = subprocess.call("ping -c 1 %s" % PING_IP,
shell=True,
stdout=open('/dev/null', 'w'),
stderr=subprocess.STDOUT)
if ret == 0:
return True
else:
return False
def show_window():
""" update screen window when currimage changes is set """
global UPDATE_UI
global WINDOW
global CURR_IMG
global LAST_CURRIMG
if LAST_CURRIMG != CURR_IMG:
if not UPDATE_UI:
UPDATE_UI = wx.App()
if not WINDOW:
WINDOW = UpdateFrame(CURR_IMG)
UPDATE_UI.ExitMainLoop()
while(UPDATE_UI.IsMainLoopRunning()):
pass
WINDOW.Destroy()
WINDOW = UpdateFrame(CURR_IMG)
WINDOW.Show(True)
UPDATE_UI.MainLoop()
LAST_CURRIMG = CURR_IMG
print 'changed'
def in_updatemode():
return ping_ip()
while True:
try:
if not in_updatemode():
print 'waiting for connect'
time.sleep(3)
if NOTIFIER:
NOTIFIER.stop()
else:
if not HANDLER:
HANDLER = EventHandler()
if not NOTIFIER:
NOTIFIER = pyinotify.ThreadedNotifier(WM, HANDLER)
NOTIFIER.start()
if not WDD:
WDD = WM.add_watch(WATCH_PATH, WM_MASK, rec=True,quiet=False)
# ip is active so show the image and start the notifier
# state = ip active
CURR_IMG = os.path.join(WATCH_PATH, CONNECTED_IMG)
show_window()
print 'here'
except KeyboardInterrupt:
print 'out'
NOTIFIER.stop()
break
I basically found out that that the issue was indeed the fact that pyinotify's threaded notifier and wxPython's main loop dont play nice together.
The solution was to create a custom main loop (something I didn't know you could do in the first place) and place the pyinotify's threaded notifier within that loop. That way it ran as part of the wxPython main loop.
I got the idea from
http://www.java2s.com/Open-Source/Python/GUI/wxPython/wxPython-src-2.8.11.0/wxPython/samples/mainloop/mainloop.py.htm
Code below explains the concept
class CustomApp(wx.App):
def MainLoop(self):
global HANDLER
global WM
global NOTIFIER
global WDD
global UPDATE_UI
global PING_TIMER
# Create an event loop and make it active. If you are
# only going to temporarily have a nested event loop then
# you should get a reference to the old one and set it as
# the active event loop when you are done with this one...
evtloop = wx.EventLoop()
old = wx.EventLoop.GetActive()
wx.EventLoop.SetActive(evtloop)
# This outer loop determines when to exit the application,
# for this example we let the main frame reset this flag
# when it closes.
while self.keepGoing:
# At this point in the outer loop you could do
# whatever you implemented your own MainLoop for. It
# should be quick and non-blocking, otherwise your GUI
# will freeze.
# call_your_code_here()
if not HANDLER:
HANDLER = EventHandler()
if not WM:
WM = pyinotify.WatchManager() # Watch Manager
if not NOTIFIER:
NOTIFIER = pyinotify.ThreadedNotifier(WM, HANDLER)
NOTIFIER.start()
print 'notifier started'
if not WDD:
WDD = WM.add_watch(WATCH_PATH, WM_MASK, rec=True,quiet=False)
# This inner loop will process any GUI events
# until there are no more waiting.
while evtloop.Pending():
evtloop.Dispatch()
# Send idle events to idle handlers. You may want to
# throttle this back a bit somehow so there is not too
# much CPU time spent in the idle handlers. For this
# example, I'll just snooze a little...
time.sleep(0.10)
self.ProcessIdle()
wx.EventLoop.SetActive(old)
def OnInit(self):
global UPDATE_UI
if not UPDATE_UI:
UPDATE_UI = Updater()
UPDATE_UI.Show()
self.SetTopWindow(UPDATE_UI)
self.keepGoing = True
return True
"--------------------------------------------------------------------------------------"
Watcher()
app = CustomApp(False)
Additionally, I wanted to catch SIGINT in the program and I solved it using the recipe from this url
http://code.activestate.com/recipes/496735-workaround-for-missed-sigint-in-multithreaded-prog/
Hope this helps another python newbie or oldie :)