I'm trying to replace an ad-hoc logging system with Python's logging module. I'm using the logging system to output progress information for a long task on a single line so you can tail the log or watch it in a console. I've done this by having a flag on my logging function which suppresses the newline for that log message and build the line piece by piece.
All the logging is done from a single thread so there's no serialisation issues.
Is it possible to do this with Python's logging module? Is it a good idea?
If you wanted to do this you can change the logging handler terminator. I'm using Python 3.4. This was introduced in Python 3.2 as stated by Ninjakannon.
handler = logging.StreamHandler()
handler.terminator = ""
When the StreamHandler writes it writes the terminator last.
Let's start with your last question: No, I do not believe it's a good idea.
IMO, it hurts the readability of the logfile in the long run.
I suggest sticking with the logging module and using the '-f' option on your 'tail' command to watch the output from the console. You will probably end up using the FileHandler. Notice that the default argument for 'delay' is False meaning the output won't be buffered.
If you really needed to suppress newlines, I would recommend creating your own Handler.
The new line, \n, is inserted inside the StreamHandler class.
If you're really set on fixing this behaviour, then here's an example of how I solved this by monkey patching the emit(self, record) method inside the logging.StreamHandler class.
A monkey patch is a way to extend or modify the run-time code of dynamic languages without altering the original source code. This process has also been termed duck punching.
Here is the custom implementation of emit() that omits line breaks:
def customEmit(self, record):
# Monkey patch Emit function to avoid new lines between records
try:
msg = self.format(record)
if not hasattr(types, "UnicodeType"): #if no unicode support...
self.stream.write(msg)
else:
try:
if getattr(self.stream, 'encoding', None) is not None:
self.stream.write(msg.encode(self.stream.encoding))
else:
self.stream.write(msg)
except UnicodeError:
self.stream.write(msg.encode("UTF-8"))
self.flush()
except (KeyboardInterrupt, SystemExit):
raise
except:
self.handleError(record)
Then you would make a custom logging class (in this case, subclassing from TimedRotatingFileHandler).
class SniffLogHandler(TimedRotatingFileHandler):
def __init__(self, filename, when, interval, backupCount=0,
encoding=None, delay=0, utc=0):
# Monkey patch 'emit' method
setattr(StreamHandler, StreamHandler.emit.__name__, customEmit)
TimedRotatingFileHandler.__init__(self, filename, when, interval,
backupCount, encoding, delay, utc)
Some people might argue that this type of solution is not Pythonic, or whatever. It might be so, so be careful.
Also, be aware that this will globally patch SteamHandler.emit(...), so if you are using multiple logging classes, then this patch will affect the other logging classes as well!
Check out these for further reading:
What is monkey-patching?
Is monkeypatching considered good programming practice?
Monkeypatching For Humans
Hope that helps.
Python 3.5.9
class MFileHandler(logging.FileHandler):
"""Handler that controls the writing of the newline character"""
special_code = '[!n]'
def emit(self, record) -> None:
if self.special_code in record.msg:
record.msg = record.msg.replace( self.special_code, '' )
self.terminator = ''
else:
self.terminator = '\n'
return super().emit(record)
Then
fHandler = MFileHandler(...)
Example:
# without \n
log.info( 'waiting...[!n]' )
...
log.info( 'OK' )
# with \n
log.info( 'waiting...' )
...
log.info( 'OK' )
log.txt:
waiting...OK
waiting...
OK
I encountered a need to log a certain section in a single line as I iterated through a tuple, but wanted to retain the overall logger.
Collected the output first into a single string and later sent it out to logger once I was out of the section. Example of concept
for fld in object._fields:
strX = (' {} --> {} ').format(fld, formattingFunction(getattr(obj,fld)))
debugLine += strX
logger.debug('{}'.format(debugLine))
Related
I've encountered a pretty standard issue and was wondering if my solution is the right way to go.
Every time an exception occures I want it to be caught and logged by the caller, and then re-raised.
Since I dont want to repeat logging messages every time, I created a custom exception which saves the message data and also logs.
class LoggingException(Exception):
def __init__(self, message, package_id):
# Get caller informat
caller = getframeinfo(stack()[2][0])
self.filename = caller.filename
self.function = caller.function
# Set message info and log
self.message = message
if (LogManager.log_handler is None):
print(message)
else:
LogManager.l(package_id, LogLevelEnum.ERROR, message)
Use case:
def main_func():
try:
secondary_func()
except Exception as ex:
raise LoggingException("Some log") from ex
def secondary_func():
raise LoggingException("Exception info")
The problem is im not completley sure having an exception do any operations is a good idea, and this to generic for it to not have a standard python solution.
Note: I am not using the python logging module due to product constraints.
Trying to get caller information like that is going to be unreliable. Also, there will be cases where it's not the immediate caller that you are interested in.
The idea of the exception logging itself seems sensible. To compromise, I would move the logging functionality into a separate method that you could trigger explicitly. Exceptions are, after all, mostly regular objects:
class LoggingException(Exception):
def __init__(self, message, package_id):
# Set message info
super.__init__(message)
self.package_id = package_id
def log(self, manager=None):
if manager.log_handler is None:
print(super().__str__())
else:
manager.l(self.package_id, LogLevelEnum.ERROR, super()..__str__())
Now you can trigger the logging operation whenever you want, without having to reconstruct the message:
try:
...
except LoggingException as e:
e.log(some_manager)
raise
This gives you the option of really re-raising the error, as shown here, or chaining it as in your example. I highly recommend against chaining unless you have a really good reason to do it.
I have this snippet of code I wrote in a cleanup method
for (key, target) in self.targets.items():
try:
logger.info('releasing %s: %s', key, target)
target.dispose() # Something bad can occur here
except:
logger.exception('uh oh, something has gone wrong')
The problem I am having is that in the target.dispose() call, something bad can happen, and can cause a crash in the interpreter. (The target object calls stuff in C and/or C#) I would like to flush the logs first, before calling target.dispose(), but python loggers don't have a flush() method; only handlers do.
Is there a way to notify the logging framework that something bad might happen next, and the handlers should be flushed now before proceeding?
Looking into the implementation in logging/__init__.py, this seems to be the correct way to iterate over the all the handlers of a logger:
def flush_logger(logger):
c = logger
while c:
for hdlr in c.handlers:
try:
hdlr.flush()
except AttributeError:
pass
if not c.propagate:
c = None #break out
else:
c = c.parent
I want to log some information of every single request sent to a busy http server in a formatted form. Using the logging module would create some thing I don't want to:
[I 131104 15:31:29 Sys:34]
I thought of the CSV format, but I don't know how to customize it. Python has the csv module, but I read in the manual
import csv
with open('some.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerows(someiterable)
Since it would open and close a file each time, I am afraid this way would slow down the whole server performance. What could I do?
Just use python's logging module.
You can adjust the output the way you want; take a look at Changing the format of displayed messages:
To change the format which is used to display messages, you need to specify the format you want to use:
import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logging.debug('This message should appear on the console')
logging.info('So should this')
logging.warning('And this, too')
and Formatters:
Formatter objects configure the final order, structure, and contents of the log message.
You'll find a list of the attribtus you can use here: LogRecord attributes.
If you want to produce a valid csv-file, use python's csv module, too.
Here's a simple example:
import logging
import csv
import io
class CsvFormatter(logging.Formatter):
def __init__(self):
super().__init__()
self.output = io.StringIO()
self.writer = csv.writer(self.output, quoting=csv.QUOTE_ALL)
def format(self, record):
self.writer.writerow([record.levelname, record.msg])
data = self.output.getvalue()
self.output.truncate(0)
self.output.seek(0)
return data.strip()
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logging.root.handlers[0].setFormatter(CsvFormatter())
logger.debug('This message should appear on the console')
logger.info('So should "this", and it\'s using quoting...')
logger.warning('And this, too')
Output:
"DEBUG","This message should appear on the console"
"INFO","So should ""this"", and it's using quoting..."
"WARNING","And this, too"
As sloth suggests, you can easily edit the delimiter of the log to a comma, thus producing a CSV file.
Working example:
import logging
# create logger
lgr = logging.getLogger('logger name')
lgr.setLevel(logging.DEBUG) # log all escalated at and above DEBUG
# add a file handler
fh = logging.FileHandler('path_of_your_log.csv')
fh.setLevel(logging.DEBUG) # ensure all messages are logged to file
# create a formatter and set the formatter for the handler.
frmt = logging.Formatter('%(asctime)s,%(name)s,%(levelname)s,%(message)s')
fh.setFormatter(frmt)
# add the Handler to the logger
lgr.addHandler(fh)
# You can now start issuing logging statements in your code
lgr.debug('a debug message')
lgr.info('an info message')
lgr.warn('A Checkout this warning.')
lgr.error('An error writen here.')
lgr.critical('Something very critical happened.')
I would agree that you should use the logging module, but you can't really do it properly with just a format string like some of the other answers show, as they do not address the situation where you log a message that contains a comma.
If you need a solution that will properly escape any special characters in the message (or other fields, I suppose), you would have to write a custom formatter and set it.
logger = logging.getLogger()
formatter = MyCsvFormatter()
handler = logging.FileHandler(filename, "w")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(level)
You'll obviously have to implement the MyCsvFormatter class, which should inherit from logging.Formatter and override the format() method
class MyCsvFormatter(logging.Formatter):
def __init__(self):
fmt = "%(levelname)s,%(message)s" # Set a format that uses commas, like the other answers
super(MyCsvFormatter, self).__init__(fmt=fmt)
def format(self, record):
msg = record.getMessage()
# convert msg to a csv compatible string using your method of choice
record.msg = msg
return super(MyCsvFormatter, self).format(self, record)
Note: I've done something like this before, but haven't tested this particular code sample
As far as doing the actual escaping of the message, here's one possible approach:
Python - write data into csv format as string (not file)
I don't think that is the best idea, but it is doable, and quite simple.
Manually buffer your log. Store log entries in some place, and write them to file from time to time.
If you know that your server will be constantly busy, flush your buffer when it reaches some size. If there may be big gaps in usage, I'd say that new thread (or better process, check yourself why threads suck and slow down apps) with endless (theoretically of course) loop of sleep/flush would be better call.
Also, remember to create some kind of hook that will flush buffer when server is interrupted or fails (maybe signals? or just try/except on main function - there are even more ways to do it), so you don't lose unflushed buffer data on unexpected exit.
I repeat - this is not the best idea, it's the first thing that came to my mind. You may want to consult logging implementations from Flask or some other webapp framework (AFAIR Flask has CSV logging too).
I use a internal library that prints a lot (one script could print 40000 lines in total), and I suppose it may have bad impact in performance. This is a library developed by another team in my company and do a lot of calculations, they print to debug errors (and I know this is not a good habit but it's too late because of 100 scripts already on production)
and I'm developing a script that uses 100 scripts to produce the result.
How can I decide to turn all this print off ?
I'm not asking how to print these lines to file, but completely omit it
Replace sys.stdout with an object that eats the output:
import sys
class Null:
def write(self, text):
pass
def flush(self):
pass
print "One" # This gets output OK
old_stdout = sys.stdout
sys.stdout = Null()
print "Two" # This disappears
sys.stdout = old_stdout
print "Three" # Output, back to normal
The best way is to simply remove the print statements as there
is no overhead whatsoever.
Alternatively, you can redirect the output to /dev/null, which will
effectively remove the I/O overhead but will not remove the syscall.
To spare the syscall you can replace sys.stdout with a Writer which does nothing.
For example:
class NullWriter():
def write(self, s): pass
sys.stdout = NullWriter()
Apparently, this has been asked and solved before. Also here.
In case you're using python 3, you can overload the print function as
seen in this answer:
def print(*args, **kwargs):
pass
I came across the Python with statement for the first time today. I've been using Python lightly for several months and didn't even know of its existence! Given its somewhat obscure status, I thought it would be worth asking:
What is the Python with statement
designed to be used for?
What do
you use it for?
Are there any
gotchas I need to be aware of, or
common anti-patterns associated with
its use? Any cases where it is better use try..finally than with?
Why isn't it used more widely?
Which standard library classes are compatible with it?
I believe this has already been answered by other users before me, so I only add it for the sake of completeness: the with statement simplifies exception handling by encapsulating common preparation and cleanup tasks in so-called context managers. More details can be found in PEP 343. For instance, the open statement is a context manager in itself, which lets you open a file, keep it open as long as the execution is in the context of the with statement where you used it, and close it as soon as you leave the context, no matter whether you have left it because of an exception or during regular control flow. The with statement can thus be used in ways similar to the RAII pattern in C++: some resource is acquired by the with statement and released when you leave the with context.
Some examples are: opening files using with open(filename) as fp:, acquiring locks using with lock: (where lock is an instance of threading.Lock). You can also construct your own context managers using the contextmanager decorator from contextlib. For instance, I often use this when I have to change the current directory temporarily and then return to where I was:
from contextlib import contextmanager
import os
#contextmanager
def working_directory(path):
current_dir = os.getcwd()
os.chdir(path)
try:
yield
finally:
os.chdir(current_dir)
with working_directory("data/stuff"):
# do something within data/stuff
# here I am back again in the original working directory
Here's another example that temporarily redirects sys.stdin, sys.stdout and sys.stderr to some other file handle and restores them later:
from contextlib import contextmanager
import sys
#contextmanager
def redirected(**kwds):
stream_names = ["stdin", "stdout", "stderr"]
old_streams = {}
try:
for sname in stream_names:
stream = kwds.get(sname, None)
if stream is not None and stream != getattr(sys, sname):
old_streams[sname] = getattr(sys, sname)
setattr(sys, sname, stream)
yield
finally:
for sname, stream in old_streams.iteritems():
setattr(sys, sname, stream)
with redirected(stdout=open("/tmp/log.txt", "w")):
# these print statements will go to /tmp/log.txt
print "Test entry 1"
print "Test entry 2"
# back to the normal stdout
print "Back to normal stdout again"
And finally, another example that creates a temporary folder and cleans it up when leaving the context:
from tempfile import mkdtemp
from shutil import rmtree
#contextmanager
def temporary_dir(*args, **kwds):
name = mkdtemp(*args, **kwds)
try:
yield name
finally:
shutil.rmtree(name)
with temporary_dir() as dirname:
# do whatever you want
I would suggest two interesting lectures:
PEP 343 The "with" Statement
Effbot Understanding Python's
"with" statement
1.
The with statement is used to wrap the execution of a block with methods defined by a context manager. This allows common try...except...finally usage patterns to be encapsulated for convenient reuse.
2.
You could do something like:
with open("foo.txt") as foo_file:
data = foo_file.read()
OR
from contextlib import nested
with nested(A(), B(), C()) as (X, Y, Z):
do_something()
OR (Python 3.1)
with open('data') as input_file, open('result', 'w') as output_file:
for line in input_file:
output_file.write(parse(line))
OR
lock = threading.Lock()
with lock:
# Critical section of code
3.
I don't see any Antipattern here.
Quoting Dive into Python:
try..finally is good. with is better.
4.
I guess it's related to programmers's habit to use try..catch..finally statement from other languages.
The Python with statement is built-in language support of the Resource Acquisition Is Initialization idiom commonly used in C++. It is intended to allow safe acquisition and release of operating system resources.
The with statement creates resources within a scope/block. You write your code using the resources within the block. When the block exits the resources are cleanly released regardless of the outcome of the code in the block (that is whether the block exits normally or because of an exception).
Many resources in the Python library that obey the protocol required by the with statement and so can used with it out-of-the-box. However anyone can make resources that can be used in a with statement by implementing the well documented protocol: PEP 0343
Use it whenever you acquire resources in your application that must be explicitly relinquished such as files, network connections, locks and the like.
Again for completeness I'll add my most useful use-case for with statements.
I do a lot of scientific computing and for some activities I need the Decimal library for arbitrary precision calculations. Some part of my code I need high precision and for most other parts I need less precision.
I set my default precision to a low number and then use with to get a more precise answer for some sections:
from decimal import localcontext
with localcontext() as ctx:
ctx.prec = 42 # Perform a high precision calculation
s = calculate_something()
s = +s # Round the final result back to the default precision
I use this a lot with the Hypergeometric Test which requires the division of large numbers resulting form factorials. When you do genomic scale calculations you have to be careful of round-off and overflow errors.
An example of an antipattern might be to use the with inside a loop when it would be more efficient to have the with outside the loop
for example
for row in lines:
with open("outfile","a") as f:
f.write(row)
vs
with open("outfile","a") as f:
for row in lines:
f.write(row)
The first way is opening and closing the file for each row which may cause performance problems compared to the second way with opens and closes the file just once.
See PEP 343 - The 'with' statement, there is an example section at the end.
... new statement "with" to the Python
language to make
it possible to factor out standard uses of try/finally statements.
points 1, 2, and 3 being reasonably well covered:
4: it is relatively new, only available in python2.6+ (or python2.5 using from __future__ import with_statement)
The with statement works with so-called context managers:
http://docs.python.org/release/2.5.2/lib/typecontextmanager.html
The idea is to simplify exception handling by doing the necessary cleanup after leaving the 'with' block. Some of the python built-ins already work as context managers.
Another example for out-of-the-box support, and one that might be a bit baffling at first when you are used to the way built-in open() behaves, are connection objects of popular database modules such as:
sqlite3
psycopg2
cx_oracle
The connection objects are context managers and as such can be used out-of-the-box in a with-statement, however when using the above note that:
When the with-block is finished, either with an exception or without, the connection is not closed. In case the with-block finishes with an exception, the transaction is rolled back, otherwise the transaction is commited.
This means that the programmer has to take care to close the connection themselves, but allows to acquire a connection, and use it in multiple with-statements, as shown in the psycopg2 docs:
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
curs.execute(SQL2)
conn.close()
In the example above, you'll note that the cursor objects of psycopg2 also are context managers. From the relevant documentation on the behavior:
When a cursor exits the with-block it is closed, releasing any resource eventually associated with it. The state of the transaction is not affected.
In python generally “with” statement is used to open a file, process the data present in the file, and also to close the file without calling a close() method. “with” statement makes the exception handling simpler by providing cleanup activities.
General form of with:
with open(“file name”, “mode”) as file_var:
processing statements
note: no need to close the file by calling close() upon file_var.close()
The answers here are great, but just to add a simple one that helped me:
with open("foo.txt") as file:
data = file.read()
open returns a file
Since 2.6 python added the methods __enter__ and __exit__ to file.
with is like a for loop that calls __enter__, runs the loop once and then calls __exit__
with works with any instance that has __enter__ and __exit__
a file is locked and not re-usable by other processes until it's closed, __exit__ closes it.
source: http://web.archive.org/web/20180310054708/http://effbot.org/zone/python-with-statement.htm