While I use logging to record exceptions, it occurred to me the methods of the logging library itself can throw exceptions.
For example, if a "bad" path is set for the log file, like
import logging
logging.basicConfig(filename='/bad/path/foo.log')
a FileNotFoundError will be thrown.
Suppose my goal of handling these logging exceptions is to keep the program running (and not exit, which the program would otherwise do). The first thing that comes to mind is
try:
logging.basicConfig(filename='/bad/path/foo.log')
except Exception:
pass
But this is considered by some to be an antipattern. It's also quite ugly to wrap every logging.error with try and except blocks.
What's a good pattern to handle exceptions thrown by logging methods like basicConfig(), and possibly by debug(), error(), etc.?
Unless you re-initialise your logger mid-way through your code, why not just check whether the file exists during logger initialisation:
import os
if os.path.isfile(filename):
# Initialise the logger
This should work normally, unless of course some part of the later code will attemp to delete the file, but I hope that it's not the case.
In my case, I am handling databases throughout the entire runtime of my program, and so I need to keep my 'cursor' open for the entire program. Is there anyway I can implement a termination protocol, so that when I terminate its execution or an error arises, I am able to run this quick piece of code that simply closes the cursor (I am using python sockets btw).
I would suspect I could do something like this:
if __name__ == "__main__":
Menu()
cursor.close()
However, the only reason that this doesn't work in my case is that Menu is simply starting up threads, and so its execution continues on, returning me back to cursor.close() whilst my program continues to run.
I'm not sure if there is a way to get around this problem.
Yes, you could use the signal library in python to achieve some of this functionality, in particular, capturing program termination as well interrupts to the program like ctrl + c. Example:
# a function to register the signal handlers
# once the program terminates or is halted by an interrupt like ctrl + c it executes the quit_gracefully function
def register_signal_handler():
signal.signal(signal.SIGINT, quit_gracefully)
signal.signal(signal.SIGTERM, quit_gracefully)
return
def quit_gracefully():
# close connections etc.
in case of a different error you could use a try-except block which handles the error and runs the quit_gracefully function in the except.
try:
# some code
except:
quit_gracefully()
EDIT:
this is a good post on signal. How do I capture SIGINT in Python?
You can also use the atexit module: https://docs.python.org/3/library/atexit.html.
Something like this:
import atexit
#atexit.register
def close_cursor():
print("Closing cursor before exiting.")
cursor.close()
I have the following function,
def load():
with open(PATH_CONFIG, 'r') as file:
return json.loads(file.read())
Will there be a file.close() called? I know that the with keyword normally calls the close() method for the file at the end of the indented block, but at the same time the return keyword means that the rest of the function does not run.
Just like try/finally, anything that exits the with block (return, break/continue that affects a loop surrounding it, exception thrown, sys.exit called, etc.) will perform appropriate cleanup as execution bubbles out of the with block.
The only exceptions are:
When there are actual bugs (in the interpreter, or in misuse of intrinsically dangerous tools like ctypes) where the interpreter itself crashes or otherwise exits "forcefully" (e.g. due to a segfault)
Calling os._exit bypasses all cleanup procedures (that's why it should never be used in anything but forked worker processes)
Return exits the with block like a normal dedent
Yes.
If an exception is raised, a context manager has the option of changing its behavior, but there's no difference between a return and falling off the end of the statement body, and with few exceptions most context managers will perform their cleanup and allow the exception to propagate.
The idea is that it's comparable to a finally statement, and will be executed no matter how the block is exited. The contextmanager class from the standard library makes this analogy concrete.
from contextlib import contextmanager
#contextmanager
def example():
print('enter')
try:
yield
print('unexceptional return')
finally:
print('exit')
We can use with example(): in various ways to see how the with statement performs in a more visible example than closing a file.
All the docs tell us is,
Raised when the user hits the interrupt key (normally Control-C or Delete). During execution, a check for interrupts is made regularly.
But from the point of the code, when can I see this exception? Does it occur during statement execution? Only between statements? Can it happen in the middle of an expression?
For example:
file_ = open('foo')
# <-- can a KeyboardInterrupt be raised here, after the successful
# completion of open but prior to the try? -->
try:
# try some things with file_
finally:
# cleanup
Will this code leak during a well-timed KeyboardInterrupt? Or is it raised during the execution of some statements or expressions?
According to a note in the unrelated PEP 343:
Even if you write bug-free code, a KeyboardInterrupt exception can still cause it to exit between any two virtual machine opcodes.
So it can occur essentially anywhere. It can indeed occur during evaluation of a single expression. (This shouldn't be surprising, since an expression can include function calls, and pretty much anything can happen inside a function call.)
Yes, a KeyboardInterrupt can occur in the place you marked.
To deal with this, you should use a with block:
with open('foo') as file_:
# do some things
raise KeyboardInterrupt
# file resource is closed no matter what, even if a KeyboardInterrupt is raised
However, the exception could occur even between the open() call and the assignment to file_. It's probably not worth worrying about this, because usually a ctrl-c will mean your program is about to end, so the "leaked" file handle will be cleaned up by the OS. But if you know that it is important, you can use a signal handler to catch the signal that raises KeyboardInterrupt (SIGINT).
I came across the Python with statement for the first time today. I've been using Python lightly for several months and didn't even know of its existence! Given its somewhat obscure status, I thought it would be worth asking:
What is the Python with statement
designed to be used for?
What do
you use it for?
Are there any
gotchas I need to be aware of, or
common anti-patterns associated with
its use? Any cases where it is better use try..finally than with?
Why isn't it used more widely?
Which standard library classes are compatible with it?
I believe this has already been answered by other users before me, so I only add it for the sake of completeness: the with statement simplifies exception handling by encapsulating common preparation and cleanup tasks in so-called context managers. More details can be found in PEP 343. For instance, the open statement is a context manager in itself, which lets you open a file, keep it open as long as the execution is in the context of the with statement where you used it, and close it as soon as you leave the context, no matter whether you have left it because of an exception or during regular control flow. The with statement can thus be used in ways similar to the RAII pattern in C++: some resource is acquired by the with statement and released when you leave the with context.
Some examples are: opening files using with open(filename) as fp:, acquiring locks using with lock: (where lock is an instance of threading.Lock). You can also construct your own context managers using the contextmanager decorator from contextlib. For instance, I often use this when I have to change the current directory temporarily and then return to where I was:
from contextlib import contextmanager
import os
#contextmanager
def working_directory(path):
current_dir = os.getcwd()
os.chdir(path)
try:
yield
finally:
os.chdir(current_dir)
with working_directory("data/stuff"):
# do something within data/stuff
# here I am back again in the original working directory
Here's another example that temporarily redirects sys.stdin, sys.stdout and sys.stderr to some other file handle and restores them later:
from contextlib import contextmanager
import sys
#contextmanager
def redirected(**kwds):
stream_names = ["stdin", "stdout", "stderr"]
old_streams = {}
try:
for sname in stream_names:
stream = kwds.get(sname, None)
if stream is not None and stream != getattr(sys, sname):
old_streams[sname] = getattr(sys, sname)
setattr(sys, sname, stream)
yield
finally:
for sname, stream in old_streams.iteritems():
setattr(sys, sname, stream)
with redirected(stdout=open("/tmp/log.txt", "w")):
# these print statements will go to /tmp/log.txt
print "Test entry 1"
print "Test entry 2"
# back to the normal stdout
print "Back to normal stdout again"
And finally, another example that creates a temporary folder and cleans it up when leaving the context:
from tempfile import mkdtemp
from shutil import rmtree
#contextmanager
def temporary_dir(*args, **kwds):
name = mkdtemp(*args, **kwds)
try:
yield name
finally:
shutil.rmtree(name)
with temporary_dir() as dirname:
# do whatever you want
I would suggest two interesting lectures:
PEP 343 The "with" Statement
Effbot Understanding Python's
"with" statement
1.
The with statement is used to wrap the execution of a block with methods defined by a context manager. This allows common try...except...finally usage patterns to be encapsulated for convenient reuse.
2.
You could do something like:
with open("foo.txt") as foo_file:
data = foo_file.read()
OR
from contextlib import nested
with nested(A(), B(), C()) as (X, Y, Z):
do_something()
OR (Python 3.1)
with open('data') as input_file, open('result', 'w') as output_file:
for line in input_file:
output_file.write(parse(line))
OR
lock = threading.Lock()
with lock:
# Critical section of code
3.
I don't see any Antipattern here.
Quoting Dive into Python:
try..finally is good. with is better.
4.
I guess it's related to programmers's habit to use try..catch..finally statement from other languages.
The Python with statement is built-in language support of the Resource Acquisition Is Initialization idiom commonly used in C++. It is intended to allow safe acquisition and release of operating system resources.
The with statement creates resources within a scope/block. You write your code using the resources within the block. When the block exits the resources are cleanly released regardless of the outcome of the code in the block (that is whether the block exits normally or because of an exception).
Many resources in the Python library that obey the protocol required by the with statement and so can used with it out-of-the-box. However anyone can make resources that can be used in a with statement by implementing the well documented protocol: PEP 0343
Use it whenever you acquire resources in your application that must be explicitly relinquished such as files, network connections, locks and the like.
Again for completeness I'll add my most useful use-case for with statements.
I do a lot of scientific computing and for some activities I need the Decimal library for arbitrary precision calculations. Some part of my code I need high precision and for most other parts I need less precision.
I set my default precision to a low number and then use with to get a more precise answer for some sections:
from decimal import localcontext
with localcontext() as ctx:
ctx.prec = 42 # Perform a high precision calculation
s = calculate_something()
s = +s # Round the final result back to the default precision
I use this a lot with the Hypergeometric Test which requires the division of large numbers resulting form factorials. When you do genomic scale calculations you have to be careful of round-off and overflow errors.
An example of an antipattern might be to use the with inside a loop when it would be more efficient to have the with outside the loop
for example
for row in lines:
with open("outfile","a") as f:
f.write(row)
vs
with open("outfile","a") as f:
for row in lines:
f.write(row)
The first way is opening and closing the file for each row which may cause performance problems compared to the second way with opens and closes the file just once.
See PEP 343 - The 'with' statement, there is an example section at the end.
... new statement "with" to the Python
language to make
it possible to factor out standard uses of try/finally statements.
points 1, 2, and 3 being reasonably well covered:
4: it is relatively new, only available in python2.6+ (or python2.5 using from __future__ import with_statement)
The with statement works with so-called context managers:
http://docs.python.org/release/2.5.2/lib/typecontextmanager.html
The idea is to simplify exception handling by doing the necessary cleanup after leaving the 'with' block. Some of the python built-ins already work as context managers.
Another example for out-of-the-box support, and one that might be a bit baffling at first when you are used to the way built-in open() behaves, are connection objects of popular database modules such as:
sqlite3
psycopg2
cx_oracle
The connection objects are context managers and as such can be used out-of-the-box in a with-statement, however when using the above note that:
When the with-block is finished, either with an exception or without, the connection is not closed. In case the with-block finishes with an exception, the transaction is rolled back, otherwise the transaction is commited.
This means that the programmer has to take care to close the connection themselves, but allows to acquire a connection, and use it in multiple with-statements, as shown in the psycopg2 docs:
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
curs.execute(SQL2)
conn.close()
In the example above, you'll note that the cursor objects of psycopg2 also are context managers. From the relevant documentation on the behavior:
When a cursor exits the with-block it is closed, releasing any resource eventually associated with it. The state of the transaction is not affected.
In python generally “with” statement is used to open a file, process the data present in the file, and also to close the file without calling a close() method. “with” statement makes the exception handling simpler by providing cleanup activities.
General form of with:
with open(“file name”, “mode”) as file_var:
processing statements
note: no need to close the file by calling close() upon file_var.close()
The answers here are great, but just to add a simple one that helped me:
with open("foo.txt") as file:
data = file.read()
open returns a file
Since 2.6 python added the methods __enter__ and __exit__ to file.
with is like a for loop that calls __enter__, runs the loop once and then calls __exit__
with works with any instance that has __enter__ and __exit__
a file is locked and not re-usable by other processes until it's closed, __exit__ closes it.
source: http://web.archive.org/web/20180310054708/http://effbot.org/zone/python-with-statement.htm