Below is a hypothetical piece of code
with dbengine.connect(**details) as db:
cur = db.exec(sql_string)
results = cur.fetchall()
return results
In this case I would expect that when tabbed out of that with block db.close() is called and db is marked for garbage collection.
In work I've started seeing this code crop up.
with something() as myobj:
logger.info('I got an obj!')
return myobj
I don't know if you should be using with like the new keyword in java. Could someone direct me to any good docs that might explain what you can/can't should/shouldn't do when using with?
P.S Log messages are actually that lame :-)
The target name the with statement binds the contextmanager __enter__ return value to (the name after as) is not scoped to just the with statement. Like for loop variable, the as target name is scoped in the current function or module namespace. The name does not disappear or is otherwise cleared when the with suite ends.
As such, return myobj outside of the with statement is perfectly legal, if somewhat nonsensical. All that the with statement guarantees is that the something().__exit__() method will have been called when the block completes (be that by reaching the end of the block, or because of a continue, break or return statement, or because an exception has been raised).
That said, you'd be better off just moving the return inside the with statement:
with something() as myobj:
logger.info('I got an obj!')
return myobj
and
with dbengine.connect(**details) as db:
cur = db.exec(sql_string)
return cur.fetchall()
The context manager will still be cleaned up properly, but now the return statement looks like it is a logical part of the with block. The execution order is not altered; something().__exit__() is called, then the function returns.
As always, the Python documentation on the with syntax is excellent. You could also review the documentation on context managers and the original proposal, PEP-343.
Related
I know this is a super basic question, but I need to know if I understand.
If I have this function:
def fn():
# connections to a db and builts tables and runs things externally
return "all the tables that were built"
if I write:
x = fn()
do all those external operations still happen? like the tables being built and all that?
or do I run this:
fn()
to have them built?
Thank you
The x = fn() might do less. Imagine one of your returned tables only commits its actions when it gets deleted:
def fn():
class Table:
def __del__(self):
print('commit')
return Table()
x = fn()
print('last thing')
Output:
last thing
commit
Since we hold on to the table, it won't get deleted right away and its actions won't be committed right away. Only at the very end of the program does that happen then, when Python shuts down and deletes everything. Or maybe not even then, see Silvio's comment and the doc ("It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits"), or if Python or your computer crashes.
Output if I remove the x =:
commit
last thing
Now we don't hold on to the table, so nothing references it anymore and it gets deleted right away and the commit happens right away.
If I am correct, with statement doesn't introduce a local scope for the with statement.
These are examples from Learning Python:
with open(r'C:\misc\data') as myfile:
for line in myfile:
print(line)
...more code here...
and
lock = threading.Lock() # After: import threading
with lock:
# critical section of code
...access shared resources...
Is the second example equivalent to the following rewritten in a way similar to the first example?
with threading.Lock() as lock:
# critical section of code
...access shared resources...
What are their differences?
Is the first example equivalent to the following rewritten in a way similar to the second example?
myfile = open(r'C:\misc\data')
with myfile:
for line in myfile:
print(line)
...more code here...
What are their differences?
When with enters a context, it calls a hook on the context manager object, called __enter__, and the return value of that hook can optionally be assigned to a name using as <name>. Many context managers return self from their __enter__ hook. If they do, then you can indeed take your pick between creating the context manager on a separate line or capturing the object with as.
Out of your two examples, only the file object returned from open() has an __enter__ hook that returns self. For threading.Lock(), __enter__ returns the same value as Lock.acquire(), so a boolean, not the lock object itself.
You'll need to look for explicit documentation that confirms this; this is not always that clear however. For Lock objects, the relevant section of the documentation states:
All of the objects provided by this module that have acquire() and release() methods can be used as context managers for a with statement. The acquire() method will be called when the block is entered, and release() will be called when the block is exited.
and for file objects, the IOBase documentation is rather on the vague side and you have to infer from the example that the file object is returned.
The main thing to take away is that returning self is not mandatory, nor is it always desired. Context managers are entirely free to return something else. For example, many database connection objects are context managers that let you manage the transaction (roll back or commit automatically, depending on whether or not there was an exception), where entering returns a new cursor object bound to the connection.
To be explicit:
for your open() example, the two examples are for all intents and purposes exactly the same. Both call open(), and if that does not raise an exception, you end up with a reference to that file object named myfile. In both cases the file object will be closed after the with statement is done. The name continues to exist after the with statement is done.
There is a difference, but it is mostly technical. For with open(...) as myfile:, the file object is created, has it's __enter__ method called and then myfile is bound. For the myfile = open(...) case, myfile is bound first, __enter__ called later.
For your with threading.Lock() as lock: example, using as lock will set lock to a True (locking always either succeeds or blocks indefinitely this way). This differs from the lock = threading.Lock() case, where lock is bound to the lock object.
Here's a good explanation. I'll paraphrase the key part:
The with statement could be thought of like this code:
set things up
try:
do something
finally:
tear things down
Here, “set things up” could be opening a file, or acquiring some sort of external resource, and “tear things down” would then be closing the file, or releasing or removing the resource. The try-finally construct guarantees that the “tear things down” part is always executed, even if the code that does the work doesn’t finish.
Lets say I have a function myFunc defined as
def myFunc(value):
return value if isinstance(value, int) else None
Now wherever in my project I use myFunc the enclosing funciton should return automatically if the value returned from myFunc is None and should continue if some integer value is returned
For example:
def dumbFunc():
# some code
# goes here..
result = myFunc('foo')
# some code
# goes here..
This funciton should automatically behave like..
def dumbFunc():
# some code
# goes here..
result = myFunc('foo')
if not result:
return
# some code
# goes here..
PS - I don't know whether this thing even possible or not.
This is simply not possible.
Apart from exceptions, you cannot give a called function the ability to impact the control flow of the calling scope. So a function call foo() can never interrupt the control flow without throwing an exception. As a consumer of the function, you (the calling function) always have the responsibility yourself to handle such cases and decide about your own control flow.
And it is a very good idea to do it like that. Just the possibility that a function call might interrupt my control flow without having a possibility to react on it first sounds like a pure nightmare. Just alone for the ability to release and cleanup resources, it is very important that the control flow is not taken from me.
Exceptions are the notable exception from this, but of course this is a deeply rooted language feature which also still gives me the ability to act upon it (by catching exceptions, and even by having finally blocks to perform clean up tasks). Exceptions are deliberately not silent but very loud, so that interruptions from the deterministic control flow are clearly visible and have a minimum impact when properly handled.
But having a silent feature that does neither give any control nor feedback would be just a terrible idea.
If myFunc is used at 100 places in my project, everywhere I need to put an if condition after it.
If your code is like that that you could just return nothing from any function that calls myFunc without having to do anything, then either you are building an unrealistic fantasy project, or you simply are not aware of the implications this can have to the calling code of the functions that would be returned that way.
ok, I'll bite.
on the one hand, this isn't really possible. if you want to check something you have to have a line in your code that checks it.
there are a few ways you could achieve something like this, but i think you may have already found the best one.
you already have this function:
def myFunc(value):
return value if isinstance(value, int) else None
I would probably have done:
def myFunc(value):
return isinstance(value, int)
but either way you could use it:
def dumb_func():
value = do_something()
if myFunc(value):
return
do_more()
return value
alternately you could use try and except
I would raise a TypeError, seeing as that seems to be what you are checking:
def myFunc(value):
if not isinstance(value, int):
raise TypeError('myFunc found that {} is not an int'.format(value))
then you can use this as such
def dumb_func():
value = do_something()
try:
myFunc(value):
Except TypeError as e:
print e # some feedback that this has happened, but no error raised
return
do_more()
return value
for bonus points you could define a custom exception (which is safer because then when you catch that specific error you know it wasn't raised by anything else in your code, also if you did that you could be lazier eg:)
Class CustomTypeError(TypeError):
pass
def dumb_func():
try:
value = do_something()
myFunc(value):
do_more()
return value
Except CustomTypeError as e:
print e # some feedback that this has happened, but no error raised
return
but none of this gets around the fact that if you want to act based on the result of a test, you have to check that result.
Python has a ternary conditional operator, and the syntax you used is right, so this will work:
def myFunc(value):
return value if isinstance(value, int) else None
def dumbFunc():
print("Works?")
result = myFunc(5)
print(result)
dumbFunc()
Result:
Works?
5
I want the function to return automatically in that case
This is not possible. To do that, you have to check the return value of myFunc() and act upon it.
PS: You could do that with a goto statement, but Python, fortunately, doesn't support this functionality.
This question already has answers here:
What is the python "with" statement designed for?
(11 answers)
Closed 9 years ago.
I am new to Python. In one tutorial of connecting to mysql and fetching data, I saw the with statement. I read about it and it was something related to try-finally block. But I couldn't find a simpler explanation that I could understand.
with statements open a resource and guarantee that the resource will be closed when the with block completes, regardless of how the block completes. Consider a file:
with open('/etc/passwd', 'r') as f:
print f.readlines()
print "file is now closed!"
The file is guaranteed to be closed at the end of the block -- even if you have a return, even if you raise an exception.
In order for with to make this guarantee, the expression (open() in the example) must be a context manager. The good news is that many python expressions are context managers, but not all.
According to a tutorial I found, MySQLdb.connect() is, in fact, a context manager.
This code:
conn = MySQLdb.connect(...)
with conn:
cur = conn.cursor()
cur.do_this()
cur.do_that()
will commit or rollback the sequence of commands as a single transaction. This means that you don't have to worry so much about exceptions or other unusual code paths -- the transaction will be dealt with no matter how you leave the code block.
Fundamentally it's a object that demarcates a block of code with custom logic that is called on entrance and exit and can take arguments in it's construction. You can define a custom context manager with a class:
class ContextManager(object):
def __init__(self, args):
pass
def __enter__(self):
# Entrance logic here, called before entry of with block
pass
def __exit__(self, exception_type, exception_val, trace):
# Exit logic here, called at exit of with block
return True
The entrance then gets passed an instance of the contextmanager class and can reference anything created in the __init__ method (files, sockets, etc). The exit method also receives any exceptions raised in the internal block and the stack trace object or Nones if the logic completed without raising.
We could then use it like so:
with ContextManager(myarg):
# ... code here ...
This is useful for many things like managing resource lifetimes, freeing file descriptors, managing exceptions and even more complicated uses like building embedded DSLs.
An alternative (but equivalent) method of construction is to the contextlib decorator which uses a generator to separate the entrance and exit logic.
from contextlib import contextmanager
#contextmanager
def ContextManager(args):
# Entrance logic here
yield
# Exit logic here
Think of with as creating a "supervisor" (context manager) over a code block. The supervisor can even be given a name and referenced within the block. When the code block ends, either normally or via an exception, the supervisor is notified and it can take appropriate action based on what happened.
I have observed that after an exception I have an object for which constructor is not called, which causes a lock to be held. What is the best way to improve the situation? Would calling del in an except block be the solution?
b=BigHash(DB_DIR, url)
meta = bdecode(b.get())
return meta
b holds a lock which is released on destruction (it's a C++ object)
an exception is thrown by b.get().
No matter what, you want the lock to be released - whether or not an exception is thrown. In that case, it's probably best to release the lock/delete b in a finally: clause:
b=BigHash(DB_DIR, url)
try:
meta = bdecode(b.get())
finally:
del b # or whatever you need to do to release the lock
return meta
You could also use a context manager - http://docs.python.org/library/stdtypes.html#typecontextmanager. Simply add code to free the lock in the BigHash.__exit__ function, which will be called after leaving the with block in the following code:
with BigHash(DB_DIR, url) as b:
meta = bdecode(b.get())
return meta
You need to do something like this to make sure b in unlocked
b=BigHash(DB_DIR, url)
try:
meta = bdecode(b.get())
return meta
finally:
#unlock b here
A cleaner way would be if BigHash can work as a context, so you can write
with b as BigHash(DB_DIR, url):
meta = bdecode(b.get())
return meta
You might have to add some code to BigHash to make it work as a context though
Calling del on a name is something you prettymuch never should do. Calling del does not guarantee anything useful about what will happen to the underlying object. You should never depend on a __del__ method for something you need to happen.
del only gets rid of one reference to an object, which can be confusing when you may have made more without thinking. Therefore, del is useful for cleaning up a namespace, not for controlling the lifetime of objects, and it's not even great for that—the proper way to control a name's lifetime is to put it in a function and have it go out of scope or put it in a with block.
You need to equip BigHash with the ability to release the lock explicitly, with an release or unlock or close method. If you want to use this with a context manager (with), you can define __exit__, which will get called at a predictable, useful time.