Develop Reference

Exception is not getting caught, while the function was converted into lambda and executed in another function

As you can see, the connect function, is converting the _connect function into a lambda through convert and it's getting passed to run_api function. The exception thrown in _connect is not getting caught by the except in the run_api function. Is anything that needs to be done with respect to lambda?
The code looks good but still am not able to figure out why the exception is not getting caught incase of failure.
Here is my code.
def run_api(function, retry_count):
count = 0
while count < retry_count:
count += 1
try:
function()
return True
except (BleTestFail, BleTestError):
if count == retry_count:
return False
def convert(func):
return lambda: func
def _connect(self, target_id):
result = self.device.ble_central.connect(target_id)
self.logger.debug('Connect output %s', result)
if result['op'] != 'ok':
self.logger.error('Connect command execution failed')
raise ble_utils.BleTestFail('Failed to connect')
return True
def connect(self, target_ids, retry_count=1):
connected = []
unconnected = []
if not isinstance(target_ids, list):
target_ids = [target_ids]
for target_id in target_ids:
connect_function = ble_utils.convert(self._connect(target_id))
connect_status = ble_utils.run_api(connect_function, retry_count,
'connecting device %s' % target_id,
self.logger)
if connect_status:
connected.append(target_id)
else:
unconnected.append(target_id)
if connected:
self.logger.info('Connected to %s devices: %s', len(connected), connected)
if unconnected:
self.logger.error('Unable to connect %s devices: %s', len(unconnected),
unconnected)
return connected, unconnected

So, to clarify, we have the example "API" to which we want to provide a callback:
def run_api(function, retry_count):
count = 0
while count < retry_count:
count += 1
try:
function()
return True
except (BleTestFail, BleTestError):
if count == retry_count:
return False
And a method that we want to be called in that API, with a specific argument:
class Example:
# other stuff omitted...
def _connect(self, target_id):
result = self.device.ble_central.connect(target_id)
self.logger.debug('Connect output %s', result)
if result['op'] != 'ok':
self.logger.error('Connect command execution failed')
raise ble_utils.BleTestFail('Failed to connect')
return True
connection = Example()
So now we want to call run_api with connection._connect, but somehow provide the target_id information.
This is called binding, and the most elegant way to do it is with the standard library functools.partial:
from functools import partial
# This is how we can make the `convert` function from before:
def convert(func, param):
return partial(func, param)
# But there is no point to this, since we can just use `partial` directly.
# There was no hope for the original approach, because you were calling the
# function ahead of time and passing the returned result to `convert`.
# So, the process looks like this:
# target_id = 1, retry_count = 2
run_api(partial(connection._connect, 1), 2)
You can make it work with lambda, but I don't recommend it - functools.partial is more explicit, and elegantly handles more advanced use cases that have some unexpected gotchas (in particular, if you want to make multiple callbacks in a loop; you may find they all unexpectedly bind with the same value, or else you have to use a very ugly workaround). But for the sake of completion, that looks like so:
def convert(func, param):
return lambda: func(param)

Is it possible to write arbitrary depth delegated generators in Python?

I would like to write a class with the following interface.
class Automaton:
""" A simple automaton class """
def iterate(self, something):
""" yield something and expects some result in return """
print("Yielding", something)
result = yield something
print("Got \"" + result + "\" in return")
return result
def start(self, somefunction):
""" start the iteration process """
yield from somefunction(self.iterate)
raise StopIteration("I'm done!")
def first(iterate):
while iterate("what do I do?") != "over":
continue
def second(iterate):
value = yield from iterate("what do I do?")
while value != "over":
value = yield from iterate("what do I do?")
# A simple driving process
automaton = Automaton()
#generator = automaton.start(first) # This one hangs
generator = automaton.start(second) # This one runs smoothly
next_yield = generator.__next__()
for step in range(4):
next_yield = generator.send("Continue...({})".format(step))
try:
end = generator.send("over")
except StopIteration as excp:
print(excp)
The idea is that Automaton will regularly yield values to the caller which will in turn send results/commands back to the Automaton.
The catch is that the decision process "somefunction" will be some user defined function I have no control over. Which means that I can't really expect it to call the iterate method will a yield from in front. Worst, it could be that the user wants to plug some third-party function he has no control over inside this Automaton class. Meaning that the user might not be able to rewrite his somefunction for it to include yield from in front of iterate calls.
To be clear: I completely understand why using the first function hangs the automaton. I am just wondering if there is a way to alter the definition of iterate or start that would make the first function work.

How to best remove redundant calls from a code block, that should be executed when the block exists successfully?

I'm trying to simplify the following code (removing the redundant prints), but can't find a satisfying way to do this:
original code
def main():
if expression1:
print("1")
print("always_do_this")
return
if expression2:
print("2")
print("always_do_this")
return
# ... possibly more expressions and thus redundancy
print("always_do_this")
# do something else
My first idea was a try-(except-)else combination, but the else is not executed on a return in the try-block.
Solution 1 - extracting into a separate function
def func():
if expression1:
print("1")
return True
if expression2:
print("2")
return True
return False
def main():
result = func()
print("always_do_this")
if result:
return
# do something else
Solution 2 - workaround using finally
def main():
error = False
try:
if expression1:
print("1")
return
if expression2:
print("2")
return
except:
error = True
raise
finally:
if not error:
print("always_do_this")
# do something else
Surely there must be a better way to achieve this in python?
PS: Also any ideas for a better title would be appreciated...
PPS: I'm not directly asking about (subjective) codestyle, but wondering if there is a way to write this that I didn't consider (e.g. a language construct/pattern, that also makes the code more concise/cleaner; obviously there are a lot worse ways).

Check if your flow did not enter the first two if blocks by checking for the opposite of the first two if statements joined by an andso that you can execute "do something else" only if the first two if statements failed. Return at the end instead of in the middle of the if statements.
def main():
expression1 = True
expression2 = False
if expression1:
print("1")
elif expression2:
print("2")
print("always_do_this")
if not expression1 and not expression2:
# do something else
return

If the thing you always want to do is closing a file, I would use a with statement. In a more general case you can create your own context manager to have full control of what gets run at the end.
Sample code:
class my_closer:
def __enter__(self):
return True
def __exit__(self, type, value, traceback):
if type is None:
print("always_do_this")
else
print("An exception was raised: {}".format(type))
def main():
with my_closer() as c:
if someexpr:
print("1")
return
if someexpr:
print("2")
return
I added a superfluous else to print something about the exception in case of error, but leave it out to reproduce your original code more accurately.
This code is not shorter than yours with the trivial print statement, but I like this method for more complex "closer" code.
You can also define the context manager this way using the contextlib library:
from contextlib import contextmanager
#contextmanager
def my_closer(*args, **kwds):
try:
yield True
except:
#print("some error happened")
raise
else:
print("always_do_this")
References:
http://effbot.org/zone/python-with-statement.htm
https://docs.python.org/3/library/stdtypes.html#typecontextmanager
https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager

Python polling MSSQL at set interval

I have a need to poll a MSSQL database to watch the status of a running job. I want to run a status check every X seconds to see if the status=done. I am trying to use the threading module. I have tested the threading module with some simple print statements and it seems to work, but when I try inside my pymssql script it does not.
def watcher_query(cursor):
print 'Watching'
return cursor.execute(""" select *
from some_table' """)
def is_it_done(row):
if row['status'] == 'done':
return row['the_id'], True
else:
return row['the_id'], False
def d(cur):
watcher_query(cur)
for row in cur:
return is_it_done(row)[1]
threading.Timer(100, d).start()
def job_watch(server):
with pymssql.connect(server_info) as conn:
with conn.cursor(as_dict=True) as cur:
is_done = false
while is_done:
is_done = d(cur)
No matter what I set the threading.Timer to I see the 'Watching' statement print constantly. Is there a better way to set the polling timer perhaps?
I have also tried to use Twisted to set up a basic function which makes a function call every X sec until some condition is met. I haven't tried it with MSSQL yet though.

The way your code is written it doesn't seems to be in a working order:
It doesn't compile because of is_done = false,
If fixed to is_done = False, it skips the loop straight away,
Even if the loop is fixed in some reasonable way you never get to call threading.Timer(100, d).start() and don't examine any other rows as you return from d straight away after examining the first row using return is_it_done(row)[1]
It doesn't matter what the actual timed worker method does, prints to console or checks the database, should work just the same with the same timer.
What about something like this:
import threading
def is_it_done():
# get some dummy predictable results
if not hasattr(is_it_done, 'results'):
is_it_done.results = iter([False] * 3)
return next(is_it_done.results, True)
def job_watch():
is_done = False
def d():
is_done = is_it_done()
print('is_done: {}'.format(is_done))
timer = threading.Timer(3, d).start()
d()
job_watch()

Conditional if in asynchronous python program with twisted

I'm creating a program that uses the Twisted module and callbacks.
However, I keep having problems because the asynchronous part goes wrecked.
I have learned (also from previous questions..) that the callbacks will be executed at a certain point, but this is unpredictable.
However, I have a certain program that goes like
j = calc(a)
i = calc2(b)
f = calc3(c)
if s:
combine(i, j, f)
Now the boolean s is set by a callback done by calc3. Obviously, this leads to an undefined error because the callback is not executed before the s is needed.
However, I'm unsure how you SHOULD do if statements with asynchronous programming using Twisted. I've been trying many different things, but can't find anything that works.
Is there some way to use conditionals that require callback values?
Also, I'm using VIFF for secure computations (which uses Twisted): VIFF

Maybe what you're looking for is twisted.internet.defer.gatherResults:
d = gatherResults([calc(a), calc2(b), calc3(c)])
def calculated((j, i, f)):
if s:
return combine(i, j, f)
d.addCallback(calculated)
However, this still has the problem that s is undefined. I can't quite tell how you expect s to be defined. If it is a local variable in calc3, then you need to return it so the caller can use it.
Perhaps calc3 looks something like this:
def calc3(argument):
s = bool(argument % 2)
return argument + 1
So, instead, consider making it look like this:
Calc3Result = namedtuple("Calc3Result", "condition value")
def calc3(argument):
s = bool(argument % 2)
return Calc3Result(s, argument + 1)
Now you can rewrite the calling code so it actually works:
It's sort of unclear what you're asking here. It sounds like you know what callbacks are, but if so then you should be able to arrive at this answer yourself:
d = gatherResults([calc(a), calc2(b), calc3(c)])
def calculated((j, i, calc3result)):
if calc3result.condition:
return combine(i, j, calc3result.value)
d.addCallback(calculated)
Or, based on your comment below, maybe calc3 looks more like this (this is the last guess I'm going to make, if it's wrong and you'd like more input, then please actually share the definition of calc3):
def _calc3Result(result, argument):
if result == "250":
# SMTP Success response, yay
return Calc3Result(True, argument)
# Anything else is bad
return Calc3Result(False, argument)
def calc3(argument):
d = emailObserver("The argument was %s" % (argument,))
d.addCallback(_calc3Result)
return d
Fortunately, this definition of calc3 will work just fine with the gatherResults / calculated code block immediately above.

You have to put if in the callback. You may use Deferred to structure your callback.

As stated in previous answer - the preocessing logic should be handled in callback chain, below is simple code demonstration how this could work. C{DelayedTask} is a dummy implementation of a task which happens in the future and fires supplied deferred.
So we first construct a special object - C{ConditionalTask} which takes care of storring the multiple results and servicing callbacks.
calc1, calc2 and calc3 returns the deferreds, which have their callbacks pointed to C{ConditionalTask}.x_callback.
Every C{ConditionalTask}.x_callback does a call to C{ConditionalTask}.process which checks if all of the results have been registered and fires on a full set.
Additionally - C{ConditionalTask}.c_callback sets a flag of wheather or not the data should be processed at all.
from twisted.internet import reactor, defer
class DelayedTask(object):
"""
Delayed async task dummy implementation
"""
def __init__(self,delay,deferred,retVal):
self.deferred = deferred
self.retVal = retVal
reactor.callLater(delay, self.on_completed)
def on_completed(self):
self.deferred.callback(self.retVal)
class ConditionalTask(object):
def __init__(self):
self.resultA=None
self.resultB=None
self.resultC=None
self.should_process=False
def a_callback(self,result):
self.resultA = result
self.process()
def b_callback(self,result):
self.resultB=result
self.process()
def c_callback(self,result):
self.resultC=result
"""
Here is an abstraction for your "s" boolean flag, obviously the logic
normally would go further than just setting the flag, you could
inspect the result variable and do other strange stuff
"""
self.should_process = True
self.process()
def process(self):
if None not in (self.resultA,self.resultB,self.resultC):
if self.should_process:
print 'We will now call the processor function and stop reactor'
reactor.stop()
def calc(a):
deferred = defer.Deferred()
DelayedTask(5,deferred,a)
return deferred
def calc2(a):
deferred = defer.Deferred()
DelayedTask(5,deferred,a*2)
return deferred
def calc3(a):
deferred = defer.Deferred()
DelayedTask(5,deferred,a*3)
return deferred
def main():
conditional_task = ConditionalTask()
dFA = calc(1)
dFB = calc2(2)
dFC = calc3(3)
dFA.addCallback(conditional_task.a_callback)
dFB.addCallback(conditional_task.b_callback)
dFC.addCallback(conditional_task.c_callback)
reactor.run()