def connect(self):
ok = False
try:
conn = ftplib.FTP(self.hostname, self.user, self.password)
ok = True
return conn
finally:
if not ok:
logging.error('Failed to connect to %s for %s' % (self.hostname, self.user))
I'm assuming it is not a good idea to return within the try block if there is something happening in the finally block. I just want to be sure of the order of execution before I slam someones head!
I'm assuming it is not a good idea to return within the try block if there is something happening in the finally block.
You're assuming wrong. The whole point of a finally is that it always happens, even if you return or break early or raise an unhandled exception.* And it happens after the return.**
This is explained under The try statement in the documentation… but that isn't exactly the most novice-friendly section of the docs.
That being said, you seem to be using a finally block to fake an except block. Instead of having a flag that you check at finally time to see if there was an exception, just use the exception itself tell you that there was an exception:
def connect(self):
try:
conn = ftplib.FTP(self.hostname, self.user, self.password)
return conn
except Exception as e:
logging.error('Failed to connect to %s for %s' % (self.hostname, self.user))
raise
* That "always" is really only true within limits. If your program calls some C code that segfaults the interpreter, or you explicitly call _exit, or someone pulls the power cord in the middle of the program, the finally code obviously won't run.
** It might more precise to say it's happening in the middle of the return, but I think that just adds confusion; unless you're actually working on the guts of the interpreter, you can think of return as a single step.
Rest assured the finally block is always executed. The return in the try block doesn't change that. No head slamming required ;-)
Related
I have below script which generates report for large size data.
Due to large size data request call times out.
I have added exception to handle this situation which works fine to get keep script running.
Issue I am having is after exception it goes to next project and skips the projects where it timed out.
I want it to start from same project again.
How can I achieve this ?
if __name__ = ‘__main__’
for project in AllProjectData['value']:
try:
project_name = project['name']
** code to be executed
except:
requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout
continue
You are catching exceptions in a very odd way. I've never seen it done like this. I believe this code is catching all exceptions. For example:
try:
1/0
except:
ZeroDivisionError
pass
Works fine, but so does (it should raise IndexError):
try:
a = []
print(a[1])
except:
ZeroDivisionError
pass
So you shouldn't write except statements this way. What you should have is something along the lines:
success = False
while not success:
try:
# Your execution code
except (requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout):
continue
else:
success = True
Also you should try and not put so much code in your except statement as it is confusing as to what you're trying to catch and where. Also, you're completely missing some possibilities like a KeyError when there's no id field in project and others.
Try this -
def myfunc():
# Write the code given in question completely
# Then add this in your except statement
except:
requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout
# Call your function so that it will jump back to the same function
myfunc()
# You don't need continue keyword because it jumps to the same function
The simplest way would be to use a while loop with a counter variable. Let me demonstrate:
i = 0
while i < len(AllProjectData['value']):
try:
project = AllProjectData['value'][i]
# code to be executed
i += 1 # increment counter on success
except:
requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout
continue
This way, you will go to the next project only if work on the previous project was executed, as the loop variable is incremented only in the try block.
Note: I have assumed that your iterable is indexable. If it's not, just pass it inside the list() constructor.
I have the strangest problem I have ever met in my life.
I have a part of my code that looks like this:
class AzureDevOpsServiceError(Exception):
pass
skip = ["auto"]
def retrieve_results():
print(variable_not_defined)
... # some useful implementation
if not "results" in skip:
try:
print("before")
retrieve_results()
print("after")
except AzureDevOpsServiceError as e:
print(f"Error raised: {e}")
Obviously, this shall raise an error because variable_not_defined is, well, not defined.
However, for some strange reasons, the code executes correctly and prints
before
after
I have tried to call the function with an argument (retrieve_results(1234)) or adding an argument in the function (def retrieve_results(arg1) and retrieve_results()): both modifications will trigger an exception, so obviously the function is called.
Anyone has got a similar issue and knows what happens?
FYI: this is actually what my implementation looks like:
from azure.devops.exceptions import AzureDevOpsServiceError
import logging
def _retrieve_manual_results(connect: Connectivity, data: DataForPickle) -> None:
"""Retrieve the list of Test Results"""
print("G" + ggggggggggggggggggggggggggggggggggggg)
logger = connect.logger
data.run_in_progress = [165644]
if __name__ == "__main__":
p = ...
connect = ...
data = ...
if not "results" in p.options.skip:
try:
print("........B.........")
_retrieve_manual_results(connect, data)
print("........A.........")
except AzureDevOpsServiceError as e:
logging.error(f"E004: Error while retrieving Test Results: {e}")
logging.debug("More details below...", exc_info=True)
As highlighted by #gmds, it was a problem of cache.
Deleting the .pyc file didn't do much.
However, I have found a solution:
Renaming the function (e.g. adding _)
Running the program
Renaming back (i.e. removing _ in the previous example)
Now, the issue is solved.
If anyone knows what is going behind the scene I am very interested.
I am trying to implement the retry decorator on a serial query. A general idea of my code is shown below. I am struggling to get it to retry when the decorator is one method up in the hierarchy. How can I have the method be retried when it's one method up from the method that throws the exception?
One complication that is frustrating is my increment time per retry depends on the actual command. Some commands require more time than others. That's why I have the extra_time_per_retry passed in, and couldn't implement the retry decorator using the traditional #retry style.
FYI the _serial is created in the class on init via pySerial.
I got it to work with the retry decorator directly above the method that throws the exception. I would like it to be two above to keep my code clean.
I have tried feeding the retry decorator the exact exception type, but couldn't get it to work.
def _query_with_retries(self, cmd, extra_time_per_retry):
_retriable_query = retry(stop_max_attempt_number=5,
wait_incrementing_start=self._serial.timeout + extra_time_per_retry,
wait_incrementing_increment=10)(self._query)
return _retriable_query(cmd)
def _query(self, cmd):
cmd_msg = cmd + '\r'
self._serial.reset_input_buffer()
self._serial.reset_output_buffer()
self._serial.write(cmd_msg)
return self._readlines()
def _readlines(self):
response_str = self._serial.read_until('\r', 256) # Max 256 bytes
# Parse response here, if a bad one set bad_response = true
if bad_response:
raise ResponseError("Response had custom error xyz")
I guess you could pack your call to _readlines() into an exception handling block, reraising the error:
python 3.x
#retry
def _query(self, cmd):
cmd_msg = cmd + '\r'
self._serial.reset_input_buffer()
self._serial.reset_output_buffer()
self._serial.write(cmd_msg)
try:
answer = self._readlines()
except Exception as e:
raise e
return answer
python 2.x
#retry
def _query(self, cmd):
cmd_msg = cmd + '\r'
self._serial.reset_input_buffer()
self._serial.reset_output_buffer()
self._serial.write(cmd_msg)
try:
answer = self._readlines()
except Exception:
t, v, tb = sys.exc_info()
raise t, v, tb
return answer
This way, you catch the exception directly when it occurs, and raise it inside the method which will be retried. I am not sure whether this declutters enough for you, however it should work.
Some might complain about using a blank except Exception, however since I am reraising it always immediately, I do not see any harm.
I've been thinking about switching from nose to behave for testing (mocha/chai etc have spoiled me). So far so good, but I can't seem to figure out any way of testing for exceptions besides:
#then("It throws a KeyError exception")
def step_impl(context):
try:
konfigure.load_env_mapping("baz", context.configs)
except KeyError, e:
assert (e.message == "No baz configuration found")
With nose I can annotate a test with
#raises(KeyError)
I can't find anything like this in behave (not in the source, not in the examples, not here). It sure would be grand to be able to specify exceptions that might be thrown in the scenario outlines.
Anyone been down this path?
I'm pretty new to BDD myself, but generally, the idea would be that the tests document what behaves the client can expect - not the step implementations. So I'd expect the canonical way to test this would be something like:
When I try to load config baz
Then it throws a KeyError with message "No baz configuration found"
With steps defined like:
#when('...')
def step(context):
try:
# do some loading here
context.exc = None
except Exception, e:
context.exc = e
#then('it throws a {type} with message "{msg}"')
def step(context, type, msg):
assert isinstance(context.exc, eval(type)), "Invalid exception - expected " + type
assert context.exc.message == msg, "Invalid message - expected " + msg
If that's a common pattern, you could just write your own decorator:
def catch_all(func):
def wrapper(context, *args, **kwargs):
try:
func(context, *args, **kwargs)
context.exc = None
except Exception, e:
context.exc = e
return wrapper
#when('... ...')
#catch_all
def step(context):
# do some loading here - same as before
This try/catch approach by Barry works, but I see some issues:
Adding a try/except to your steps means that errors will be hidden.
Adding an extra decorator is inelegant. I would like my decorator to be a modified #where
My suggestion is to
have the expect exception before the failing statement
in the try/catch, raise if the error was not expected
in the after_scenario, raise error if expected error not found.
use the modified given/when/then everywhere
Code:
def given(regexp):
return _wrapped_step(behave.given, regexp) #pylint: disable=no-member
def then(regexp):
return _wrapped_step(behave.then, regexp) #pylint: disable=no-member
def when(regexp):
return _wrapped_step(behave.when, regexp) #pylint: disable=no-member
def _wrapped_step(step_function, regexp):
def wrapper(func):
"""
This corresponds to, for step_function=given
#given(regexp)
#accept_expected_exception
def a_given_step_function(context, ...
"""
return step_function(regexp)(_accept_expected_exception(func))
return wrapper
def _accept_expected_exception(func):
"""
If an error is expected, check if it matches the error.
Otherwise raise it again.
"""
def wrapper(context, *args, **kwargs):
try:
func(context, *args, **kwargs)
except Exception, e: #pylint: disable=W0703
expected_fail = context.expected_fail
# Reset expected fail, only try matching once.
context.expected_fail = None
if expected_fail:
expected_fail.assert_exception(e)
else:
raise
return wrapper
class ErrorExpected(object):
def __init__(self, message):
self.message = message
def get_message_from_exception(self, exception):
return str(exception)
def assert_exception(self, exception):
actual_msg = self.get_message_from_exception(exception)
assert self.message == actual_msg, self.failmessage(exception)
def failmessage(self, exception):
msg = "Not getting expected error: {0}\nInstead got{1}"
msg = msg.format(self.message, self.get_message_from_exception(exception))
return msg
#given('the next step shall fail with')
def expect_fail(context):
if context.expected_fail:
msg = 'Already expecting failure:\n {0}'.format(context.expected_fail.message)
context.expected_fail = None
util.show_gherkin_error(msg)
context.expected_fail = ErrorExpected(context.text)
I import my modified given/then/when instead of behave, and add to my environment.py initiating context.expected fail before scenario and checking it after:
def after_scenario(context, scenario):
if context.expected_fail:
msg = "Expected failure not found: %s" % (context.expected_fail.message)
util.show_gherkin_error(msg)
The try / except approach you show is actually completely correct because it shows the way that you would actually use the code in real life. However, there's a reason that you don't completely like it. It leads to ugly problems with things like the following:
Scenario: correct password accepted
Given that I have a correct password
When I attempt to log in
Then I should get a prompt
Scenario: incorrect password rejected
Given that I have an incorrect password
When I attempt to log in
Then I should get an exception
If I write the step definition without try/except then the second scenario will fail. If I write it with try/except then the first scenario risks hiding an exception, especially if the exception happens after the prompt has already been printed.
Instead those scenarios should, IMHO, be written as something like
Scenario: correct password accepted
Given that I have a correct password
When I log in
Then I should get a prompt
Scenario: correct password accepted
Given that I have a correct password
When I try to log in
Then I should get an exception
The "I log in" step should not use try; The "I try to log in" matches neatly to try and gives away the fact that there might not be success.
Then there comes the question about code reuse between the two almost, but not quite identical steps. Probably we don't want to have two functions which both login. Apart from simply having a common other function you call, you could also do something like this near the end of your step file.
#when(u'{who} try to {what}')
def step_impl(context):
try:
context.execute_steps("when" + who + " " + what)
context.exception=None
except Exception as e:
context.exception=e
This will automatically convert all steps containing the word "try to" into steps with the same name but with try to deleted and then protect them with a try/except.
There are some questions about when you actually should deal with exceptions in BDD since they aren't user visible. It's not part of the answer to this question though so I've put them in a separate posting.
Behave is not in the assertion matcher business. Therefore, it does not provide a solution for this. There are already enough Python packages that solve this problem.
SEE ALSO: behave.example: Select an assertion matcher library
Sometimes I need the following pattern within a for loop. At times more than once in the same loop:
try:
# attempt to do something that may diversely fail
except Exception as e:
logging.error(e)
continue
Now I don't see a nice way to wrap this in a function as it can not return continue:
def attempt(x):
try:
raise random.choice((ValueError, IndexError, TypeError))
except Exception as e:
logging.error(e)
# continue # syntax error: continue not properly in loop
# return continue # invalid syntax
return None # this sort of works
If I return None than I could:
a = attempt('to do something that may diversely fail')
if not a:
continue
But I don't feel that does it the justice. I want to tell the for loop to continue (or fake it) from within attempt function.
Python already has a very nice construct for doing just this and it doesn't use continue:
for i in range(10):
try:
r = 1.0 / (i % 2)
except Exception, e:
print(e)
else:
print(r)
I wouldn't nest any more than this, though, or your code will soon get very ugly.
In your case I would probably do something more like this as it is far easier to unit test the individual functions and flat is better than nested:
#!/usr/bin/env python
def something_that_may_raise(i):
return 1.0 / (i % 2)
def handle(e):
print("Exception: " + str(e))
def do_something_with(result):
print("No exception: " + str(result))
def wrap_process(i):
try:
result = something_that_may_raise(i)
except ZeroDivisionError, e:
handle(e)
except OverflowError, e:
handle(e) # Realistically, this will be a different handler...
else:
do_something_with(result)
for i in range(10):
wrap_process(i)
Remember to always catch specific exceptions. If you were not expecting a specific exception to be thrown, it is probably not safe to continue with your processing loop.
Edit following comments:
If you really don't want to handle the exceptions, which I still think is a bad idea, then catch all exceptions (except:) and instead of handle(e), just pass. At this point wrap_process() will end, skipping the else:-block where the real work is done, and you'll go to the next iteration of your for-loop.
Bear in mind, Errors should never pass silently.
The whole idea of exceptions is that they work across multiple levels of indirection, i.e., if you have an error (or any other exceptional state) deep inside your call hierarchy, you can still catch it on a higher level and handle it properly.
In your case, say you have a function attempt() which calls the functions attempt2() and attempt3() down the call hierarchy, and attempt3() may encounter an exceptional state which should cause the main loop to terminate:
class JustContinueException(Exception):
pass
for i in range(0,99):
try:
var = attempt() # calls attempt2() and attempt3() in turn
except JustContinueException:
continue # we don't need to log anything here
except Exception, e:
log(e)
continue
foo(bar)
def attempt3():
try:
# do something
except Exception, e:
# do something with e, if needed
raise # reraise exception, so we catch it downstream
You can even throw a dummy exception yourself, that would just cause the loop to terminate, and wouldn't even be logged.
def attempt3():
raise JustContinueException()
Apart from the context I just want to answer the question in a brief fashion. No, a function cannot continue a loop it may be called in. That is because it has no information about this context. Also, it would raise a whole new class of questions like what shall happen if that function is called without a surrounding loop to handle that continue?
BUT a function can signal by various means that it wants the caller to continue any loop it currently performs. One means of course is the return value. Return False or None to signal this for example. Another way of signaling this is to raise a special Exception:
class ContinuePlease(Exception): pass
def f():
raise ContinuePlease()
for i in range(10):
try:
f()
except ContinuePlease:
continue
Maybe you want to do continuations? You could go and look at how Eric Lippert explains them (if you are ready to have your mind blown, but in Python it could look a bit like this:
def attempt(operation, continuation):
try:
operation()
except:
log('operation failed!')
continuation()
Inside your loop you could do:
attempt(attempt_something, lambda: foo(bar)) # attempt_something is a function
You could use this:
for l in loop:
attempt() and foo(bar)
but you should make sure attempt() returns True or False.
Really, though, Johnsyweb's answer is probably better.
Think that you are mapping foo on all items where attempt worked. So attempt is a filter and it's easy to write this as a generator:
def attempted( items ):
for item in items:
try:
yield attempt( item )
except Exception, e:
log(e)
print [foo(bar) for bar in attempted( items )]
I wouldn't normally post a second answer, but this is an alternative approach if you really don't like my first answer.
Remember that a function can return a tuple.
#!/usr/bin/env python
def something_that_mail_fail(i):
failed = False
result = None
try:
result = 1.0 / (i % 4)
except:
failed = True # But we don't care
return failed, result
for i in range(20):
failed, result = something_that_mail_fail(i)
if failed:
continue
for rah in ['rah'] * 3:
print(rah)
print(result)
I maintain that try ... except ... else is the way to go, and you shouldn't silently ignore errors though. Caveat emptor and all that.
Try the for loop outside the try, except block
This answer had Python 3.4 in mind however there are better ways in newer versions. Here is my suggestion
import sys
if '3.4' in sys.version:
from termcolor import colored
def list_attributes(module_name):
'''Import the module before calling this func on it.s '''
for index, method in enumerate(dir(module_name)):
try:
method = str(method)
module = 'email'
expression = module + '.' + method
print('*' * len(expression), '\n')
print( str(index).upper() + '. ',colored( expression.upper(), 'red'),
' ', eval( expression ).dir() , '...' , '\n'2 )
print('' * len(expression), '\n')
print( eval( expression + '.doc' ), '\n'*4,
'END OF DESCRIPTION FOR: ' + expression.upper(), '\n'*4)
except (AttributeError, NameError):
continue
else:
pass
finally:
pass
Edit: Removed all that stupidity I said...
The final answer was to rewrite the whole thing, so that I don't need to code like that.