Develop Reference

Python Mock: Raising Error for function of Mocked Class

I'm trying to test some code that create or changes directories and files based on some inputs. My issue is raising exceptions when a method of a mocked object is called. For example, say I have some code:
def create_if_not_exists(dest):
dir = os.path.dirname(dest)
if not dir:
# create if it doesn't exist
try:
os.makedirs(os.path.dirname(dest))
except OSError:
pass # Nevermind what goes here, it's unimportant to the question
and a unit test:
#patch('my.package.os')
def test_create_dir_if_not_exists(self, mock_os):
mock_os.path.dirname.return_value = None
mock_os.makedirs.raiseError.side_effect = OSError()
with self.assertRaises(OSError)
create_if_not_exists('test')
This setup returns AssertionError: OSError not raised but my understanding is it should raise the error when the makedirs call is made in the actual (non-test) method. Is my understanding incorrect?

Try with the following patch in the test file:
#patch('my.package.os.path.dirname')
#patch('my.package.os.makedirs')
def test_create_dir_if_not_exists(self, mock_os_makedirs, mock_os_path_dirname):
mock_os_path_dirname.return_value = None
mock_os_makedirs.raiseError.side_effect = OSError()
with self.assertRaises(OSError)
create_if_not_exists('test')

How to jump to same function again after exception in python

I have below script which generates report for large size data.
Due to large size data request call times out.
I have added exception to handle this situation which works fine to get keep script running.
Issue I am having is after exception it goes to next project and skips the projects where it timed out.
I want it to start from same project again.
How can I achieve this ?
if __name__ = ‘__main__’
for project in AllProjectData['value']:
try:
project_name = project['name']
** code to be executed
except:
requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout
continue

You are catching exceptions in a very odd way. I've never seen it done like this. I believe this code is catching all exceptions. For example:
try:
1/0
except:
ZeroDivisionError
pass
Works fine, but so does (it should raise IndexError):
try:
a = []
print(a[1])
except:
ZeroDivisionError
pass
So you shouldn't write except statements this way. What you should have is something along the lines:
success = False
while not success:
try:
# Your execution code
except (requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout):
continue
else:
success = True
Also you should try and not put so much code in your except statement as it is confusing as to what you're trying to catch and where. Also, you're completely missing some possibilities like a KeyError when there's no id field in project and others.

Try this -
def myfunc():
# Write the code given in question completely
# Then add this in your except statement
except:
requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout
# Call your function so that it will jump back to the same function
myfunc()
# You don't need continue keyword because it jumps to the same function

The simplest way would be to use a while loop with a counter variable. Let me demonstrate:
i = 0
while i < len(AllProjectData['value']):
try:
project = AllProjectData['value'][i]
# code to be executed
i += 1 # increment counter on success
except:
requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout
continue
This way, you will go to the next project only if work on the previous project was executed, as the loop variable is incremented only in the try block.
Note: I have assumed that your iterable is indexable. If it's not, just pass it inside the list() constructor.

suggestions for to improve my python function to parse Wordpress/config.php

I am writing a python script function to backup Wordpress. As part of the script i wrote a function to fetch database details from the config.php file.
Working of my function
function takes Wordpress installation location as an argument and using regex to match db_user,db_host,db_user,db_password from that file, the function will exist if can not find "config.php". I am using sys.exit(1) to exit from the function is that the proper way to exit from a function? I am pasting my function code snippet.
def parsing_db_info(location):
config_path = os.path.normpath(location+'/config.php')
if os.path.exists(config_path):
try:
regex_db = r'define\(\s*?\'DB_NAME\'\s*?,\s*?\'(.*?)\'\s*?'.group(1)
regex_user = r'define\(\s*?\'DB_USER\'\s*?,\s*?\'(.*?)\'\s*?'.group(1)
regex_pass = r'define\(\s*?\'DB_PASSWORD\'\s*?,\s*?\'(.*?)\'\s*?'.group(1)
regex_host = r'define\(\s*?\'DB_HOST\'\s*?,\s*?\'(.*?)\'\s*?'.group(1)
db_name = re.match(regex_db,config_path).group(1)
db_user = re.match(regex_user,config_path).group(1)
db_pass = re.match(regex_pass,config_path).group(1)
db_host = re.match(regex_host,config_path).group(1)
return {'dbname':db_name , 'dbuser':db_user , 'dbpass':db_pass , 'dbhost':db_host}
except exception as ERROR:
print(ERROR)
sys.exit(1)
else:
print('Not Found:',config_path)
sys.exit(1)
AFTER EDITING
def parsing_db_info(location):
config_path = os.path.normpath(location+'/wp-config.php')
try:
with open(config_path) as fh:
content = fh.read()
regex_db = r'define\(\s*?\'DB_NAME\'\s*?,\s*?\'(.*?)\'\s*?'
regex_user = r'define\(\s*?\'DB_USER\'\s*?,\s*?\'(.*?)\'\s*?'
regex_pass = r'define\(\s*?\'DB_PASSWORD\'\s*?,\s*?\'(.*?)\'\s*?'
regex_host = r'define\(\s*?\'DB_HOST\'\s*?,\s*?\'(.*?)\'\s*?'
db_name = re.search(regex_db,content).group(1)
db_user = re.search(regex_user,content).group(1)
db_pass = re.search(regex_pass,content).group(1)
db_host = re.search(regex_host,content).group(1)
return {'dbname':db_name , 'dbuser':db_user , 'dbpass':db_pass , 'dbhost':db_host}
except FileNotFoundError:
print('File Not Found,',config_path)
sys.exit(1)
except PermissionError:
print('Unable To read Permission Denied,',config_path)
sys.exit(1)
except AttributeError:
print('Parsing Error wp-config seems to be corrupt,')
sys.exit(1)

To answer your question, you shouldn't normally use sys.exit inside a function like that. Rather, get it to raise an exception in the case where it fails. Preferably, it should be an exception detailing what went wrong, or you could just let the existing exceptions propagate.
The normal rule in Python is this: deal with exceptions at the place you know how to deal with them.
In your code, you catch an exception, and then don't know what to do, so call sys.exit. Instead of this, you should:
let an exception propagate up to a top-level function which can catch it, and then call sys.exit if appropriate
wrap the exception in something more specific, and re-raise, so that a higher level function will have a specific exception to catch. For example, your function might raise a custom ConfigFileNotFound exception or ConfigFileUnparseable exception.
Also, you have put except exception, you probably mean except Exception. However, this is extremely broad, and will mask other programming errors. Instead, catch the specific exception class you expect.

Handling Exceptions in Python Behave Testing framework

I've been thinking about switching from nose to behave for testing (mocha/chai etc have spoiled me). So far so good, but I can't seem to figure out any way of testing for exceptions besides:
#then("It throws a KeyError exception")
def step_impl(context):
try:
konfigure.load_env_mapping("baz", context.configs)
except KeyError, e:
assert (e.message == "No baz configuration found")
With nose I can annotate a test with
#raises(KeyError)
I can't find anything like this in behave (not in the source, not in the examples, not here). It sure would be grand to be able to specify exceptions that might be thrown in the scenario outlines.
Anyone been down this path?

I'm pretty new to BDD myself, but generally, the idea would be that the tests document what behaves the client can expect - not the step implementations. So I'd expect the canonical way to test this would be something like:
When I try to load config baz
Then it throws a KeyError with message "No baz configuration found"
With steps defined like:
#when('...')
def step(context):
try:
# do some loading here
context.exc = None
except Exception, e:
context.exc = e
#then('it throws a {type} with message "{msg}"')
def step(context, type, msg):
assert isinstance(context.exc, eval(type)), "Invalid exception - expected " + type
assert context.exc.message == msg, "Invalid message - expected " + msg
If that's a common pattern, you could just write your own decorator:
def catch_all(func):
def wrapper(context, *args, **kwargs):
try:
func(context, *args, **kwargs)
context.exc = None
except Exception, e:
context.exc = e
return wrapper
#when('... ...')
#catch_all
def step(context):
# do some loading here - same as before

This try/catch approach by Barry works, but I see some issues:
Adding a try/except to your steps means that errors will be hidden.
Adding an extra decorator is inelegant. I would like my decorator to be a modified #where
My suggestion is to
have the expect exception before the failing statement
in the try/catch, raise if the error was not expected
in the after_scenario, raise error if expected error not found.
use the modified given/when/then everywhere
Code:
def given(regexp):
return _wrapped_step(behave.given, regexp) #pylint: disable=no-member
def then(regexp):
return _wrapped_step(behave.then, regexp) #pylint: disable=no-member
def when(regexp):
return _wrapped_step(behave.when, regexp) #pylint: disable=no-member
def _wrapped_step(step_function, regexp):
def wrapper(func):
"""
This corresponds to, for step_function=given
#given(regexp)
#accept_expected_exception
def a_given_step_function(context, ...
"""
return step_function(regexp)(_accept_expected_exception(func))
return wrapper
def _accept_expected_exception(func):
"""
If an error is expected, check if it matches the error.
Otherwise raise it again.
"""
def wrapper(context, *args, **kwargs):
try:
func(context, *args, **kwargs)
except Exception, e: #pylint: disable=W0703
expected_fail = context.expected_fail
# Reset expected fail, only try matching once.
context.expected_fail = None
if expected_fail:
expected_fail.assert_exception(e)
else:
raise
return wrapper
class ErrorExpected(object):
def __init__(self, message):
self.message = message
def get_message_from_exception(self, exception):
return str(exception)
def assert_exception(self, exception):
actual_msg = self.get_message_from_exception(exception)
assert self.message == actual_msg, self.failmessage(exception)
def failmessage(self, exception):
msg = "Not getting expected error: {0}\nInstead got{1}"
msg = msg.format(self.message, self.get_message_from_exception(exception))
return msg
#given('the next step shall fail with')
def expect_fail(context):
if context.expected_fail:
msg = 'Already expecting failure:\n {0}'.format(context.expected_fail.message)
context.expected_fail = None
util.show_gherkin_error(msg)
context.expected_fail = ErrorExpected(context.text)
I import my modified given/then/when instead of behave, and add to my environment.py initiating context.expected fail before scenario and checking it after:
def after_scenario(context, scenario):
if context.expected_fail:
msg = "Expected failure not found: %s" % (context.expected_fail.message)
util.show_gherkin_error(msg)

The try / except approach you show is actually completely correct because it shows the way that you would actually use the code in real life. However, there's a reason that you don't completely like it. It leads to ugly problems with things like the following:
Scenario: correct password accepted
Given that I have a correct password
When I attempt to log in
Then I should get a prompt
Scenario: incorrect password rejected
Given that I have an incorrect password
When I attempt to log in
Then I should get an exception
If I write the step definition without try/except then the second scenario will fail. If I write it with try/except then the first scenario risks hiding an exception, especially if the exception happens after the prompt has already been printed.
Instead those scenarios should, IMHO, be written as something like
Scenario: correct password accepted
Given that I have a correct password
When I log in
Then I should get a prompt
Scenario: correct password accepted
Given that I have a correct password
When I try to log in
Then I should get an exception
The "I log in" step should not use try; The "I try to log in" matches neatly to try and gives away the fact that there might not be success.
Then there comes the question about code reuse between the two almost, but not quite identical steps. Probably we don't want to have two functions which both login. Apart from simply having a common other function you call, you could also do something like this near the end of your step file.
#when(u'{who} try to {what}')
def step_impl(context):
try:
context.execute_steps("when" + who + " " + what)
context.exception=None
except Exception as e:
context.exception=e
This will automatically convert all steps containing the word "try to" into steps with the same name but with try to deleted and then protect them with a try/except.
There are some questions about when you actually should deal with exceptions in BDD since they aren't user visible. It's not part of the answer to this question though so I've put them in a separate posting.

Behave is not in the assertion matcher business. Therefore, it does not provide a solution for this. There are already enough Python packages that solve this problem.
SEE ALSO: behave.example: Select an assertion matcher library

Try and finally in python

def connect(self):
ok = False
try:
conn = ftplib.FTP(self.hostname, self.user, self.password)
ok = True
return conn
finally:
if not ok:
logging.error('Failed to connect to %s for %s' % (self.hostname, self.user))
I'm assuming it is not a good idea to return within the try block if there is something happening in the finally block. I just want to be sure of the order of execution before I slam someones head!

I'm assuming it is not a good idea to return within the try block if there is something happening in the finally block.
You're assuming wrong. The whole point of a finally is that it always happens, even if you return or break early or raise an unhandled exception.* And it happens after the return.**
This is explained under The try statement in the documentation… but that isn't exactly the most novice-friendly section of the docs.
That being said, you seem to be using a finally block to fake an except block. Instead of having a flag that you check at finally time to see if there was an exception, just use the exception itself tell you that there was an exception:
def connect(self):
try:
conn = ftplib.FTP(self.hostname, self.user, self.password)
return conn
except Exception as e:
logging.error('Failed to connect to %s for %s' % (self.hostname, self.user))
raise
* That "always" is really only true within limits. If your program calls some C code that segfaults the interpreter, or you explicitly call _exit, or someone pulls the power cord in the middle of the program, the finally code obviously won't run.
** It might more precise to say it's happening in the middle of the return, but I think that just adds confusion; unless you're actually working on the guts of the interpreter, you can think of return as a single step.

Rest assured the finally block is always executed. The return in the try block doesn't change that. No head slamming required ;-)