Develop Reference

What is the most Pythonic Way to Catch Thrown Library Errors From Deep Inside Code

I have code that looks like this:
# foo.py
from internallib.a import A
A.some(new_schema)
# a.py
from internallib.b import B
class A:
#staticmethod
def some(schema_name):
B.thing(schema_name, other_parameter)
# b.py
import marshmallow
class B:
#staticmethod
def thing(schema_name, other_parameter):
marshmallow.class_registry.get_class(schema_name)
Marshmallow throws an error that I'd like to catch and print nicely for the user and then exit (instead of just printing a stack trace). The specifics here are that Marshmallow throws a RegistryError when get_class is called and the schema hasn't been registered.
However, this is a completely expected condition - it's ok to TRY to add the schema, and I want to present that error to the user - think of it kind like a 'file not found' error. A stack trace feels way too low level to present.
However, A.some() is called in many places in the code. What's the pythonic way of doing this? My thoughts:
Wrap the marshmallow() in a try-except, print something out and then exit()
Go everywhere I call A.some(new_schema) and wrap that (it's not my preference, but if necessary)
Something else?

Solution
Option 1 - uniform behavior for all A.some() callers: Wrap B.thing() call in A.some() in a try except catching very specific exceptions while using logger to log error in the except block prior to exiting.
Option 2 - varying behavior needed for A.some() callers: If you wanted varying behavior depending on the context of where A.some() is being called, then you would need to refactor to catch the same error from Option 1 and raise your own internal error named FileNotFound, which each caller to A.some() would be able to handle behavior individually.

How can I use unittest.mock to remove side effects from code?

I have a function with several points of failure:
def setup_foo(creds):
"""
Creates a foo instance with which we can leverage the Foo virtualization
platform.
:param creds: A dictionary containing the authorization url, username,
password, and version associated with the Foo
cluster.
:type creds: dict
"""
try:
foo = Foo(version=creds['VERSION'],
username=creds['USERNAME'],
password=creds['PASSWORD'],
auth_url=creds['AUTH_URL'])
foo.authenticate()
return foo
except (OSError, NotFound, ClientException) as e:
raise UnreachableEndpoint("Couldn't find auth_url {0}".format(creds['AUTH_URL']))
except Unauthorized as e:
raise UnauthorizedUser("Wrong username or password.")
except UnsupportedVersion as e:
raise Unsupported("We only support Foo API with major version 2")
and I'd like to test that all the relevant exceptions are caught (albeit not handled well currently).
I have an initial test case that passes:
def test_setup_foo_failing_auth_url_endpoint_does_not_exist(self):
dummy_creds = {
'AUTH_URL' : 'http://bogus.example.com/v2.0',
'USERNAME' : '', #intentionally blank.
'PASSWORD' : '', #intentionally blank.
'VERSION' : 2
}
with self.assertRaises(UnreachableEndpoint):
foo = osu.setup_foo(dummy_creds)
but how can I make my test framework believe that the AUTH_URL is actually a valid/reachable URL?
I've created a mock class for Foo:
class MockFoo(Foo):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
and my thought is mock the call to setup_foo and remove the side effect of raising an UnreachableEndpoint exception. I know how to add side-effects to a Mock with unittest.mock, but how can I remove them?

Assuming your exceptions are being raised from foo.authenticate(), what you want to realize here is that it does not necessarily matter whether the data is in fact really valid in your tests. What you are trying to say really is this:
When this external method raises with something, my code should behave accordingly based on that something.
So, with that in mind, what you want to do is have different test methods where you pass what should be valid data, and have your code react accordingly. The data itself does not matter, but it provides a documented way of showing how the code should behave with data that is passed in that way.
Ultimately, you should not care how the nova client handles the data you give it (nova client is tested, and you should not care about it). What you care about is what it spits back at you and how you want to handle it, regardless of what you gave it.
In other words, for the sake of your tests, you can actually pass a dummy url as:
"this_is_a_dummy_url_that_works"
For the sake of your tests, you can let that pass, because in your mock, you will raise accordingly.
For example. What you should be doing here is actually mocking out Client from novaclient. With that mock in hand, you can now manipulate whatever call within novaclient so you can properly test your code.
This actually brings us to the root of your problem. Your first exception is catching the following:
except (OSError, NotFound, ClientException)
The problem here, is that you are now catching ClientException. Almost every exception in novaclient inherits from ClientException, so no matter what you try to test beyond that exception line, you will never reach those exceptions. You have two options here. Catch ClientException, and just raise a custom exception, or, remote ClientException, and be more explicit (like you already are).
So, let us go with removing ClientException and set up our example accordingly.
So, in your real code, you should be now setting your first exception line as:
except (OSError, NotFound) as e:
Furthermore, the next problem you have is that you are not mocking properly. You are supposed to mock with respect to where you are testing. So, if your setup_nova method is in a module called your_nova_module. It is with respect to that, that you are supposed to mock. The example below illustrates all this.
#patch("your_nova_module.Client", return_value=Mock())
def test_setup_nova_failing_unauthorized_user(self, mock_client):
dummy_creds = {
'AUTH_URL': 'this_url_is_valid',
'USERNAME': 'my_bad_user. this should fail',
'PASSWORD': 'bad_pass_but_it_does_not_matter_what_this_is',
'VERSION': '2.1',
'PROJECT_ID': 'does_not_matter'
}
mock_nova_client = mock_client.return_value
mock_nova_client.authenticate.side_effect = Unauthorized(401)
with self.assertRaises(UnauthorizedUser):
setup_nova(dummy_creds)
So, the main idea with the example above, is that it does not matter what data you are passing. What really matters is that you are wanting to know how your code will react when an external method raises.
So, our goal here is to actually raise something that will get your second exception handler to be tested: Unauthorized
This code was tested against the code you posted in your question. The only modifications were made were with module names to reflect my environment.

If you wish to mock out http servers from bogus urls, I suggest you check out HTTPretty. It mocks out urls at a socket level so it can trick most Python HTTP libraries that it's a valid url.
I suggest the following setup for your unittest:
class FooTest(unittest.TestCase):
def setUp(self):
httpretty.register_uri(httpretty.GET, "http://bogus.example.com/v2.0",
body='[{"response": "Valid"}]',
content_type="application/json")
#httpretty.activate
def test_test_case(self):
resp = requests.get("http://bogus.example.com/v2.0")
self.assertEquals(resp.status_code, 200)
Note that the mock will only apply to stacks that are decorated with http.activate decorator, so it won't leak to other places in your code that you don't want to mock. Hope that makes sense.

Custom exception codes and messages in Python 3.4

I'd like to have something like a custom error code/message database and use it when raising exceptions (in Python 3.4). So I did the following:
class RecipeError(Exception):
# Custom error codes
ERRBADFLAVORMIX = 1
ERRNOINGREDIENTS = ERRBADFLAVORMIX + 1
# Custom messages
ERRMSG = {ERRBADFLAVORMIX: "Bad flavor mix",
ERRNOINGREDIENTS: "No ingredients to mix"}
raise RecipeError(RecipeError.ERRMSG[RecipeError.ERRBADFLAVORMIX])
This works as expected, but the raise statement is just monstrous. Sure, I could have stored the values in a more compact way, but what I really want to know is: Can I just do something like raise RecipeError(code) and leave the work of getting the message to RecipeError?

Sure. Exception classes are just normal classes, so you can define your own __init__ that calls super appropriately:
class RecipeError(BaseException):
# existing stuff
def __init__(self, code):
super().__init__(self, RecipeError.ERRMSG[code])
You might also want to save the code:
class RecipeError(BaseException):
# existing stuff
def __init__(self, code):
msg = RecipeError.ERRMSG[code]
super().__init__(self, msg)
self.code, self.msg = code, msg
Take a look at the information stored in the standard library's exceptions (which are pretty decent in 3.4, although there are still more changes to come…) to see what kinds of things might be useful to stash.
Some side notes:
First, it may be better to use subclasses instead of error codes. For example, if someone wants to write code that catches an ERRBADFLAVORMIX but not an ERRNOINGREDIENTS, they have to do this:
try:
follow_recipe()
except RecipeError as e:
if e != RecipeError.ERRBADFLAVORMIX:
raise
print('Bad flavor, bad!')
Or, if you'd used subclasses:
try:
follow_recipe():
except BadFlavorRecipeError as e:
print('Bad flavor, bad!')
That's exactly why Python no longer has a monolithic OSError with an errno value that you have to switch on, and instead has separate subclasses like FileNotFoundError.
If you do want to use error codes, you might want to consider using an Enum, or maybe one of the fancier enum types on PyPI that make it easier to attach a custom string to each one.
You almost never want to inherit from BaseException, unless you're specifically trying to make sure your exception doesn't get caught.

Python django how to properly test if a queryset returned a result

I haven't had a very thorough training in python and sometimes don't know the correct way of doing things. One of these things is testing if my resultQuery returned a result or not. I find myself doing this a lot:
try:
user = User.objects.get(email=email)
except DoesNotExist:
user = User()
I don't know about python but try catches in other languages are supposed to be for exceptions and not for normal program flow. How would you do this with an if else?
I guess I want something similar to:
if request.GET.get('email','') is not None:
email = request.GET['email'])

Your exception example is most commonly the preferred way to do things. Wikipedia has quite a nice description about this:
Python style calls for the use of exceptions whenever an error
condition might arise. Rather than testing for access to a file or
resource before actually using it, it is conventional in Python to
just go ahead and try to use it, catching the exception if access is
rejected.
Exceptions are often used as an alternative to the if-block
(...). A commonly-invoked motto is EAFP, or "It is Easier to Ask
for Forgiveness than Permission."
Exceptions aren't necessarily costly in Python. Again, quoting the Wikipedia example:
if hasattr(spam, 'eggs'):
ham = spam.eggs
else:
handle_error()
... versus:
try:
ham = spam.eggs
except AttributeError:
handle_error()
These two code samples have the same effect, although there will be performance differences. When spam has the attribute eggs, the EAFP sample will run faster. When spam does not have the attribute eggs (the "exceptional" case), the EAFP sample will run slower.
Specifically for Django, I would use the exception example. It is the standard way to do things in that framework, and following standards is never a bad thing :).

You can use .filter() instead of .get(), and also make use of django's optimized queryset evaluation.
You can do
qs = User.objects.filter(email=email)
if qs :
user = qs[0]
...

I agree that catching the exception is the normal pattern here. If you do need to check for whether or not a queryset returns results, the exists method is one way to do so that I don't think has been mentioned yet. The docs run through the various performance implications, most notably that if you know the queryset will be evaluated it's better to just check the boolean interpretation of the queryset.

Django defines a class EmptyQuerySet that can be used for special implementations when strict isinstance(...) testing is necessary.
A straightforward way is as follows:
try:
user = User.objects.get(email=email)
if user:
# ...
else:
# ...
except DoesNotExist:
user = User()
Django provides a get method to directly retrieve one single Model instance or will raise an Exception. In this case the normal way is to check for Exceptions such as DoesNotExist
To retrieve entire rows from a database query as you normally would, use the user.objects.filter(...) method. The returned QuerySet instance provides you with .count(), .exists() methods.

Is it reasonable to declare an exception type for a single function?

Say I have this code:
def wait_for_x(timeout_at=None):
while condition_that_could_raise_exceptions
if timeout_at is not None and time.time() > timeout_at:
raise SOMEEXCEPTIONHERE
do_some_stuff()
try:
foo()
wait_for_x(timeout_at=time.time() + 10)
bar()
except SOMEEXCEPTIONHERE:
# report timeout, move on to something else
How do I pick an exception type SOMEEXCEPTIONHERE for the function? Is it reasonable to create a unique exception type for that function, so that there's no danger of condition_that_could_raise_exceptions raising the same exception type?
wait_for_x.Timeout = type('Timeout', (Exception,), {})

If distinguishing exceptions from wait_for_x from those from condition_that_could_raise_exceptions is important enough, then sure, define a new exception type. After all, the type is the main way of distinguishing different kinds of exceptions, and parsing the message tends to get messy pretty quickly.

Yes, you should certainly define your own exception class whenever none of the built-in exception types are appropriate. In some cases (say, if you're building some kind of HTML munger on top of LXML or BeautifulSoup) it might also be appropriate to use an exception from some other module.

Python Standard Library defines a lot of its own custom exceptions. It seems good practice to do that as well for personal modules or functions.