In Django 3.1, suppose I have some model M, and I have a QuerySet over M that I expect has either one or zero elements. How do I branch on whether it is one or zero elements (throwing an exception if it has two or more), and in the branch with one element, how do I get the one M object:
try:
my_query_set = M.objects.filter(some_filter_expr)
if ???:
m = ??? # the one M object
else:
on_zero_objects()
except ???:
more_than_one_object()
Use .first() (or last() )
one_or_none = M.objects.filter(some_filter_expr).first()
Thus, the variable one_or_none will have either an instance of model M or None
You can handle the more than mone element condition or separate conditions are in the following way,
my_query_set = M.objects.filter(some_filter_expr)
try:
my_query_set[1]
handle_more_than_one_element()
except IndexError:
try:
my_query_set[0]
handle_only_one_element()
except IndexError:
handle_no_element()
You can use .exists() to check if there is any data exists in my_queryset, if yes, then use .get() to get the one object. If there is multiple objects, then it will raise MultipleObjectsReturned error. In except block, you can handle it with more_than_one_object.
my_query_set = M.objects.filter(some_filter_expr)
try:
if my_query_set.exists()
m = my_query_set.get()
else:
on_zero_objects()
except MultipleObjectsReturned:
more_than_one_object()
You can use .get() to get a single element, and handle the exceptions for the DoesNotExist case and MultipleObjectsReturned cases.
from django.core.exceptions import MultipleObjectsReturned
...
try:
model_object = M.objects.get(some_filter_expr)
except M.DoesNotExist:
on_zero_objects()
except MultipleObjectsReturned:
more_than_one_object()
Since too many python operations return ValueError, how can we differentiate between them?
Example: I expect an iterable to have a single element, and I want to get it
a, = [1, 2]: ValueError: too many values to unpack
a, = []: ValueError: too few values to unpack
How can I differentiate between those two cases?? eg
try:
a, = lst
except ValueError as e:
if e.too_many_values:
do_this()
else:
do_that()
I realise that in this particular case I could find a work-around using length/indexing, but the point is similar cases come up often, and I want to know if there's a general approach. I also realise I could check the error message for if 'too few' in message but it seems a bit crude.
try:
raise ValueError('my error')
except ValueError as e:
# use str(), not repr(), see
# https://stackoverflow.com/a/45532289/7919597
x = getattr(e, 'message', str(e))
if 'my error' in x:
print('got my error')
(see also How to get exception message in Python properly)
But this might not be a clean solution after all.
The best thing would be to narrow the scope of your try block so that only one was possible. Or don't depend on exceptions to detect those error cases.
This isn't really an answer, because it only applies if you have some control over how the exceptions are raised. Since exceptions are just objects, you can just tack on other objects / flags to them. Not saying that this is a great thing to do or a great way of doing it:
from enum import Enum
class ValueErrorType(Enum):
HelloType = 0,
FooType = 1
def some_func(string):
if "Hello" in string:
error = ValueError("\"Hello\" is not allowed in my strings!!!!")
error.error_type = ValueErrorType.HelloType
raise error
elif "Foo" in string:
error = ValueError("\"Foo\" is also not allowed!!!!!!")
error.error_type = ValueErrorType.FooType
raise error
try:
some_func("Hello World!")
except ValueError as error:
error_type_map = {
ValueErrorType.HelloType: lambda: print("It was a HelloType"),
ValueErrorType.FooType: lambda: print("It was a FooType")
}
error_type_map[error.error_type]()
I'd be curious to know if there is some way you can achieve this with exceptions where you have no control over how they're raised.
This is a homework problem. I've been trying to solve it but couldn't get the right result.
This is the question:
Write a function string2int that attempts to convert a string to an integer. If the string does represent a positive integer then that integer should be returned. If the string does not represent a positive integer then you should raise a syntax exception (raise SyntaxError('not an integer')).
You may choose to use the (already defined) function all_digits that takes a string and returns True if all the characters of the string are digits and False otherwise.
What I've got so far is:
try all_digits is True:
return int(num)
except all_digits is False:
raise SyntaxError('not an integer')
Because I'm using an already defined function, I didn't define a function (or did I get it wrong?).
Can anyone have a look at my code please? Much appreciated.
I can guess, but you might want to tell us what kind of error you get when you execute the code (just a heads up for the next time you ask a question).
There's a couple of mistakes:
1) The syntax of
try all_digits is True:
is wrong. The "try" statement should look like this:
try:
<your code>
except <type of exception to catch>:
<error handling code>
2) You said "all_digits" is a function. Therefore, the code
all_digits is True
should be
if all_digits(num):
Putting it all together:
def string2int(num):
if all_digits(num):
return int(num)
raise SyntaxError('not an integer')
In addition to Rawing's excellent answer, note that the usual time to use try/except is when you can handle the error thrown in the try block and continue as usual. For instance:
def add_three(x) -> int:
try:
return x + 3
except TypeError:
# someone passed a non-int/float to the function!
# let's try and coerce it.
return int(x) + 3
# if this still throws an exception, we'll let it
# raise its own TypeError exception!
In your case it looks like you're just doing regular conditionals, so use if all_digits(num): return int(num) else: raise TypeError('not an integer')
all_digits(string) function:
First, it's good to understand what does the pre-defined all_digits(string) function do. Following is a sample implementation of that function, which works as desired by your description. It checks whether each letter of the string is a digit and returns a boolean, True or False, accordingly:
def all_digits(string):
''' only returns True if all characters of the string are Integers '''
for l in string:
if l.isdigit(): pass
else: return False
return True
string2num(string) function with raise statement:
Now, we can use this function in our error handling block of the string2num(string) function. Since your problem requires you to only raise a specific exception and not to continue with an alternate block of code, you do not need the try: ... except: block.
With the proper syntax of the raise statement, we can write:
def string2num( string = '-23'):
if all_digits(string):
return int('23')
raise SyntaxError("not an integer")
and we get:
>>> string2num()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 4, in string2num
SyntaxError: not an integer
with try: ... except: ... block:
But if you do want to execute an alternate block of code when the exception is raised, you can use the try: ... except: block syntax. You may need it, for instance, if you want to further check if the string is a negative integer and if so then return the negative integer:
def string2num( string = '-23'):
try:
if all_digits(string):
return int(string)
raise SyntaxError("not an integer")
except SyntaxError:
#alternate code goes here#
try:
return int(string)
except ValueError:
print "string contains an alphabet"
This will produce:
>>> string2num()
-23
>>> string2num('ab2')
string contains an alphabet
Style for if statement:
As a side note on your style, you don't have to explicitly write whether an expression evaluates to True or False in an if statement, like so:
if all_digits(string) is True:
Since all_digits(string) returns a boolean, you can equivalently just say if True, like so:
if all_digits(string):
How do you get around this error while using .pop? I get that when it tries to return a number but there isn't one an error is raised but how do you get around it so the program keeps running?
def remove_element(self,integer):
self.integer = integer
self.members.pop()
Just check if self.members is not empty:
if self.members:
self.members.pop()
or, catch KeyError via try/except:
try:
self.members.pop()
except KeyError:
# do smth
You can use try/except to catch the KeyError raised by an_empty_set.pop(), or check the set first to make sure it's not empty:
if s:
value = s.pop()
else:
# whatever you want to do if the set is empty
Single line solution
res = self.members.pop() if self.members else None
This is a follow-up to Handle an exception thrown in a generator and discusses a more general problem.
I have a function that reads data in different formats. All formats are line- or record-oriented and for each format there's a dedicated parsing function, implemented as a generator. So the main reading function gets an input and a generator, which reads its respective format from the input and delivers records back to the main function:
def read(stream, parsefunc):
for record in parsefunc(stream):
do_stuff(record)
where parsefunc is something like:
def parsefunc(stream):
while not eof(stream):
rec = read_record(stream)
do some stuff
yield rec
The problem I'm facing is that while parsefunc can throw an exception (e.g. when reading from a stream), it has no idea how to handle it. The function responsible for handling exceptions is the main read function. Note that exceptions occur on a per-record basis, so even if one record fails, the generator should continue its work and yield records back until the whole stream is exhausted.
In the previous question I tried to put next(parsefunc) in a try block, but as turned out, this is not going to work. So I have to add try-except to the parsefunc itself and then somehow deliver exceptions to the consumer:
def parsefunc(stream):
while not eof(stream):
try:
rec = read_record()
yield rec
except Exception as e:
?????
I'm rather reluctant to do this because
it makes no sense to use try in a function that isn't intended to handle any exceptions
it's unclear to me how to pass exceptions to the consuming function
there going to be many formats and many parsefunc's, I don't want to clutter them with too much helper code.
Has anyone suggestions for a better architecture?
A note for googlers: in addition to the top answer, pay attention to senderle's and Jon's posts - very smart and insightful stuff.
You can return a tuple of record and exception in the parsefunc and let the consumer function decide what to do with the exception:
import random
def get_record(line):
num = random.randint(0, 3)
if num == 3:
raise Exception("3 means danger")
return line
def parsefunc(stream):
for line in stream:
try:
rec = get_record(line)
except Exception as e:
yield (None, e)
else:
yield (rec, None)
if __name__ == '__main__':
with open('temp.txt') as f:
for rec, e in parsefunc(f):
if e:
print "Got an exception %s" % e
else:
print "Got a record %s" % rec
Thinking deeper about what would happen in a more complex case kind of vindicates the Python choice of avoiding bubbling exceptions out of a generator.
If I got an I/O error from a stream object the odds of simply being able to recover and continue reading, without the structures local to the generator being reset in some way, would be low. I would somehow have to reconcile myself with the reading process in order to continue: skip garbage, push back partial data, reset some incomplete internal tracking structure, etc.
Only the generator has enough context to do that properly. Even if you could keep the generator context, having the outer block handle the exceptions would totally flout the Law of Demeter. All the important information that the surrounding block needs to reset and move on is in local variables of the generator function! And getting or passing that information, though possible, is disgusting.
The resulting exception would almost always be thrown after cleaning up, in which case the reader-generator will already have an internal exception block. Trying very hard to maintain this cleanliness in the brain-dead-simple case only to have it break down in almost every realistic context would be silly. So just have the try in the generator, you are going to need the body of the except block anyway, in any complex case.
It would be nice if exceptional conditions could look like exceptions, though, and not like return values. So I would add an intermediate adapter to allow for this: The generator would yield either data or exceptions and the adapter would re-raise the exception if applicable. The adapter should be called first-thing inside the for loop, so that we have the option of catching it within the loop and cleaning up to continue, or breaking out of the loop to catch it and and abandon the process. And we should put some kind of lame wrapper around the setup to indicate that tricks are afoot, and to force the adapter to get called if the function is adapting.
That way each layer is presented errors that it has the context to handle, at the expense of the adapter being a tiny bit intrusive (and perhaps also easy to forget).
So we would have:
def read(stream, parsefunc):
try:
for source in frozen(parsefunc(stream)):
try:
record = source.thaw()
do_stuff(record)
except Exception, e:
log_error(e)
if not is_recoverable(e):
raise
recover()
except Exception, e:
properly_give_up()
wrap_up()
(Where the two try blocks are optional.)
The adapter looks like:
class Frozen(object):
def __init__(self, item):
self.value = item
def thaw(self):
if isinstance(value, Exception):
raise value
return value
def frozen(generator):
for item in generator:
yield Frozen(item)
And parsefunc looks like:
def parsefunc(stream):
while not eof(stream):
try:
rec = read_record(stream)
do_some_stuff()
yield rec
except Exception, e:
properly_skip_record_or_prepare_retry()
yield e
To make it harder to forget the adapter, we could also change frozen from a function to a decorator on parsefunc.
def frozen_results(func):
def freezer(__func = func, *args, **kw):
for item in __func(*args, **kw):
yield Frozen(item)
return freezer
In which case we we would declare:
#frozen_results
def parsefunc(stream):
...
And we would obviously not bother to declare frozen, or wrap it around the call to parsefunc.
Without knowing more about the system, I think it's difficult to tell what approach will work best. However, one option that no one has suggested yet would be to use a callback. Given that only read knows how to deal with exceptions, might something like this work?
def read(stream, parsefunc):
some_closure_data = {}
def error_callback_1(e):
manipulate(some_closure_data, e)
def error_callback_2(e):
transform(some_closure_data, e)
for record in parsefunc(stream, error_callback_1):
do_stuff(record)
Then, in parsefunc:
def parsefunc(stream, error_callback):
while not eof(stream):
try:
rec = read_record()
yield rec
except Exception as e:
error_callback(e)
I used a closure over a mutable local here; you could also define a class. Note also that you can access the traceback info via sys.exc_info() inside the callback.
Another interesting approach might be to use send. This would work a little differently; basically, instead of defining a callback, read could check the result of yield, do a lot of complex logic, and send a substitute value, which the generator would then re-yield (or do something else with). This is a bit more exotic, but I thought I'd mention it in case it's useful:
>>> def parsefunc(it):
... default = None
... for x in it:
... try:
... rec = float(x)
... except ValueError as e:
... default = yield e
... yield default
... else:
... yield rec
...
>>> parsed_values = parsefunc(['4', '6', '5', '5h', '22', '7'])
>>> for x in parsed_values:
... if isinstance(x, ValueError):
... x = parsed_values.send(0.0)
... print x
...
4.0
6.0
5.0
0.0
22.0
7.0
On it's own this is a bit useless ("Why not just print the default directly from read?" you might ask), but you could do more complex things with default inside the generator, resetting values, going back a step, and so on. You could even wait to send a callback at this point based on the error you receive. But note that sys.exc_info() is cleared as soon as the generator yields, so you'll have to send everything from sys.exc_info() if you need access to the traceback.
Here's an example of how you might combine the two options:
import string
digits = set(string.digits)
def digits_only(v):
return ''.join(c for c in v if c in digits)
def parsefunc(it):
default = None
for x in it:
try:
rec = float(x)
except ValueError as e:
callback = yield e
yield float(callback(x))
else:
yield rec
parsed_values = parsefunc(['4', '6', '5', '5h', '22', '7'])
for x in parsed_values:
if isinstance(x, ValueError):
x = parsed_values.send(digits_only)
print x
An example of a possible design:
from StringIO import StringIO
import csv
blah = StringIO('this,is,1\nthis,is\n')
def parse_csv(stream):
for row in csv.reader(stream):
try:
yield int(row[2])
except (IndexError, ValueError) as e:
pass # don't yield but might need something
# All others have to go up a level - so it wasn't parsable
# So if it's an IOError you know why, but this needs to catch
# exceptions potentially, just let the major ones propogate
for record in parse_csv(blah):
print record
I like the given answer with the Frozen stuff. Based on that idea I came up with this, solving two aspects I did not yet like. The first was the patterns needed to write it down. The second was the loss of the stack trace when yielding an exception. I tried my best to solve the first by using decorators as good as possible. I tried keeping the stack trace by using sys.exc_info() instead of the exception alone.
My generator normally (i.e. without my stuff applied) would look like this:
def generator():
def f(i):
return float(i) / (3 - i)
for i in range(5):
yield f(i)
If I can transform it into using an inner function to determine the value to yield, I can apply my method:
def generator():
def f(i):
return float(i) / (3 - i)
for i in range(5):
def generate():
return f(i)
yield generate()
This doesn't yet change anything and calling it like this would raise an error with a proper stack trace:
for e in generator():
print e
Now, applying my decorators, the code would look like this:
#excepterGenerator
def generator():
def f(i):
return float(i) / (3 - i)
for i in range(5):
#excepterBlock
def generate():
return f(i)
yield generate()
Not much change optically. And you still can use it the way you used the version before:
for e in generator():
print e
And you still get a proper stack trace when calling. (Just one more frame is in there now.)
But now you also can use it like this:
it = generator()
while it:
try:
for e in it:
print e
except Exception as problem:
print 'exc', problem
This way you can handle in the consumer any exception raised in the generator without too much syntactic hassle and without losing stack traces.
The decorators are spelled out like this:
import sys
def excepterBlock(code):
def wrapper(*args, **kwargs):
try:
return (code(*args, **kwargs), None)
except Exception:
return (None, sys.exc_info())
return wrapper
class Excepter(object):
def __init__(self, generator):
self.generator = generator
self.running = True
def next(self):
try:
v, e = self.generator.next()
except StopIteration:
self.running = False
raise
if e:
raise e[0], e[1], e[2]
else:
return v
def __iter__(self):
return self
def __nonzero__(self):
return self.running
def excepterGenerator(generator):
return lambda *args, **kwargs: Excepter(generator(*args, **kwargs))
(I answered the other question linked in the OP but my answer applies to this situation as well)
I have needed to solve this problem a couple of times and came upon this question after a search for what other people have done.
One option- which will probably require refactoring things a little bit- would be to simply create an error handling generator, and throw the exception in the generator (to another error handling generator) rather than raise it.
Here is what the error handling generator function might look like:
def err_handler():
# a generator for processing errors
while True:
try:
# errors are thrown to this point in function
yield
except Exception1:
handle_exc1()
except Exception2:
handle_exc2()
except Exception3:
handle_exc3()
except Exception:
raise
An additional handler argument is provided to the parsefunc function so it has a place to put the errors:
def parsefunc(stream, handler):
# the handler argument fixes errors/problems separately
while not eof(stream):
try:
rec = read_record(stream)
do some stuff
yield rec
except Exception as e:
handler.throw(e)
handler.close()
Now just use almost the original read function, but now with an error handler:
def read(stream, parsefunc):
handler = err_handler()
for record in parsefunc(stream, handler):
do_stuff(record)
This isn't always going to be the best solution, but it's certainly an option, and relatively easy to understand.
About your point of propagating exception from generator to consuming function,
you can try to use an error code (set of error codes) to indicate the error.
Though not elegant that is one approach you can think of.
For example in the below code yielding a value like -1 where you were expecting
a set of positive integers would signal to the calling function that there was
an error.
In [1]: def f():
...: yield 1
...: try:
...: 2/0
...: except ZeroDivisionError,e:
...: yield -1
...: yield 3
...:
In [2]: g = f()
In [3]: next(g)
Out[3]: 1
In [4]: next(g)
Out[4]: -1
In [5]: next(g)
Out[5]: 3
Actually, generators are quite limited in several aspects. You found one: the raising of exceptions is not part of their API.
You could have a look at the Stackless Python stuff like greenlets or coroutines which offer a lot more flexibility; but diving into that is a bit out of scope here.