I'm setting up a logger in my script like shown at the bottom. This works fine for my purposes and logs my __main__ log messages and those of any modules I use to stdout and a log file.
During program execution a module call that I'm using xarray.open_dataset(file, engine="cfgrib") raises an Error in some conditions and produces the following log output:
2023-02-18 10:02:06,731 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
...
How can I access this output during program execution?
The raised error in the cfgrib module is handled there gracefully and program execution can continue, but the logic of my program requires that I access the error message, in particular the part saying shortName='i10fg' in order to handle the error exhaustively.
Here is how my logger is set up:
def init_log():
"""initialize logging
returns logger using log settings from the config file (settings.toml)
"""
# all settings from a settings file with reasonable defaults
lg.basicConfig(
level=settings.logging.log_level,
format=settings.logging.format,
filemode=settings.logging.logfile_mode,
filename=settings.logging.filename,
)
mylogger = lg.getLogger(__name__)
stream = lg.StreamHandler()
mylogger.addHandler(stream)
clg.install(
level=settings.logging.log_level,
logger=mylogger,
fmt="%(asctime)s %(levelname)s:\t%(message)s",
)
return mylogger
# main
log = init_log()
log.info('...reading files...')
I went through the python logging documentation and cookbook. While this contains ample examples on how to modify logging for various purposes, I could not find an example for accessing and reacting to a log message during program execution.
The Exception in my logs look this:
2023-02-20 12:22:37,209 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 660, in build_dataset_components
dict_merge(variables, coord_vars)
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 591, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='time' value=Variable(dimensions=('time',), data=array([1640995200, 1640998800, 1641002400, ..., 1672520400, 1672524000,
1672527600])) new_value=Variable(dimensions=('time',), data=array([1640973600, 1641016800, 1641060000, 1641103200, 1641146400,
1641189600, 1641232800, 1641276000, 1641319200, 1641362400,
I cannot catch the Exception directly for some reason:
...
import sys
from cfgrib.dataset import DatasetBuildError
...
try:
df = xr.open_dataset(file, engine="cfgrib").to_dataframe()
# triggering error manually like with the two lines below works as expected
# raise Exception()
# raise DatasetBuildError()
except Exception as e:
print('got an Exception')
print(e)
print(e.args)
except BaseException as e:
print('got a BaseException')
print(e.args)
except DatasetBuildError as e:
print(e)
except:
print('got any and all exception')
type, value, traceback = sys.exc_info()
print(type)
print(value)
print(traceback)
Unless I uncomment the two lines where I raise the exception manually, the except clauses are never triggered, event though I can see the DatabaseBuildError in my logs.
Not sure if this has any bearing, but while I can see the Exception as quoted above in my file log, it is not printed to stdout.
How do I raise an exception in Python so that it can later be caught via an except block?
How do I manually throw/raise an exception in Python?
Use the most specific Exception constructor that semantically fits your issue.
Be specific in your message, e.g.:
raise ValueError('A very specific bad thing happened.')
Don't raise generic exceptions
Avoid raising a generic Exception. To catch it, you'll have to catch all other more specific exceptions that subclass it.
Problem 1: Hiding bugs
raise Exception('I know Python!') # Don't! If you catch, likely to hide bugs.
For example:
def demo_bad_catch():
try:
raise ValueError('Represents a hidden bug, do not catch this')
raise Exception('This is the exception you expect to handle')
except Exception as error:
print('Caught this error: ' + repr(error))
>>> demo_bad_catch()
Caught this error: ValueError('Represents a hidden bug, do not catch this',)
Problem 2: Won't catch
And more specific catches won't catch the general exception:
def demo_no_catch():
try:
raise Exception('general exceptions not caught by specific handling')
except ValueError as e:
print('we will not catch exception: Exception')
>>> demo_no_catch()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in demo_no_catch
Exception: general exceptions not caught by specific handling
Best Practices: raise statement
Instead, use the most specific Exception constructor that semantically fits your issue.
raise ValueError('A very specific bad thing happened')
which also handily allows an arbitrary number of arguments to be passed to the constructor:
raise ValueError('A very specific bad thing happened', 'foo', 'bar', 'baz')
These arguments are accessed by the args attribute on the Exception object. For example:
try:
some_code_that_may_raise_our_value_error()
except ValueError as err:
print(err.args)
prints
('message', 'foo', 'bar', 'baz')
In Python 2.5, an actual message attribute was added to BaseException in favor of encouraging users to subclass Exceptions and stop using args, but the introduction of message and the original deprecation of args has been retracted.
Best Practices: except clause
When inside an except clause, you might want to, for example, log that a specific type of error happened, and then re-raise. The best way to do this while preserving the stack trace is to use a bare raise statement. For example:
logger = logging.getLogger(__name__)
try:
do_something_in_app_that_breaks_easily()
except AppError as error:
logger.error(error)
raise # just this!
# raise AppError # Don't do this, you'll lose the stack trace!
Don't modify your errors... but if you insist.
You can preserve the stacktrace (and error value) with sys.exc_info(), but this is way more error prone and has compatibility problems between Python 2 and 3, prefer to use a bare raise to re-raise.
To explain - the sys.exc_info() returns the type, value, and traceback.
type, value, traceback = sys.exc_info()
This is the syntax in Python 2 - note this is not compatible with Python 3:
raise AppError, error, sys.exc_info()[2] # avoid this.
# Equivalently, as error *is* the second object:
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]
If you want to, you can modify what happens with your new raise - e.g. setting new args for the instance:
def error():
raise ValueError('oops!')
def catch_error_modify_message():
try:
error()
except ValueError:
error_type, error_instance, traceback = sys.exc_info()
error_instance.args = (error_instance.args[0] + ' <modification>',)
raise error_type, error_instance, traceback
And we have preserved the whole traceback while modifying the args. Note that this is not a best practice and it is invalid syntax in Python 3 (making keeping compatibility much harder to work around).
>>> catch_error_modify_message()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in catch_error_modify_message
File "<stdin>", line 2, in error
ValueError: oops! <modification>
In Python 3:
raise error.with_traceback(sys.exc_info()[2])
Again: avoid manually manipulating tracebacks. It's less efficient and more error prone. And if you're using threading and sys.exc_info you may even get the wrong traceback (especially if you're using exception handling for control flow - which I'd personally tend to avoid.)
Python 3, Exception chaining
In Python 3, you can chain Exceptions, which preserve tracebacks:
raise RuntimeError('specific message') from error
Be aware:
this does allow changing the error type raised, and
this is not compatible with Python 2.
Deprecated Methods:
These can easily hide and even get into production code. You want to raise an exception, and doing them will raise an exception, but not the one intended!
Valid in Python 2, but not in Python 3 is the following:
raise ValueError, 'message' # Don't do this, it's deprecated!
Only valid in much older versions of Python (2.4 and lower), you may still see people raising strings:
raise 'message' # really really wrong. don't do this.
In all modern versions, this will actually raise a TypeError, because you're not raising a BaseException type. If you're not checking for the right exception and don't have a reviewer that's aware of the issue, it could get into production.
Example Usage
I raise Exceptions to warn consumers of my API if they're using it incorrectly:
def api_func(foo):
'''foo should be either 'baz' or 'bar'. returns something very useful.'''
if foo not in _ALLOWED_ARGS:
raise ValueError('{foo} wrong, use "baz" or "bar"'.format(foo=repr(foo)))
Create your own error types when apropos
"I want to make an error on purpose, so that it would go into the except"
You can create your own error types, if you want to indicate something specific is wrong with your application, just subclass the appropriate point in the exception hierarchy:
class MyAppLookupError(LookupError):
'''raise this when there's a lookup error for my app'''
and usage:
if important_key not in resource_dict and not ok_to_be_missing:
raise MyAppLookupError('resource is missing, and that is not ok.')
Don't do this. Raising a bare Exception is absolutely not the right thing to do; see Aaron Hall's excellent answer instead.
It can't get much more Pythonic than this:
raise Exception("I know Python!")
Replace Exception with the specific type of exception you want to throw.
See the raise statement documentation for Python if you'd like more information.
In Python 3 there are four different syntaxes for raising exceptions:
raise exception
raise exception (args)
raise
raise exception (args) from original_exception
1. Raise exception vs. 2. raise exception (args)
If you use raise exception (args) to raise an exception then the args will be printed when you print the exception object - as shown in the example below.
# Raise exception (args)
try:
raise ValueError("I have raised an Exception")
except ValueError as exp:
print ("Error", exp) # Output -> Error I have raised an Exception
# Raise exception
try:
raise ValueError
except ValueError as exp:
print ("Error", exp) # Output -> Error
3. Statement raise
The raise statement without any arguments re-raises the last exception.
This is useful if you need to perform some actions after catching the exception and then want to re-raise it. But if there wasn't any exception before, the raise statement raises a TypeError Exception.
def somefunction():
print("some cleaning")
a=10
b=0
result=None
try:
result=a/b
print(result)
except Exception: # Output ->
somefunction() # Some cleaning
raise # Traceback (most recent call last):
# File "python", line 8, in <module>
# ZeroDivisionError: division by zero
4. Raise exception (args) from original_exception
This statement is used to create exception chaining in which an exception that is raised in response to another exception can contain the details of the original exception - as shown in the example below.
class MyCustomException(Exception):
pass
a=10
b=0
reuslt=None
try:
try:
result=a/b
except ZeroDivisionError as exp:
print("ZeroDivisionError -- ",exp)
raise MyCustomException("Zero Division ") from exp
except MyCustomException as exp:
print("MyException",exp)
print(exp.__cause__)
Output:
ZeroDivisionError -- division by zero
MyException Zero Division
division by zero
For the common case where you need to throw an exception in response to some unexpected conditions, and that you never intend to catch, but simply to fail fast to enable you to debug from there if it ever happens — the most logical one seems to be AssertionError:
if 0 < distance <= RADIUS:
#Do something.
elif RADIUS < distance:
#Do something.
else:
raise AssertionError("Unexpected value of 'distance'!", distance)
Read the existing answers first, this is just an addendum.
Notice that you can raise exceptions with or without arguments.
Example:
raise SystemExit
exits the program, but you might want to know what happened. So you can use this.
raise SystemExit("program exited")
This will print "program exited" to standard error before closing the program.
Just to note: there are times when you do want to handle generic exceptions. If you're processing a bunch of files and logging your errors, you might want to catch any error that occurs for a file, log it, and continue processing the rest of the files. In that case, a
try:
foo()
except Exception as e:
print(e) # Print out handled error
block is a good way to do it. You'll still want to raise specific exceptions so you know what they mean, though.
Another way to throw an exception is using assert. You can use assert to verify a condition is being fulfilled. If not, then it will raise AssertionError. For more details have a look here.
def avg(marks):
assert len(marks) != 0, "List is empty."
return sum(marks)/len(marks)
mark2 = [55,88,78,90,79]
print("Average of mark2:", avg(mark2))
mark1 = []
print("Average of mark1:", avg(mark1))
You might also want to raise custom exceptions. For example, if you're writing a library, it's a very good practice to make a base exception class for your module, and then have custom sub-exceptions to be more specific.
You can achieve that like this:
class MyModuleBaseClass(Exception):
pass
class MoreSpecificException(MyModuleBaseClass):
pass
# To raise custom exceptions, you can just
# use the raise keyword
raise MoreSpecificException
raise MoreSpecificException('message')
If you're not interested in having a custom base class, you can just inherit your custom exception classes from an ordinary exception class like Exception, TypeError, ValueError, etc.
If you don't care about which error to raise, you could use assert to raise an AssertionError:
>>> assert False, "Manually raised error"
Traceback (most recent call last):
File "<pyshell#24>", line 1, in <module>
assert False, "Manually raised error"
AssertionError: Manually raised error
>>>
The assert keyword raises an AssertionError if the condition is False. In this case, we specified False directly, so it raises the error, but to have it have a text we want it to raise to, we add a comma and specify the error text we want. In this case, I wrote Manually raised error and this raises it with that text.
You should learn the raise statement of Python for that.
It should be kept inside the try block.
Example -
try:
raise TypeError # Replace TypeError by any other error if you want
except TypeError:
print('TypeError raised')
Suppose the following code.
try:
some_code_1
except: # will it be called twice, if an error occures in finally?
some_code_2
finally:
some_code_3
Suppose an exception occurs in some_code_3. Do I need an extra try-except clause around some_code_3 (see below) or will the exception with some_code_2 be called again, which in principle could cause an infinite loop?
Is this saver?
try:
some_code_1
except: # will it be called twice, if an error occures in finally?
some_code_2
finally:
try:
some_code_3
except:
pass
Just give it a try:
try:
print(abc) #Will raise NameError
except:
print("In exception")
finally:
print(xyz) #Will raise NameError
Output:
In exception
Traceback (most recent call last):
File "Z:/test/test.py", line 7, in <module>
print(xyz)
NameError: name 'xyz' is not defined
So no, it doesn't end up in an infinite loop
python doesn't go back in the execution flow, but rather statement by statement.
By the time it reaches finally, if an error is thrown there, it needs yet another handle
The finally in your sample code will not catch exception from some_code_3.
whether it's needed to catch exception from some_code_3 depends on your design.
Consider this example, with a deliberate typo:
try:
print("Hello!")
raise ValueError("?")
except ValueErro:
print("Error!")
finally:
print("World!")
The handling of the explicitly raised ValueError results in a NameError:
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
ValueError: ?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
NameError: name 'ValueErro' is not defined
However, if a ValueError is not raised:
try:
print("Hello!")
except ValueErro:
print("Error!")
finally:
print("World!")
then no exception occurs; Hello! and World! are simply printed.
Why is there no NameError caused by the typo ValueErro (no such name exists)? Shouldn't this have been detected ahead of time?
Can I use a different syntax to ensure that the problem is caught ahead of time?
After a comment from #DYZ, I found the right search terms to get the answer.
https://docs.python.org/3/tutorial/errors.html#handling-exceptions
The try statement works as follows.
First, the try clause (the statement(s) between the try and except keywords) is executed.
If no exception occurs, the except clause is skipped and execution of the try statement is finished.
Another resource exploring this problem.
https://dbaktiar-on-python.blogspot.com/2009/07/python-lazy-evaluation-on-exception.html
-
My Solution moving forward:
# Explicitly bind the Exception Names in a non-lazy fashion.
errors = (KeyboardInterrupt, ValueErro) # Caught!
try:
print("Hello!")
raise ValueError("?")
except errors:
print("Error!")
finally:
print("World!")
-
tl;dr - The except clauses are entirely skipped if the try clauses executes without exception.
Exceptions can be defined at run time. Example:
myexcept.py contains:
class ValueErro(Exception):
pass
Your program slightly modified:
import_myexcept = False
if import_myexcept:
from myexcept import ValueErro
try:
print("Hello!")
raise ValueError("?")
except ValueErro:
print("Error!")
finally:
print("World!")
This behaves just like your program: NameError: name 'ValueErro' is not defined.
But change: import_myexcept = True, and by the time the except ValueErro: statement is encountered, the ValueErro exception is defined and no NameError occurs.
I am doing this:
try: self.failUnless(sel.is_text_present("F!")) #sel.is_text_present("F!") is false
except AssertionError, e:
print("x"+e+"y")
sys.exit()
it is printing nothing except xy. no class name or anything else. what does the error in AssertionError normally contain?
edit: apparently the user provides its own message. selenium generated many of these:
except AssertionError, e: self.verificationErrors.append(str(e))
without sending in a message at all, so it appends a bunch of empty strings to verificationErrors.
Don't catch the errors from assertions. All the assertions in the unittest module take a final parameter, msg, which is the message to be raised if the assertion fails. Put your debugging there if necessary.
Standard assert statement doesn't put anything into the AssertionError, it's the traceback that matters. There is a assert expr, msg variant that sets the error message, if you're using unittest, then the second argument of assertTrue (failUnless is deprecated) will do it.
Sounds like you want to use your assertions as debug statements in the event of a failure. This should help...
import traceback
try:
assert 1 == 2
except AssertionError:
traceback.print_exc()
This prints:
Traceback (most recent call last):
File "./foo.py", line 4, in <module>
assert 1 == 2