I'm trying to find the cause of a crash in of our python scripts.
The main structure is this:
def main()
try:
dostuff
except Exception as ex:
import traceback
tb = traceback.format_exc()
import platform
node = platform.node()
sendMail([DEBUG_EMAIL], "Alarm exception on %s" % node, str(tb), [])
I get this stacktrace in our main error handling, not in an error email which I'm supposed to.
Traceback (most recent call last):
File "/usr/lib/python2.6/logging/__init__.py", line 799, in emit
stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 66: ordinal not in range(128)
From what I see all of the write-calls to logger are inside the try-block, but since it's not caught and handled in my email sending exception block it seems I've missed something. I've checked and the sendMail function doesn't use the logging module at all. So the exception shouldn't originate in my except-block.
I tried adding
sys.tracebacklimit = 10
at the top of the file see where the exception originates but that didn't affect anything. And now I'm out of ideas on how to find where the problem originates.
The script runs once every hour and only crashes about once per week, which makes me assume that it's something related to the input data, but that's only handled by dostuff().
UPDATE:
I've figured out why I only get one row of the stacktrace. Inside the emit() I found this.
try:
... doing stuff, something goes boom with encoding...
except UnicodeError:
stream.write(fs % msg.encode("UTF-8")) Here it goes Boom again
self.flush()
except (KeyboardInterrupt, SystemExit):
raise
except:
self.handleError(record) Which means it ends up here
And the relevant part of the handleError function looks like this:
ei = sys.exc_info()
try:
traceback.print_exception(ei[0], ei[1], ei[2], None, sys.stderr)
Which only prints the last part of the stacktrace.
Basically your problem is twofold
One log stream does not accept 8-bit strings with extended characters, and throws UnicodeError
There is a silly bug in logging module which make it lose the original traceback
The exact cause of the exception is this:
>>> 'ä'.encode('UTF-8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
But this exception is not the real problem. This is the part of the 2.6 logging code; the 799 is the last line of the block. The last line is the one causing the problems. Basically something logs a message in a 8-bit byte string, UTF-8 encoded, containing Latin-1 extended letters; but the stream does not like that, and throws an UnicodeError within the try block;
try:
if (isinstance(msg, unicode) and
getattr(stream, 'encoding', None)):
# .... the string is NOT an unicode instance, so ignored
# for brevity
else:
# this line throws the original exception
# fs is a bytestring "%s\n", and msg is a bytestring
# with extended letters, most probably Latin 1.
# stream.write spits out an UnicodeError on these values
stream.write(fs % msg)
except UnicodeError:
# now we get a useless exception report from this code
stream.write(fs % msg.encode("UTF-8"))
So to debug this you'd want to set up a breakpoint on the aforementioned line 799, and try all loggers if they accept the following string:
logging.getLogger(name).critical('Testing logger: ä')
If you hit line 799 then get the backtrace of the exception, it can shed some light on what is happening...
Related
I'm setting up a logger in my script like shown at the bottom. This works fine for my purposes and logs my __main__ log messages and those of any modules I use to stdout and a log file.
During program execution a module call that I'm using xarray.open_dataset(file, engine="cfgrib") raises an Error in some conditions and produces the following log output:
2023-02-18 10:02:06,731 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
...
How can I access this output during program execution?
The raised error in the cfgrib module is handled there gracefully and program execution can continue, but the logic of my program requires that I access the error message, in particular the part saying shortName='i10fg' in order to handle the error exhaustively.
Here is how my logger is set up:
def init_log():
"""initialize logging
returns logger using log settings from the config file (settings.toml)
"""
# all settings from a settings file with reasonable defaults
lg.basicConfig(
level=settings.logging.log_level,
format=settings.logging.format,
filemode=settings.logging.logfile_mode,
filename=settings.logging.filename,
)
mylogger = lg.getLogger(__name__)
stream = lg.StreamHandler()
mylogger.addHandler(stream)
clg.install(
level=settings.logging.log_level,
logger=mylogger,
fmt="%(asctime)s %(levelname)s:\t%(message)s",
)
return mylogger
# main
log = init_log()
log.info('...reading files...')
I went through the python logging documentation and cookbook. While this contains ample examples on how to modify logging for various purposes, I could not find an example for accessing and reacting to a log message during program execution.
The Exception in my logs look this:
2023-02-20 12:22:37,209 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 660, in build_dataset_components
dict_merge(variables, coord_vars)
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 591, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='time' value=Variable(dimensions=('time',), data=array([1640995200, 1640998800, 1641002400, ..., 1672520400, 1672524000,
1672527600])) new_value=Variable(dimensions=('time',), data=array([1640973600, 1641016800, 1641060000, 1641103200, 1641146400,
1641189600, 1641232800, 1641276000, 1641319200, 1641362400,
I cannot catch the Exception directly for some reason:
...
import sys
from cfgrib.dataset import DatasetBuildError
...
try:
df = xr.open_dataset(file, engine="cfgrib").to_dataframe()
# triggering error manually like with the two lines below works as expected
# raise Exception()
# raise DatasetBuildError()
except Exception as e:
print('got an Exception')
print(e)
print(e.args)
except BaseException as e:
print('got a BaseException')
print(e.args)
except DatasetBuildError as e:
print(e)
except:
print('got any and all exception')
type, value, traceback = sys.exc_info()
print(type)
print(value)
print(traceback)
Unless I uncomment the two lines where I raise the exception manually, the except clauses are never triggered, event though I can see the DatabaseBuildError in my logs.
Not sure if this has any bearing, but while I can see the Exception as quoted above in my file log, it is not printed to stdout.
The script I am writing should exit back to the shell prompt with a helpful message if the data to be processed is not exactly right. The user should fix the problems flagged until the script is happy and no longer exits with error messages. I am developing the script with TTD, so I write a pytest test before I write the function.
The most heavily up-voted answer here suggests that scripts be edited by calling sys.exit or raising SystemExit.
The function:
def istext(file_to_test):
try:
open(file_to_test).read(512)
except UnicodeDecodeError:
sys.exit('File {} must be encoded in UTF-8 (Unicode); try converting.'.format(file_to_test))
passes this test (where _non-text.png is a PNG file, i.e., not encoded in UTF-8):
def test_istext():
with pytest.raises(SystemExit):
istext('_non-text.png')
However, the script continues to run, and statements placed after the try/except block execute.
I want the script to completely exit every time so that the user can debug the data until it is correct, and the script will do what it is supposed to do (which is to process a directory full of UTF-8 text files, not PNG, JPG, PPTX... files).
Also tried:
The following also passes the test above by raising an exception that is a sub-class of SystemExit, but it also does not exit the script:
def istext(file_to_test):
class NotUTF8Error(SystemExit): pass
try:
open(file_to_test).read(512)
except UnicodeDecodeError:
raise NotUTF8Error('File {} must be UTF-8.'.format(file_to_test))
You can use raise Exception from exception syntax:
class MyException(SystemExit):
pass
def istext(file_to_test):
try:
open(file_to_test).read(512)
except UnicodeDecodeError as exception:
raise MyException(f'File {file_to_test} must be encoded in UTF-8 (Unicode); try converting.') \
from exception
I this case you doesn't change original error message and add your own message.
The try...except block is for catching an error and handling it internally. What you want to do is to re-raise the error.
def istext(file_to_test):
try:
open(file_to_test).read(512)
except UnicodeDecodeError:
print(('File {} must be encoded in UTF-8 (Unicode); try converting.'.format(file_to_test)))
raise
This will print your message, then automatically re-raise the error you've caught.
Instead of just re-raising the old error, you might want to change the error type as well. For this case, you specify raise further, e.g.:
raise NameError('I'm the shown error message')
You problem is not how to exit a program (sys.exit() works fine). You problem is that your test scenario is not raising a UnicodeDecodeError.
Here's a simplified version of your example. It works as expected:
import pytest
import sys
def foo(n):
try:
1/n
except ZeroDivisionError as e:
sys.exit('blah')
def test_foo():
# Assertion passes.
with pytest.raises(SystemExit):
foo(0)
# Assertion fails: "DID NOT RAISE <type 'exceptions.SystemExit'>"
with pytest.raises(SystemExit):
foo(9)
Add some diagnostic printing to your code to learn more. For example:
def istext(file_to_test):
try:
content = open(file_to_test).read(512)
# If you see this, no error occurred. Maybe your source
# file needs different content to trigger UnicodeDecodeError.
print('CONTENT len()', len(content))
except UnicodeDecodeError:
sys.exit('blah')
except Exception as e:
# Maybe some other type of error should also be handled?
...
In the end, what worked is similar to what #ADR proposed, with one difference: I was not able to get the formatted string syntax shown above to work correctly (f'File {file_to_test} must...'), nor could I find documentation of the f prefix for strings.
My slightly less elegant solution, then, for the (renamed) function:
def is_utf8(file):
class NotUTF8Error(SystemExit): pass
try:
open(file).read(512)
except UnicodeDecodeError as e:
raise NotUTF8Error('File {} not UTF-8: convert or delete, then retry.'.format(file)) from e
passes the pytest:
def test_is_utf81():
with pytest.raises(SystemExit):
is_utf8('/Users/tbaker/github/tombaker/mklists/mklists/_non-text.png')
I am working with Twilio's API to return information about phone numbers. Some of the phone numbers are invalid and return an error such as
Traceback (most recent call last):
File "test_twilio.py", line 17, in <module>
number = client.lookups.phone_numbers("(4154) 693-
6078").fetch(type="carrier")
File "/Users/jawnsano/anaconda/lib/python2.7/site-
packages/twilio/rest/lookups/v1/phone_number.py", line 158, in fetch
params=params,
File "/Users/jawnsano/anaconda/lib/python2.7/site-
packages/twilio/base/version.py", line 82, in fetch
raise self.exception(method, uri, response, 'Unable to fetch
record')
twilio.base.exceptions.TwilioRestException:
HTTP Error Your request was:
GET /PhoneNumbers/(4154) 693-6078
Twilio returned the following information:
Unable to fetch record: The requested resource /PhoneNumbers/(4154)
693-6078 was not found
More information may be available here:
https://www.twilio.com/docs/errors/20404
If an error like the one shown above is returned, I want to print 'There is an error.' However, for my if statement, is there a way to make Python print that for when there is a traceback error/error in general? I think there is probably a better way than setting making it like
if returned_value = (super long error message):
etc...
You use try and except to catch errors.
from twilio.base.exceptions import TwilioRestException
try:
... your code
except TwilioRestException:
print("whatever")
For this specific exception:
try:
the_function_that_raises_the_exception()
except twilio.base.exceptions.TwilioRestException as e:
print("Oops, exception encountered:\n" + str(e))
Note that you probably need to call import twilio.base.exceptions first.
For any exception:
try:
the_function_that_raises_the_exception()
except Exception as e:
print(e)
Be careful when using the second approach though - this catches ALL exceptions, and might mask a bigger problem if not addressed properly. Unless you know where the exception originates from (but if that is the case, you know the type and you can filter only that type), sometimes this approach can be used:
try:
the_function_that_can_raise_numerous_exceptions()
except Exception as e:
with open("exceptions.txt", "a") as f:
f.write(e)
# or even send an email here
raise
This makes sure that the exception is caught (by except), then written to a file and then reraised. This will still cause the script to fail, but will have a log to look into.
MWE of my code distribution:
main.py
../func1.py
From main.py I call func1.py with:
data_list = [elem1, .., elemN] # Data input.
params = [1., 2., 5.] # Some parameters.
for elem in data_list:
try:
func1(elem, params) # Call function.
except Exception:
print traceback.format_exc()
This way if the function fails for some element, the main code keeps running executing the remaining elements in the list.
I want to insert a custom error message for a given block of func1, so I've defined:
try:
# try something
except ValueError:
raise ValueError('Custom error message.')
When a ValueError occurs in func1 the output I get with this, before jumping to the next element in data_list, is:
Traceback (most recent call last):
File "/main.py", line 44, in main
func1(params)
File "/func1.py", line 68, func1
raise ValueError('Custom error message.')
ValueError: Custom error message.
Why is the custom error message being printed twice?
The exception is not raised twice. There can be only one exception "up in the air".
What you are seeing is the whole traceback, an extraordinary help when it comes to finding out why and where your programme crashed, It prints line by line all the frames and ends in the line where the exception was thrown. Therefore you can read your message "again".
If you catch it and print it, you will only see the exception itself. For instance
>>> try:
... raise ValueError('Custom error message.')
... except ValueError as exc:
... print exc
...
Custom error message.
>>>
raise ValueError('Custom error message.')
ValueError: Custom error message.
"Why is the custom error message being printed twice?"
this is the last code executed in the Tracedback:
raise ValueError('Custom error message.')
This is the actual exception being echoed on your REPL:
ValueError: Custom error message.
It is not.
You are seeing the top printing because of the traceback
The first time it's printed, the interpreter is quoting the code that raised the exception, namely line 68 of func1.py.
The second time (the last line of your output), it printed the exception message.
This is a followup to my question Hang in Python script using SQLAlchemy and multiprocessing. As discussed in that question, pickling exceptions is problematic in Python. This is usually not a issue, but one case when it is, is when errors occur in the python multiprocessing module. Since multiprocessing moves objects around by pickling, if an error occurs inside a multiprocessing process, the entire process may hang, as demonstrated in that question.
One possible approach is to fix all the problematic exceptions, as discussed in that question. This is not easy, since one cannot easily know in advance which exceptions may be called. An alternative approach, which was suggested by lbolla in an answer to the question, is to catch the exception, construct an equivalent harmless exception, and then rethrow.
However, I'm not sure of exactly how to do this. Consider the following code.
class BadExc(Exception):
def __init__(self, message, a):
'''Non-optional param in the constructor.'''
Exception.__init__(self, message)
self.a = a
import sys
try:
try:
#print foo
raise BadExc("bad exception error message", "a")
except Exception, e:
raise Exception(e.__class__.__name__ + ": " +str(e)), None, sys.exc_info()[2]
except Exception, f:
pass
import cPickle
a = cPickle.dumps(f)
l = cPickle.loads(a)
print "raising error"
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]
This code pickles and unpickles the exception, and then throws it, giving the error
raising error
Traceback (most recent call last):
File "<stdin>", line 11, in <module>
Exception: BadExc: bad exception error message
Credits to Glenn Maynard's answer to "“Inner exception” (with traceback) in Python?".
This has the important stuff, namely the traceback, the error message, and the exception type, so this might be the best one can do. But ideally I'd like something that looks exactly like the original exception, namely
Traceback (most recent call last):
File "<stdin>", line 11, in <module>
__main__.BadExc: bad exception error message
or more generally, with the name of the exception in the front, rather than Exception. Is this possible?
Alternatively, instead of the BadExc class, one can use the print foo statement instead, which gives a NameError. However, this exception does not require special handling.
You can override sys.excepthook to achieve what you want. It at least works for this example, but it's pretty hacky so please test and no promises :-)
import sys
def excepthook_wrapper(type, value, traceback):
if len(value.args) == 2:
name, msg = value.args
value.args = (msg,)
sys.__excepthook__(name, value, traceback)
else:
sys.__excepthook__(type, value, traceback)
sys.excepthook = excepthook_wrapper
(Edit: I'm not really happy with this because now 'normal' Exceptions with two arguments will get handled differently too. Possible solution, 'tag' your special Exceptions by passing "PICKLED" as a first argument and then check for that, instead of checking for the length of the args.)
And then create the Exception with two arguments, the name (__module__.__class__) and the Exception message (str(e)):
try:
try:
#print foo
raise BadExc("bad exception error message", "a")
except Exception, e:
cls = e.__class__
if hasattr(cls, '__module__'):
name = '{0}.{1}'.format(cls.__module__, cls.__name__)
else:
name = cls.__name__
raise Exception(name, str(e)), None, sys.exc_info()[2]
except Exception, f:
pass
Then this:
import cPickle
a = cPickle.dumps(f)
l = cPickle.loads(a)
print "raising error"
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]
Prints:
raising error
Traceback (most recent call last):
File "test.py", line 18, in <module>
raise BadExc("bad exception error message", "a")
__main__.BadExc: bad exception error message