The following snippet:
import traceback
def a():
b()
def b():
try:
c()
except:
traceback.print_exc()
def c():
assert False
a()
Produces this output:
Traceback (most recent call last):
File "test.py", line 8, in b
c()
File "test.py", line 13, in c
assert False
AssertionError
What should I use if I want the complete stack trace including the call to a?
If it matters I have Python 2.6.6
edit: What I'd like to get is the same information I'd get if I left the try except out and let the exception propagate to the top level. This snippet for example:
def a():
b()
def b():
c()
def c():
assert False
a()
Produces this output:
Traceback (most recent call last):
File "test.py", line 10, in <module>
a()
File "test.py", line 2, in a
b()
File "test.py", line 5, in b
c()
File "test.py", line 8, in c
assert False
AssertionError
Here's a function based on this answer. It will also work when no exception is present:
def full_stack():
import traceback, sys
exc = sys.exc_info()[0]
stack = traceback.extract_stack()[:-1] # last one would be full_stack()
if exc is not None: # i.e. an exception is present
del stack[-1] # remove call of full_stack, the printed exception
# will contain the caught exception caller instead
trc = 'Traceback (most recent call last):\n'
stackstr = trc + ''.join(traceback.format_list(stack))
if exc is not None:
stackstr += ' ' + traceback.format_exc().lstrip(trc)
return stackstr
print full_stack() will print the full stack trace up to the top, including e.g. IPython's interactiveshell.py calls, since there is (to my knowledge) no way of knowing who would catch exceptions. It's probably not worth figuring out anyway...
If print full_stack() is called from within an except block, full_stack will include the stack trace down to the raise. In the standard Python interpreter, this will be identical to the message you receive when not catching the exception (Which is why that del stack[-1] is there, you don't care about the except block but about the try block).
I don't know if there is a better way, but here's what I did:
import traceback
import sys
def format_exception(e):
exception_list = traceback.format_stack()
exception_list = exception_list[:-2]
exception_list.extend(traceback.format_tb(sys.exc_info()[2]))
exception_list.extend(traceback.format_exception_only(sys.exc_info()[0], sys.exc_info()[1]))
exception_str = "Traceback (most recent call last):\n"
exception_str += "".join(exception_list)
# Removing the last \n
exception_str = exception_str[:-1]
return exception_str
def main1():
main2()
def main2():
try:
main3()
except Exception as e:
print "Printing only the traceback above the current stack frame"
print "".join(traceback.format_exception(sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]))
print
print "Printing the full traceback as if we had not caught it here..."
print format_exception(e)
def main3():
raise Exception()
if __name__ == '__main__':
main1()
And here's the output I get:
Printing only the traceback above the current stack frame
Traceback (most recent call last):
File "exc.py", line 22, in main2
main3()
File "exc.py", line 31, in main3
raise Exception()
Exception
Printing the full traceback as if we had not caught it here...
Traceback (most recent call last):
File "exc.py", line 34, in <module>
main1()
File "exc.py", line 18, in main1
main2()
File "exc.py", line 22, in main2
main3()
File "exc.py", line 31, in main3
raise Exception()
Exception
Use
traceback.print_stack()
http://docs.python.org/library/traceback.html#traceback.print_stack
suxmac2 $ python out.py
File "out.py", line 16, in <module>
a()
File "out.py", line 5, in a
b()
File "out.py", line 11, in b
traceback.print_stack()
Here is a bit better variant of Tobias Kienzler answer. It works same, but can be called not right in except block, but somewhere deeper.
In other words, this variant will print same stacks, when called like
try:
...
except Exception:
print full_stack()
or
def print_full_stack():
print full_stack()
try:
...
except Exception:
print_full_stack()
Here is code:
def full_stack():
import traceback, sys
exc = sys.exc_info()[0]
if exc is not None:
f = sys.exc_info()[-1].tb_frame.f_back
stack = traceback.extract_stack(f)
else:
stack = traceback.extract_stack()[:-1] # last one would be full_stack()
trc = 'Traceback (most recent call last):\n'
stackstr = trc + ''.join(traceback.format_list(stack))
if exc is not None:
stackstr += ' ' + traceback.format_exc().lstrip(trc)
return stackstr
Related
I'm utilizing the following code to print exceptions from a python module I'm running inside of my script:
except Exception as e:
logging.info('yt-dlp initated a break', exc_info=True)
Currently it outputs much more information than I need.
Something along the lines of:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/web_ripper.py", line 58, in subscriptions_loop
ydl.download([url])
File "/app/venv/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 3113, in download
self.__download_wrapper(self.extract_info)(
File "/app/venv/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 3086, in wrapper
res = func(*args, **kwargs)
File "/app/venv/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 1344, in extract_info
return self.__extract_info(url, self.get_info_extractor(ie_key), download, extra_info, process)
File "/app/venv/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 1371, in wrapper
self.report_error(str(e), e.format_traceback())
File "/app/venv/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 912, in report_error
self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
File "/app/venv/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 853, in trouble
raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: [youtube:tab] Videogamedunkey: Unable to download API page: HTTP Error 404: Not Found
What's the most efficient way to simply print the last line of the exception?
You can use a custom Formatter that will process the exc_text of the record you are trying to log.
The following MyFormatter will check if the record has an exception text. If not, it will create it using self.formatException. Then, since record.exc_text is a string storing the trace as text, with \n as separators between lines, you can keep the last line using .split("\n")[-1].
import logging
class MyFormatter(logging.Formatter):
def format(self, record):
# https://github.com/python/cpython/blob/main/Lib/logging/__init__.py#L711
if not record.exc_text:
record.exc_text = self.formatException(record.exc_info)
record.exc_text = record.exc_text.split("\n")[-1]
return super().format(record)
Then, add the following code to use your new MyFormatter:
def a(x):
return b(x)
def b(x):
return c(x)
def c(x):
return d(x)
def d(x):
return x / 0
logger = logging.getLogger("foobar")
logger.setLevel(logging.INFO)
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(MyFormatter(fmt="%(name)s - %(levelname)s - %(message)s"))
logger.addHandler(stream_handler)
try:
a(1)
except Exception as e:
logger.info('yt-dlp initated a break', exc_info=True)
and you get:
foobar - INFO - yt-dlp initated a break
ZeroDivisionError: division by zero
instead of the previous:
foobar - INFO - yt-dlp initated a break
Traceback (most recent call last):
File "70704112.py", line 31, in <module>
a(1)
File "70704112.py", line 12, in a
return b(x)
File "70704112.py", line 15, in b
return c(x)
File "70704112.py", line 18, in c
return d(x)
File "70704112.py", line 21, in d
return x / 0
ZeroDivisionError: division by zero
I am trying to understand how to use the pdb.post_mortem() method.
for this given file
# expdb.py
import pdb
import trace
def hello():
a = 6 * 9
b = 7 ** 2
c = a * b
d = 4 / 0
print(c)
tracer = trace.Trace()
Command prompt
'''
# first Try
λ python -i expdb.py
>>> pdb.post_mortem()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Anaconda3\lib\pdb.py", line 1590, in post_mortem
raise ValueError("A valid traceback must be passed if no "
ValueError: A valid traceback must be passed if no exception is being handled
'''
'''
# Second Try
λ python -i expdb.py
>>> pdb.post_mortem(traceback=tracer.run('hello()') )
--- modulename: trace, funcname: _unsettrace
trace.py(77): sys.settrace(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Anaconda3\lib\trace.py", line 500, in run
self.runctx(cmd, dict, dict)
File "C:\Program Files\Anaconda3\lib\trace.py", line 508, in runctx
exec(cmd, globals, locals)
File "<string>", line 1, in <module>
File "expdb.py", line 8, in hello
d = 4 / 0
ZeroDivisionError: division by zero
>>>
The post_mortem method wants a traceback object, not a Trace object. Traceback objects can be acquired from sys.exc_info()[2] inside of an except block, or you can simply call pdb.post_mortem() with no arguments directly (in the except block).
But either way, you must catch the exception before you can debug it.
I read Get last function's call arguments from traceback? but it is not specific enough to answer my problem.
This is really bothering me since not having the call arguments is slowing me down and I am pretty sure that it is possible to get this kind of info from Python.
Here is an example to illustrate the problem:
# -*- coding: utf-8 -*-
import sys
import traceback
import inspect
import logging as log
def fl(x):
# exception is happening here
y = 5/x
return y
def fm(x):
return fl(x-3)
def fn(a, b, c=None):
return fm(c)
def main():
try:
print fn(1, 2, c=3)
except Exception as e:
log.error('Unexpected problem.')
log.error(e)
traceback.print_exc()
### what I need to see is are the call arguments of the last / deepest call: ###
### def fl(x) was called with arguments: [(x, 3)] ###
# this does not cut it:
tb = sys.exc_info()[2]
traceback.print_tb(tb)
# this is broken:
#frames = inspect.getinnerframes(tb)
#log.error('Argvalues: %s', inspect.getargvalues(frames))
# not sure:
frames = inspect.trace()
argvalues = inspect.getargvalues(frames[0][0])
log.error('Argvalues: %s', inspect.formatargvalues(*argvalues))
if __name__ == '__main__':
main()
so I get details, but the call arguments are not contained:
ERROR:root:Unexpected problem.
ERROR:root:integer division or modulo by zero
Traceback (most recent call last):
File "sample.py", line 24, in main
print fn(1, 2, c=3)
File "sample.py", line 18, in fn
return fm(c)
File "sample.py", line 14, in fm
return fl(x-3)
File "sample.py", line 9, in fl
y = 5/x
ZeroDivisionError: integer division or modulo by zero
File "sample.py", line 24, in main
print fn(1, 2, c=3)
File "sample.py", line 18, in fn
return fm(c)
File "sample.py", line 14, in fm
return fl(x-3)
File "sample.py", line 9, in fl
y = 5/x
ERROR:root:Argvalues: ()
frames[0][0] represents main function. main is called without arguments, thats why you get empty tuple. Change it to frames[-1][0] to get last frame.
I'm using gevent on Python 2.7 to do some multithreaded work. However, I am unable to catch exceptions that are raised in the spawned methods. I am using the .get() method on the returned AsyncResult object (returned when calling the .spawn() method on the thread pool).
Per the gevent documentation: http://www.gevent.org/gevent.event.html#gevent.event.AsyncResult.get:
get(block=True, timeout=None)
Return the stored value or raise the exception.
If this instance already holds a value or an exception, return or raise it immediately.
However, instead of returning the exception, .get() returns NoneType. Additionally, in the console, the exception details are printed out. Instead, .get() should return the Exception back.
Here is some sample code and corresponding output that showcases this:
from gevent import threadpool
def this_will_fail():
raise Exception('This will fail')
result_list = []
for i in range(1,5):
tpool = threadpool.ThreadPool(5)
result_list.append(tpool.spawn(this_will_fail))
tpool.join()
for result in result_list:
try:
returned_object = result.get()
print type(returned_object)
except Exception as e:
print 'This is not running for some reason...'
Output:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
value = func(*args, **kwargs)
File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c79e50 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
value = func(*args, **kwargs)
File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c99210 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
value = func(*args, **kwargs)
File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c99410 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
value = func(*args, **kwargs)
File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c99610 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception
<type 'NoneType'>
<type 'NoneType'>
<type 'NoneType'>
<type 'NoneType'>
Am I missing something obvious, or is this a bug in gevent?
EDIT: concurrent.futures doesn't have this problem. While that is an acceptable solution for my use case, I would still like to understand why gevent doesn't properly return the exception.
Here we had the same problem with the gevent threadpool. We are using a solution which wrap the function to return exceptions as result, and raise it again when we want to use the result from the AsynResult.
class _ExceptionWrapper:
def __init__(self, exception, error_string, tb):
self.exception = exception
self.error_string = error_string
self.tb = tb
class wrap_errors(object):
def __init__(self, func):
self.func = func
def __call__(self, *args, **kwargs):
func = self.func
try:
return func(*args, **kwargs)
except:
return _ExceptionWrapper(*sys.exc_info())
def __str__(self):
return str(self.func)
def __repr__(self):
return repr(self.func)
def __getattr__(self, item):
return getattr(self.func, item)
def get_with_exception(g, block=True, timeout=None):
result = g._get(block, timeout)
if isinstance(result, _ExceptionWrapper):
# raise the exception using the caller context
raise result.error_string, None, result.tb
else:
return result
def spawn(fn, *args, **kwargs):
# wrap the function
fn = wrap_errors(fn)
g = threadpool.spawn(fn, *args, **kwargs)
# and the asynresult
g._get = g.get
g.get = types.MethodType(get_with_exception, g)
return g
Here is my code (almost full version for #cdhowie :)):
def getResult(method, argument=None):
result = None
while True:
print('### loop')
try:
print ('### try hard...')
if argument:
result = method(argument)
else:
result = method()
break
except Exception as e:
print('### GithubException')
if 403 == e.status:
print('Warning: ' + str(e.data))
print('I will try again after 10 minutes...')
else:
raise e
return result
def getUsernames(locations, gh):
usernames = set()
for location in locations:
print location
result = getResult(gh.legacy_search_users, location)
for user in result:
usernames.add(user.login)
print user.login,
return usernames
# "main.py"
gh = Github()
locations = ['Washington', 'Berlin']
# "main.py", line 12 is bellow
usernames = getUsernames(locations, gh)
The problem is, that exception is raised, but I can't handle it. Here is an output:
### loop
### try hard...
Traceback (most recent call last):
File "main.py", line 12, in <module>
usernames = getUsernames(locations, gh)
File "/home/ciembor/projekty/github-rank/functions.py", line 39, in getUsernames
for user in result:
File "/usr/lib/python2.7/site-packages/PyGithub-1.8.0-py2.7.egg/github/PaginatedList.py", line 33, in __iter__
newElements = self.__grow()
File "/usr/lib/python2.7/site-packages/PyGithub-1.8.0-py2.7.egg/github/PaginatedList.py", line 45, in __grow
newElements = self._fetchNextPage()
File "/usr/lib/python2.7/site-packages/PyGithub-1.8.0-py2.7.egg/github/Legacy.py", line 37, in _fetchNextPage
return self.get_page(page)
File "/usr/lib/python2.7/site-packages/PyGithub-1.8.0-py2.7.egg/github/Legacy.py", line 48, in get_page
None
File "/usr/lib/python2.7/site-packages/PyGithub-1.8.0-py2.7.egg/github/Requester.py", line 69, in requestAndCheck
raise GithubException.GithubException(status, output)
github.GithubException.GithubException: 403 {u'message': u'API Rate Limit Exceeded for 11.11.11.11'}
Why it doesn't print ### handling exception?
Take a close look at the stack trace in the exception:
Traceback (most recent call last):
File "main.py", line 12, in <module>
usernames = getUsernames(locations, gh)
File "/home/ciembor/projekty/github-rank/functions.py", line 39, in getUsernames
for user in result:
File "/usr/lib/python2.7/site-packages/PyGithub-1.8.0-py2.7.egg/github/PaginatedList.py", line 33, in __iter__
newElements = self.__grow()
...
The exception is being thrown from code being called by the line for user in result: after getResult finishes executing. This means that the API you're using is using lazy evaluation, so the actual API request doesn't quite happen when you expect it to.
In order to catch and handle this exception, you'll need to wrap the code inside the getUsernames function with a try/except handler.