Develop Reference

Get line number of a python function call in exception

I have a function like this
def try_strip(s):
try:
return s.strip()
except Exception as e:
print(e)
# (I've tried inspect, traceback, logging, sys)
and if I call it somewhere like this
try_strip('could be a string or not')
then the exception line number would be the one in which the try_strip is defined.
Is there a way to get info about where was it called? Thank you in advance.

The Traceback module which is included with Python provides this functionality. According to its documentation it:
provides a standard interface to extract, format and print stack traces of Python programs. It exactly mimics the behaviour of the Python interpreter when it prints a stack trace.
The function traceback.format_stack() will return the stack trace information you need as a list of strings while the function traceback.print_stack() will print the stack trace information to the console. Below I have included some code which shows how you might use this in the example you provided:
import traceback
def try_strip(s):
try:
return s.strip()
except Exception as e:
traceback.print_stack()
stack_trace_info = traceback.format_stack()
# Code that write stack_trace_info to a log could go here
try_strip(5) # This will cause an error at runtime
For additional info on the Traceback module, please see https://docs.python.org/3/library/traceback.html .

Jupyter magic to handle notebook exceptions

I have a few long-running experiments in my Jupyter Notebooks. Because I don't know when they will finish, I add an email function to the last cell of the notebook, so I automatically get an email, when the notebook is done.
But when there is a random exception in one of the cells, the whole notebook stops executing and I never get any email. So I'm wondering if there is some magic function that could execute a function in case of an exception / kernel stop.
Like
def handle_exception(stacktrace):
send_mail_to_myself(stacktrace)
%%in_case_of_notebook_exception handle_exception # <--- this is what I'm looking for
The other option would be to encapsulate every cell in try-catch, right? But that's soooo tedious.
Thanks in advance for any suggestions.

Such a magic command does not exist, but you can write it yourself.
from IPython.core.magic import register_cell_magic
#register_cell_magic('handle')
def handle(line, cell):
try:
exec(cell)
except Exception as e:
send_mail_to_myself(e)
raise # if you want the full trace-back in the notebook
It is not possible to load the magic command for the entire notebook automatically, you have to add it at each cell where you need this feature.
%%handle
some_code()
raise ValueError('this exception will be caught by the magic command')

#show0k gave the correct answer to my question (in regards to magic methods). Thanks a lot! :)
That answer inspired me to dig a little deeper and I came across an IPython method that lets you define a custom exception handler for the whole notebook.
I got it to work like this:
from IPython.core.ultratb import AutoFormattedTB
# initialize the formatter for making the tracebacks into strings
itb = AutoFormattedTB(mode = 'Plain', tb_offset = 1)
# this function will be called on exceptions in any cell
def custom_exc(shell, etype, evalue, tb, tb_offset=None):
# still show the error within the notebook, don't just swallow it
shell.showtraceback((etype, evalue, tb), tb_offset=tb_offset)
# grab the traceback and make it into a list of strings
stb = itb.structured_traceback(etype, evalue, tb)
sstb = itb.stb2text(stb)
print (sstb) # <--- this is the variable with the traceback string
print ("sending mail")
send_mail_to_myself(sstb)
# this registers a custom exception handler for the whole current notebook
get_ipython().set_custom_exc((Exception,), custom_exc)
So this can be put into a single cell at the top of any notebook and as a result it will do the mailing in case something goes wrong.
Note to self / TODO: make this snippet into a little python module that can be imported into a notebook and activated via line magic.
Be careful though. The documentation contains a warning for this set_custom_exc method: "WARNING: by putting in your own exception handler into IPython’s main execution loop, you run a very good chance of nasty crashes. This facility should only be used if you really know what you are doing."

Since notebook 5.1 you can use a new tag: raises-exception
This will indicate that exception in the specific cell is expected and jupyter will not stop the execution.
(In order to set a tag you have to choose from the main menu: View -> Cell Toolbar -> Tags)

Why exec is not always the solution
It's some years later and I had a similar issue trying to handle errors with Jupyter magics. However, I needed variables to persist in the actual Jupyter notebook.
%%try_except print
a = 12
raise ValueError('test')
In this example, I want the error to print (but could be anything such as e-mail as in the opening post), but also a == 12 to be true in the next cell. For that reason, the method exec suggested does not work when you define the magic in a different file. The solution I found is to use the IPython functionalities.
How you can solve it
from IPython.core.magic import line_magic, cell_magic, line_cell_magic, Magics, magics_class
#magics_class
class CustomMagics(Magics):
#cell_magic
def try_except(self, line, cell):
""" This magic wraps a cell in try_except functionality """
try:
self.shell.ex(cell) # This executes the cell in the current namespace
except Exception as e:
if ip.ev(f'callable({how})'): # check we have a callable handler
self.shell.user_ns['error'] = e # add error to namespace
ip.ev(f'{how}(error)') # call the handler with the error
else:
raise e
# Register
from IPython import get_ipython
ip = get_ipython()
ip.register_magics(CustomMagics)

I don't think there is an out-of-the-box way to do that not using a try..except statement in your cells. AFAIK a 4 years old issue mentions this, but is still in open status.
However, the runtools extension may do the trick.

Can I force debugging python on AssertionError?

Suppose I have a python program where assert has been used to define how things should be, and I would like to capture anomalies with the read-eval-loop rather than having AssertionError be thrown.
Granted, I could have
if (reality!=expectation):
print("assertion failed");
import pdb; pdb.set_trace();
but that's far more ugly in the code than a plain assert(reality==expectation).
I could have pdb.set_trace() called in an except: block at top-level, but then I'd have lost all the context of the failure, right ? (I mean, stacktrace could be recovered from the exception object, but not argument values, etc.)
Is there anything like a --magic command-line flag that could turn the python3 interpreter into what I need ?

Mainly taken from this great snippet:
import sys
def info(type, value, tb):
if hasattr(sys, 'ps1') or not sys.stderr.isatty() or type != AssertionError:
# we are in interactive mode or we don't have a tty-like
# device, so we call the default hook
sys.__excepthook__(type, value, tb)
else:
import traceback, pdb
# we are NOT in interactive mode, print the exception...
traceback.print_exception(type, value, tb)
print
# ...then start the debugger in post-mortem mode.
pdb.pm()
sys.excepthook = info
When you initialize your code with this, all AssertionErrors should invoke pdb.

Have a look at the nose project. You can use it with the --pdb option to drop into the debugger on errors.

Launch an IPython shell on exception

Is there a way to launch an IPython shell or prompt when my program runs a line that raises an exception?
I'm mostly interested in the context, variables, in the scope (and subscopes) where the exception was raised. Something like Visual Studio's debugging, when an exception is thrown but not caught by anyone, Visual Studio will halt and give me the call stack and the variables present at every level.
Do you think there's a way to get something similar using IPython?
EDIT: The -pdb option when launching IPython doesn't seem do what I want (or maybe I don't know how to use it properly, which is entirely possible). I run the following script :
def func():
z = 2
g = 'b'
raise NameError("This error will not be caught, but IPython still"
"won't summon pdb, and I won't be able to consult"
"the z or g variables.")
x = 1
y = 'a'
func()
Using the command :
ipython -pdb exceptionTest.py
Which stops execution when the error is raised, but brings me an IPython prompt where I have access to the global variables of the script, but not the local variables of function func. pdb is only invoked when I directly type a command in ipython that causes an error, i.e. raise NameError("This, sent from the IPython prompt, will trigger pdb.").
I don't necessarily need to use pdb, I'd just like to have access to the variables inside func.
EDIT 2: It has been a while, IPython's -pdb option is now working just as I want it to. That means when I raise an exception I can go back in the scope of func and read its variables z and g without any problem. Even without setting the -pdb option, one can run IPython in interactive mode then call the magic function %debug after the program has exit with error -- that will also drop you into an interactive ipdb prompt with all scopes accessibles.

Update for IPython v0.13:
import sys
from IPython.core import ultratb
sys.excepthook = ultratb.FormattedTB(mode='Verbose',
color_scheme='Linux', call_pdb=1)

Doing:
ipython --pdb -c "%run exceptionTest.py"
kicks off the script after IPython initialises and you get dropped into the normal IPython+pdb environment.

You can try this:
from ipdb import launch_ipdb_on_exception
def main():
with launch_ipdb_on_exception():
# The rest of the code goes here.
[...]

ipdb integrates IPython features into pdb. I use the following code to throw my apps into the IPython debugger after an unhanded exception.
import sys, ipdb, traceback
def info(type, value, tb):
traceback.print_exception(type, value, tb)
ipdb.pm()
sys.excepthook = info

#snapshoe's answer does not work on newer versions of IPython.
This does however:
import sys
from IPython import embed
def excepthook(type, value, traceback):
embed()
sys.excepthook = excepthook

#Adam's works like a charm except that IPython loads a bit slowly(800ms on my machine). Here I have a trick to make the load lazy.
class ExceptionHook:
instance = None
def __call__(self, *args, **kwargs):
if self.instance is None:
from IPython.core import ultratb
self.instance = ultratb.FormattedTB(mode='Verbose',
color_scheme='Linux', call_pdb=1)
return self.instance(*args, **kwargs)
sys.excepthook = ExceptionHook()
Now we don't need to wait at the very beginning. Only when the program crashes will cause IPython to be imported.

You can do something like the following:
import sys
from IPython.Shell import IPShellEmbed
ipshell = IPShellEmbed()
def excepthook(type, value, traceback):
ipshell()
sys.excepthook = excepthook
See sys.excepthook and Embedding IPython.

If you want to both get the traceback and open a IPython shell with the environment at the point of the exception:
def exceptHook(*args):
'''A routine to be called when an exception occurs. It prints the traceback
with fancy formatting and then calls an IPython shell with the environment
of the exception location.
'''
from IPython.core import ultratb
ultratb.FormattedTB(call_pdb=False,color_scheme='LightBG')(*args)
from IPython.terminal.embed import InteractiveShellEmbed
import inspect
frame = inspect.getinnerframes(args[2])[-1][0]
msg = 'Entering IPython console at {0.f_code.co_filename} at line {0.f_lineno}'.format(frame)
savehook = sys.excepthook # save the exception hook
InteractiveShellEmbed()(msg,local_ns=frame.f_locals,global_ns=frame.f_globals)
sys.excepthook = savehook # reset IPython's change to the exception hook
import sys
sys.excepthook = exceptHook
Note that it is necessary to pull than namespace information from the last frame referenced by the traceback (arg[2])

This man page says iPython has --[no]pdb option to be passed at command line to start iPython for uncaught exceptions. Are you looking for more?
EDIT:
python -m pdb pythonscript.py can launch pdb. Not sure about similar thing with iPython though. If you are looking for the stack trace and general post-mortem of the abnormal exit of program, this should work.

Do you actually want to open a pdb session at every exception point? (as I think a pdb session opened from ipython is the same as the one open in the normal shell). If that's the case, here's the trick:
http://code.activestate.com/recipes/65287-automatically-start-the-debugger-on-an-exception/

How to detect errors from compileall.compile_dir?

How do I detect an error when compiling a directory of python files using compile_dir?
Currently I get something on stderr, but no way to detect it in my app.
py_compile.compile() takes a doraise argument, but nothing here.
Or is there a better way to do this from a python script?
Edit:
I fixed it with os.walk and calling py_compile.compile for each file. But the question remains.

I don't see a better way. The code is designed to support the command-line program, and the API doesn't seem fully meant to be used as a library.
If you really had to use the compileall then you could fake it out with this hack, which notices that "quiet" is tested for boolean-ness while in the caught exception handler. I can override that with nonzero, check the exception state to see if it came from py_compile (quiet is tested in other contexts) and do something with that information:
import sys
import py_compile
import compileall
class ReportProblem:
def __nonzero__(self):
type, value, traceback = sys.exc_info()
if type is not None and issubclass(type, py_compile.PyCompileError):
print "Problem with", repr(value)
raise type, value, traceback
return 1
report_problem = ReportProblem()
compileall.compile_dir(".", quiet=report_problem)
Förresten, det finns GothPy på första måndagen varje månad, om du skulle ta sällskap med andra Python-användare i Gbg.

works fine for me. Could it be that you're not setting doraise to True somehow?