In Java/C# you can easily step through code to trace what might be going wrong, and IDE's make this process very user friendly.
Can you trace through python code in a similar fashion?
Yes! There's a Python debugger called pdb just for doing that!
You can launch a Python program through pdb by using pdb myscript.py or python -m pdb myscript.py.
There are a few commands you can then issue, which are documented on the pdb page.
Some useful ones to remember are:
b: set a breakpoint
c: continue debugging until you hit a breakpoint
s: step through the code
n: to go to next line of code
l: list source code for the current file (default: 11 lines including the line being executed)
u: navigate up a stack frame
d: navigate down a stack frame
p: to print the value of an expression in the current context
If you don't want to use a command line debugger, some IDEs like Pydev, Wing IDE or PyCharm have a GUI debugger. Wing and PyCharm are commercial products, but Wing has a free "Personal" edition, and PyCharm has a free community edition.
By using Python Interactive Debugger 'pdb'
First step is to make the Python interpreter to enter into the debugging mode.
A. From the Command Line
Most straight forward way, running from command line, of python interpreter
$ python -m pdb scriptName.py
> .../pdb_script.py(7)<module>()
-> """
(Pdb)
B. Within the Interpreter
While developing early versions of modules and to experiment it more iteratively.
$ python
Python 2.7 (r27:82508, Jul 3 2010, 21:12:11)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb_script
>>> import pdb
>>> pdb.run('pdb_script.MyObj(5).go()')
> <string>(1)<module>()
(Pdb)
C. From Within Your Program
For a big project and long-running module, can start the debugging from inside the program using
import pdb and set_trace()
like this :
#!/usr/bin/env python
# encoding: utf-8
#
import pdb
class MyObj(object):
count = 5
def __init__(self):
self.count= 9
def go(self):
for i in range(self.count):
pdb.set_trace()
print i
return
if __name__ == '__main__':
MyObj(5).go()
Step-by-Step debugging to go into more internal
Execute the next statement… with “n” (next)
Repeating the last debugging command… with ENTER
Quitting it all… with “q” (quit)
Printing the value of variables… with “p” (print)
a) p a
Turning off the (Pdb) prompt… with “c” (continue)
Seeing where you are… with “l” (list)
Stepping into subroutines… with “s” (step into)
Continuing… but just to the end of the current subroutine… with “r” (return)
Assign a new value
a) !b = "B"
Set a breakpoint
a) break linenumber
b) break functionname
c) break filename:linenumber
Temporary breakpoint
a) tbreak linenumber
Conditional breakpoint
a) break linenumber, condition
Note:**All these commands should be execute from **pdb
For in-depth knowledge, refer:-
https://pymotw.com/2/pdb/
https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/
There is a module called 'pdb' in python. At the top of your python script you do
import pdb
pdb.set_trace()
and you will enter into debugging mode. You can use 's' to step, 'n' to follow next line similar to what you would do with 'gdb' debugger.
Starting in Python 3.7, you can use the breakpoint() built-in function to enter the debugger:
foo()
breakpoint() # drop into the debugger at this point
bar()
By default, breakpoint() will import pdb and call pdb.set_trace(). However, you can control debugging behavior via sys.breakpointhook() and use of the environment variable PYTHONBREAKPOINT.
See PEP 553 for more information.
ipdb (IPython debugger)
ipdb adds IPython functionality to pdb, offering the following HUGE improvements:
tab completion
show more context lines
syntax highlight
Much like pdg, ipdb is still far from perfect and completely rudimentary if compared to GDB, but it is already a huge improvement over pdb.
Usage is analogous to pdb, just install it with:
python3 -m pip install --user ipdb
and then add to the line you want to step debug from:
__import__('ipdb').set_trace(context=21)
You likely want to add a shortcut for that from your editor, e.g. for Vim snipmate I have:
snippet ipd
__import__('ipdb').set_trace(context=21)
so I can type just ipd<tab> and it expands to the breakpoint. Then removing it is easy with dd since everything is contained in a single line.
context=21 increases the number of context lines as explained at: How can I make ipdb show more lines of context while debugging?
Alternatively, you can also debug programs from the start with:
ipdb3 main.py
but you generally don't want to do that because:
you would have to go through all function and class definitions as Python reads those lines
I don't know how to set the context size there without hacking ipdb. Patch to allow it: https://github.com/gotcha/ipdb/pull/155
Or alternatively, as in raw pdb 3.2+ you can set some breakpoints from the command line:
ipdb3 -c 'b 12' -c 'b myfunc' ~/test/a.py
although -c c is broken for some reason: https://github.com/gotcha/ipdb/issues/156
python -m module debugging has been asked at: How to debug a Python module run with python -m from the command line? and since Python 3.7 can be done with:
python -m pdb -m my_module
Serious missing features of both pdb and ipdb compared to GDB:
persistent command history across sessions: Save command history in pdb
ipdb specific annoyances:
multithreading does not work well if you don't hack some settings...
ipdb, multiple threads and autoreloading programs causing ProgrammingError
https://github.com/gotcha/ipdb/issues/51
Tested in Ubuntu 16.04, ipdb==0.11, Python 3.5.2.
VSCode
If you want to use an IDE, this is a good alternative to PyCharm.
Install VSCode
Install the Python extension, if it's not already installed
Create a file mymodule.py with Python code
To set a breakpoint, hover over a line number and click the red dot, or press F9
Hit F5 to start debugging and select Python File
It will stop at the breakpoint and you can do your usual debugging stuff like inspecting the values of variables, either at the tab VARIABLES (usually on the left) or by clicking on Debug Console (usually at the bottom next to your Terminal):
This screenshot shows VSCodium.
More information
Python debugging in VS Code
Getting Started with Python in VS Code
Debugging in Visual Studio Code
There exist breakpoint() method nowadays, which replaces import pdb; pdb.set_trace().
It also has several new features, such as possible environment variables.
Python Tutor is an online single-step debugger meant for novices. You can put in code on the edit page then click "Visualize Execution" to start it running.
Among other things, it supports:
hiding variables, e.g. to hide a variable named x, put this at the end:
#pythontutor_hide: x
saving/sharing
a few other languages like Java, JS, Ruby, C, C++
However it also doesn't support a lot of things, for example:
Reading/writing files - use io.StringIO and io.BytesIO instead: demo
Code that is too large, runs too long, or defines too many variables or objects
Command-line arguments
Lots of standard library modules like argparse, csv, enum, html, os, sys, weakref...
Python 3.7+
Let's take look at what breakpoint() can do for you in 3.7+.
I have installed ipdb and pdbpp, which are both enhanced debuggers, via
pip install pdbpp
pip install ipdb
My test script, really doesn't do much, just calls breakpoint().
#test_188_breakpoint.py
myvars=dict(foo="bar")
print("before breakpoint()")
breakpoint() # 👈
print(f"after breakpoint myvars={myvars}")
breakpoint() is linked to the PYTHONBREAKPOINT environment variable.
CASE 1: disabling breakpoint()
You can set the variable via bash as usual
export PYTHONBREAKPOINT=0
This turns off breakpoint() where it does nothing (as long as you haven't modified sys.breakpointhook() which is outside of the scope of this answer).
This is what a run of the program looks like:
(venv38) myuser#explore$ export PYTHONBREAKPOINT=0
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
after breakpoint myvars={'foo': 'bar'}
(venv38) myuser#explore$
Didn't stop, because I disabled breakpoint. Something that pdb.set_trace() can't do 😀😀😀!
CASE 2: using the default pdb behavior:
Now, let's unset PYTHONBREAKPOINT which puts us back to normal, enabled-breakpoint behavior (it's only disabled when 0 not when empty).
(venv38) myuser#explore$ unset PYTHONBREAKPOINT
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
[0] > /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
-> print(f"after breakpoint myvars={myvars}")
(Pdb++) print("pdbpp replaces pdb because it was installed")
pdbpp replaces pdb because it was installed
(Pdb++) c
after breakpoint myvars={'foo': 'bar'}
It stopped, but I actually got pdbpp because it replaces pdb entirely while installed. If I unistalled pdbpp, I'd be back to normal pdb.
Note: a standard pdb.set_trace() would still get me pdbpp
CASE 3: calling a custom debugger
But let's call ipdb instead. This time, instead of setting the environment variable, we can use bash to set it only for this one command.
(venv38) myuser#explore$ PYTHONBREAKPOINT=ipdb.set_trace py test_188_breakpoint.py
before breakpoint()
> /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
5 breakpoint()
----> 6 print(f"after breakpoint myvars={myvars}")
7
ipdb> print("and now I invoked ipdb instead")
and now I invoked ipdb instead
ipdb> c
after breakpoint myvars={'foo': 'bar'}
Essentially, what it does, when looking at $PYTHONBREAKPOINT:
from ipdb import set_trace # function imported on the right-most `.`
set_trace()
Again, much cleverer than a plain old pdb.set_trace() 😀😀😀
in practice? I'd probably settle on a debugger.
Say I want ipdb always, I would:
export it via .profile or similar.
disable on a command by command basis, without modifying the normal value
Example (pytest and debuggers often make for unhappy couples):
(venv38) myuser#explore$ export PYTHONBREAKPOINT=ipdb.set_trace
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
(venv38) myuser#explore$ PYTHONBREAKPOINT=0 pytest test_188_breakpoint.py
=================================== test session starts ====================================
platform darwin -- Python 3.8.6, pytest-5.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/myuser/kds2/wk/explore
plugins: celery-4.4.7, cov-2.10.0
collected 0 items
================================== no tests ran in 0.03s ===================================
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
p.s.
I'm using bash under macos, any posix shell will behave substantially the same. Windows, either powershell or DOS, may have different capabilities, especially around PYTHONBREAKPOINT=<some value> <some command> to set a environment variable only for one command.
If you come from Java/C# background I guess your best bet would be to use Eclipse with Pydev. This gives you a fully functional IDE with debugger built in. I use it with django as well.
https://wiki.python.org/moin/PythonDebuggingTools
pudb is a good drop-in replacement for pdb
PyCharm is an IDE for Python that includes a debugger. Watch this YouTube video for an introduction on using it to step through code:
PyCharm Tutorial - Debug python code using PyCharm (the debugging starts at 6:34)
Note: PyCharm is a commercial product, but the company does provide a free license to students and teachers, as well as a "lightweight" Community version that is free and open-source.
If you want an IDE with integrated debugger, try PyScripter.
Programmatically stepping and tracing through python code is possible too (and its easy!). Look at the sys.settrace() documentation for more details. Also here is a tutorial to get you started.
Visual Studio with PTVS could be an option for you: http://www.hanselman.com/blog/OneOfMicrosoftsBestKeptSecretsPythonToolsForVisualStudioPTVS.aspx
There is a high level logic error deep within my Python script, and pdb doesn't help to debug it. Is there any other way to see what is being executed after I run my script?
NOTE: Using pdb is too slow and inconvenient - I wish I could grep over all cases when my function is executed, instead of inspecting manually each and every call, set/unset breakpoints. The state is lost when I exit pdb and its user interface is more confusing than helpful - requires docs at hand.
UPDATE: made it clear that pdb is not an option, so the most popular Python debugging tips can not be applied
I would recommend using pdb. You can use
import pdb
at the top of your script, and then add the line
pdb.set_trace()
somewhere in the code where you want to trace the problem. When the script gets to that line, you will have an interactive console where you can check variable values, run your own checks, and see what is going on. You can use n to execute the next line, or c to continue to the next occurrence of set_trace(). Full documentation is here: http://docs.python.org/2/library/pdb.html.
Let me know if you have any specific questions!
No, there's no magic formula.
My best suggestion is to get a good IDE with a debugger, like JetBrains' PyCharm, and step through your code to see where you went wrong.
Most of the time these situations happen because you make assumptions about behavior that aren't true. Get a debugger, step through, and check your assumptions.
I found a way to do this using excellent trace module that comes with Python.
An example how to troubleshoot module installation problem:
python -m trace -t setup.py install > execution.log
This will dump all source line of setup.py install execution to execution.log. I find this more useful than pdb, because usability of pdb command line interface is very poor.
I'd like to have my debugger run post_mortem() any time an exception is encountered, without having to modify the source that I'm working on. I see lots of examples that involve wrapping code in a try/except block, but I'd like to have it always run, regardless of what I'm working on.
I worked on a python wrapper script but that got to be ugly and pretty much unusable.
I use pudb, which is API-equivalent to pdb, so a pdb-specific answer is fine. I run code from within my editor (vim) and would like to have the pm come up any time an exception is encountered.
It took a few months of not doing anything about it, but I happened to stumble upon a solution. I'm sure this is nothing new for the more experienced.
I have the following in my environment:
export PYTHONUSERBASE=~/.python
export PYTHONPATH=$PYTHONPATH:$PYTHONUSERBASE
And I have the following file:
~/.python/lib/python2.7/site-packages/usercustomize.py
With the following contents:
import traceback
import sys
try:
import pudb as debugger
except ImportError:
import pdb as debugger
def drop_debugger(type, value, tb):
traceback.print_exception(type, value, tb)
debugger.pm()
sys.excepthook = drop_debugger
__builtins__['debugger'] = debugger
__builtins__['st'] = debugger.set_trace
Now, whether interactively or otherwise, the debugger always jumps in upon an exception. It might be nice to smarten this up some.
It's important to make sure that you have no no-global-site-packages.txt in your site-packages. This will disable the usercustomize module with the default site.py (my virtualenv had a no-global-site-packages.txt)
Just in case it would help others, I left in the bit about modifying __builtins__. I find it quite handy to always be able to rely on some certain tools being available.
Flavor to taste.
A possible solution is to invoke pdb (I don't know about pudb, but I'll just assume it works the same) as a script:
python -m pdb script.py
Quoting the the documentation:
When invoked as a script, pdb will automatically enter post-mortem
debugging if the program being debugged exits abnormally. After
post-mortem debugging (or after normal exit of the program), pdb will
restart the program.
A solution for pdb since Python 3.2 is to start the program under the debugger via -m pdb and tell pdb to continue via -c c:
python3 -m pdb -c c program.py
Quoting the pdb documentation:
When invoked as a script, pdb will automatically enter post-mortem debugging if the program being debugged exits abnormally. After post-mortem debugging (or after normal exit of the program), pdb will restart the program.
As-of pudb 2019.2: according to the pudb documentation, the official way involves changing your code a bit (and even then, I don't end up in post-mortem mode if I just run python3 program.py!):
To start the debugger without actually pausing use:
from pudb import set_trace; set_trace(paused=False)
at the top of your code. This will start the debugger without breaking, and run it until a predefined breakpoint is hit. You can also press b on a set_trace call inside the debugger, and it will prevent it from stopping there.
Although it is possible to start the program properly under the debugger via python3 -m pudb.run program.py, pudb's command-line args do not support anything like pdb's -c c. The pudb's --pre-run=COMMAND is for external commands, not pudb commands.
What I currently do is run python3 -m pudb.run program.py without mentioning pudb or set_tracein program.py at all and press c on the keyboard. This way pudb enters its post-mortem mode upon any unhandled exception. This however only works well when I know that the exception will be reproduced. For hunting down sporadically occuring exceptions I go back to the pdb solution.
I'm developing a django app using aquamacs as my ide. Pdb isn't working since upgrading to emacs 23.2.1 using python 2.6.1. When I invoke pdb like this:
M-x pdb
Run pdb (like this): pdb ./manage.py runserver
The gud-manage.py frame appears with this message (and nothing more) -
Current directory is /path/to/my/source/
It isn't responsive to keyboard input, though I can right-click and send a quit or kill signal. It seems like emacs isn't capturing the pdb output correctly.
Has anyone seen this and (hopefully) fixed it? I believe it has something to do with the gud-pdb-marker-regexp variable (see point #2 in link).
Related issues
Seems to have been around since 2007
One person a solution for this problem on Windows (adding -u to the python command in the pdb script). I tried it anyway, but this didn't work for me.
Same issue (Current directory is ...) to me with emacs 23.2 (9). As you mentioned, it is caused by a CR/LF ending and can be fixed by setting the gud-pdb-marker-regexp.
I added the CR (\r) to the gud-pdb-marker-regexp. May you want to add the following line to your .emacs file and give it a try.
(setq gud-pdb-marker-regexp "^> \\([-axx-zA-Z0-9_/.:\\]*\\|<string>\\)(\\([0-9]+\\))\\([a-zA-Z0-9_]*\\|\\?\\|<module>\\)()\\(->[^\n\r]*\\)?[\n\r]")
Not sure if that is the case for you, but just to mention it: PDB hangs for me in Emacs when the source code path contains a space - when I move the python file to a directory without a space in the name, it works (on Emacs 23.1.1).
I have been having this same issue. I had it fixed in 23.1 (http://debbugs.gnu.org/db/56/5653.html) but now in 23.2 that fix no longer works, or at least it doesn't appear to for me. I've just submitted a bug to Emacs explaining the problem in detail and hopefully it will get resolved.
A workaround for this is to execute pdb from the emacs shell:
Open the shell: M-x shell
Enter this in the shell: pdb
This will get pdb working properly within the shell.
How do I debug python programs in emacs? I'm using python-mode.el
I've found references suggesting:
import pdb; pdb.set_trace();
but I'm not sure how to use it.
Type M-x cd to change directory to the location of the program you wish to debug.
Type M-x pdb. You'll be prompted with Run pdb (like this): pdb. Enter the name of the program (e.g. test.py).
At the (Pdb) prompt, type help to learn about how to use pdb.
Alternatively, you can put
import pdb
pdb.set_trace()
right inside your program (e.g. test.py). Now type M-x shell to get a shell prompt. When you run your program, you'll be dumped into pdb at the point where pdb.set_trace() is executed.
For me, I needed to replace the default "pdb" with
python -m pdb myscript.py
The realgud package (available from MELPA) supports PDB (among a gazillion other debuggers), and has a host of neat features that Emac's PDB doesn't have.
The one I like best is the shortkeys mode. Once you start debugging a program, you can press n, s, c etc. right in the source window, instead of having to type these commands in the PDB buffer. It also supports Visual-Studio style keybindings with function keys (f10, f11, f5, etc).
After installing RealGUD, you need to run M-x load-feature realgud to load it, and you can start pdb with M-x realgud:pdb.