I'm running my Python program and have a point where it would be useful to jump in and see what's going on, and then step out again. Sort of like a temporary console mode.
In Matlab, I'd use the keyboard command to do this, but I'm not sure what the command is in python.
Is there a way to do this?
For instance:
for thing in set_of_things:
enter_interactive_mode_here()
do_stuff_to(thing)
When enter_interactive_mode() calls, I'd like to go there, look around, and then leave and have the program continue running.
code.interact() seems to work somehow:
>>> import code
>>> def foo():
... a = 10
... code.interact(local=locals())
... return a
...
>>> foo()
Python 3.6.5 (default, Apr 1 2018, 05:46:30)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> a
10
Ctrl+Z returns to the "main" interpreter.
You can read the locals, but modifying them doesn't seem to work this way.
python -i myapp.py
This will execute myapp.py and drop you in the interactive shell. From there you can execute functions and check their output, with the whole environment (imports, etc.) of myapp.py loaded.
For something more sophisticated - it would be better to use a debugger like pdb, setting a breakpoint. Also, most IDEs (PyDev, PyCharm, Komodo...) have graphical debuggers.
I use pdb for this purpose. I realize Emil already mentioned this in his answer, but he did not include an example or elaborate on why it answers your question.
for thing in set_of_things:
import pdb; pdb.set_trace()
do_stuff_to(thing)
You can read and set variables by starting your command with an exclamation point. You can also move up and down the stack (commands u and d), which InteractiveConsole does not have built-in mechanisms to do.
To have the program continue executing, use the c command. In the above example it will enter the debugger every loop iteration, so you might want to wrap the set_trace() call in an if sentence.
You have options -- Python standard library or IPython.
The Python standard library has a code module which has an InteractiveConsole class whose purpose is to "Closely emulate the behavior of the interactive Python interpreter." This would probably be able to do what you want, but the documentation doesn't have any examples on how to use this, and I don't have any suggestions on where to go.
IPython, which is a more advanced Python terminal, has the option to embed a console at any point in your program built in. According to their documentation, you can simply do
from IPython import embed
for thing in set_of_things:
embed()
do_stuff_to(thing)
From Python 3.7 onwards, you can also use breakpoint() to get into the debugger, e.g.:
for thing in set_of_things:
breakpoint()
do_stuff_to(thing)
This is a little easier to remember and write, and will open your code in pdb by default.
However, it's also possible to set the PYTHONBREAKPOINT environment to the name of a callable, which could be another debugger such as pudb or ipdb, or it could be IPython's embed, or anything else.
Most comfortable tool for me is ipdb.
ipdb exports functions to access the IPython debugger, which features tab completion, syntax highlighting, better tracebacks, better introspection with the same interface as the pdb module.
Completion and handy introspection is especially useful for debugging.
You can use ipdb.
To set your breakpoints, add import ipdb; ipdb.set_trace() where you want to jump into the debugger. Once you reach a breakpoint, you’ll be given an interactive shell and a few lines of code around your breakpoint for context.
https://www.safaribooksonline.com/blog/2014/11/18/intro-python-debugger/
Related
I am trying to run code in the python interpreter from a python script (on windows, using the terminal build in to vsc), but I can't make anything work. I have spent a lot of time using the subprocess,and have also tried os module, but the issue with those, is that they cannot run code in the interpreter. So, I can make them start the interpreter, and I can enter code myself, which my script can get the result of (stdout and stderr), but it cannot enter code into the interpreter. I have tried running multiple commands in a row, using \n\r in the commands, and a few other attempts, but it always runs the second command/line after I manually quit() the interpreter. I have tried almost all of the functions from the subprocess module, and have tried numerous configrations for stdin, stdout, and stderr.
So, my qyuestion is: How can I have a script enter code into the interpreter?
It would also be nice to collect the results in real time, so my script does not have to start and quit an instance of the interpreter every time it wants the results, but that is not a priority.
Example of the issue with the OS module (but the issue is more or less the same with subprocess:
My code:
import os
print(os.popen("python").read())
print(os.popen("1 + 1").read())
Result:
Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> 1 + 2 #entered by me
>>> quit() #entered by me
3 #what the print statement returns
'1' is not recognized as an internal or external command,
operable program or batch file.
P.S. I am aware there is another question about this issue, but the only answer it has does not work for me. (When using the module they say, python cannot find the module after I installed it)
EDIT: my code with subprocess:
import subprocess as sp
c = sp.Popen("python", text=True, stdin=sp.PIPE, stdout=sp.PIPE, stderr=sp.PIPE)
c.stdin.write("1 + 1")
c.stdin.close()
print(c.stdout.read())
Use the suprocess library like this.
import sys
import subprocess
p = subprocess.run(sys.executable, text=True, capture_output=True,
input='print(1+1)')
print(p.stdout)
print(p.stderr)
If you want to reuse a single child process, you have to implement a client and server system. One easy method is to implement a remote call with multiprocessing.Manager. See the example in the documentation.
As a side note, I don't recommend these if you don't have a good reason for spawning a child process, such as sandboxing an execution environment. Just use eval() in the parent process, because the child process will do the same work as what will be done by eval() if it has been done by the parent process.
In Java/C# you can easily step through code to trace what might be going wrong, and IDE's make this process very user friendly.
Can you trace through python code in a similar fashion?
Yes! There's a Python debugger called pdb just for doing that!
You can launch a Python program through pdb by using pdb myscript.py or python -m pdb myscript.py.
There are a few commands you can then issue, which are documented on the pdb page.
Some useful ones to remember are:
b: set a breakpoint
c: continue debugging until you hit a breakpoint
s: step through the code
n: to go to next line of code
l: list source code for the current file (default: 11 lines including the line being executed)
u: navigate up a stack frame
d: navigate down a stack frame
p: to print the value of an expression in the current context
If you don't want to use a command line debugger, some IDEs like Pydev, Wing IDE or PyCharm have a GUI debugger. Wing and PyCharm are commercial products, but Wing has a free "Personal" edition, and PyCharm has a free community edition.
By using Python Interactive Debugger 'pdb'
First step is to make the Python interpreter to enter into the debugging mode.
A. From the Command Line
Most straight forward way, running from command line, of python interpreter
$ python -m pdb scriptName.py
> .../pdb_script.py(7)<module>()
-> """
(Pdb)
B. Within the Interpreter
While developing early versions of modules and to experiment it more iteratively.
$ python
Python 2.7 (r27:82508, Jul 3 2010, 21:12:11)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb_script
>>> import pdb
>>> pdb.run('pdb_script.MyObj(5).go()')
> <string>(1)<module>()
(Pdb)
C. From Within Your Program
For a big project and long-running module, can start the debugging from inside the program using
import pdb and set_trace()
like this :
#!/usr/bin/env python
# encoding: utf-8
#
import pdb
class MyObj(object):
count = 5
def __init__(self):
self.count= 9
def go(self):
for i in range(self.count):
pdb.set_trace()
print i
return
if __name__ == '__main__':
MyObj(5).go()
Step-by-Step debugging to go into more internal
Execute the next statement… with “n” (next)
Repeating the last debugging command… with ENTER
Quitting it all… with “q” (quit)
Printing the value of variables… with “p” (print)
a) p a
Turning off the (Pdb) prompt… with “c” (continue)
Seeing where you are… with “l” (list)
Stepping into subroutines… with “s” (step into)
Continuing… but just to the end of the current subroutine… with “r” (return)
Assign a new value
a) !b = "B"
Set a breakpoint
a) break linenumber
b) break functionname
c) break filename:linenumber
Temporary breakpoint
a) tbreak linenumber
Conditional breakpoint
a) break linenumber, condition
Note:**All these commands should be execute from **pdb
For in-depth knowledge, refer:-
https://pymotw.com/2/pdb/
https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/
There is a module called 'pdb' in python. At the top of your python script you do
import pdb
pdb.set_trace()
and you will enter into debugging mode. You can use 's' to step, 'n' to follow next line similar to what you would do with 'gdb' debugger.
Starting in Python 3.7, you can use the breakpoint() built-in function to enter the debugger:
foo()
breakpoint() # drop into the debugger at this point
bar()
By default, breakpoint() will import pdb and call pdb.set_trace(). However, you can control debugging behavior via sys.breakpointhook() and use of the environment variable PYTHONBREAKPOINT.
See PEP 553 for more information.
ipdb (IPython debugger)
ipdb adds IPython functionality to pdb, offering the following HUGE improvements:
tab completion
show more context lines
syntax highlight
Much like pdg, ipdb is still far from perfect and completely rudimentary if compared to GDB, but it is already a huge improvement over pdb.
Usage is analogous to pdb, just install it with:
python3 -m pip install --user ipdb
and then add to the line you want to step debug from:
__import__('ipdb').set_trace(context=21)
You likely want to add a shortcut for that from your editor, e.g. for Vim snipmate I have:
snippet ipd
__import__('ipdb').set_trace(context=21)
so I can type just ipd<tab> and it expands to the breakpoint. Then removing it is easy with dd since everything is contained in a single line.
context=21 increases the number of context lines as explained at: How can I make ipdb show more lines of context while debugging?
Alternatively, you can also debug programs from the start with:
ipdb3 main.py
but you generally don't want to do that because:
you would have to go through all function and class definitions as Python reads those lines
I don't know how to set the context size there without hacking ipdb. Patch to allow it: https://github.com/gotcha/ipdb/pull/155
Or alternatively, as in raw pdb 3.2+ you can set some breakpoints from the command line:
ipdb3 -c 'b 12' -c 'b myfunc' ~/test/a.py
although -c c is broken for some reason: https://github.com/gotcha/ipdb/issues/156
python -m module debugging has been asked at: How to debug a Python module run with python -m from the command line? and since Python 3.7 can be done with:
python -m pdb -m my_module
Serious missing features of both pdb and ipdb compared to GDB:
persistent command history across sessions: Save command history in pdb
ipdb specific annoyances:
multithreading does not work well if you don't hack some settings...
ipdb, multiple threads and autoreloading programs causing ProgrammingError
https://github.com/gotcha/ipdb/issues/51
Tested in Ubuntu 16.04, ipdb==0.11, Python 3.5.2.
VSCode
If you want to use an IDE, this is a good alternative to PyCharm.
Install VSCode
Install the Python extension, if it's not already installed
Create a file mymodule.py with Python code
To set a breakpoint, hover over a line number and click the red dot, or press F9
Hit F5 to start debugging and select Python File
It will stop at the breakpoint and you can do your usual debugging stuff like inspecting the values of variables, either at the tab VARIABLES (usually on the left) or by clicking on Debug Console (usually at the bottom next to your Terminal):
This screenshot shows VSCodium.
More information
Python debugging in VS Code
Getting Started with Python in VS Code
Debugging in Visual Studio Code
There exist breakpoint() method nowadays, which replaces import pdb; pdb.set_trace().
It also has several new features, such as possible environment variables.
Python Tutor is an online single-step debugger meant for novices. You can put in code on the edit page then click "Visualize Execution" to start it running.
Among other things, it supports:
hiding variables, e.g. to hide a variable named x, put this at the end:
#pythontutor_hide: x
saving/sharing
a few other languages like Java, JS, Ruby, C, C++
However it also doesn't support a lot of things, for example:
Reading/writing files - use io.StringIO and io.BytesIO instead: demo
Code that is too large, runs too long, or defines too many variables or objects
Command-line arguments
Lots of standard library modules like argparse, csv, enum, html, os, sys, weakref...
Python 3.7+
Let's take look at what breakpoint() can do for you in 3.7+.
I have installed ipdb and pdbpp, which are both enhanced debuggers, via
pip install pdbpp
pip install ipdb
My test script, really doesn't do much, just calls breakpoint().
#test_188_breakpoint.py
myvars=dict(foo="bar")
print("before breakpoint()")
breakpoint() # 👈
print(f"after breakpoint myvars={myvars}")
breakpoint() is linked to the PYTHONBREAKPOINT environment variable.
CASE 1: disabling breakpoint()
You can set the variable via bash as usual
export PYTHONBREAKPOINT=0
This turns off breakpoint() where it does nothing (as long as you haven't modified sys.breakpointhook() which is outside of the scope of this answer).
This is what a run of the program looks like:
(venv38) myuser#explore$ export PYTHONBREAKPOINT=0
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
after breakpoint myvars={'foo': 'bar'}
(venv38) myuser#explore$
Didn't stop, because I disabled breakpoint. Something that pdb.set_trace() can't do 😀😀😀!
CASE 2: using the default pdb behavior:
Now, let's unset PYTHONBREAKPOINT which puts us back to normal, enabled-breakpoint behavior (it's only disabled when 0 not when empty).
(venv38) myuser#explore$ unset PYTHONBREAKPOINT
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
[0] > /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
-> print(f"after breakpoint myvars={myvars}")
(Pdb++) print("pdbpp replaces pdb because it was installed")
pdbpp replaces pdb because it was installed
(Pdb++) c
after breakpoint myvars={'foo': 'bar'}
It stopped, but I actually got pdbpp because it replaces pdb entirely while installed. If I unistalled pdbpp, I'd be back to normal pdb.
Note: a standard pdb.set_trace() would still get me pdbpp
CASE 3: calling a custom debugger
But let's call ipdb instead. This time, instead of setting the environment variable, we can use bash to set it only for this one command.
(venv38) myuser#explore$ PYTHONBREAKPOINT=ipdb.set_trace py test_188_breakpoint.py
before breakpoint()
> /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
5 breakpoint()
----> 6 print(f"after breakpoint myvars={myvars}")
7
ipdb> print("and now I invoked ipdb instead")
and now I invoked ipdb instead
ipdb> c
after breakpoint myvars={'foo': 'bar'}
Essentially, what it does, when looking at $PYTHONBREAKPOINT:
from ipdb import set_trace # function imported on the right-most `.`
set_trace()
Again, much cleverer than a plain old pdb.set_trace() 😀😀😀
in practice? I'd probably settle on a debugger.
Say I want ipdb always, I would:
export it via .profile or similar.
disable on a command by command basis, without modifying the normal value
Example (pytest and debuggers often make for unhappy couples):
(venv38) myuser#explore$ export PYTHONBREAKPOINT=ipdb.set_trace
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
(venv38) myuser#explore$ PYTHONBREAKPOINT=0 pytest test_188_breakpoint.py
=================================== test session starts ====================================
platform darwin -- Python 3.8.6, pytest-5.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/myuser/kds2/wk/explore
plugins: celery-4.4.7, cov-2.10.0
collected 0 items
================================== no tests ran in 0.03s ===================================
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
p.s.
I'm using bash under macos, any posix shell will behave substantially the same. Windows, either powershell or DOS, may have different capabilities, especially around PYTHONBREAKPOINT=<some value> <some command> to set a environment variable only for one command.
If you come from Java/C# background I guess your best bet would be to use Eclipse with Pydev. This gives you a fully functional IDE with debugger built in. I use it with django as well.
https://wiki.python.org/moin/PythonDebuggingTools
pudb is a good drop-in replacement for pdb
PyCharm is an IDE for Python that includes a debugger. Watch this YouTube video for an introduction on using it to step through code:
PyCharm Tutorial - Debug python code using PyCharm (the debugging starts at 6:34)
Note: PyCharm is a commercial product, but the company does provide a free license to students and teachers, as well as a "lightweight" Community version that is free and open-source.
If you want an IDE with integrated debugger, try PyScripter.
Programmatically stepping and tracing through python code is possible too (and its easy!). Look at the sys.settrace() documentation for more details. Also here is a tutorial to get you started.
Visual Studio with PTVS could be an option for you: http://www.hanselman.com/blog/OneOfMicrosoftsBestKeptSecretsPythonToolsForVisualStudioPTVS.aspx
In Java/C# you can easily step through code to trace what might be going wrong, and IDE's make this process very user friendly.
Can you trace through python code in a similar fashion?
Yes! There's a Python debugger called pdb just for doing that!
You can launch a Python program through pdb by using pdb myscript.py or python -m pdb myscript.py.
There are a few commands you can then issue, which are documented on the pdb page.
Some useful ones to remember are:
b: set a breakpoint
c: continue debugging until you hit a breakpoint
s: step through the code
n: to go to next line of code
l: list source code for the current file (default: 11 lines including the line being executed)
u: navigate up a stack frame
d: navigate down a stack frame
p: to print the value of an expression in the current context
If you don't want to use a command line debugger, some IDEs like Pydev, Wing IDE or PyCharm have a GUI debugger. Wing and PyCharm are commercial products, but Wing has a free "Personal" edition, and PyCharm has a free community edition.
By using Python Interactive Debugger 'pdb'
First step is to make the Python interpreter to enter into the debugging mode.
A. From the Command Line
Most straight forward way, running from command line, of python interpreter
$ python -m pdb scriptName.py
> .../pdb_script.py(7)<module>()
-> """
(Pdb)
B. Within the Interpreter
While developing early versions of modules and to experiment it more iteratively.
$ python
Python 2.7 (r27:82508, Jul 3 2010, 21:12:11)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb_script
>>> import pdb
>>> pdb.run('pdb_script.MyObj(5).go()')
> <string>(1)<module>()
(Pdb)
C. From Within Your Program
For a big project and long-running module, can start the debugging from inside the program using
import pdb and set_trace()
like this :
#!/usr/bin/env python
# encoding: utf-8
#
import pdb
class MyObj(object):
count = 5
def __init__(self):
self.count= 9
def go(self):
for i in range(self.count):
pdb.set_trace()
print i
return
if __name__ == '__main__':
MyObj(5).go()
Step-by-Step debugging to go into more internal
Execute the next statement… with “n” (next)
Repeating the last debugging command… with ENTER
Quitting it all… with “q” (quit)
Printing the value of variables… with “p” (print)
a) p a
Turning off the (Pdb) prompt… with “c” (continue)
Seeing where you are… with “l” (list)
Stepping into subroutines… with “s” (step into)
Continuing… but just to the end of the current subroutine… with “r” (return)
Assign a new value
a) !b = "B"
Set a breakpoint
a) break linenumber
b) break functionname
c) break filename:linenumber
Temporary breakpoint
a) tbreak linenumber
Conditional breakpoint
a) break linenumber, condition
Note:**All these commands should be execute from **pdb
For in-depth knowledge, refer:-
https://pymotw.com/2/pdb/
https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/
There is a module called 'pdb' in python. At the top of your python script you do
import pdb
pdb.set_trace()
and you will enter into debugging mode. You can use 's' to step, 'n' to follow next line similar to what you would do with 'gdb' debugger.
Starting in Python 3.7, you can use the breakpoint() built-in function to enter the debugger:
foo()
breakpoint() # drop into the debugger at this point
bar()
By default, breakpoint() will import pdb and call pdb.set_trace(). However, you can control debugging behavior via sys.breakpointhook() and use of the environment variable PYTHONBREAKPOINT.
See PEP 553 for more information.
ipdb (IPython debugger)
ipdb adds IPython functionality to pdb, offering the following HUGE improvements:
tab completion
show more context lines
syntax highlight
Much like pdg, ipdb is still far from perfect and completely rudimentary if compared to GDB, but it is already a huge improvement over pdb.
Usage is analogous to pdb, just install it with:
python3 -m pip install --user ipdb
and then add to the line you want to step debug from:
__import__('ipdb').set_trace(context=21)
You likely want to add a shortcut for that from your editor, e.g. for Vim snipmate I have:
snippet ipd
__import__('ipdb').set_trace(context=21)
so I can type just ipd<tab> and it expands to the breakpoint. Then removing it is easy with dd since everything is contained in a single line.
context=21 increases the number of context lines as explained at: How can I make ipdb show more lines of context while debugging?
Alternatively, you can also debug programs from the start with:
ipdb3 main.py
but you generally don't want to do that because:
you would have to go through all function and class definitions as Python reads those lines
I don't know how to set the context size there without hacking ipdb. Patch to allow it: https://github.com/gotcha/ipdb/pull/155
Or alternatively, as in raw pdb 3.2+ you can set some breakpoints from the command line:
ipdb3 -c 'b 12' -c 'b myfunc' ~/test/a.py
although -c c is broken for some reason: https://github.com/gotcha/ipdb/issues/156
python -m module debugging has been asked at: How to debug a Python module run with python -m from the command line? and since Python 3.7 can be done with:
python -m pdb -m my_module
Serious missing features of both pdb and ipdb compared to GDB:
persistent command history across sessions: Save command history in pdb
ipdb specific annoyances:
multithreading does not work well if you don't hack some settings...
ipdb, multiple threads and autoreloading programs causing ProgrammingError
https://github.com/gotcha/ipdb/issues/51
Tested in Ubuntu 16.04, ipdb==0.11, Python 3.5.2.
VSCode
If you want to use an IDE, this is a good alternative to PyCharm.
Install VSCode
Install the Python extension, if it's not already installed
Create a file mymodule.py with Python code
To set a breakpoint, hover over a line number and click the red dot, or press F9
Hit F5 to start debugging and select Python File
It will stop at the breakpoint and you can do your usual debugging stuff like inspecting the values of variables, either at the tab VARIABLES (usually on the left) or by clicking on Debug Console (usually at the bottom next to your Terminal):
This screenshot shows VSCodium.
More information
Python debugging in VS Code
Getting Started with Python in VS Code
Debugging in Visual Studio Code
There exist breakpoint() method nowadays, which replaces import pdb; pdb.set_trace().
It also has several new features, such as possible environment variables.
Python Tutor is an online single-step debugger meant for novices. You can put in code on the edit page then click "Visualize Execution" to start it running.
Among other things, it supports:
hiding variables, e.g. to hide a variable named x, put this at the end:
#pythontutor_hide: x
saving/sharing
a few other languages like Java, JS, Ruby, C, C++
However it also doesn't support a lot of things, for example:
Reading/writing files - use io.StringIO and io.BytesIO instead: demo
Code that is too large, runs too long, or defines too many variables or objects
Command-line arguments
Lots of standard library modules like argparse, csv, enum, html, os, sys, weakref...
Python 3.7+
Let's take look at what breakpoint() can do for you in 3.7+.
I have installed ipdb and pdbpp, which are both enhanced debuggers, via
pip install pdbpp
pip install ipdb
My test script, really doesn't do much, just calls breakpoint().
#test_188_breakpoint.py
myvars=dict(foo="bar")
print("before breakpoint()")
breakpoint() # 👈
print(f"after breakpoint myvars={myvars}")
breakpoint() is linked to the PYTHONBREAKPOINT environment variable.
CASE 1: disabling breakpoint()
You can set the variable via bash as usual
export PYTHONBREAKPOINT=0
This turns off breakpoint() where it does nothing (as long as you haven't modified sys.breakpointhook() which is outside of the scope of this answer).
This is what a run of the program looks like:
(venv38) myuser#explore$ export PYTHONBREAKPOINT=0
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
after breakpoint myvars={'foo': 'bar'}
(venv38) myuser#explore$
Didn't stop, because I disabled breakpoint. Something that pdb.set_trace() can't do 😀😀😀!
CASE 2: using the default pdb behavior:
Now, let's unset PYTHONBREAKPOINT which puts us back to normal, enabled-breakpoint behavior (it's only disabled when 0 not when empty).
(venv38) myuser#explore$ unset PYTHONBREAKPOINT
(venv38) myuser#explore$ python test_188_breakpoint.py
before breakpoint()
[0] > /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
-> print(f"after breakpoint myvars={myvars}")
(Pdb++) print("pdbpp replaces pdb because it was installed")
pdbpp replaces pdb because it was installed
(Pdb++) c
after breakpoint myvars={'foo': 'bar'}
It stopped, but I actually got pdbpp because it replaces pdb entirely while installed. If I unistalled pdbpp, I'd be back to normal pdb.
Note: a standard pdb.set_trace() would still get me pdbpp
CASE 3: calling a custom debugger
But let's call ipdb instead. This time, instead of setting the environment variable, we can use bash to set it only for this one command.
(venv38) myuser#explore$ PYTHONBREAKPOINT=ipdb.set_trace py test_188_breakpoint.py
before breakpoint()
> /Users/myuser/kds2/wk/explore/test_188_breakpoint.py(6)<module>()
5 breakpoint()
----> 6 print(f"after breakpoint myvars={myvars}")
7
ipdb> print("and now I invoked ipdb instead")
and now I invoked ipdb instead
ipdb> c
after breakpoint myvars={'foo': 'bar'}
Essentially, what it does, when looking at $PYTHONBREAKPOINT:
from ipdb import set_trace # function imported on the right-most `.`
set_trace()
Again, much cleverer than a plain old pdb.set_trace() 😀😀😀
in practice? I'd probably settle on a debugger.
Say I want ipdb always, I would:
export it via .profile or similar.
disable on a command by command basis, without modifying the normal value
Example (pytest and debuggers often make for unhappy couples):
(venv38) myuser#explore$ export PYTHONBREAKPOINT=ipdb.set_trace
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
(venv38) myuser#explore$ PYTHONBREAKPOINT=0 pytest test_188_breakpoint.py
=================================== test session starts ====================================
platform darwin -- Python 3.8.6, pytest-5.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/myuser/kds2/wk/explore
plugins: celery-4.4.7, cov-2.10.0
collected 0 items
================================== no tests ran in 0.03s ===================================
(venv38) myuser#explore$ echo $PYTHONBREAKPOINT
ipdb.set_trace
p.s.
I'm using bash under macos, any posix shell will behave substantially the same. Windows, either powershell or DOS, may have different capabilities, especially around PYTHONBREAKPOINT=<some value> <some command> to set a environment variable only for one command.
If you come from Java/C# background I guess your best bet would be to use Eclipse with Pydev. This gives you a fully functional IDE with debugger built in. I use it with django as well.
https://wiki.python.org/moin/PythonDebuggingTools
pudb is a good drop-in replacement for pdb
PyCharm is an IDE for Python that includes a debugger. Watch this YouTube video for an introduction on using it to step through code:
PyCharm Tutorial - Debug python code using PyCharm (the debugging starts at 6:34)
Note: PyCharm is a commercial product, but the company does provide a free license to students and teachers, as well as a "lightweight" Community version that is free and open-source.
If you want an IDE with integrated debugger, try PyScripter.
Programmatically stepping and tracing through python code is possible too (and its easy!). Look at the sys.settrace() documentation for more details. Also here is a tutorial to get you started.
Visual Studio with PTVS could be an option for you: http://www.hanselman.com/blog/OneOfMicrosoftsBestKeptSecretsPythonToolsForVisualStudioPTVS.aspx
so far I used the Komodo IDE for Python development, but I'm now testing Eclipse with PyDev. Everything works fine, but there is one Komodo feature that I'm missing.
In Komodo I can inspect the running application in a debugger shell. I.e. after hitting a breakpoint I can not only read the content of variables, but I can execute arbitrary Python code (e.g. changing the value of variables) and then continue program execution.
PyDev has also some interactive shell during debugging, but I can only read variables and not change their content. Is this feature not available in PyDev or am I missing something here?
Many thanks,
Axel
As you've seen, you can just use the console directly:
http://pydev.org/manual_adv_debug_console.html
Now, you can also connect the interactive console (which is a bit more advanced) by selecting a stack frame to attach it:
http://pydev.org/manual_adv_interactive_console.html
Yes you can do that. Just type in the console what ever commands you want :). I usually have to right click then
Debug As >> Python run
PyDev is a little bit quirky, but you get used to it.
I just want to see the state of the process, is it possible to attach a console into the process, so I can invoke functions inside the process and see some of the global variables.
It's better the process is running without being affected(of course performance can down a little bit)
This will interrupt your process (unless you start it in a thread), but you can use the code module to start a Python console:
import code
code.interact()
This will block until the user exits the interactive console by executing exit().
The code module is available in at least Python v2.6, probably others.
I tend to use this approach in combination with signals for my Linux work (for Windows, see below). I slap this at the top of my Python scripts:
import code
import signal
signal.signal(signal.SIGUSR2, lambda sig, frame: code.interact())
And then trigger it from a shell with kill -SIGUSR2 <PID>, where <PID> is the process ID. The process then stops whatever it is doing and presents a console:
Python 2.6.2 (r262:71600, Oct 9 2009, 17:53:52)
[GCC 3.4.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
Generally from there I'll load the server-side component of a remote debugger like the excellent WinPDB.
Windows is not a POSIX-compliant OS, and so does not provide the same signals as Linux. However, Python v2.2 and above expose a Windows-specific signal SIGBREAK (triggered by pressing CTRL+Pause/Break). This does not interfere with normal CTRL+C (SIGINT) operation, and so is a handy alternative.
Therefore a portable, but slightly ugly, version of the above is:
import code
import signal
signal.signal(
vars(signal).get("SIGBREAK") or vars(signal).get("SIGUSR2"),
lambda sig, frame: code.interact()
)
Advantages of this approach:
No external modules (all standard Python stuff)
Barely consumes any resources until triggered (2x import)
Here's the code I use in my production environment which will load the server-side of WinPDB (if available) and fall back to opening a Python console.
# Break into a Python console upon SIGUSR1 (Linux) or SIGBREAK (Windows:
# CTRL+Pause/Break). To be included in all production code, just in case.
def debug_signal_handler(signal, frame):
del signal
del frame
try:
import rpdb2
print
print
print "Starting embedded RPDB2 debugger. Password is 'foobar'"
print
print
rpdb2.start_embedded_debugger("foobar", True, True)
rpdb2.setbreak(depth=1)
return
except StandardError:
pass
try:
import code
code.interact()
except StandardError as ex:
print "%r, returning to normal program flow" % ex
import signal
try:
signal.signal(
vars(signal).get("SIGBREAK") or vars(signal).get("SIGUSR1"),
debug_signal_handler
)
except ValueError:
# Typically: ValueError: signal only works in main thread
pass
If you have access to the program's source-code, you can add this functionality relatively easily.
See Recipe 576515: Debugging a running python process by interrupting and providing an interactive prompt (Python)
To quote:
This provides code to allow any python
program which uses it to be
interrupted at the current point, and
communicated with via a normal python
interactive console. This allows the
locals, globals and associated program
state to be investigated, as well as
calling arbitrary functions and
classes.
To use, a process should import the
module, and call listen() at any point
during startup. To interrupt this
process, the script can be run
directly, giving the process Id of the
process to debug as the parameter.
Another implementation of roughly the same concept is provided by rconsole. From the documentation:
rconsole is a remote Python console
with auto completion, which can be
used to inspect and modify the
namespace of a running script.
To invoke in a script do:
from rfoo.utils import rconsole
rconsole.spawn_server()
To attach from a shell do:
$ rconsole
Security note: The rconsole listener
started with spawn_server() will
accept any local connection and may
therefore be insecure to use in shared
hosting or similar environments!
Use pyrasite-shell. I can't believe it works so well, but it does. "Give it a pid, get a shell".
$ sudo pip install pyrasite
$ echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope # If YAMA activated, see below.
$ pyrasite-shell 16262
Pyrasite Shell 2.0
Connected to 'python my_script.py'
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> globals()
>>> print(db_session)
>>> run_some_local_function()
>>> some_existing_local_variable = 'new value'
This launches the python shell with access to the globals() and locals() variables of that running python process, and other wonderful things.
Only tested this personally on Ubuntu but seems to cater for OSX too.
Adapted from this answer.
Note: The line switching off the ptrace_scope property is only necessary for kernels/systems that have been built with CONFIG_SECURITY_YAMA on. Take care messing with ptrace_scope in sensitive environments because it could introduce certain security vulnerabilities. See here for details.
Why not simply using the pdb module? It allows you to stop a script, inspect elements values, and execute the code line by line. And since it is built upon the Python interpreter, it also provides the features provided by the classic interpreter. To use it, just put these 2 lines in your code, where you wish to stop and inspect it:
import pdb
pdb.set_trace()
Another possibility, without adding stuff to the python scripts, is described here:
https://wiki.python.org/moin/DebuggingWithGdb
Unfortunately, this solution also requires some forethought, at least to the extent that you need to be using a version of python with debugging symbols in it.
pdb_attach worked well for us for attaching the Python debugger to a long-running process.
The author describes it as follows:
This package was made in response to frustration over debugging long running processes. Wouldn't it be nice to just attach pdb to a running python program and see what's going on? Well that's exactly what pdb-attach does.
Set it up as follows in your main module:
import pdb_attach
pdb_attach.listen(50000) # Listen on port 50000.
When the program is running, attach to it by calling pdb_attach from the command line with the PID of the program and the port passed to pdb_attach.listen():
$ python -m pdb_attach <PID> 50000
(Pdb) # Interact with pdb as you normally would
You can use my project madbg. It is a python debugger that allows you to attach to a running python program and debug it in your current terminal. It is similar to pyrasite and pyringe, but supports python3, doesn't require gdb, and uses IPython for the debugger (which means pdb with colors and autocomplete).
For example, to see where your script is stuck, you could run:
madbg attach <pid>
After that you will have a pdb shell, in which you can invoke functions and inspect variables.
Using PyCharm, I was getting a failure to connect to process in Ubuntu. The fix for this is to disable YAMA. For more info see askubuntu
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope