I am running Emacs 23.2 with python.el and debugging some Python code with pdb.
My code spawns a sibling thread using the threading module and I set a breakpoint at the start of the run() method, but the break is never handled by pdb even though the code definitely runs and works for all intents and purposes.
I was under the impression that I could use pdb to establish breakpoints in any thread, even though full multi-threaded debugging is in fact not supported.
Am I wrong in assuming pdb within an M-x pdb invocation can break within any thread? If you don't believe me try this minimal example for yourself.
import threading
class ThreadTest(threading.Thread):
def __init__(self,):
threading.Thread.__init__(self)
def run(self):
print "Type M-x pdb, set a breakpoint here then type c <RET>..."
print "As you can see it does not break!"
if __name__ == '__main__':
tt = ThreadTest()
tt.start()
Thanks to Pierre and the book text he refers to, I tried the option to include pdb.set_trace() as follows:
def run(self):
import pdb; pdb.set_trace()
print "Set a breakpoint here then M-x pdb and type c..."
But this only breaks and offers pdb controls for step, next, continue etcetera, if it is executed from a console and run directly within the Python interpreter, and crucially not via M-x pdb - at least with my Emacs and pdb configuration.
So my original question could do with being rephrased:
Is there a way to invoke a Python program from within Emacs, where that program uses inlined invocation of pdb (thereby supporting breaks in multi-threaded applications), and for there to be a pdb comint control buffer established auto-magically?
or
If I run my Python application using M-x pdb and it contains an inline invocation of pdb, how best to handle the fact that this results in a pdb-session-within-a-pdb-session with the associated loss of control?
Are you using the default python.el? I've given up on that and started using python-mode.el. Then type M-x shell, from the prompt type python myproblem.py (replace with your program name of course) and it will stop at the set_trace line. It works out of the box with pdb integration. (And it works on your program).
See http://heather.cs.ucdavis.edu/~matloff/158/PLN/ParProcBook.pdf, there's a section on multithreaded debugging.
3.6.1 Using PDB to Debug Threaded Programs
Using PDB is a bit more complex when threads are involved. One cannot, for instance, simply do something
like this:
pdb.py buggyprog.py
because the child threads will not inherit the PDB process from the main thread. You can still run PDB in
the latter, but will not be able to set breakpoints in threads.
What you can do, though, is invoke PDB from within the function which is run by the thread, by calling
pdb.set trace() at one or more points within the code:
import pdb
pdb.set_trace()
In essence, those become breakpoints.
For example, in our program srvr.py in Section 3.1.1, we could add a PDB call at the beginning of the loop
in serveclient():
while 1:
import pdb
pdb.set_trace()
# receive letter from client, if it is still connected
k = c.recv(1)
if k == ’’: break
You then run the program directly through the Python interpreter as usual, NOT through PDB, but then the
program suddenly moves into debugging mode on its own. At that point, one can then step through the code
using the n or s commands, query the values of variables, etc.
PDB’s c (“continue”) command still works. Can one still use the b command to set additional breakpoints?
Yes, but it might be only on a one-time basis, depending on the context. A breakpoint might work only once,
due to a scope problem. Leaving the scope where we invoked PDB causes removal of the trace object. Thus
I suggested setting up the trace inside the loop above.
Related
I have attempted in a few different ways to perform Pool.starmap. I have tried various different suggestions and answers, and to no avail. Below is a sample of the code I am trying to run, however it gets caught and never terminates. What am I doing wrong here?
Side note: I am on python version 3.9.8
if __name__ == '__main__':
with get_context("spawn").Pool() as p:
tasks = [(1,1),(2,2),(3,3)]
print(p.starmap(add,tasks))
p.close()
p.join()
Multiprocessing in python has some complexity you should be aware of that make it dependent on how you run your script in addition to what OS, and python version you're using.
One of the big issues I see very often is the fact that Jupyter and other "notebook" style python environments don't always play nice with multiprocessing. There are technically some ways around this, but I typically just suggest executing your code from a more normal system terminal. The common thread is "interactive" interpreters don't work very well because there needs to be a "main" file, and in interactive mode there's no file; it just waits for user input.
I can't know exactly what your issue is here, as you haven't provided all your code, what OS you're using, and what IDE you're using but I can at least leave you with a working (on my setup) example. (windows 10; python 3.9; Spyder IDE with run settings -> execute in an external system terminal)
import multiprocessing as mp
def add(a, b): #I'm assuming your "add" function looks a bit like this...
return a+b
if __name__ == "__main__":
#this is critical when using "spawn" so code doesn't run when the file is imported
#you should only define functions, classes, and static data outside this (constants)
#most critically, it shouldn't be possible for a new child process to start outside this
ctx = mp.get_context("spawn")
#This is the only context available on windows, and the default for MacOS since python 3.8.
# Contexts are an important topic somewhat unique to python multiprocessing, and you should
# absolutely do some additional reading about "spawn" vs "fork". tldr; "spawn" starts a new
# process with no knowledge of the old one, and must `import` everything from __main__.
# "fork" on the other hand copies the existing process and all its memory before branching. This is
# faster than re-starting the interpreter, and re-importing everything, but sometimes things
# get copied that shouldn't, and other things that should get copied don't.
with ctx.Pool() as p:
#using `with` automatically shuts down the pool (forcibly) at the end of the block so you don't have to call `close` or `join`.
# It was also pointed out that due to the forcible shutdown, async calls like `map_async` may not finish unless you wait for the results
# before the end of the `with` block. `starmap` already waits for the results in this case however, so extra waiting is not needed.
tasks = [(1,1),(2,2),(3,3)]
print(p.starmap(add, tasks))
I have found that the pycharm debugger will not stop at breakpoints set in work functions of custom python GR blocks. However, it will stop at breakpoints set in a block's constructor.
Is there a way to stop at breakpoints in a work function? I know that the function is being run because when I put print statements in the work function, I see the results of the print when the flowgraph is run.
That's to be expected: the work() is called from C++ context; your debugger simply doesn't "control" the execution.
I'm sure it's possible to allow attaching breakpoints, but I don't know of a specific way. It would be relatively involved, because you need to instrument python in a GNU Radio-specific way, I guess.
I'm trying to find a way to pause a script's execution (one that is not timeout) that would allow me to e.g. check variable values at that point, input additional commands from the command line and then resume the execution once I'm ready. In IDL (from which I switched to Python very recently) I was able to do that by including "stop" at the chosen code line and then typing ".cont". Does something similar exist in Python?
Many thanks in advance for the help!
(This is my first post here, so I hope the above description meets the standards but if not I'd welcome any critical feedback!)
I believe you are looking for the Python Debugger (pdb):
The module pdb defines an interactive source code debugger for Python programs. It supports setting (conditional) breakpoints and single stepping at the source line level, inspection of stack frames, source code listing, and evaluation of arbitrary Python code in the context of any stack frame. It also supports post-mortem debugging and can be called under program control.
I'd suggest you to start with set_trace():
The typical usage to break into the debugger from a running program is to insert
import pdb; pdb.set_trace()
at the location you want to break into the debugger. You can then step through the code following this statement, and continue running without the debugger using the continue command.
Be sure to take a look at the other ways to enter the Debugger and the Debugger Commands section.
I'm using the PyDev for Eclipse plugin, and I'm trying to set a break point in some code that gets run in a background thread. The break point never gets hit even though the code is executing. Here's a small example:
import thread
def go(count):
print 'count is %d.' % count # set break point here
print 'calling from main thread:'
go(13)
print 'calling from bg thread:'
thread.start_new_thread(go, (23,))
raw_input('press enter to quit.')
The break point in that example gets hit when it's called on the main thread, but not when it's called from a background thread. Is there anything I can do, or is that a limitation of the PyDev debugger?
Update
Thanks for the work arounds. I submitted a PyDev feature request, and it has been completed. It should be released with version 1.6.0. Thanks, PyDev team!
The problem is that there's no API in the thread module to know when a thread starts.
What you can do in your example is set the debugger trace function yourself (as Alex pointed) as in the code below (if you're not in the remote debugger, the pydevd.connected = True is currently required -- I'll change pydev so that this is not needed anymore). You may want to add a try..except ImportError for the pydevd import (which will fail if you're not running in the debugger)
def go(count):
import pydevd
pydevd.connected = True
pydevd.settrace(suspend=False)
print 'count is %d.' % count # set break point here
Now, on a second thought, I think that pydev can replace the start_new_thread method in the thread module providing its own function which will setup the debugger and later call the original function (just did that and it seems to be working, so, if you use the nightly that will be available in some hours, which will become the future 1.6.0, it should be working without doing anything special).
The underlying issue is with sys.settrace, the low-level Python function used to perform all tracing and debugging -- as the docs say,
The function is thread-specific; for a
debugger to support multiple threads,
it must be registered using settrace()
for each thread being debugged.
I believe that when you set a breakpoint in PyDev, the resulting settrace call is always happening on the main thread (I have not looked at PyDev recently so they may have added some way to work around that, but I don't recall any from the time when I did look).
A workaround you might implement yourself is, in your main thread after the breakpoint has been set, to use sys.gettrace to get PyDev's trace function, save it in a global variable, and make sure in all threads of interest to call sys.settrace with that global variable as the argument -- a tad cumbersome (more so for threads that already exist at the time the breakpoint is set!), but I can't think of any simpler alternative.
On this question, I found a way to start the command-line debugger:
import pdb; pdb.set_trace()
It's not as easy to use as the Eclipse debugger, but it's better than nothing.
For me this worked according to one of Fabio's posts, after setting the trace with setTrace("000.000.000.000") # where 0's are the IP of your computer running Eclipse/PyDev
threading.settrace(pydevd.GetGlobalDebugger().trace_dispatch)
I'm relatively new to python but have a bit of experience using IDL. I was wondering if anyone knows if there are equivalent commands in python for IDL's stop and .reset commands.
If I'm running some IDL script I wrote that I put a stop command in, essentially what it does is stop the script there and give me access to the command line in the middle of the script. So I have access to all the functions and variables that I defined before the stop command, which I find really useful for debugging.
The .reset command I find extremely useful too. What it does is reset the the IDL environment (clears all variables, functions, etc.). It's as if I closed that session and opened a new one, but without having to exit and restart IDL. I find that if I'm trying to debug a script I wrote it's useful sometimes to start from scratch and not have to reset IDL (or python now). It would be useful also in python to be able to un-import any modules I had previously imported.
Any help with these issues would be greatly appreciated.
Cheers
Related
Python Drop into REPL
Is it possible to go into ipython from code?
IPython (aside from being a far nicer REPL than the standard python interpreter) may do what you want:
from IPython.Shell import IPShellEmbed
start_shell = IPShellEmbed()
def times_2(x):
return 2*x
a = 5
start_shell()
# now in IPython shell
# a -> 5
# times_2(a) -> 10
Note that any changes you make in the shell will not be sent back to the main python process on exit - if you set a = 10 in IPython (using the above example), a is still equal to 5 in the main python process.
edit: post on IPython-user mailing list where I first saw this technique.
stop sounds equivalent to use of the code module. .reset doesn't have an equivalent in Python short of gratuitous use of del.
Use pdb, as in this short script. Run it at the command line, and the PDB prompt will magically appear allowing single stepping, evaluation of arbitrary expressions, etc.
#!/usr/bin/env python
import pdb;
print 1
print 2
pdb.set_trace()
print 3
print 4
You could do %reset from within an IPython shell.
For stops, just add pydebug breakpoints as mentioned
pdb.set_trace() breaking out of code apparently does not allow you to ".continue" (IDL command) from that point (from http://pythondammit.blogspot.fr/2012/04/equivalent-of-idls-stop-command.html)
An update to redacted's solution.
The interactive IPython is much more powerful and convenient than pdb.set_trace.
Try this script
from IPython import embed
a = 1
b = 2
print('before')
embed()
print('after')
c = 3
Put embed() where you want to interrupt.
Run the script you will enter an interactive IPython Shell, you can view and modify the variables.
The script will continue after you exiting the shell.
You probably just want to use a Python debugger for this.
Welcome to the Python community! I'm still learning, but imo Python's nicer than the Interactive Data Language.
Anyway, Ignacio's answer about using the code module looks like it may provide what you want, at least as far as a parallel to IDL's stop.
Another thing you may find useful is to go into Python's interactive mode and import your program. You can then interact with it by running functions, etc. (Admittedly, I'm no expert at this.) If you do this, you'll need a main() function in the file which drives the program. For example, you'd have something like:
import sys
def main():
# do stuff
return(0)
if __name__ == '__main__':
sys.exit(main())
instead of just:
# do stuff
This prevents the execution of the program when you pull it into the Python interpreter. For more, see Guido's article about main functions.