Following a tutorial on Python's debugger, I used pdb.set_trace() to interrupt the sample code. It worked, but what if you are at the interactive prompt and want another nested breakpoint?
(Pdb) def test(): pdb.set_trace(); print "don't print this yet"
(Pdb) test()
don't print this yet
It didn't stop. Is the debugger fundamentally "one deep"? e.g. is this is a limitation of Python's hooks, or just something pdb doesn't choose to do?
is this is a limitation of Python's hooks, or just something pdb doesn't choose to do?
It appears to be a limitation of the hooks.
I did a test to see what was being called and what wasn't (putting print statements in /usr/lib/python2.7/bdb.py)
Quick inspection finds set_trace in pdb.py:
def set_trace():
Pdb().set_trace(sys._getframe().f_back)
That calls set_trace in bdb.py
def set_trace(self, frame=None):
"""Start debugging from `frame`.
If frame is not specified, debugging starts from caller's frame.
"""
if frame is None:
frame = sys._getframe().f_back
self.reset()
while frame:
frame.f_trace = self.trace_dispatch
self.botframe = frame
frame = frame.f_back
self.set_step()
sys.settrace(self.trace_dispatch)
This sets up a callback to trace_dispatch, also in bdb.py. The sys.settrace code itself is perhaps in threading.py:
def settrace(func):
global _trace_hook
_trace_hook = func
GitHub search finds no more references for _trace_hook, so presumably that's picked up magically in C code somewhere.
When test() is called, it turns out the sys.settrace() call is made...but then the call to trace_dispatch() does not happen.
Related
Suppose you have this generator :
def gen():
for i in range(10):
yield i-5
And then you do that :
try :
for a in gen():
print(5 / a) # Obviously, it will crash on a=0
except:
import pdb; pdb.xpm() # I'm using pdbpp actually
Once the debugger started (either vanilla pdb or pdbpp), is there a way to "enter" inside the generator (go to its execution frame), to inspect its internal variables, without interrupting the program (I mean, to just "observe" and being able to coninue the execution as if the debugger were never launched) ?
EDIT
If it could help, in real code, I still have a reference to the generator, so it "more" like this :
g = gen()
try :
for a in g:
print(5 / a) # Obviously, it will crash on a=0
except:
import pdb; pdb.xpm() # I'm using pdbpp actually
With code like you originally posted, you can't inspect the generator.
The generator's already gone at the point you launch the debugger. The only reference to the generator was on the value stack of the frame object where the loop over the generator occurred, and that reference gets cleared as the exception propagates. Here's a program demonstrating this fact:
class DelPrintIterator:
def __iter__(self):
return self
def __next__(self):
return 1
def __del__(self):
print('Deleting iterator')
try:
for i in DelPrintIterator():
1/0
except ZeroDivisionError:
print('Too late')
This prints
Deleting iterator
Too late
showing that the iterator becomes unreachable before any code in the except block runs.
Now, with a reference to the generator, something like
g = gen()
try:
for a in gen():
print(5 / a)
except ZeroDivisionError:
...
then... well, I don't know of any debuggers that would let you "step into" the generator's stack frame when it's not on the stack, but you can manually retrieve the frame with g.gi_frame and then start looking through the same attributes the debugger uses, like frame.f_locals or frame.f_lineno. (Frame object attributes are listed in the inspect module docs.) It'll be less convenient than the debugger's interface, but the same information is still there.
I've seen numerous question on this site, where people ask how user-written code, say a function foo, can be executed in tkinter's mainloop, e.g. this or this. Two alternatives exist: Use the after method, or use threading. I'd like to know more about how the after method actually works.
More precisely, inspired by this excellent article, where a very high-level description of how the GIL in Python works is given, I'd like to know more how the after method works in terms of the Python interpreter processing foo inside tkinter's mainloop.
I'm particularly confused how the CPython interpreter steps through the code, when I insert code using after. How is foo ending up being executed by tkinter's mainloop?
What I found so far:
Bryan Oakley, quoted from the first link, says: "after does not create another thread of execution. Tkinter is single-threaded. after merely adds a function to a queue."
But inspecting the source code
def after(self, ms, func=None, *args):
"""Call function once after given time.
MS specifies the time in milliseconds. FUNC gives the
function which shall be called. Additional parameters
are given as parameters to the function call. Return
identifier to cancel scheduling with after_cancel."""
if not func:
# I'd rather use time.sleep(ms*0.001)
self.tk.call('after', ms)
else:
def callit():
try:
func(*args)
finally:
try:
self.deletecommand(name)
except TclError:
pass
callit.__name__ = func.__name__
name = self._register(callit)
return self.tk.call('after', ms, name)
doesn't really help me, as it doesn't reveal the answers to these questions, and I'm a novice programmer, so I don't really understand how to trace this further.
I'd like to know more about how the after method actually works.
mainloop is just an endless loop which scans some internal queues to see if there are any events to process. Think of it as if it was implemented something like this:
def mainloop():
while the_window_exists():
if len(after_queue) > 0:
event = after_queue.pop()
if event.time_to_execute >= time.time():
event.command(**event.args)
if len(event_queue) > 0:
...
It's not literally implemented that way -- it's a bit more efficient and there's a little more going on, but logically it's nearly identical.
When you call after, it simply puts something on the "after" queue. Nothing more, nothing less.
Using the same analogy, after might be implemented something like this:
def after(delay, code_to_run, *args):
event = Event()
event.code_to_run = code_to_run
event.args = args
event.time_to_execute = time.time() + delay
event_queue.append(event)
That's all there is to it. You're putting something on a queue when you call after, and mainloop pulls things off of that queue. It all happens in the same thread.
To mainloop, the function you added with after is no different than a function that gets added when you move the mouse or press a button -- it's just an event object on a queue.
class MyClass():
def __init__(self):
...
def start(self):
colorThread = threading.Thread(target = self.colorIndicator())
colorThread.start()
while True:
print ('something')
...
...
I also have a print statement inside the colorIndicator(). That statement is getting printed. But the print statement inside the while loop of start() method isn't displayed on screen.
The colorIndicator() also has an infinite loop. It gets some data from the internet and updates a counter. This counter is initialized inside __init__ as self variable and I'm using that variable inside other methods.
I do not understand why print inside while is not being executed.
colorIndicator function:
def colorIndicator(self):
print ('something else')
...
while (True):
...
print ('here')
time.sleep(25)
The output I get is the following:
something else
here
here
I stopped it after that. So, the colorIndicator is clearly running completely.
I'm calling the script with import in a python interpreter (in a terminal). Then I instantiate MyClass and call the start function.
You're not actually running colorIndicator in a thread, because you called it in the main thread, rather than passing the method itself (uncalled) as the thread target. Change:
colorThread = threading.Thread(target=self.colorIndicator())
# ^^ Agh! Call parens!
to:
# Remove parens so you pass the method, without calling it
colorThread = threading.Thread(target=self.colorIndicator)
# ^ Note: No call parens
Basically, your problem is that before you ever construct the Thread, it's trying to run colorIndicator to completion so it can use its return value as the target, which is wrong in multiple ways (the method never returns, and even if it did, it wouldn't return a callable suitable for use as a target).
I was reading Roger Stuckey's wxPython Multiprocessing code to try to make a similar program myself. Full code can be found here.
The code runs fine without any modification. However, I found a parameter self.update been pass around between the GUI class MyFrame to the processing class TaskSErverMP. I have been searched throughout the entire code snippet and couldn't figure out what it is doing in the code -- it has never been initialized and used anyhow.
In the class MyFrame:
def OnStart(self, event):
...
self.taskserver.processTasks(self.update)
...
def OnStop(self, event):
...
self.taskserver.processStop(self.update)
...
def update(self, output):
"""
Get and print the results from one completed task.
"""
self.output_tc.AppendText('%s [%d] calculate(%d) = %.2f\n'...
...
# Give the user an opportunity to interact
wx.YieldIfNeeded()
In the class TaskServerMP:
def run(self):
...
self.processTasks(self.update)
...
def processTasks(self, resfunc=None):
...
def processStop(self, resfunc=None):
...
def update(self, output):
"""
Get and print the results from one completed task.
"""
sys.stdout.write('%s [%d] calculate(%d) = %.2f' % ....
So I thought that is a dependency injection practice but nothing more. I then removed it from the code and the strangest thing happened -- the program doesn't work anymore! I have the GUI displayed and I was able to start the processing. However, the GUI just hanged and then later Windows reported that the program is not responding. I have end up kill all the pythonw.exe processes manually from the Windows Task Manager.
Then I start to think if there is anything to do with the signature of the functions processTasks and processStop in the class TaskServerMP. But I really have no idea how I can associate the parameter self.update to the optional argument resfunc.
I don't think there is anything wrong in Roger's code. But it bothers me if I cannot twisted around the source to test out my understanding of the code.
I use Python 2.7 in Windows 7.
MyFrame.update is a method. You can see its definition on line 365.
So self.update is a bound method, meaning it can be called as if it were a regular function.
You can see that processTasks takes a resfunc parameter; then, at least 165, if it got a function or method as that resfunc parameter, it calls it.
The idea here is that processTasks leaves it up to the caller to decide how to print out progress updates as each task completes. One class might do it by writing them to stdout. A different class might instead update a GUI progress bar.
This is a pretty typical way to pass callbacks around in Python code.
So, why does the program hang if you take out the self.update? Well, look what's inside it, at line 372:
# Give the user an opportunity to interact
wx.YieldIfNeeded()
In wx, as in most GUI frameworks, the main thread is running an "event loop", something which has to process each event (a mouse move, a keypress, whatever) as it comes in, and then wait for the next one. You write your code as a bunch of event handlers—when someone clicks this button, run that function; etc. Your event handlers have to return quickly. Otherwise, the event loop doesn't get to pick up and dispatch the next event, so your GUI isn't responding. In wx, the Yield family of functions make life easier. As long as you Yield often enough, you don't have to return quickly. But you still have to do one or the other—either return early, or Yield—or the GUI will hang.
Here's a very simple example showing how to use bound methods:
class Foo(object):
def __init__(self, name):
self.name = name
def print_name(self):
print(self.name)
def give_me_a_printer_function(self):
return self.print_name
spam = Foo('Spam')
my_function1 = spam.print_name
my_function2 = spam.give_me_a_printer_function()
my_function1()
my_function2()
This will print Spam twice.
Functions and methods are first class values in Python—you can pass them around just like you can pass around numbers, strings, lists, and class instances. You can even print them out (although you'll get something ugly like <bound method Foo.print_name of <__main__.Foo object at 0x104629190>>).
In my python code I have this line:
try:
result = eval(command, self.globals, self.locals)
except SyntaxError:
exec(command, self.globals, self.locals)
The command variable can be any string. Hence the python debugger pdb may be started in eval/exec and still be active when eval/exec is returning. What I want to do is make sure normal program execution is resumed when returning from eval/exec. To just give you an idea, this is approximately the behavior that I want:
try:
result = eval(command, self.globals, self.locals)
try: self.globals['pdb'].run('continue')
except: pass
except SyntaxError:
exec(command, self.globals, self.locals)
try: self.globals['pdb'].run('continue')
except: pass
However the try line is shown in the debugger before it is executed, but I dont want the debugger to show my code at all. Also it doesn't really work... The reason i repeat code is to minimize debugging in my code, else I could just do it after the except block.
So how can I do this?
As a sidenote:
If you try to enter the following lines into the IPython or bpython interpreters you'll see that they have the same problem and you are able to step into their code.
import pdb
pdb.set_trace()
next
However if you do this in the standard cpython interpreter you are returned to the python prompt. The reason for this is obviously because the two former are implemented in python and the last one is not. But my wish is to get the same behavior even when all code is python.
While I'm somewhat concerned that you are eval/exec'ing a string that you don't control, I'll assume you've thought that bit through.
I think the simplest thing would be to persuade pdb to check the stack frame on each step and resume automatically when you return to the desired level. You can do that with a simple bit of hotfixing. In the code below I've simplified it down to a simple eval since all you are really asking is to have pdb resume automatically on return to a specific function. Call Pdb().resume_here() in the function that you don't want traced. N.B. the resumption is global and there's only one resumption point but I'm sure you can modify that if you wanted.
If you run the code then you'll enter the debugger in function foo() and you can then single step but as soon as you return to bar() the code continues automatically.
e.g.
import sys
from pdb import Pdb
def trace_dispatch(self, frame, event, arg):
if frame is self._resume_frame:
self.set_continue()
return
return self._original_trace_dispatch(frame, event, arg)
def resume_here(self):
Pdb._resume_frame = sys._getframe().f_back
# hotfix Pdb
Pdb._original_trace_dispatch = Pdb.trace_dispatch
Pdb.trace_dispatch = trace_dispatch
Pdb.resume_here = resume_here
Pdb._resume_frame = None
def foo():
import pdb
pdb.set_trace()
print("tracing...")
for i in range(3):
print(i)
def bar():
Pdb().resume_here()
exec("foo();print('done')")
print("returning")
bar()