I've found that under some circumstances the Eclipse python debugger can be unreliable. For example, when stepping through a memory-hungry Python program I've found that after a certain point the debugger fails to respond. The entire process hangs with 100% cpu load.
I've heard (unconfirmed) reports from developers that when this occurs it might actually not be a hang but very slow, for example a step which usually takes a micro-second might be taking five minutes.
I've observed this with both the regular eclipse debugger (e.g. when you do debug as->python run) and the pydev network debugger. The point at which things stop working seems quite arbitrary but re-producable. I'm debugging purely numerical functions so there's no obvious socket or tcp activity which could disrupt a network debugger. Nevertheless, the point at which the failure occurs seems to be quite consistent for a given workstation & task.
I get almost exactly the same effect whether I debug using debug as->python run as I might if I use the pydev's network debugger ('import pydevd;pydevd.settrace()') - where one fails the other is also likely to fail.
If I were to run the script normally (no debugger) it executes without any hangs. Furthermore, if I replace the eclipse debugger with the old-skool inline break-point ('import pdb;pdb.set_trace()') everything works just fine. Pdb almost never goes wrong, however there are times I simply cannot use it, for example when I am debugging a python function invoked from Excel - as there is no console present.
I've noticed that this failure never seems to occur with trivial programs, hence I've tentatively concluded that the fault may be memory related.
So - is anybody aware of what might be causing this unreliability. Are there any tweaks I can make to the debugger or environment that might make the debugger work more reliably?
FYI, running Python 2.4.4 on Windows XP 32bit with the latest Eclipse & Pydev. Psyco (JIT Compiler) is not available.
There are known issues with Python 2.4 (so, if possible, get a newer version), still, if you're unable to get a newer version, at least use threadframe extension (http://pypi.python.org/pypi/threadframe). If None of that's possible, it's really expected that the debugger doesn't work as well as it could (as the debugger needs features that are not available in that version).
Not sure if its related, but I had a scenario where too many debug messages will cause my PC to become very slow. You can try go to Window -> Preferences -> Run/Debug -> Console... tick Limit console output, and reduce Console buffer size (mine is set to 40000).
Related
So I installed the free version of PyCharm professional last week and have been encountering the problem where I am debugging code on a remote server and when I try to display variables, it simply says "collecting data" and then if I try to continue the debugging process PyCharm breaks.
I have been researching solutions and I have Gevent compatible enabled as well as tried all 3 variable settings; Synchronously, Asynchronously, and On Demand.
I should also note that I am running into a problem where the debugger is skipping all my break points as well and I have to restart my server connection in order to get the break points to hit (and sometimes it takes a couple of tries)
I know that it is entirely possible to see the variables that are collecting data as one of my co-workers who recommended PyCharm has no problem, and there was 1 run where I was able to see the variables, but when I re-ran the commands (with absolutely no change), I was back at square one.
I've been going through PyCharm forums and it seems as if this has been a reoccurring issue for a handful of years now, but knowing that it worked once for me, and it works for my coworker, am I simply missing something?
Just recently my PyCharm has started to behave this way as well. I researched and tried the same solutions you did, to no avail. On certain projects it simply hangs forever on "collecting data" - projects that used to work - where code hasn't changed. Pls let me know if you find anything else; I will keep researching and testing as well
EDIT: FWIW. In my particular case I isolated the cause of this (at least I think). I had a very large dataframe in memory and if this DF is not in memory, the debugger does not hang. None of my watches were explicitly on this DF, but I guess the debugger needed to inspect it upon break and the object was just too big somehow. Note that it hung even if "variables loading policy" was set to "on demand", so the debugger still must automatically investigate all variables somehow.
I apologize for not having a reproducible example. My problem is with a large base of proprietary code and I don't have an extract that shows the same behavior. Even better, it isn't my software and I know about 2% of how it works.
Simply, this Python program I'm dealing with takes about 80 seconds to complete its entire setup and get to the point where all its flask code is running and the webserver being created is up and able to respond to requests. BUT -- that's only the first time I run it on Windows after rebooting. On subsequent times starting the python script in question, it takes more like 10 seconds.
And the nutty part is, in a workgroup of 10 people, mine is the only computer that has the problem.
Things I can say:
Python 2.7.11, Windows 7, git bash version 2.9.0.windows.1.
It doesn't appear to matter whether I invoke my python program from the git bash command line or the Windows command line.
However, in git bash, saying "python" gets no response forever until I hit Ctrl-C, but saying "winpty python" opens an interactive python session as it should. I mention this because for a while I thought my main problem was related to the git bash shell bug (https://stackoverflow.com/a/32599341/5593532). But point 2 above would seem to contradict this. No such weirdness occurs in invoking a bare python interactive session from the Windows command line.
I've had trouble getting meaningful profiling output, partly because of multi-threading or child processes or something. And the web server doesn't have an exit event per se, thus I can only stop it by smacking it with a Ctrl-C in the command-line window where I ran the script, which seems to kill the part of the process that would save the profiling data.
From the fragmentary profiling info I was able to produce (with gratitude to https://ymichael.com/2014/03/08/profiling-python-with-cprofile.html), I am suspicious that something weird is happening in loading a large number of imported packages, and perhaps especially the alembic and/or werkzeug packages. (And maybe even sqlalchemy.) The profiling output didn't have much tottime in those packages, but it did have rather a lot of cumtime there.
My sys.path inside Python doesn't seem meaningfully different from anyone else's nearby. I might have one or two different items in the list, or three .egg files on the path when they've only got one, but it's mostly the same list in the same order. So much for the idea that it's taking a long time to hunt and learn where packages are and then re-using the information later.
I've got PyCharm Community Edition able to run the script and its associated junk in IDE mode, set breakpoints, and all that jazz, so I can set breakpoints and follow execution to a degree, in case that would answer a noteworthy question you could raise for me.
Anyone got a wild notion what's up? (he asked quite unreasonably)
I am running a Python script using Spyder 2.3.9. I have a fairly large script and when running it through with (300x600) iterations (a loop inside another loop), everything appears to be working fine and takes approximately 40 minutes. But when I increase the number to (500x600) iterations, after 2 hours, the output yields:
It seems the kernel died unexpectedly. Use 'Restart kernel' to continue using this console.
I've been trying to go through the code but don't see anything that might be causing this in particular. I am using Python 2.7.12 64bits, Qt 4.8.7, PyQt4 (API v2) 4.11.4. (Anaconda2-4.0.0-MacOSX-x86_64)
I'm not entirely sure what additional information is pertinent, but if you have any suggestions or questions, I'd be happy to read them.
https://github.com/spyder-ide/spyder/issues/3114
It seems this issue has been opened on their GitHub profile, should be addressed soon given the repo record.
Some possible solutions:
It may be helpful, if possible, to modify your script for faster convergence. Very often, for most practical purposes, the incremental value of iterations after a certain point is negligible.
An upgrade or downgrade of the Spyder environment may help.
Check your local firewall for blocked connections to 127.0.0.1 from pythonw.exe.
If nothing works, try using Spyder on Ubuntu.
I made this script:
from gasp import *
begin_graphics()
Circle((200, 200), 60)
Line((100, 400), (580, 200))
Box((400, 350), 120, 100)
update_when('key_pressed')
end_graphics()
When I start it from terminal, it works perfectly. When I run it from IDLE, it doesn't work, I get no answer (shell prompt (>>>) disappears but nothing happens).
In general, you can't run GUI apps in the embedded Python interpreter in IDLE, unless the library you're using is designed to integrate with IDLE. Or, worse, it may work on one machine and not on another. I'll explain why below, but first just take that on faith.
As far as I can tell, gasp's documentation doesn't address this, but similar libraries either warn you that they may not work in IDLE (easygui, early versions of graphics, etc.) or come with special instructions for how to use them in IDLE (e.g., later versions of graphics).
Now, maybe gasp should be designed to integrate with IDLE, given that it's specifically designed for novices, and many of those novices will be using the IDE that comes built in with Python. Or maybe not. But, even if that should be true, that's something for gasp to deal with. File a bug or feature request, but you'll need some way to keep working until someone gets around to writing the code.
The simplest solution here is to use a different IDE, one that runs its interactive Python interpreter in a completely separate process, exactly the same as you get when running it yourself in the terminal. There are lots of good options out there that are at least free (-as-in-beer) for non-commercial use (PyCharm, Komodo, Eclipse PyDev, emacs with your favorite collection of packages, etc.). Although Stack Overflow is not a good place for advice on picking the best one for you (if googling isn't sufficient, try asking on a mailing list or forum), almost any of them will work.
Another option: instead of using an interpreter built into an IDE, you might want to to consider running an enhanced interpreter environment (like ipython-gtk or emacs with a smaller set of packages) alongside your IDE. Of course they'll no longer be tightly integrated (the "I" in "IDE"), but in my experience, even working in an environment where the whole team uses PyCharm or PyDev, I still end up doing most of my interactive testing in ipython; you may find you prefer that as well. Or you may not, but give it a try and see.
So, why is there a problem in the first place?
First, if you don't understand what an "event loop" or "runloop" or "mainloop" is, please read either Why your GUI app freezes or the Wikipedia page or some other introduction to the idea.
Normally, when you run the interactive Python interpreter (e.g., by typing python at the bash or C: prompt in your terminal), it runs in its own process. So, it can start up a runloop and never return (until you quit), and the terminal won't get in your way.
But when you run the interactive Python interpreter inside IDLE, it's actually running in the same process as IDLE, which has its own runloop. If you start up a runloop and never return, IDLE's runloop doesn't get to run. That means it doesn't get to respond to events from the OS, like "refresh your window" or "prepare for a new window to open", so from the user's (your) point of view, IDLE and your app are both just frozen.
One way to get around this is in your code is to spawn another thread for your runloop, instead of taking over the main thread. (This doesn't work with all GUI libraries, but it works with some. This is how graphics solved the problem.) Another way is to spawn a whole new child process to run your GUI. (This works with all GUI libraries, but it's a lot more workânow you have to deal with inter-process communication. One of the novice-friendly matplotlib wrappers does this.) Finally, you can integrate your runloop with IDLE's Tkinter runloop by having one drive the other. (Not all GUI libraries can be driven this way, but Tkinter's can, and IDLE can be monkeypatched to work this way; graphics used to do this.) But none of these are even remotely simple. And they're probably things that gasp itself should be doing, not your code.
I've recently moved from Matlab to Python. Python is a much better language (from the point of view of a computer scientist), but Python IDEs all seem to lack one important thing:
A proper interactive debugger.
I'm looking for:
The ability to set breakpoints graphically by clicking next to a line of code in the editor.
The ability to run ANY CODE while stopped in the debugger, including calling functions from my code, showing new windows, playing audio, etc.
When an error occurs, the debugger should automatically open an interactive console at the error line.
Once done with the interactive console, you can resume normal execution.
Matlab has all these features and they work incredibly well, but I can't find them anywhere in Python tools.
I've tried:
PyCharm: the interactive console is clunky, often fails to appear, and crashes all the time (I've tried several different versions and OSs).
IPython: can't set breakpoints -Launching a Python console programatically: you have to stop your code, insert an extra line of code, and run again from the beginning to do this. Plus, you can't access functions already imported without re-importing them.
Being able to debug and fix problems THE FIRST TIME THEY APPEAR is very important to me, as I work in programs that often take dozens of minutes to re-run (computational neuroscience).
CONCLUSION: there is NO way to do all of these in Python at the moment. Let us hope that PyLab development accelerates.
At the top of your code, write
import pdb
And within your code, use the following statement wherever you want to debug.
pdb.set_trace()
You will have an interactive shell thus, whenever the set_trace() statement is met.
You can then use step(s), next(n), continue(c) and so on to check the execution flow, and print values of variables like print var
For more details on pdb, refer here
There are many Python IDEs. That was a topic here: What IDE to use for Python?
"The ability to set breakpoints graphically by clicking next to a line of code in the editor."
PyDev has this. Double-click in the gray margin bar.
"The ability to run ANY CODE while stopped in the debugger, including calling functions from my code, showing new windows, playing audio, etc."
PyDev has this. It's not the only one. PyScripter's stated features seem to include this.
"When an error occurs, the debugger should automatically open an interactive console at the error line."
PyDev does this. (I think. Or at worst do you need to double-click on the console message that states the error's location in the code?)
"Once done with the interactive console, you can resume normal execution."
PyDev has this. It's called "resume". It's what the green "play" triangle in a toolbar does. Some other software calls this feature "continue".
I've been searching for the same, but unfortunately Python IDEs are not as well-featured as Matlab's at this point. For scientific programming, you will also want graphics/plotting to run in an entirely different thread, so IPython integration is essential. As far as I can tell, the Matlab IDE feature to change the workspace from the debugger, which then affects code running subsequently, is quite unique. Each of the features exist in some IDE, but none exist in all:
Spyder has good integration with scientific tools, but its debugging is limited to the built-in pdb, which lacks the requirement to execute any code and have this code affect the namespace after continuing.
PyDev and PyCharm, and quite a few others, have decent debugging features, but I don't think it has good integration with scientific tools. That means, if you plot, you lose access to your prompt. Not good.
As far as I've experienced, the closest comes Wing IDE. It is a propietry product, but if you're making the transition from Matlab 89$/year for non-commercial-use should be acceptable (you can evaluate it first). But for me, I've ultimately settled to alter my workflow, and not using any sophisticated IDE at all. When I looked was some years ago, so perhaps the situation has improved.
You might also be interested in this article from April 2013, Evaluating IDEs for Scientific Python. It doesn't really reach a conclusion either.
Seeing as you are comming from Matlab, I would suggest you give a look at
Python(x,y)
The page decribes it as follows:
Python(x,y) is a free scientific and engineering development software for numerical computations, data analysis and data visualization based on Python programming language, Qt graphical user interfaces and Spyder interactive scientific development environment.
It will not cater to all your wishes, but it certainly made me feel comfortable when I started out with Python, coming from Matlab.
Hope it helps
You can do all this in the iPython Notebook. Use the magic command %pdb to stop on error.
If using the command line,
alias ipythondebug='ipython --InteractiveShell.pdb true'
in your ~/.profile will give you debug on error like Matlab. This of course requires ipython installed.
Not sure about the resuming part.
You can also edit the ipython config file if you want the debug on error to be permanent. See
https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-pdb