Develop Reference

Preventing python script from using all ram

I use a jupyter notebook to execute a pyton script. The script calls the association_rules function from the mlxtend framework. Calling this function the ram literally explodes from 500 MB used to over 32 GB. But that would not be the problem. The Problem is if i execute the script locally on my windows 10 PC the ram maxes out but everything is still running. When i do the same on a unix server (Xfce) the server crashes. Is there something i can do prevent the server from crashing and to guarantee that the script continues?
Upadate:
I basically missed the fact that windows is swapping ram all the time, the only difference is that windows does not crash. I'm quite sure this would be solved on linux by fixing the swapping configuration. So basically the question is obselete.
Update:
I have made some wrong assumptions. The windows PC was already swapping, and the swapping partition went out of memory as well. So on all machine the same Problem appeared and all them crashed. In the end it was a mistake on the data preprocessing. Sorry for the unconvenience and please see this question as not relevant any more.

Run the script using the nice command to assign priority.

PyCharm Debugger Stuck on "Collecting Data"

So I installed the free version of PyCharm professional last week and have been encountering the problem where I am debugging code on a remote server and when I try to display variables, it simply says "collecting data" and then if I try to continue the debugging process PyCharm breaks.
I have been researching solutions and I have Gevent compatible enabled as well as tried all 3 variable settings; Synchronously, Asynchronously, and On Demand.
I should also note that I am running into a problem where the debugger is skipping all my break points as well and I have to restart my server connection in order to get the break points to hit (and sometimes it takes a couple of tries)
I know that it is entirely possible to see the variables that are collecting data as one of my co-workers who recommended PyCharm has no problem, and there was 1 run where I was able to see the variables, but when I re-ran the commands (with absolutely no change), I was back at square one.
I've been going through PyCharm forums and it seems as if this has been a reoccurring issue for a handful of years now, but knowing that it worked once for me, and it works for my coworker, am I simply missing something?

Just recently my PyCharm has started to behave this way as well. I researched and tried the same solutions you did, to no avail. On certain projects it simply hangs forever on "collecting data" - projects that used to work - where code hasn't changed. Pls let me know if you find anything else; I will keep researching and testing as well
EDIT: FWIW. In my particular case I isolated the cause of this (at least I think). I had a very large dataframe in memory and if this DF is not in memory, the debugger does not hang. None of my watches were explicitly on this DF, but I guess the debugger needed to inspect it upon break and the object was just too big somehow. Note that it hung even if "variables loading policy" was set to "on demand", so the debugger still must automatically investigate all variables somehow.

Python code freeze on different pc

I got a Neural Network implemented with numpy (Python 2.7) and a machine to test it faster. Lately my code on this machine got freeze but if a test it on my notebook (less cpu, ram, etc) it run without problem (only slower).
Which can be the problem? I thought that it was my code but if it works on a slower pc, so I think that machine have a trouble.
edit: Also, sometimes it works without problems.
edit 2: Both pcs are Ubuntu 16.04
edit 3: It happens event with same input and parameters

If it doesn't always occurs and is confined to one machine it could very well be a hardware problem.
The problem is that they are often hard to test, because they generally leave little evidence in the way of log files.
Try testing the RAM.
If that doesn't turn up errors, try logging the CPU temperatures to check that it doesn't get too hot.
Also, log the different voltages. It could be that the power supply is on the way out.

Try compiling code on the same machine where your code gets frozen. Each machine (more precisely microprocessor) has different instruction set. The flaws in instruction set may be covered by using Microcode. This could be the place where the problem may exist.

Python starts extremely slowly the first time after I reboot Windows

I apologize for not having a reproducible example. My problem is with a large base of proprietary code and I don't have an extract that shows the same behavior. Even better, it isn't my software and I know about 2% of how it works.
Simply, this Python program I'm dealing with takes about 80 seconds to complete its entire setup and get to the point where all its flask code is running and the webserver being created is up and able to respond to requests. BUT -- that's only the first time I run it on Windows after rebooting. On subsequent times starting the python script in question, it takes more like 10 seconds.
And the nutty part is, in a workgroup of 10 people, mine is the only computer that has the problem.
Things I can say:
Python 2.7.11, Windows 7, git bash version 2.9.0.windows.1.
It doesn't appear to matter whether I invoke my python program from the git bash command line or the Windows command line.
However, in git bash, saying "python" gets no response forever until I hit Ctrl-C, but saying "winpty python" opens an interactive python session as it should. I mention this because for a while I thought my main problem was related to the git bash shell bug (https://stackoverflow.com/a/32599341/5593532). But point 2 above would seem to contradict this. No such weirdness occurs in invoking a bare python interactive session from the Windows command line.
I've had trouble getting meaningful profiling output, partly because of multi-threading or child processes or something. And the web server doesn't have an exit event per se, thus I can only stop it by smacking it with a Ctrl-C in the command-line window where I ran the script, which seems to kill the part of the process that would save the profiling data.
From the fragmentary profiling info I was able to produce (with gratitude to https://ymichael.com/2014/03/08/profiling-python-with-cprofile.html), I am suspicious that something weird is happening in loading a large number of imported packages, and perhaps especially the alembic and/or werkzeug packages. (And maybe even sqlalchemy.) The profiling output didn't have much tottime in those packages, but it did have rather a lot of cumtime there.
My sys.path inside Python doesn't seem meaningfully different from anyone else's nearby. I might have one or two different items in the list, or three .egg files on the path when they've only got one, but it's mostly the same list in the same order. So much for the idea that it's taking a long time to hunt and learn where packages are and then re-using the information later.
I've got PyCharm Community Edition able to run the script and its associated junk in IDE mode, set breakpoints, and all that jazz, so I can set breakpoints and follow execution to a degree, in case that would answer a noteworthy question you could raise for me.
Anyone got a wild notion what's up? (he asked quite unreasonably)

How can I make the Eclipse Python debugger more reliable?

I've found that under some circumstances the Eclipse python debugger can be unreliable. For example, when stepping through a memory-hungry Python program I've found that after a certain point the debugger fails to respond. The entire process hangs with 100% cpu load.
I've heard (unconfirmed) reports from developers that when this occurs it might actually not be a hang but very slow, for example a step which usually takes a micro-second might be taking five minutes.
I've observed this with both the regular eclipse debugger (e.g. when you do debug as->python run) and the pydev network debugger. The point at which things stop working seems quite arbitrary but re-producable. I'm debugging purely numerical functions so there's no obvious socket or tcp activity which could disrupt a network debugger. Nevertheless, the point at which the failure occurs seems to be quite consistent for a given workstation & task.
I get almost exactly the same effect whether I debug using debug as->python run as I might if I use the pydev's network debugger ('import pydevd;pydevd.settrace()') - where one fails the other is also likely to fail.
If I were to run the script normally (no debugger) it executes without any hangs. Furthermore, if I replace the eclipse debugger with the old-skool inline break-point ('import pdb;pdb.set_trace()') everything works just fine. Pdb almost never goes wrong, however there are times I simply cannot use it, for example when I am debugging a python function invoked from Excel - as there is no console present.
I've noticed that this failure never seems to occur with trivial programs, hence I've tentatively concluded that the fault may be memory related.
So - is anybody aware of what might be causing this unreliability. Are there any tweaks I can make to the debugger or environment that might make the debugger work more reliably?
FYI, running Python 2.4.4 on Windows XP 32bit with the latest Eclipse & Pydev. Psyco (JIT Compiler) is not available.

There are known issues with Python 2.4 (so, if possible, get a newer version), still, if you're unable to get a newer version, at least use threadframe extension (http://pypi.python.org/pypi/threadframe). If None of that's possible, it's really expected that the debugger doesn't work as well as it could (as the debugger needs features that are not available in that version).

Not sure if its related, but I had a scenario where too many debug messages will cause my PC to become very slow. You can try go to Window -> Preferences -> Run/Debug -> Console... tick Limit console output, and reduce Console buffer size (mine is set to 40000).