I am running a Python 2.7 script in Spyder that can run as long as I need, but I am getting a MemoryError. Short runs of this script do not do this, and I have no reason to believe that the run is substantially different as the runtime grows arbitrarily long. I'd like to go into Spyder's settings to allow more memory to be consumed, but I cannot find where to go in the settings.
Related
I am trying to read around 20k+ text files but I run out of memory before I can process them.
How can I increase the available memory for my python script? in Pycharm. I have already tried increasing my heap memory for Pycharm by going to help->change memory settings but it didn't work. Also if I run my script in Jupyter it works perfectly fine just getting this issue in Pycharm.
I am currently currently a python code that I believe has some memory issue.
I reasoned it out last night, I believe it is due to some overheads that's been being constantly generated but I don't know what exactly they are. I have tried gccollect() method and it didn't really work. The good thing about my python code is that at each run, I can save the parameters of the partially trained model as pickle. And hopefully, I will shut down the script, and start a new python script and load my saved pickle to continue training.
My strategy is to do the following in a shell script:
for i in 1000:
run `python train_model.py` # this will give me allows a fresh environment with little memory overhead
# during the runs of `train_model.py`, some memory overhead has been accumulated
torch.save(train_model.state_dict())
shut down the train_model.py script
# I am hoping that for the next run of this python script, I can get a fresh environment
Do you think this will work? How do I do this in a shell script?
EDIT2: Stress tested the fix, it still failed after creating near 1TB of intermediary files. Changed .py code to delete intermediary files after performing necessary processes. Stress test succeeded after this change. Original issues likely to do with memory. Still no way of proving.
EDIT: I'm still not sure whether or not it was a memory issue. However I got the full process to run via .bat by breaking up the second script into 4 additional .py files and running those instead of the full second script at once. Problem solved for now.
ORIGINAL:
I am running two .py files through cmd via a .bat file. The reason I am doing this is the first .py is an arcpy file that requires use of 32 bit python, and the second .py is a combined PCI and arcpy file that requires use of 64 bit python.
The first file runs without issue. The second file gets to a certain point, the same point every time, and I am prompted with a "python.exe has stopped working" dialog box. This point is in the middle of a for loop in the PCI section of code. I have run the PCI python in PCI's interpreter without issue. I suspect this might be related to memory issues but am uncertain, and have no idea how to check.
How do I check to see if memory issues are the problem?
.bat code below:
C:/Python27/ArcGIS10.5/python.exe E:/Path/CODE/file1.py
C:/Python27/ArcGISx6410.5/python.exe E:/Path/CODE/file2.py
Is there a way of resetting the GPU through arrayfire with each run without restarting the Python console?
I have a python script which calls a number of functions to be executed on a GPU using arrayfire. The script gives the expected result when run once, but if run a second time without changing any parameters it returns NaN values. This stops once I either restart the Python console or, equivalently, run the following:
import arrayfire as af
af.device.free_device(af.device.device_mem_info()['alloc']['bytes'])
which restarts the console after clearing all allocated device memory.
I'm having a hard time seeing what could be going wrong here; the program runs well the first time, so I think that it is likely to be a command that I can't find in the docs that I am missing.
I have tried most of the functions in the docs which looks promising (device_gc, free_device, free_pinned, device.init, etc.) but have not had much luck. I have also attempted to use reload(arrayfire) too.
Could this be a bug? Any advice would be appreciated.
I'm using: nvidia quadro M4000, Python 2.7, windows 7, Spyder 3
I've looked at some questions about profiling memory usage in Python programs, but so far haven't been able to get anything to work. My program must run as root (it opens a TUN/TAP device).
First, I tried heapy; unfortunately this didn't work for me. Every time my code tried to execute hpy().heap() the program froze. Not wanting to waste too much timed I decided to try valgrind.
I tried valgrind with massif:
# valgrind --tool=massif ./my_prog.py --some-options value
I think the issue is related to profiling Python programs. I tried my program (which runs as root) and no massif output file was generated. I also wasn't able to generate an output file with another Python program (which doesn't run as root). However, a simple C test program worked fine and produced the massif file.
What are the issues preventing Valgrind and massif from working correctly with Python programs?
Instead of having the script launch the interpreter, directly calling it as a parameter to Valgrind solves the problem.
valgrind --tool=massif python my_script.py