I've looked at some questions about profiling memory usage in Python programs, but so far haven't been able to get anything to work. My program must run as root (it opens a TUN/TAP device).
First, I tried heapy; unfortunately this didn't work for me. Every time my code tried to execute hpy().heap() the program froze. Not wanting to waste too much timed I decided to try valgrind.
I tried valgrind with massif:
# valgrind --tool=massif ./my_prog.py --some-options value
I think the issue is related to profiling Python programs. I tried my program (which runs as root) and no massif output file was generated. I also wasn't able to generate an output file with another Python program (which doesn't run as root). However, a simple C test program worked fine and produced the massif file.
What are the issues preventing Valgrind and massif from working correctly with Python programs?
Instead of having the script launch the interpreter, directly calling it as a parameter to Valgrind solves the problem.
valgrind --tool=massif python my_script.py
Related
Python script generates segmentation fault
I cannot find the source of the problem.
How is it to debug a segfault simply in Python ?
Are there recommended coding practices to avoid Segmentation Faults ?
I wrote a script (1600 lines) to systematically download CSV files from a source and format the collected data. The script works very well for 100-200 files, but it systematically crashes with a segmentation fault message after a while.
-Crash always happens at the same spot in the script, with no understandable cause
-I run it on Mac OSX but the crash also happens on Ubuntu Linux and Debian 9
-If I run the Pandas Routines that crash during my script on single files, they work properly. The crash only happens when I loop the script 100-200 times.
-I checked every variable content, constructors (init) and they seem to be fine.
Code is too long to be pasted, but available on demand
Expected result should be execution to the end. Instead, it crashes after 100-200 iterations
I need to instrument a Python script using Intel Pin for ChampSim simulator.
The problem is, whenever I run the tool, the script does not seem to run as nothing is printed. Moreover, no matter how long/complex the script is, the trace always ends up with a size of 62M (this is also the case when I simply instrument the interpreter without any script).
I tried running the solution from this post, but it didn't work either. For reference, I am running the following command:
../../../pin -t obj-intel64/champsim_tracer.so -- ./python_script.py
Is it even possible to instrument a Python script? If yes, please detail the steps. Thanks!
I am running a Python 2.7 script in Spyder that can run as long as I need, but I am getting a MemoryError. Short runs of this script do not do this, and I have no reason to believe that the run is substantially different as the runtime grows arbitrarily long. I'd like to go into Spyder's settings to allow more memory to be consumed, but I cannot find where to go in the settings.
I have the situation that I want the job done from eclipse so I use eclipse's .launch configuration. I tried to make it run python directly but got error: error 193 (%1 is not a valid Win32 app). where %1 is probably my python script.
I decided to create a simple batch script that calls this big wild python animal.
I did a lot of combinations and found this the best (batch outputs some strings, runs python, waits for it, the outputs some strings again):
start /b /wait "Python_script.py" "%1" "%2" "%3" "%4" "%5"
It worked until python itself started to run exe file.
Once again I tried a lot of combinations:
os.system([exe, arg1, arg2, ...]) and
subprocess.call(..) and subprocess.check_output(..)
-> I either didn't see the output in eclipse console, or the output was delayed or there was only python / or only exe's output in console.
finally I used subprocess.Popen(...) and it's nearly allright - the only defect is that the output from python script don't wait for exe's process to finish, and when I use subprocess.Popen(...).wait() exe passes output to console but the WHOLE output from python script is delayed until the 'exe' terminates. I want to delay only the part of pythons script output that is written after the exe is called.
how to achieve this 'partly console output delay' is the main topic
advices on python and eclipse .launch configuration will be appreciated
general advices on how does the communication between this processes(?) work will be appreciated
Thanks!
It sounds to me like you have three different processes you're trying to get to work together, you've tried a whole bunch of stuff to get it working, and the code is complex enough that you can't easily post it here. That makes it pretty hard to get a good answer (Stack Overflow works much better with focused questions), but here's the general approach I'd take:
Does your script run if you try to run Python_script.py directly from the command prompt?
If it doesn't, then look into registering the .py file type in Windows.
If it does, then maybe Eclipse launch configurations don't support or don't properly support Windows registered file types. There should be no need to mess with batch files and start; just replace Python_script.py in your launch configuration with c:\Python27\python.exe Python_script.py (or similar).
Get your script working from a command prompt - able to run, with proper Python and subprocess output, and waiting for everything to terminate.
If things work from the command prompt and still don't work from Eclipse, then post a new question with a small snippet of code showing what you're trying and a description of what's wrong. subprocess.call, subprocess.check_output, and Popen all have different uses, so it's hard to give general advice besides just referring to the documentation.
I've found a lot of people having the reverse issue, but haven't yet found a question that involves IDLE not being able to run something that runs fine from the command line. I'm using a new module that I haven't used before that uses one .pyd file and one .dll, and involves a device that connects through USB. I sadly can't post in-depth code snippets since this is copyrighted code, but if anyone knows where to start on a problem like this I would be very grateful.
IDLE swaps out the sys.stdout and sys.stderr objects at the Python level this causes issues with some pyd modules. Try using another debugger.