I wrote some C code to create a python module. I wrote the code myself (did not use SWIG etc). If you're interested, the C code is at the bottom of this thread.
Q: Is there any way to (hopefully, easily) find whether my C code has memory leaks? Is there any way to use python's awesomeness for finding memory leaks in its extensions?
if you using a linux environment we can easily find the memory leaks by using debugger named valgrind.
To get the valgrind first you have to install it from the internet by using command
sudo apt-get valgrind
after the installation is completed compile your c code using this debugger you can easily find the memory leaks. This debugger shows what is the reason for the memory leak and also specifies the line at which the leak has been occured.
Related
My research requires processing memory traces of applications. For C/C++ programs, this is easy using Intel's PIN library. However, as suggested here Use Intel Pin to instrument Python scripts, I may need to instrument the Python runtime itself, which I'm not sure will represent the true memory behavior of a given python script due to some overheads(If this is not the case, please comment). Some of the existing python memory profilers only talk about the runtime memory "usage" in terms of the heap space usage, etc.
I ended up making an executable from my python script using PyInstaller and running my PINTool over it. However, I'm not sure if this is the right approach.
Is there any way(any library or any hack into the python runtime) that may help in getting the memory traces accessed by the python scripts?
I am writing a data mining script to pull information off of a program called Agisoft PhotoScan for my lab. PhotoScan uses its own Python library (and I'm not sure how to access pip for this particular build), which has caused me a few problems installing other packages. After dragging, dropping, and praying, I've gotten a few packages to work, but I'm still facing a memory leak. If there is no way around it, I can try to install some more packages to weed out the leak, but I'd like to avoid this if possible.
My understanding of Python garbage collection so far is, when an object loses its reference, it should be deleted. I used sys.getrefcount() to check all my variables, but they all stay constant. I have a hunch that the issue could be in the mysql-connector package I installed, or in PhotoScan itself, but I am not sure how to go about testing. I will be more than happy to provide code if that will help!
It turns out that the memory leak was indeed with the PhotoScan program. I've worked around it by having a separate script open and close it, running my original script once each time. Thank you all for the help!
I am writing a python program that includes an extension module written in C. The extension module defines a function that is used in my program continually.
I have a memory leak somewhere in my program, but I don't know how to find it. I have tried
1) Installing Valgrind. I can't get this to work however since I have OSX Mavericks (10.9) and Valgrind only supports OXS 10.7 (and 10.8 somewhat unstably).
2) Using gdb. I can't seem to get gdb to run python scripts, as I'd need to set a breakpoint in the c function that gets called from my python code. I believe i need to install python-debuginfo in order to do that, but I have not been able to.
3) Using pdb. I had, however, no idea what to do to debug c code using pdb.
Is there a good tool to use to debug memory leaks in my program?
The debugger that currently ships with the Mac development tools is now lldb, it replaces gdb. The interface is somewhat similar to gdb. You can install gdb using Homebrew.
The way to debug C extensions is just like any other shared library. Debug the executable (python) that loads the shared library (your extension), and set breakpoints that would become valid after the executable loads the shared library.
Run lldb:
lldb /path/to/python -- your_python_script.py
set some breakpoints (replace with your relevant breakpoints)
(lldb) b some_source_file.c:4343
then run, and the debugger will stop at the breakpoint
(lldb) r
now you can debug the C extension normally.
Is there any other way to debug swig extensions except for doing
gdb python stuff.py
?
I have wrapped the legacy library libkdtree++ and followed all the swig related memory managemant points (borrowed ref vs. own ref, etc.). But still, I am not sure whether my binding is not eating up memory. It would be helpful to be able to just debug step by step each publicized function: starting from Python then going to via the C glue binding into C space, and returning back.
Is there already such a possibility?
gdb 7.0 supports python scripting. It might help you in this particular case.
Well, for debugging, you use a debugger ;-).
When debugging, it may be a good idea to configure Python with '--with-pydebug' and recompile. It does additional checks then.
If you are looking for memory leaks, there is a simple way:
Run your code over and over in a loop, and look for Python's memory consumption.
I develop an extension module for Guile, written in C. This extension module embeds a Python interpreter.
Since this extension module invokes the Python interpreter, I need to verify that it properly manages the memory occupied by Python objects.
I found that the Python interpreter is well-behaved in its own memory handling, so that by running valgrind I can find memory leaks due to bugs in my own Python interpreter embedding code, if there are no other interfering factors.
However, when I run Guile under valgrind, valgrind reports memory leaks. Such memory leaks obscure any memory leaks due to my own code.
The question is what can I do to separate memory leaks due to bugs in my code from memory leaks reported by valgrind as due to Guile. Another tool instead of valgrind? Special valgrind options? Give up and rely upon manual code walkthrough?
You've got a couple options. One is to write a supressions file for valgrind that turns off reporting of stuff that you're not working on. Python has such a file, for example:
http://svn.python.org/projects/python/trunk/Misc/valgrind-python.supp
If valgrind doesn't like your setup, another possibility is using libmudflap; you compile your program with gcc -fmudflap -lmudflap, and the resulting code is instrumented for pointer debugging. Described in the gcc docs, and here: http://gcc.gnu.org/wiki/Mudflap_Pointer_Debugging