I have a Python-program that uses ctypes and a C-shared library (dll-file). As an IDE, I am using Eclipse, where both projects will be developed (the C-shared library and the python program that uses it).
My idea is: when I start the Python-program in Debug-mode, can I somehow debug the shared library, which is written in C, too? Meaning: Can I set breakpoints and when the Python-program reaches that breakpoint in the shared library, executing stops and I can change variable values etc.?
Actually, it is a fairly simple thing to do using the CDT and PyDev environments in Eclipse.
I assume here that you have already configured the projects correctly, so you can build and debug each one seperately.
Basically, you simply need to start the Python project in Debug mode and then to attach the CDT debugger to the running python process. To make it easier I'll try to describe it step by step:
Run your Python project in debug mode. Put a breakpoint somewhere after the loading of the dll using ctypes. Make note of the pid of the python process created (you should see a first line in the console view stating the pid. something like: pydev debugger: starting (pid: 1234))
Create a Debug configuration for your CDT project, choosing the type "C/C++ Attach to Application". You can use the default configuration.
Debug your project using the configuration you've created. A window should appear, asking you which process you want to attach to. Choose the python process having the right pid.
You can now add breakpoints to you C code.
You'll have two debuggers in the debug perspective, as if they were two different processes. You should always make sure the C/C++ debugging session is running when you work with the python debugger - as long as the C/C++ debugging session is suspended, the python debugger will be unresponsive.
As far as I know, not in eclipse.
However, Python tools for visual studio has this capability:
https://pytools.codeplex.com/wikipage?title=Mixed-mode%20debugging
It is also possible to get this for free. From the Microsoft website, you'll need (as well as a copy of Windows)
1) Visual Studio (paid Pro+ version or free Express versions (starting w 2.1Beta))
2) PTVS extension (this gives VS Python support)
3) A Python interpreter and Python Libraries (these are not bundled with PTVS)
This means that you can debug python and c side-by-side. Calls to libraries written in c are captured by the debugger, provided that they were compiled with symbols by visual studio.
Note that the capabilities of mixed mode debugging tend to be less featured for native python, but it is still capable of using the regular native debugger.
Related
We are trying to debug C++ code called by a Python script (using subprocess.call). Is there a way to do this with PyDev? I am a Java developer and we are developing an Eclipse RCP application, and can write custom code as needed to make this work.
If you want to debug c++ code, you need a c++ debugger (and PyDev won't really help you there).
My suggestion is looking for a c++ debugger in this case (you can use gdb in windows and linux -- on windows you can also use visual c++ if you want -- you should be able to attach to a running process on both cases -- I think that CDT also has some gdb integration which you can use if you set it up properly).
I have followed the documentation provided for the Python comtypes module (http://pythonhosted.org/comtypes/server.html) and built a simple COM server.
It seems to be working correctly. I can see that the typelib and interface are registered (using Oleviewer) and from a Python script I can call the (single) method that adds two integers.
However, I cannot make it work with Excel (Office 2007, running on Win7). The following simple function does not work (execution stops when MyMethod is called)
Function test(a, b)
Dim x As New MyTypeLib.MyObject
ab = x.MyMethod(a, b)
Debug.Print "d" & CStr(ab)
test = ab
End Function
Is there any way to debug this?
More comments
There is indeed light at the end of the tunnel. The suggestion to use VBScript has been helpful. The following short script does work (providing independent confirmation that the COM server works).
Dim x,ab
Set x = CreateObject("MyTypeLib.MyObject")
ab = x.MyMethod(1, 2)
MsgBox CStr(ab)
However, VBScript is calling the server using late binding and I was trying to get the early binding to work.
So, I changed my Excel VBA function to use late binding and expected things to work, but they do not.
Here is the modified VBA:
Function test(a, b)
Dim x As Object
Set x = CreateObject("MyTypeLib.MyObject")
gadd = x.MyMethod(1, 5)
End Function
So my server can be used with late binding from Python and VBScript, but not Excel!
Using Process Monitor I can see that in both late and early binding cases Excel tries unsuccessfully to load msvcr90.dll, even though this DLL is installed on the system.
Here is a screenshot from Process Monitor, showing where Office Excel 2007 on 64-bit Windows 7 Home Edition starts to try to load msvcr90.dll
When I posted this question, I needed to know a bit more about how to debug COM servers (well a lot more actually!). It turns out that there were several problems that stopped my server from working. I think finding out about the tools is probably as much of interest as the actual problems, so this answer will try to cover both.
Initially, it was helpful to use VBScript to provide an independent verification that the server was registered and working (the simple code is shown above). VBScript uses late binding and I wanted early binding, but it was nevertheless helpful because I then realised that Excel did not work with the late binding VBA code either (on machine A)!
Something odd happened too with VBScript. When I first wrote scripts they ran by default (clicking on the file in Windows Explorer), but then for no apparent reason they stopped working. I found that I needed to specify the full path to the 32-bit version (C:\WINDOWS\SysWOW64\cscript.exe) for them to work.
Also worth noting is that by using cscript.exe, instead of wscript.exe, debugging messages embedded in my Python modules appear on the Windows CMD window.
The next breakthrough came by using the Dependency Walker program depends.exe. This is often mentioned in posts like mine where people are trying to figure out why DLLs are not working.
Dependency Walker is available from it's own website, but that version does not work properly with Windows 7 (see Profiling x86 executable with Dependency Walker hangs on Windows 7 x64) so I found that I needed to install WDK 8.1 (from here: http://msdn.microsoft.com/en-US/windows/hardware/gg454513).
DW is very powerful and a bit overwhelming at first. However, by reading its logging output it became clear that Excel 2007 uses msvcr80.dll and that the server's attempt to load msvcr90.dll was failing (even though the DLL was properly installed on the machine). This was not a problem on machine B because Excel 2013 uses msvcr90.dll.
Now, in these tests I was creating the server using Python comtypes interactively (there is no server DLL involved). But there is also the possibility of creating a DLL by using the py2exe tool. I was trying to do this too, the DLL failed to register using regsvr32.
DW helped debug this problem too. Happily, in the logging output I noticed a stack trace of a Python exception. It was the GetModule() in the example. It turns out that this call is needed only once, to create some Python classes that are then cached on the machine. Removing this command fixed the problem.
Also, although it may not have caused a problem, the official current version of py2exe is 0.6.9 but this caused some files that are no longer available to be bundled (specifically zlib.pyd). The unofficial version 0.6.10 (available here: http://www.lfd.uci.edu/~gohlke/pythonlibs/#py2exe) solved that issue.
So I now have the possibility of compiling a COM server DLL, in which I can include the required msvcr90.dll, and, yes, it works on both machines :-)
In summary then. Windows 7, with its mixture of 32-bit and 64-bit libraries is a nightmare. VBScript helps, by providing an independent environment for testing a server. Dependency Walker really is amazing, but takes ages to learn to use (other useful tools in the WDK include oleviewer, process-monitor and process-explorer).
I am writing a python program that includes an extension module written in C. The extension module defines a function that is used in my program continually.
I have a memory leak somewhere in my program, but I don't know how to find it. I have tried
1) Installing Valgrind. I can't get this to work however since I have OSX Mavericks (10.9) and Valgrind only supports OXS 10.7 (and 10.8 somewhat unstably).
2) Using gdb. I can't seem to get gdb to run python scripts, as I'd need to set a breakpoint in the c function that gets called from my python code. I believe i need to install python-debuginfo in order to do that, but I have not been able to.
3) Using pdb. I had, however, no idea what to do to debug c code using pdb.
Is there a good tool to use to debug memory leaks in my program?
The debugger that currently ships with the Mac development tools is now lldb, it replaces gdb. The interface is somewhat similar to gdb. You can install gdb using Homebrew.
The way to debug C extensions is just like any other shared library. Debug the executable (python) that loads the shared library (your extension), and set breakpoints that would become valid after the executable loads the shared library.
Run lldb:
lldb /path/to/python -- your_python_script.py
set some breakpoints (replace with your relevant breakpoints)
(lldb) b some_source_file.c:4343
then run, and the debugger will stop at the breakpoint
(lldb) r
now you can debug the C extension normally.
complete Python noob here (and rusty in C).
I am using a Mac with Lion OS. Trying to use NFCpy, which uses USBpy, which uses libUSB. libUSB is crashing due to a null pointer but I have no idea how to debug that since there are so many parts involved.
Right now I am using xcode to view the code highlighted but I run everything from bash. I can switch to Windows or Linux if this is going to be somehow easier with a different environment.
Any suggestions on how to debug this would be much appreciated ;-)
PS: It would be just fine if I could see the prints I put in C in the bash where I run the Python script
You should see your printf() you put in C in your terminal, something is already wrong here. Are you sure that you're using the latest compiled library? To be sure, instead of print, you can use use assert(0) (you need to include assert.h).
Anyway, you can debug your software using gdb:
gdb --args python yourfile.py
# type "run" to start the program
# if you put assert() in your code, gdb will stop at the assert, or you can put
# manual breakpoint by using "b filename:lineno" before "run"
Enable core dumps (ulimit -Sc unlimited) and crash the program to produce a core file. Examine the core file with gdb to learn more about the conditions leading up to the crash. Inspect the functions and local variables on the call stack for clues.
Or run the program under gdb to begin with and inspect the live process after it crashes and gdb intercepts the signal (SIGSEGV, SIGBUS, whatever).
Both of these approaches will be easier if you make sure all relevant native code (Python, libUSB, etc) have debugging symbols available.
Isolating the problem in a program which is as small as you can manage to make it, as Tio suggested, will also make this process easier.
PS: It would be just fine if I could see the prints I put in C in the bash where I run the Python script
You didn't mention anything about adding prints "in C" elsewhere in your question. Did you modify libUSB to add debugging prints? If so, did you rebuild it? What steps did you take to ensure that your new build would be used instead of the previously available libUSB? You may need to adjust your dylib-related environment variables to get the dynamic linker to prefer your version over the system version. If you did something else, explain what. :)
I have a C++ application (on Windows XP) that invokes some Python code. I currently use Winpdb as my python debugger.
Using winpdb as the debugger has some disadvantages since firstly, I need to add a pdb statement, and there is little control afforded to me during the execution since I add dynamic breakpoints.
Does anyone know of tools that can be used that work in a similar fashion to Visual Studio 2010 (or Visual Studio in general)? Most important for me is to be able to set up and remove breakpoints while the application is executing, much like we can do using Visual Studio.
I have looked at ActiveState Komodo IDE6 and Eclipse Python PyDev, but I do not think either one actually fits the bill. This is because I have various python modules that are initialized
using the C++ function call "Py_InitModule" with a name that might be different from the name of the .py file. There modules are not recognized by Komodo and Eclipse Python PyDev.
Please let me know if you have any suggestions. Thank you for your help.
Well, for me it seems that it'd work in PyDev... The only thing is that you have to attach the tracing to the debugger -- use pydevd.settrace(suspend=False) on the thread you want to trace and after that it should 'synchronize' the current breakpoints and their additions/removals ( more references: http://pydev.org/manual_adv_remote_debugger.html ) -- note that you could use pydevd.settrace(suspend=True) to work as a breakpoint as in in pdb.
I didn't really understand why you said it wouldn't work. Can you post your specific example of what's not working to see how that would be solvable? (some things may be customized in pydevd_file_utils.py to help in translating breakpoint paths).