I have PyOpenCL code with OpenCL C kernel code. I catch segmentation fault error when I run my app. How to debug such error with some debugger or some other development tool? I don't know what exactly to do to find out the problem. I have in mind option with printf or something but I want use more powerful stuff.
I believe that error in kernel code, so I want to debug kernel code firstly.
UPD. I'm on linux (Arch Linux, 3.6.11), python 2 or 3, PyOpenCl 2012.1
Kernel debugging is an implementation-dependent affair. On Linux, the best I've found is to use AMD's CL implementation on the CPU, compile the kernel with -g, and use gdb. They've got instructions on this in their programming guide, here:
AMD CL Documentation page
If you're using nvidia instead of ATI/AMD GPU, the OpenCL support in nvidia SDK is...less than desired.
Intel provides a CPU-based OpenCL SDK for their recent processors, see http://software.intel.com/en-us/vcsource/tools/opencl-sdk-2013 -- (to use the RPM packages they provide on Ubuntu, you need to run "fakeroot alien --to-deb" on each package, then "dpkg -i").
With that SDK, you need to add the "-g" and "-s filename" flags to the compiler options in build(). (If your kernel only exists as a string in your program, you can add code to save it to a file just before you run it.) Then try "gdb --args python-cmd", you can start debugging by typing "break mykernel", answer Y when asked if you want to wait for the "mykernel" symbol to be dynamically loaded, then type "run".
Once you have the debugger running manually typing the command, I suggest making an executable shell script to launch your favorite .py file with the debugger (which will also be a convenient place to add hackery to your application's launch, e.g. python -m unittest, PYTHONPATH, virtualenv, LD_LIBRARY_PATH, LD_PRELOAD, etc.).
I wouldn't jump into conclusions without fully testing your software suite. You are running the last released version of PyOpenCl. Chances are you are passing in something to the module that isn't being populated correctly and the backend module doesn't do the necessary error-checking before using something that isn't properly populated (impossible to really help you debug without any code being made available)
Have you tried using the python debugger to set different breakpoints (import pdb; pdb.set_trace()) right before different pyopencl calls to even see where in your code it is seg faulting? This should definitely be your first task. When you find out where it is seg faulting, you need to closely look at pyopencl examples/api to see why you've errored out.
Segmentation fault is usually because of a bad memory access in your kernel. There is a cool tool to detect bad memory accesses, similar to valgrind: https://github.com/jrprice/Oclgrind . In combination with some printfs in the kernel code it makes it quite a bit easier to localize the problem.
Related
I have a C++ application that uses machine learning from Python and my current approach is making a single file executable with pyInstaller and then just running it from C++. This have obvious drawbacks, notably interapplication communication. At the moment I'm using an intermediate JSON file to talk to each other but this is massively suboptimal for my future requirements. What's beautiful about this, is that is working on all major platforms without too much hassle.
Section 1.6. from Python's manual reads "Compiling and Linking under Unix-like systems"
Does this mean that Python interpreter will be inside my application binary and target system doesn't need to have Python installed as the program will always use embedded interpreter ? If so, whats with python libraries ? Can I embed a whole conda enviroment ?
Also, whats with:
"(...) under Unix-like systems"
Does this means this approach is not multiplatform ?
Thanks in advance.
Embedding the Python interpreter is possible on all platforms. However it will only be the interpreter. Embedding any libraries will be a lot harder or even impossible.
But since you seem to deploy the Python libs already, you can use them just fine from the embedded interpreter. And then you could bridge C++ and Python without IPC, since they are both running in the same process.
pybind11 is very nice for embedding and generating C++ <-> Python interfaces.
A possible alternative, depending on the libraries in use, may be to export the model and use a C++ library to load and use it (for instance Tensorflow -> ONNX -> ONNX runtime).
It means that cpython (Python interpreter) will be inside your application. You will be able to run Python code and observe and manipulate virtual machine state directly from C++ code (good entry point C API reference is here). Your application might have some additional dynamic library dependencies (which ones depends on compilation options of embedded Python). Also interpreter isn't completely self contained and depends on some external .py modules normally shipped with Python distribution (as standard library). If you plan to import external modules that expect standard library, you will have to ship it with your application. There are ways how to build modules into binary too (freeze) but you might run into issues specially with modules that rely on filesystem.
As far as I tried, this procedure works on UNIX like systems and Windows (where easiest way is to link against DLL which you then ship with your application). On Windows you also need to make sure that you compile with same compiler as was used to compile DLL (or you compile Python DLL from source). Here is additional information about embedding on Windows: https://docs.python.org/3/faq/windows.html#how-can-i-embed-python-into-a-windows-application
Just note that embedding Python and shipping 3rd party modules with your application might have some licensing consequences.
I'm in a systems programming class, and the python-based test driver for one of our assignments needs to run Valgrind on a C-compiled executable.
I'm trying to put Valgrind on my laptop. I'm using Ubuntu 18.04.4 LTS. I was reading though the Ubuntu wiki and the page on Valgrind said to "have packages with debug symbols installed." I checked the page they reference about this, and I still have no idea which packages I need to install with debug symbols.
I was reading though some stack overflow answers, and this one mentions that "Valgrind is readily usable for C/C++ code, but can even be used for other languages when configured." The Valgrind quickstart guide also give no warnings about debug symbols. It just says to use the flag -g when compiling.
Does the Ubuntu wiki's warning about debug symbols just apply to other programs that are not compiled from C?
Wiki says about any libraries your code links with. In order in analyze stracktraces from valgrind it is good to link with these libraries debug versions. What these libraries are depends only on your application.
Imagine you develop some application. Then you want to trace some memory leak or segmentation fault. So you run your compiled program under valgrind. You compile your program with -g flag to have nice messages from valgrind when it prints out stack traces of allocations. Thanks to this flag you see more information in stack traces like exact function names of functions you coded.
What wiki says. It says about any libraries which your code links with. E.g. you link with boost Stacktraces may include also function calls coming from boost. In order to have in stacktraces also more debug information about function names from boost you need to link with boost compiled with debug symbols.
What you found on wiki is an advice, that if you link with e.g. some library from xserver-xorg-core, then it may be good for you to install also xserver-xorg-core-dbg, link with it and run such a linked app under valgrind. Your stacktraces will then look nicer even in parts not coded by you but only used by you from libraries provided by xserver-xorg-core(-dbg). That's all.
The page they reference says:
If you want to debug a crash from an application provided by Ubuntu,
you are developing yourself, provided by a third-party, or need the
debug symbols for particular libraries very often, it is helpful to
install the relevant debugging packages.
For many, but not all, packages, one can simply add a -dbg suffix to
the package name to install it. For example:
sudo apt-get install xserver-xorg-core-dbg
This means the same what I wrote. I just tried to elaborate more.
I'm working on an (Windows and Mac) application that uses Python as an embedded scripting language.
The application includes an internal text editor, implemented using Scintilla, and I'm using Jedi for autocompletion, which generally works great.
However, when attempting autocompletion on a computer that does not have a separate installation of Python, Jedi raises an error:
jedi.api.environment.InvalidPythonEnvironment:
Could not get version information for 'python':
FileNotFoundError(2, 'The system cannot find the file specified', None, 2, None)
Digging into the code, I can see that the underlying code that is throwing the FileNotFoundError is when Jedi attempts to run python using subprocess.Popen. Python is not installed on the computer, so this fails.
I can also reproduce the same issue on a computer that does have Python installed by editing my Path environment variable not to include the location of python.exe.
Ideally, we don't want users of our application to have to install Python just to get autocompletion working.
My questions:
Is it possible to get Jedi not to spawn subprocesses, and instead run its code inside the same instance of Python within which it itself is running? I couldn't find anything about this in the documentation or the source code that deals with Environments, and extrapolating from the discussion here I suspect the answer might be no.
Is it possible somehow to get Jedi to use the same python37.dll that our application is using for its functionality, instead of looking for a .exe file that does not exist?
Is there any way we could make some kind of minimal Python installation within our existing app installation that uses the same DLLs/Python Lib etc? How could I go about doing this?
Is there any other way to get Jedi autocompletion working in our app without requiring the user to install Python, or including a full Python installer as part of our build process?
Is it possible to get Jedi not to spawn subprocesses, and instead run its code inside the same instance of Python within which it itself is running? I couldn't find anything about this in the documentation or the source code that deals with Environments, and extrapolating from the discussion here I suspect the answer might be no.
This is definitely possible. All the tools are there. There are discussions ongoing here: https://github.com/davidhalter/jedi-vim/issues/870.
IMO a patch to Jedi is needed that uses an jedi.api.environment.InterpreterEnvironment in some cases like yours. It's definitely possible, it's just buggy at the moment.
I am writing a python program that includes an extension module written in C. The extension module defines a function that is used in my program continually.
I have a memory leak somewhere in my program, but I don't know how to find it. I have tried
1) Installing Valgrind. I can't get this to work however since I have OSX Mavericks (10.9) and Valgrind only supports OXS 10.7 (and 10.8 somewhat unstably).
2) Using gdb. I can't seem to get gdb to run python scripts, as I'd need to set a breakpoint in the c function that gets called from my python code. I believe i need to install python-debuginfo in order to do that, but I have not been able to.
3) Using pdb. I had, however, no idea what to do to debug c code using pdb.
Is there a good tool to use to debug memory leaks in my program?
The debugger that currently ships with the Mac development tools is now lldb, it replaces gdb. The interface is somewhat similar to gdb. You can install gdb using Homebrew.
The way to debug C extensions is just like any other shared library. Debug the executable (python) that loads the shared library (your extension), and set breakpoints that would become valid after the executable loads the shared library.
Run lldb:
lldb /path/to/python -- your_python_script.py
set some breakpoints (replace with your relevant breakpoints)
(lldb) b some_source_file.c:4343
then run, and the debugger will stop at the breakpoint
(lldb) r
now you can debug the C extension normally.
complete Python noob here (and rusty in C).
I am using a Mac with Lion OS. Trying to use NFCpy, which uses USBpy, which uses libUSB. libUSB is crashing due to a null pointer but I have no idea how to debug that since there are so many parts involved.
Right now I am using xcode to view the code highlighted but I run everything from bash. I can switch to Windows or Linux if this is going to be somehow easier with a different environment.
Any suggestions on how to debug this would be much appreciated ;-)
PS: It would be just fine if I could see the prints I put in C in the bash where I run the Python script
You should see your printf() you put in C in your terminal, something is already wrong here. Are you sure that you're using the latest compiled library? To be sure, instead of print, you can use use assert(0) (you need to include assert.h).
Anyway, you can debug your software using gdb:
gdb --args python yourfile.py
# type "run" to start the program
# if you put assert() in your code, gdb will stop at the assert, or you can put
# manual breakpoint by using "b filename:lineno" before "run"
Enable core dumps (ulimit -Sc unlimited) and crash the program to produce a core file. Examine the core file with gdb to learn more about the conditions leading up to the crash. Inspect the functions and local variables on the call stack for clues.
Or run the program under gdb to begin with and inspect the live process after it crashes and gdb intercepts the signal (SIGSEGV, SIGBUS, whatever).
Both of these approaches will be easier if you make sure all relevant native code (Python, libUSB, etc) have debugging symbols available.
Isolating the problem in a program which is as small as you can manage to make it, as Tio suggested, will also make this process easier.
PS: It would be just fine if I could see the prints I put in C in the bash where I run the Python script
You didn't mention anything about adding prints "in C" elsewhere in your question. Did you modify libUSB to add debugging prints? If so, did you rebuild it? What steps did you take to ensure that your new build would be used instead of the previously available libUSB? You may need to adjust your dylib-related environment variables to get the dynamic linker to prefer your version over the system version. If you did something else, explain what. :)