How to see C++ function invocations behind the SWIG interface, TensorFlow - python

I'm working on TensorFlow and I want to know the relationship between each Python function and the correspondent C++ functions behind the SWIG interface.
In other words, I want to be able to know exactly which C++ functions are invoked for every line of Python code of my TensorFlow application.
I already saw how to debug the Python code here and how to display at which line of code a segmentation fault happens here, but in these ways I'm able to see only where an error is, while instead I want to be able to know every C++ function invocation, even when there are no bugs in the code (At the moment debugging with gdb I'm able to see the system calls and the dynamic library calls, but not the C++ function invocations).

The bulk of the code that most people write is for graph construction. Almost of all of graph construction happens completely in Python, which simply builds up the (Python) data structures such as Operation and Graph defined in ops.py. The one exception is shape inference, which happens when you create every operation. Shape inference calls out to C++ via the interface defined in cpp_shape_inference.i. After you built up the computational graph, you execute it by creating a Session and calling sess.run. These are all Python function wrapping TensorFlow's C API. The wrappers can be found in tf_session.i.

Related

Return a python function from Python C-API

I am currently in the process of writing a small python module using the Python API, that will speed up some of the slower python code, that is repeatedly run in a in a simulation of sorts. My issue is that currently this code is takes a bunch of arguments, that in many use cases won't change. For example the function signature will be like: func(x,a,b,c,d,e), but after an initialisation only x will change. I therefore will have the python code littered with lambda x : func(x,a,b,c,d,e) where I wrap these before use. I have observed that this actually introduces quite a bit of calling overhead.
My idea to fix this was to create a PyObject* that is essentially C++ lambda instead of the python one. The main issue with this is that I have not found a way to create PyObjects from C++ lambdas, or even lower level functions. Since functions/lambdas in python can be passed as arguments I assume it is possible, but is there a clean way I'm missing.
I would seriously consider using swig instead of pybind11 for example. It's just peace of mind. If you don't want to use swig directly, you can at least see what swig does to wrap up features like proxy objects.
http://www.swig.org/Doc2.0/SWIGPlus.html#SWIGPlus_nn38

Calling Python member functions from C++

I need to test the feasibility of calling Python member functions (running in one process)from within C++. This is for testing interfacing C++ to an existing Python application. I need to minimize the modifications to the Python code as that's run by a separate team. Therefore I do not have control of when the Python objects are created on the C++ side. For my test I'd like to try and:
See if I can determine how many instances of a specified Python class have been created
If that number is > 0, then I would like to test calling a member function on one of the instantiated Python objects from C++
I can do a simple call from C++ to a global, non member Python function, but can't figure out how to do the above 2 steps from the C++ side.
I'd also like to try and do this without pulling in the Boost Python interop. library (but will if that's the only way this can be achieved).
Thanks if anyone can advise.

Python support for Qt dll

I have my own C++ library project(with source) written in Qt and it uses QTcpsocket, QUdpSocket, QSerialPort signals and slots.
I would like to support this library in Python as well.
What is the preferred way to do this?
Writing a wrapper in Python, if so does it have obstacles?
Dont know if PyQt is just for this purpose?
Or do you thnink is it better to rewrite the lib in Python by just implementing the logic used in C++ library project.
As this is library is part of a SDK, same applies for supporting QT dll with .NET as well in fact, as a second step after supporting Python.
Example API of Qt.
quint16 SendCommandAsync(CmdBaseSpv1* pcommand,
ConnectionArg connectionArg,
emitLogOptions::emitLogOption logOption,
LogInfo &txLogInfo,
LogInfo &rxLogInfo);
I want to call this function from Python.
Function parameters CmdBaseSpv1, ConnectionArg, emitLogOption, LogInfo are all Qt classes.
Some of these arguments are using the QOBJECT base class.
As you see from the function name; it is an Asynchronous function call. Result will emit a signal so I need to get async result as well.
I'll write down what I know about wrapping C++ libraries and will try to source it, but as a huge disclaimer, I have only used this for something very, very simple myself.
You ask about rewriting the library in Python. I would say this depends. If the code is trivial, then I don't see why not. If it is larger and has to be kept up-to-date with other code (as you imply with .Net), I wouldn't. It makes sense to reuse the same code for both.
 My suggestion
From what I see of your code I would try to wrap it using boost::python or SWIG.
How to wrap
The main trouble is going to be to create CmdBaseSpv1, ConnectionArg, etc. in Python.
If you don't need any Qt-classes to instantiate your classes, this should be straightforward. However, in case you need the Qt types inside of Python (e.g. because the constructor of CmdBaseSpv1 requires a QString), your task is a lot more complicated because you need a way to convert a Python-string into a QString. If you can, you should only use stl-types.
Everything in Python
The simplest way to wrap a small C library is to use the cffi module (or ctypes). You can write the full binding in Python. However, this is a lot of manual work if your API is large and can get difficult.
There is another problem: ctypes is only compatible with C, not C++. So you'd need to change your interface to be compatible with C, internally you could still use C++ and Qt.
Wrap by hand
An alternative is to wrap the library calls yourself. You can either do this by using the Python API. There are also a few libraries that help you create the bindings. Boost::python seems especially promising and works with C++.
Binding generators
If your API is very large, you should use a binding generator which parses the C++ code and generates the bindings itself. For example sip is one of them. It is used to create the bindings for the whole Qt library. There are a few binding generators out there, one mentioned in the Python docs is SWIG. PySide uses Shiboken and also has a nice description of it on their website.
SWIG has the additional advantage, that you can create bindings for multiple languages, including C#.
PyQt
PyQt is a binding generated from Qt using sip. You'll probably not need it, unless you need to access the full power of Qt from inside Python. If this is the case, consider using sipfor generating the bindings, so things like the signal-slot mechanism are compatible between your library and PyQt.
Challenges with bindings
Bindings come with a few challenges because Python and C++ are different in some key areas.
Memory-management
Memory management in Python is almost automatic, in C++ you're required to do it manually. For example
def myfunc():
mywidget = QWidget()
at the end of myfunc() mywidget gets garbage collected. In C++ however
void myfunc() {
auto mywidget = new QWidget();
}
mywidget is still around. This means that even when inside Python, you need to take care of the C++ memory management. The problems I've seen are memory leaks and dangling pointers. Watch out for this when using callbacks, you don't want Python to garbage collect the callback while C++ thinks it's still alive.
Exceptions
Not all programming languages have exceptions or deal with them the same way. For example, it would be nice if an exception inside C++ can be caught inside Python.
Links to related question
How to wrap a c++ library for python? (example of boost::python)
Exposing a C++ API to Python (discussion about boost::python, SWIG and more)
https://stackoverflow.com/a/5686873 (discusses Cython, another choice)

Run c++ class from python

Is there a way to run a c++ class from python without using any external libraries like Boost.Python, SWING ect? I don't want to pass any arguments to this class or call a specific method and in my c++ class I have only a void main method, I just want to run the main and that is all.
Or if this is not possible a saw this tutorial http://intermediate-and-advanced-software-carpentry.readthedocs.io/en/latest/c++-wrapping.html#manual-wrapping. But I didn't understand if I should put the hello_wrapper function in the same c++ class where I have the original hello function. And also how can I create a modulo in Python(second part in the tutorial) and where should I put this code
DL_EXPORT(void) inithello(void)
{
Py_InitModule("hello", HelloMethods);
}
Thanks
is there a way to run a c++ class
you don't run C++ classes. They are data types!
Boost.Python, SWING
It's called SWIG, not SWING :)
You can add your own C wrapper code that initializes a PyObject. I'd recommend reading the CPython docs and the examples in the tutorial on extending python. Since you didn't specify a version, I can't give you a discrete link.
Note that python is C, and C++ isn't; which means that you'll have to export several things with a C ABI, i.e. by using external "C" in your code. That might not be something for the uninitiated, and you should certainly evaluate whether not using external wrapper generators is really worth the trouble – especially since using e.g. SWIG properly (which is really a pain) you can get Python objects that really behave like python objects, e.g. you can extend them with python etc.

Getting a C pointer to a function generated by Theano?

I would like to use a Theano function from C/Fortran code (in particular, I want to use an implicit ODE solver written in Fortran with a function created in Theano). Are there any examples/resources on how to do that?
You've tagged your question with ffi/cffi but that's for calling foreign code from Python. However it sounds like you actually want to call Python/Theano code from C/Fortran. For that, the documentation on Embedding Python in Another Application might be helpful.
In principle you could just run Theano Python code from your C/Fortran code via facilities in Python.h.
Although Theano compiles some operations via C code, I don't believe it produces an natively executable function/library for the entire computation graph that could then be linked in by some other, non-Python, application.
Update: via the thread on the Theano mailing list... apparently a prototype for having Theano create a linkable library was done some time ago but isn't currently integrated into Theano.

Categories

Resources