I have a Python application written in Kivy that uses a C++ program for a high speed calculation, then it returns a value and my Python application uses that.
The C++ program is wrapped in PyBind11 and imported into the application and then called from Python.
My issue is when the C++ program is executed, my application stops for a short while and I'd still like things to be going on in the background.
I naively thought this could be solved by threading the C++ call, but on second thoughts I think the issue lies in the GIL. Must I unlock the GIL and how could I achieve this?
Without seeing any code, I can only deduce that your Python code is waiting for the C++ code to complete before doing anything else. Which can mean either or both of the following:
you are not unlocking the GIL in the C++ code
According to Global Interpreter Lock (GIL) — Miscellaneous — pybind11 2.2.3 documentation, with pybind, this is supposed to be done like this:
py::gil_scoped_release release;
long_running_method();
py::gil_scoped_acquire acquire;
Note that you need the GIL to access any Python machinery (including returning the result). So before releasing it, make sure to convert all the data you need from Python types to C++ types.
you don't have any other active Python threads, so there's no other Python activity programmed in to do anything while the C++ call is in progress
Related
Using Py_Initialize(), we can start the python interpreter in a C++ program.
However as the function does not return anything, we cannot use the same interpreter in a different program.
Is there any way of calling Py_Initialize() in one C++ program, make the interpreter persistent and use it in a different C++ program(without calling Py_Initialize() again)?
Edit: To be more specific, Is there a way can get hold of an instance of a python interpreter and pass it to another execution as a parameter and use it to run python scripts.
No. The CPython interpreter itself does not work like that. There is no distinct interpreter object, but rather a floating set of globals with a stateful API. What's worse, Python code can load arbitrary other libraries, whose state can definitely not be persisted (in general).
What you can do is to pickle the existing variables. That can sometimes bring you somewhere close. That is not really a hosting problem, but a Python problem. Naturally, though, you could make sure that your C code hosting Python made sure to execute the serializations steps after the "real" Python code has finished executing. Something like How can I save all the variables in the current python session? might be a starting point.
I have a kivy application in python which uses some threads.
As python is not able to run these threads on different Cores due to the Global Interpreter Lock, I would have liked to try to use PyPy for it and see if I can make the threads run faster of different cores since PyPy is different and offers stackless (what ever that is? :).
Does somebody have some information to share on how to make a simple python program, which launches some threads by the module threading, running with the pypy interpreter such that it uses this stackless feature?
Pypy won't resolve Python problems of running a single-thread each time, since it also makes use of the GIL - http://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-why
Besides that, Kivy is a complex project embedding Python itself - although I don't know it very well, I doubt it is possible to switch the Python used in it for Pypy.
Depending on what you are doing, you may want to use the multiprocessing module instead of threading - it is a drop-in replacement that will make transparent inter-process calls to Python functions, and can therefore take advantage of multiple-cores.
https://docs.python.org/3/library/multiprocessing.html
This is standard in cPython and can likely be used from within Kivy, if (and only if) all code in the subprocess just take care of number-crunching, and so on, and all user interaction and display updates are made on the main process.
I'll keep my question short and simple.
Assume I have a python program which calls C++ code from a DLL compiled in C/C++.
-Will the speed/performance of the executing code be preserved?
Assume I have a python program ... has a binding to a C++ library (for example - GTK or Wx).
-Is the speed going to match that of the library as if it was compiled with a C++ program?
Thank you.
When Python calls into C++ code, the code it executes is the machine code generated by the C++ compiler. You will have some cost at the interface level, as you have to marshal Python types into C++ types and vice versa, but the C++ code itself will run at pretty much the same speed as if it were called from C++; any differences will be due to different locality of dynamically allocated memory due to different memory use patterns (which would cause your C++ code to run at different speeds depending on which C++ application called it as well).
I got curious and have been reading about GUI development using Python for the past hour. After reading documentation of wxPython, PyQt, Nokia's Python bindings for Qt along with Tkinter a question came to my mind.
When I create a console application with Python, it runs using the embedded Python interpreter (which I assume is usually if not always in my case cpython).
So I was wondering, what's the case with these "widget toolkits"?
How is the Python code executed and what interprets it (or executed it)?
Which part of my Python code is interpreted using the Python
interpreter?
Or does the Python code get lexically analysed and then parsed by the widget's
toolkit which then interpretes and executes (or compile during build)?
I am looking forward to understanding what goes on in the background in comparison with Python applications' (a bit simpler to understand) interpretation with the Python interpreter.
Thank you.
PS. To whichever genius thinks that this question deserves to be closed;
A lot of people wonder the internals of external libraries and systems. Especially those which are not as simple as they look. There currently is not any question explaining this on SE.
This is just a really generalized high-level explanation about "GUI toolkits"...
Lets say you decide to use the Qt framework. This framework is written in C++. There are two different python bindings that can be used, allowing one to write a GUI application in python against the same API as the C++ version.
The python bindings provide a wrapping around calls into the C++ code. PyQt4 for instance uses sip, while PySide uses shiboken. These are just language wrapping tools that take specifications for how to map between the C++ objects and their intended python interface.
Ok, so you start using PyQt... All of the code you write has to pass through the python interpreter. Some of it may be pure python. Some of it will call into C++ libs to create things like your widgets. In Qt, there will be a C++ pointer associated with the python instance counterpart.
It is the C++ layer that is then communicating with the window manager of your platform, to turn platform-independent API calls into something platform specific, like how to exactly draw a button or menu.
Whether you create a console only or GUI based python application, it all goes through the python interpreter to interpret your python code. Something must interpret the python language for you.
I'm trying to test a multi-threaded C++ DLL. This DLL is supposed to be thread-safe. I have it wrapped with boost.python, and I'd like to create multiple python threads to exercise the DLL through the boost.python wrapper. I'm actually trying to cause threading problems.
What I can't seem to find good documentation on is whether the python interpreter will support two of its threads (on different cores, say) calling into an imported module concurrently, and whether the GIL needs tending at all given that I don't want any added safety above what the DLL is supposed to provide.
Can anyone describe or refer me to a description of python calling DLL modules from multiple threads and how the GIL is suppsed to be used in this case?
How to release GIL when calling a C++ function from Python via Boost.Pyhton:
http://wiki.python.org/moin/boost.python/HowTo#Multithreading_Support_for_my_function
The answer is no, the GIL will never truly multi-thread unless the DLL manually releases the lock. Python allows exactly one thread to run at a time unless the extension manually says, "I'm blocked, carry on without me." This is commonly done with the Py_BEGIN_ALLOW_THREADS macro (and undone with Py_END_ALLOW_THREADS) defined in python's include/ceval.h. Once an extension does this, python will allow another thread to run, and the first thread doing any python stuff will likely cause problems (as the comment question notes.) It's really meant for blocking on I/O or going into heavy compute time.