Does anybody know if I can safely compile multiple extensions concurrently with threading?
I realise this might not speed things up (although compilers are run in subprocesses, so maybe!), but I'm in a situation where GUI actions can start a simulation which may involve a compilation step, so I'd like to know if I need to prevent multiple simulations from compiling at the same time, or if this is fine
Not only they're not thread-safe — they're not process-safe: you cannot call setup() multiple times in one process. See an example of errors from this.
To work around the limitation you have to run python setup.py or pip in a subprocess and then the question of thread-safety no longer applies.
I'm wondering why we can have Tensorflow run in a multi-thread fashion while python can only execute one thread at a time due to GIL?
The GIL's restriction is slightly more subtle: only one thread at a time can be executing Python bytecode.
Extensions using Python's C API (like tensorflow) can release the GIL if they don't need it. I/O operations like using files or sockets also tend to release the GIL because they generally involve lots of waiting.
So threads executing extensions or waiting for I/O can run while another thread is executing Python bytecode.
Most of the tensorflow core is written in C++ and the python APIs are just the wrappers around it. While running the C++ code regular python restrictions do not apply.
I have a Python application written in Kivy that uses a C++ program for a high speed calculation, then it returns a value and my Python application uses that.
The C++ program is wrapped in PyBind11 and imported into the application and then called from Python.
My issue is when the C++ program is executed, my application stops for a short while and I'd still like things to be going on in the background.
I naively thought this could be solved by threading the C++ call, but on second thoughts I think the issue lies in the GIL. Must I unlock the GIL and how could I achieve this?
Without seeing any code, I can only deduce that your Python code is waiting for the C++ code to complete before doing anything else. Which can mean either or both of the following:
you are not unlocking the GIL in the C++ code
According to Global Interpreter Lock (GIL) — Miscellaneous — pybind11 2.2.3 documentation, with pybind, this is supposed to be done like this:
py::gil_scoped_release release;
long_running_method();
py::gil_scoped_acquire acquire;
Note that you need the GIL to access any Python machinery (including returning the result). So before releasing it, make sure to convert all the data you need from Python types to C++ types.
you don't have any other active Python threads, so there's no other Python activity programmed in to do anything while the C++ call is in progress
I have a kivy application in python which uses some threads.
As python is not able to run these threads on different Cores due to the Global Interpreter Lock, I would have liked to try to use PyPy for it and see if I can make the threads run faster of different cores since PyPy is different and offers stackless (what ever that is? :).
Does somebody have some information to share on how to make a simple python program, which launches some threads by the module threading, running with the pypy interpreter such that it uses this stackless feature?
Pypy won't resolve Python problems of running a single-thread each time, since it also makes use of the GIL - http://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-why
Besides that, Kivy is a complex project embedding Python itself - although I don't know it very well, I doubt it is possible to switch the Python used in it for Pypy.
Depending on what you are doing, you may want to use the multiprocessing module instead of threading - it is a drop-in replacement that will make transparent inter-process calls to Python functions, and can therefore take advantage of multiple-cores.
https://docs.python.org/3/library/multiprocessing.html
This is standard in cPython and can likely be used from within Kivy, if (and only if) all code in the subprocess just take care of number-crunching, and so on, and all user interaction and display updates are made on the main process.
A reliable coder friend told me that Python's current multi-threading implementation is seriously buggy - enough to avoid using altogether. What can said about this rumor?
Python threads are good for concurrent I/O programming. Threads are swapped out of the CPU as soon as they block waiting for input from file, network, etc. This allows other Python threads to use the CPU while others wait. This would allow you to write a multi-threaded web server or web crawler, for example.
However, Python threads are serialized by the GIL when they enter interpreter core. This means that if two threads are crunching numbers, only one can run at any given moment. It also means that you can't take advantage of multi-core or multi-processor architectures.
There are solutions like running multiple Python interpreters concurrently, using a C based threading library. This is not for the faint of heart and the benefits might not be worth the trouble. Let's hope for an all Python solution in a future release.
The standard implementation of Python (generally known as CPython as it is written in C) uses OS threads, but since there is the Global Interpreter Lock, only one thread at a time is allowed to run Python code. But within those limitations, the threading libraries are robust and widely used.
If you want to be able to use multiple CPU cores, there are a few options. One is to use multiple python interpreters concurrently, as mentioned by others. Another option is to use a different implementation of Python that does not use a GIL. The two main options are Jython and IronPython.
Jython is written in Java, and is now fairly mature, though some incompatibilities remain. For example, the web framework Django does not run perfectly yet, but is getting closer all the time. Jython is great for thread safety, comes out better in benchmarks and has a cheeky message for those wanting the GIL.
IronPython uses the .NET framework and is written in C#. Compatibility is reaching the stage where Django can run on IronPython (at least as a demo) and there are guides to using threads in IronPython.
The GIL (Global Interpreter Lock) might be a problem, but the API is quite OK. Try out the excellent processing module, which implements the Threading API for separate processes. I am using that right now (albeit on OS X, have yet to do some testing on Windows) and am really impressed. The Queue class is really saving my bacon in terms of managing complexity!
EDIT: it seemes the processing module is being included in the standard library as of version 2.6 (import multiprocessing). Joy!
As far as I know there are no real bugs, but the performance when threading in cPython is really bad (compared to most other threading implementations, but usually good enough if all most of the threads do is block) due to the GIL (Global Interpreter Lock), so really it is implementation specific rather than language specific. Jython, for example, does not suffer from this due to using the Java thread model.
See this post on why it is not really feasible to remove the GIL from the cPython implementation, and this for some practical elaboration and workarounds.
Do a quick google for "Python GIL" for more information.
If you want to code in python and get great threading support, you might want to check out IronPython or Jython. Since the python code in IronPython and Jython run on the .NET CLR and Java VM respectively, they enjoy the great threading support built into those libraries. In addition to that, IronPython doesn't have the GIL, an issue that prevents CPython threads from taking full advantage of multi-core architectures.
I've used it in several applications and have never had nor heard of threading being anything other than 100% reliable, as long as you know its limits. You can't spawn 1000 threads at the same time and expect your program to run properly on Windows, however you can easily write a worker pool and just feed it 1000 operations, and keep everything nice and under control.