As the GIL is a lock that surrounds the interpreter does it affect compiled Python? I'm wondering whether it is possible to get past the inherent multi-threading limitations of cpython by simply compiling my python before executing it.
Hopefully that makes sense and I'm not missing something obvious or misinterpreting how the GIL works/affects execution.
Thanks
As Daniel said in the comments, it depends on how you "compile" the code.
For example, running the code using Jython does indeed get around the limitations imposed by the GIL.
On the other hand, using something like py2exe makes no difference, since this effectively just packages CPython alongside your code.
Jython does not have a GIL.
IronPython does not have a GIL.
You can compile your python code with cython, and then whether it uses the GIL or not depends. If you convert all you python variables into cython types, you can run your code in a with nogil block and you will have no GIL because you are expressly releasing the GIL. If you are not running in a nogil block, you will be affected by cpython's GIL. More in the cython docs: http://docs.cython.org/src/userguide/external_C_code.html#acquiring-and-releasing-the-gil
For more on python and the GIL, read up here: http://www.jeffknupp.com/blog/2013/06/30/pythons-hardest-problem-revisited/
Related
I have a pure-Python module that compiles with Cython into .so, for performance.
The project currently uses Python 3.8 and everything works fine.
I've read CPython 3.11 comes with potentially great performance benefits, so I'm wondering the cost/benefit effort of upgrading.
Do any of the CPython 3.11 optimizations translate to Cython-compiled code? Which ones? How much?
What is the expected overall effect of Cython on 3.11 relative to 3.8 code?
Basically none of it.
Start-up applies to starting the interpreter (before you load any Cython modules)
"Cheaper, lazy Python frames" - Cython doesn't create frames (apart from when an exception is raised, and even then I doubt it applies)
"Adaptive interpreter" is for interpreted code. It specializes bytecode based on past results - so if an attribute lookup bytecode instruction in a certain function turns out to be looking up in a class, it assumes the same will apply next time. Since Cython doesn't use bytecode, these optimizations just don't apply. (However, if you've applied static types correctly then Cython may be doing similar optimizations already)
As ever, if you're really interested then you should measure it. But I wouldn't expect many benefits to Cython code
I'm trying to test a multi-threaded C++ DLL. This DLL is supposed to be thread-safe. I have it wrapped with boost.python, and I'd like to create multiple python threads to exercise the DLL through the boost.python wrapper. I'm actually trying to cause threading problems.
What I can't seem to find good documentation on is whether the python interpreter will support two of its threads (on different cores, say) calling into an imported module concurrently, and whether the GIL needs tending at all given that I don't want any added safety above what the DLL is supposed to provide.
Can anyone describe or refer me to a description of python calling DLL modules from multiple threads and how the GIL is suppsed to be used in this case?
How to release GIL when calling a C++ function from Python via Boost.Pyhton:
http://wiki.python.org/moin/boost.python/HowTo#Multithreading_Support_for_my_function
The answer is no, the GIL will never truly multi-thread unless the DLL manually releases the lock. Python allows exactly one thread to run at a time unless the extension manually says, "I'm blocked, carry on without me." This is commonly done with the Py_BEGIN_ALLOW_THREADS macro (and undone with Py_END_ALLOW_THREADS) defined in python's include/ceval.h. Once an extension does this, python will allow another thread to run, and the first thread doing any python stuff will likely cause problems (as the comment question notes.) It's really meant for blocking on I/O or going into heavy compute time.
Why is it that you can run Jython and IronPython without the need for a GIL but Python (CPython) requires a GIL?
Parts of the Interpreter aren't threadsafe, though mostly because making them all threadsafe by massive lock usage would slow single-threaded extremely (source). This seems to be related to the CPython garbage collector using reference counting (the JVM and CLR don't, and therefore don't need to lock/release a reference count every time). But even if someone thought of an acceptable solution and implemented it, third party libraries would still have the same problems.
Note that extensions written in C can in fact get rid of the GIL: http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock
My guess, because the C libraries that CPython is built upon aren't thread-safe. Whereas Jython and IronPython are built against the Java and .Net respectively.
I often see people talking that the GIL is per Python Interpreter (even here on stackoverflow).
But what I see in the source code it seems to be that the GIL is a global variable and therefore there is one GIL for all Interpreters in each python process. I know they did this because there is no interpreter object passed around like lua or TCL does, it was just not designed well in the beginning. And thread local storage seems to be not portable for the python guys to use.
Is this correct? I had a short look at the 2.4 version I'm using in a project here.
Had this changed in later versions, especially in 3.0?
The GIL is indeed per-process, not per-interpreter. This is unchanged in 3.x.
Perhaps the confusion comes about because most people assume Python has one interpreter per process. I recall reading that the support for multiple interpreters via the C API was largely untested and hardly ever used. (And when I gave it a go, didn't work properly.)
I believe it is true (at least as of Python 2.6) that each process may have at most one CPython interpreter embedded (other runtimes may have different constraints). I'm not sure if this is an issue with the GIL per se, but it is likely due to global state, or to protect from conflicting global state in third-party C modules. From the CPython API Docs:
[Py___Initialize()] is a no-op when called for a second time (without calling Py_Finalize() first). There is no return value; it is a fatal error if the initialization fails.
You might be interested in the Unladen Swallow project, which aims eventually to remove the GIL entirely from CPython. Other Python runtimes don't have the GIL at all, like (I believe) Stackless Python, and certainly Jython.
Also note that the GIL is still present in CPython 3.x.
I am relatively new to Python, and I have always used the standard cpython (v2.5) implementation.
I've been wondering about the other implementations though, particularly Jython and IronPython. What makes them better? What makes them worse? What other implementations are there?
I guess what I'm looking for is a summary and list of pros and cons for each implementation.
Jython and IronPython are useful if you have an overriding need to interface with existing libraries written in a different platform, like if you have 100,000 lines of Java and you just want to write a 20-line Python script. Not particularly useful for anything else, in my opinion, because they are perpetually a few versions behind CPython due to community inertia.
Stackless is interesting because it has support for green threads, continuations, etc. Sort of an Erlang-lite.
PyPy is an experimental interpreter/compiler that may one day supplant CPython, but for now is more of a testbed for new ideas.
An additional benefit for Jython, at least for some, is it lacks the GIL (the Global Interpreter Lock) and uses Java's native threads. This means that you can run pure Python code in parallel, something not possible with the GIL.
All of the implementations are listed here:
https://wiki.python.org/moin/PythonImplementations
CPython is the "reference implementation" and developed by Guido and the core developers.
Pros: Access to the libraries available for JVM or CLR.
Cons: Both naturally lag behind CPython in terms of features.
IronPython and Jython use the runtime environment for .NET or Java and with that comes Just In Time compilation and a garbage collector different from the original CPython. They might be also faster than CPython thanks to the JIT, but I don't know that for sure.
A downside in using Jython or IronPython is that you cannot use native C modules, they can be only used in CPython.
PyPy is a Python implementation written in RPython wich is a Python subset.
RPython can be translated to run on a VM or, unlike standard Python, RPython can be statically compiled.