I often see people talking that the GIL is per Python Interpreter (even here on stackoverflow).
But what I see in the source code it seems to be that the GIL is a global variable and therefore there is one GIL for all Interpreters in each python process. I know they did this because there is no interpreter object passed around like lua or TCL does, it was just not designed well in the beginning. And thread local storage seems to be not portable for the python guys to use.
Is this correct? I had a short look at the 2.4 version I'm using in a project here.
Had this changed in later versions, especially in 3.0?
The GIL is indeed per-process, not per-interpreter. This is unchanged in 3.x.
Perhaps the confusion comes about because most people assume Python has one interpreter per process. I recall reading that the support for multiple interpreters via the C API was largely untested and hardly ever used. (And when I gave it a go, didn't work properly.)
I believe it is true (at least as of Python 2.6) that each process may have at most one CPython interpreter embedded (other runtimes may have different constraints). I'm not sure if this is an issue with the GIL per se, but it is likely due to global state, or to protect from conflicting global state in third-party C modules. From the CPython API Docs:
[Py___Initialize()] is a no-op when called for a second time (without calling Py_Finalize() first). There is no return value; it is a fatal error if the initialization fails.
You might be interested in the Unladen Swallow project, which aims eventually to remove the GIL entirely from CPython. Other Python runtimes don't have the GIL at all, like (I believe) Stackless Python, and certainly Jython.
Also note that the GIL is still present in CPython 3.x.
Related
This is an obvious question, that I haven't been able to find a concrete answer to.
Is the Python Byte-Code and Python Code itself interpreter independent,
Meaning by this, that If I take a CPython, PyPy, Jython, IronPython, Skulpt, etc, Interpreter, and I attempt to run, the same piece of code in python or bytecode, will it run correctly? (provided that they implement the same language version, and use modules strictly written in Python or standard modules)
If so, is there is a benchmark, or place where I can compare performance comparison from many interpreters?
I've been playing for a while with CPython, and now I want to explore new interpreters.
And also a side question, What are the uses for the others implementations of python?
Skulpt I get it, browsers, but the rest? Is there a specific industry or application that requires a different interpreter (which)?
From https://docs.python.org/3/library/dis.html#module-dis
Bytecode is an implementation detail of the CPython interpreter. No
guarantees are made that bytecode will not be added, removed, or
changed between versions of Python. Use of this module should not be
considered to work across Python VMs or Python releases.
On the other hand, Jython "consists of a compiler to compile Python source code down to Java bytecodes which can run directly on a JVM" and IronPython compiles to CIL to run on the .NET VM.
The purpose is to better integrate into your programming environment. CPython allows you to write C extensions, but this is not necessarily true of other implementations. Jython allows you to interact with Java code. I'm sure similar is true of IronPython.
If so, is there is a benchmark, or place where I can compare
performance comparison from many interpreters?
speed.pypy.org compares pypy to cpython
As the GIL is a lock that surrounds the interpreter does it affect compiled Python? I'm wondering whether it is possible to get past the inherent multi-threading limitations of cpython by simply compiling my python before executing it.
Hopefully that makes sense and I'm not missing something obvious or misinterpreting how the GIL works/affects execution.
Thanks
As Daniel said in the comments, it depends on how you "compile" the code.
For example, running the code using Jython does indeed get around the limitations imposed by the GIL.
On the other hand, using something like py2exe makes no difference, since this effectively just packages CPython alongside your code.
Jython does not have a GIL.
IronPython does not have a GIL.
You can compile your python code with cython, and then whether it uses the GIL or not depends. If you convert all you python variables into cython types, you can run your code in a with nogil block and you will have no GIL because you are expressly releasing the GIL. If you are not running in a nogil block, you will be affected by cpython's GIL. More in the cython docs: http://docs.cython.org/src/userguide/external_C_code.html#acquiring-and-releasing-the-gil
For more on python and the GIL, read up here: http://www.jeffknupp.com/blog/2013/06/30/pythons-hardest-problem-revisited/
I'm looking to call Python code from Ruby. There are a few existing tools to do this and a few questions on this site recommending http://rubypython.rubyforge.org/, which works by embedding the Python interpreter in Ruby. I'm working on an app that uses libraries unique to Python (namely graph-tool, which I have reasons for using over, say RGL), but the final project is in Rails so having Ruby code do the controlling work would be ideal. I want it to be speedy so I'm using PyPy. Is there a way to get the PyPy interpreter embedded in Ruby code, or to make the Python interpreter in rubypython run PyPy?
No. Well, not without a lot of work.
First, RubyPython doesn't really include an embedded Python interpreter; it just wraps the interpreter at runtime. As shown in the docs, you can run it with any Python you want, e.g.:
>> RubyPython.start(:python_exe => "python2.6")
So, what happens when you try?
>> RubyPython.start(:python_exe => "/usr/local/bin/pypy")
RubyPython::InvalidInterpreter: An invalid interpreter was specified.
from /Library/Ruby/Gems/1.8/gems/rubypython-0.6.3/lib/rubypython.rb:67:in `start'
from /Library/Ruby/Gems/1.8/gems/rubypython-0.6.3/lib/rubypython/python.rb:10:in `synchronize'
from /Library/Ruby/Gems/1.8/gems/rubypython-0.6.3/lib/rubypython/python.rb:10:in `synchronize'
from /Library/Ruby/Gems/1.8/gems/rubypython-0.6.3/lib/rubypython.rb:54:in `start'
from (irb):4
Unfortunately, it requires CPython 2.4-2.7. It doesn't work with CPython 3.x, PyPy, Jython, etc. Again, from the docs:
RubyPython has been tested with the C-based Python interpreter (cpython), versions 2.4 through 2.7. Work is planned to enable Python 3 support, but has not yet been started. If you’re interested in helping us enable Python 3 support, please let us know.
Without looking at the code, I'm guessing rubypython is using rubyffi to either:
* Wrap the CPython embedding APIs, or
* Directly call CPython VM internals via its dll/so/dylib exports.
If it's the former, the project might be doable, but still a lot of work. PyPy doesn't support CPython's embedding APIs. If it had its own embedded APIs, you could potentially rewrite rubypython's lower level to wrap those instead, and leave the higher-level code alone. But embedding PyPy at all is still a work in progress, (See http://mail.python.org/pipermail/pypy-dev/2012-March/009661.html for the state of affairs 6 months ago.) So, you'd need to first help get PyPy embedding ready for prime time and stable, and then port the lower level of rubypython to use the different APIs.
If it's the latter, you're pretty much SOL. PyPy will never support the CPython internals, and much of what's internal for CPython is actually written in RPython or Python and then compiled for PyPy, so it's not even possible in principle. You'd have to drastically rewrite all of rubypython to find some way to make it work, instead of just porting the lower level.
One alternative is to port Ruby to RPython and use PyPy to build a Ruby interpreter and a Python interpreter that can talk to each other at a higher level; then, writing something like rubypython for PyRuby and PyPy would be trivial. But that first step is a doozy.
Why is it that you can run Jython and IronPython without the need for a GIL but Python (CPython) requires a GIL?
Parts of the Interpreter aren't threadsafe, though mostly because making them all threadsafe by massive lock usage would slow single-threaded extremely (source). This seems to be related to the CPython garbage collector using reference counting (the JVM and CLR don't, and therefore don't need to lock/release a reference count every time). But even if someone thought of an acceptable solution and implemented it, third party libraries would still have the same problems.
Note that extensions written in C can in fact get rid of the GIL: http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock
My guess, because the C libraries that CPython is built upon aren't thread-safe. Whereas Jython and IronPython are built against the Java and .Net respectively.
I am relatively new to Python, and I have always used the standard cpython (v2.5) implementation.
I've been wondering about the other implementations though, particularly Jython and IronPython. What makes them better? What makes them worse? What other implementations are there?
I guess what I'm looking for is a summary and list of pros and cons for each implementation.
Jython and IronPython are useful if you have an overriding need to interface with existing libraries written in a different platform, like if you have 100,000 lines of Java and you just want to write a 20-line Python script. Not particularly useful for anything else, in my opinion, because they are perpetually a few versions behind CPython due to community inertia.
Stackless is interesting because it has support for green threads, continuations, etc. Sort of an Erlang-lite.
PyPy is an experimental interpreter/compiler that may one day supplant CPython, but for now is more of a testbed for new ideas.
An additional benefit for Jython, at least for some, is it lacks the GIL (the Global Interpreter Lock) and uses Java's native threads. This means that you can run pure Python code in parallel, something not possible with the GIL.
All of the implementations are listed here:
https://wiki.python.org/moin/PythonImplementations
CPython is the "reference implementation" and developed by Guido and the core developers.
Pros: Access to the libraries available for JVM or CLR.
Cons: Both naturally lag behind CPython in terms of features.
IronPython and Jython use the runtime environment for .NET or Java and with that comes Just In Time compilation and a garbage collector different from the original CPython. They might be also faster than CPython thanks to the JIT, but I don't know that for sure.
A downside in using Jython or IronPython is that you cannot use native C modules, they can be only used in CPython.
PyPy is a Python implementation written in RPython wich is a Python subset.
RPython can be translated to run on a VM or, unlike standard Python, RPython can be statically compiled.