If I have a script, or in this case just a function or two, written in Jython -- is there a way to interact with that code from my Python3 project?
No, not until Jython catches up with CPython enough for your whole Python 3 project to run in Jython. You can't run part of a Python application with one interpreter and the rest with another. You might be able to juggle multiple processes using remote procedure calls using pickle, but it'll be complex and brittle, not to mention slow (has to copy all data involved). If it's pure Python, just port those two functions to Python 3 (likely easy), or port your project to Python 2.5 (probably much harder). If it uses Jython's JVM interop, there are alternatives working with CPython, though possibly less mature. Depending on what you need Java for, there might be a pure Python alternative.
Related
I need to integrate a body of Python code into an existing OSGi (Apache Felix) deployment.
I assume, or at least hope, that packages exist to help with this effort.
If it helps, the Python code is still relatively new and small, so can probably re re-architected to meet whatever constraints are needed. However, it must remain in Python, because of dependencies on third-party libraries.
What are suggested best practices?
The trick is to make this an extender, see 1 and 2. You want your Python code to be separate from the code that handles the interaction with the interpreter. So what you do is wrap the Python code and any native libraries in a bundle. This is trivial since it is just a zip file.
You then develop a bundle that listens to starting bundle (see the BundleTracker) that have python code. A manifest is often used but you can also look in a directory in the JAR. If you detect this code, you extract any native libraries and run the code in the interpeter of your choice.
If can use JYthon then that would be highly recommended. You can then carry the interpreter as an OSGi bundle that runs on the VM. If you need to use a native compiler your life is less rosy. You can rely on the environment to provide you with an interpreter but then why use OSGi in the first place. You basically lose the write once run anywhere advantage. You could go the full monty by creating bundles that contain Python installers for all platforms you support. Can be done, not even that hard, but a maintenance nightmare. Believe me, native code suck, it only does it a bit faster than Java.
I have 2 python modules where one only supports Python 2.x and the other one 3.x.
Unfortunately I need both for a project.
My workaround for now is to have them run on their own as separate programs and building up their communication via the socket module.
I will be ending up with 2 executables, what I would like to avoid.
The "connection" between both modules has to be as fast as possible.
So my question is if there is a way to somehow combine both to one executable at the end and if there is a better solution for a fast communication as the client-server construction I have now.
There really is no good way to avoid that workaround.
Conceptually, there's no reason that you couldn't embed two interpreters into the same process. But practically, the CPython interpreter depends on some static/global state. While 3.7 is much better about that than, say, 3.0 or 2.6 was, that state still hasn't nearly been eliminated.1 And, the way C linkage works, there's no way to get around that without changing the interpreter.
Also, embedding CPython isn't hard, but it's not trivial, in the way that running an interpreter as a subprocess is trivial—and it may be harder than coming up with an efficient way to pass or share state between subprocesses.
Of course there are other interpreters besides CPython. But the other major implementation with both 2.7 and 3.x versions isn't easily embeddable (PyPy), and the two that are easily embeddable don't have 3.x versions, and also can only be embedded in another VM, and can't run C extension modules (Jython and IronPython). It is possible to do something like using JEP to embed CPython 3.7 via JNI in a JVM while also using Jython 2.7 natively in that same JVM, but I doubt that approach will work for you.
Meanwhile, I mentioned that passing or sharing data between processes generally isn't that hard.
If you don't have that much data, you can usually just pass it pickled over a pipe.
If you do have a ton of data, it usually is, or could be, stored in memory in some structured form—numpy arrays, big hunks of ASCII or UTF-8 text, arrays of ctypes structs, etc.—that you can overlay on an mmap or shared memory segment.
Or, of course, you can come up with your own protocol and communicate with it over a (UNIX or IP) socket. But you don't necessarily have to jump right to that option.
Notice that multiprocessing supports both of the first two—although to take advantage of it with independent interpreters, you have to dig into its source and pull out the bits you need. And there are also third-party libraries that can help. (For example, if you need to pickle things that don't pickle natively, the answer is often as simple as "replace pickle with dill".)
1. Running multiple subinterpreters in various restricted ways does sort of work with things like mod_wsgi, and PEP 554 aims to get things to the state where you can easily and cleanly run multiple 3.7 subinterpreters in the same process, but still nothing like completely independent embeddings of CPython—the subinterpreters share a GIL, a cycle collector, an atexit handler, etc.
I'm looking to call Python code from Ruby. There are a few existing tools to do this and a few questions on this site recommending http://rubypython.rubyforge.org/, which works by embedding the Python interpreter in Ruby. I'm working on an app that uses libraries unique to Python (namely graph-tool, which I have reasons for using over, say RGL), but the final project is in Rails so having Ruby code do the controlling work would be ideal. I want it to be speedy so I'm using PyPy. Is there a way to get the PyPy interpreter embedded in Ruby code, or to make the Python interpreter in rubypython run PyPy?
No. Well, not without a lot of work.
First, RubyPython doesn't really include an embedded Python interpreter; it just wraps the interpreter at runtime. As shown in the docs, you can run it with any Python you want, e.g.:
>> RubyPython.start(:python_exe => "python2.6")
So, what happens when you try?
>> RubyPython.start(:python_exe => "/usr/local/bin/pypy")
RubyPython::InvalidInterpreter: An invalid interpreter was specified.
from /Library/Ruby/Gems/1.8/gems/rubypython-0.6.3/lib/rubypython.rb:67:in `start'
from /Library/Ruby/Gems/1.8/gems/rubypython-0.6.3/lib/rubypython/python.rb:10:in `synchronize'
from /Library/Ruby/Gems/1.8/gems/rubypython-0.6.3/lib/rubypython/python.rb:10:in `synchronize'
from /Library/Ruby/Gems/1.8/gems/rubypython-0.6.3/lib/rubypython.rb:54:in `start'
from (irb):4
Unfortunately, it requires CPython 2.4-2.7. It doesn't work with CPython 3.x, PyPy, Jython, etc. Again, from the docs:
RubyPython has been tested with the C-based Python interpreter (cpython), versions 2.4 through 2.7. Work is planned to enable Python 3 support, but has not yet been started. If you’re interested in helping us enable Python 3 support, please let us know.
Without looking at the code, I'm guessing rubypython is using rubyffi to either:
* Wrap the CPython embedding APIs, or
* Directly call CPython VM internals via its dll/so/dylib exports.
If it's the former, the project might be doable, but still a lot of work. PyPy doesn't support CPython's embedding APIs. If it had its own embedded APIs, you could potentially rewrite rubypython's lower level to wrap those instead, and leave the higher-level code alone. But embedding PyPy at all is still a work in progress, (See http://mail.python.org/pipermail/pypy-dev/2012-March/009661.html for the state of affairs 6 months ago.) So, you'd need to first help get PyPy embedding ready for prime time and stable, and then port the lower level of rubypython to use the different APIs.
If it's the latter, you're pretty much SOL. PyPy will never support the CPython internals, and much of what's internal for CPython is actually written in RPython or Python and then compiled for PyPy, so it's not even possible in principle. You'd have to drastically rewrite all of rubypython to find some way to make it work, instead of just porting the lower level.
One alternative is to port Ruby to RPython and use PyPy to build a Ruby interpreter and a Python interpreter that can talk to each other at a higher level; then, writing something like rubypython for PyRuby and PyPy would be trivial. But that first step is a doozy.
I need to send code to remote clients to be executed in them but security is a concern for me right now. I don't want unsafe code to be executed there so I would like to control what a program is doing. I mean for example, know if is making connections, where is connecting to, if is reading local files, etc. Is this possible with Python?
EDIT: I'm thinking in something similar to Android permission system. I want to know what a code will do and if it does something different, stop it.
You could use a different Python runtime:
if you run your script using Jython; you can exploit Java's permission system
with Pypy's sandboxed version you can choose what is allowed to run in your controller script
There used to be a module in Python called bastian, but that was deprecated as it wasn't that secure. There's also I believe something called RPython, but I don't know too much about that.
I would in this case use Pyro and write the code on the target server. That way you know clients can only execute written and tested code.
edit - it's probably worth noting that Pyro also supports http://en.wikipedia.org/wiki/Privilege_separation - although I've not had to use it for that.
I think you are looking for a sandboxed python. There used to be an effort to implement this, but it has been abolished a couple of years ago.
Sandboxed python in the python wiki offers a nice overview of possible options for your usecase.
The most rigourous (but probably the slowest) way is to run Python on a bare OS in an emulator.
Depending on the OS you use, there are several ways of running programs with restrictions, but without the overhead of an emulator:
FreeBSD has a nice integrated solution in the form of jails.
These grew out of the chroot system call.
Linux-VServer aims to do more or less the same on Linux.
Is there a way to limit the abilities of python scripts running under an embedded interpretor? Specifically I wish to prevent the scripts from doing things like the following:
Importing python extension modules (ie .pyd modules), except those specifically allowed by the application.
Manipulating processes in any way (ie starting new processes, or terminating the application).
Any kind of networking.
Manipulating the file system (eg creating, modifying and deleting files).
No. There's no easy way to prevent those things on CPython. Your options are:
Edit CPython source code and remove things you don't want - provide mocking methods for all those things. Very error-prone and hard to do. This is the approach of Google's App Engine.
Use Restricted Python. However, with it you can't prevent your user from exhausting the memory available or running infinite eat-all-cpu loops.
Use another python implementation. PyPy has a sandbox mode you can use. Jython runs under java and I guess java can be sandboxed.
Maybe this can be helpful. You have an example provided on how to work with the ast.
What you want it Google's Unladen Swallow project that Python version of App Engine runs on.
Modules are severely restricted, ctypes are not allowed, sockets are matched against some policy or other, in other words you get a sandboxed version of Python, in line with their Java offering.
I'd like to point out that this makes the system almost useless. Well useless for anything cooler than yet another [App Engine] App. Forget monkey-patching system modules, and even access to own stack is restricted. Totally un-dynamic-like.
OT: games typically embed LUA for scripting, perhaps you should check it out.