I have a library written in Python complete with unit tests. I'm about to start porting of the functionality to Scala to run on Spark and as part of an Android application. But I'm loath to reproduce the unit tests in scala.
Is there a method for exposing the to-be-written Scala library to external interrogation from Python? I can rewrite the tests to use a command line interface, but I wondered if there were other ways.
I have ruled out Jython because it is not compatible with my existing Python 3 library and unit tests.
You might be able to do this by using Py4J to call your Scala code from Python. Py4J works in a regular cPython interpreter and is compatible with Python 3.
You will essentially be calling your Scala code from Java, so you'll have to be somewhat aware of how your Scala constructs map onto Java (here's a decent summary of some of the basic conventions). Depending on how your code is written, you might find it easier to implement a Java helper class that exposes a Java-friendly interface to your Scala class / application and call that new Java class from Py4J; this is the approach that we took for PySpark, Spark's Python API (more details here).
Short answer: Maybe, but don't.
Eventually common sense prevailed. I now aim to either use pyspark or port the unit tests across to Scala along with the rest of the library.
Exposing Scala to Python was achievable by generating Scala code that prints specifically what I wanted. Then from Python calling SBT to compile that code and run it and capture the stdout.
I started mocking the scala version of the same api in Python to create these custom scala scripts and by adding in a compilation step for each query it was getting very slow. As I started considering a command line interface or socket based api for my Mocked Scala classes the reality sunk in.
To answer the actual question of running my existing Python unittests, while I think it could still be possible it is not a very good idea.
Related
I have a python app and java app. The python app generates input for the java app and invokes it on the command line.
I'm sure there must be a more elegant solution to this; just like using JNI to invoke C code from Java.
Any pointers?
(FYI I'm v. new to Python)
Clarification (at the cost of a long question: apologies)
The py app (which I don't own) takes user input in the form of a number of configuration files. It then interprits these and farms work off to a number of (hidden) tools via a plugin mechanism. I'm looking to add support for the functionality provided by the legacy Java app.
So it doesn't make sense to call the python app from the java app and I can't run the py app in a jython environment (on the JVM).
Since there is no obvious mechanism for this I think the simple CL invocation is the best solution.
Sorry to ressurect the thread, but there was no accepted answer...
You could also use Py4J. There is an example on the frontpage and lots of documentation, but essentially, you just call Java methods from your python code as if they were python methods:
>>> from py4j.java_gateway import JavaGateway
>>> gateway = JavaGateway() # connect to the JVM
>>> java_object = gateway.jvm.mypackage.MyClass() # invoke constructor
>>> other_object = java_object.doThat()
>>> other_object.doThis(1,'abc')
>>> gateway.jvm.java.lang.System.out.println('Hello World!') # call a static method
As opposed to Jython, Py4J runs in the Python VM so it is always "up to date" with the latest version of Python and you can use libraries that do not run well on Jython (e.g., lxml). The communication is done through sockets instead of JNI.
Disclaimer: I am the author of Py4J
Take a look at Jython. It's kind of like JNI, but replace C with Python, i.e. you can call Python from Java and vice versa. It's not totally clear what you're trying to do or why your current approach isn't what you want.
Wrap your Java-Code in a Container (Servlet / EJB).
So you don´t loose time in the vm-startup and you go the way to more service-oriented.
For the wraping you can use jython (only make sense if you are familiar with python)
Choose a communication-protocoll in which python and java can use:
json (see www.json.org)
rmi (Python: JPype)
REST
SOAP (only for the brave)
Choose something you or your partners are familliar with!
If you really want to embed your Java app within your Python process, have a look at JPype. It provides access to Java through JNI.
How about using swig: http://www.swig.org/Doc1.3/Java.html ?
Give JCC a try http://pypi.python.org/pypi/JCC/2.1
JCC is a code generator for calling Java directly from CPython. It supports CPython 2.3+, several JREs (Sun JDK 1.4+, Apple JRE 1.4+, and OpenJDK 1.7) on OS X, Linux, Solaris, and Windows. It's produced by the Open Source Application Foundation (OSAF, the people making Chandler) and is released under an Apache-style license.
From the package description:
JCC is a C++ code generator for producing the glue code necessary to call into Java classes from CPython via Java's Native Invocation Interface (JNI).
JCC generates C++ wrapper classes that hide all the gory details of JNI access as well Java memory and object reference management.
JCC generates CPython types that make these C++ classes accessible from a Python interpreter. JCC attempts to make these Python types pythonic by detecting iterators and property accessors. Iterators and mappings may also be declared to JCC.
I am working on an application that uses C++ code and python code.
Data is shared between the two scripts, C++ <---> python. Right now, I achieve this via SQLite, which very roughly takes care of concurrency , and it works correctly. The problem is that it's very slow for my purpose.
My question is: how can I share data among python and C++ in a fast way?
I read that Boost can be used to implement python code in C++, but since in my python code I work with keras/tensorflow, I don't think that solution works for me.
I read that sockets are supposed to be slower than a local database, but I haven't tested this yet. Thanks!
I would like to build my project in Scala and then use it in a script in Python for my data hacking (as a module or something like that). I have seen that there are ways to integrate python code into JVM languages with Jython (only Python 2 projects though). What I want to do is the other way around though. I found no information on the net how to do this, but it seems strange that this should not be possible.
General solution -- use some RPC/IPC (sockets, protobuf, whatever).
However, you might want to look at Spark's solution -- how they translate Python code in Scala's APIs (https://www.py4j.org/) .
Recently, scalapy was created to call Python libraries from Scala.
https://github.com/shadaj/scalapy
If I have a script, or in this case just a function or two, written in Jython -- is there a way to interact with that code from my Python3 project?
No, not until Jython catches up with CPython enough for your whole Python 3 project to run in Jython. You can't run part of a Python application with one interpreter and the rest with another. You might be able to juggle multiple processes using remote procedure calls using pickle, but it'll be complex and brittle, not to mention slow (has to copy all data involved). If it's pure Python, just port those two functions to Python 3 (likely easy), or port your project to Python 2.5 (probably much harder). If it uses Jython's JVM interop, there are alternatives working with CPython, though possibly less mature. Depending on what you need Java for, there might be a pure Python alternative.
bbum posted an outline of how to do this, but I'm unable to complete the details. Where does the Python code go, and how will my Objective-C code know about it? How would I do it compiling on the command line?
Source here:
Calling Python From Objective-C
I have posted a full explanation of how to do this to my weblog as it is quite a bit longer than something I would post here.
The abstract summary remains the same: use an abstract class to provide the type information necessary to make the C compiler happy and the metadata necessary to make the bridge happy.
Unfortunately the story for using Python via PyObjC from within an Objective-C app is not very good at the moment. py2app which ships with PyObjC can compile loadable bundles (i.e. can be loaded via NSBundle), which seems like the best approach: define an NSObject subclass in python that implements a protocol (obtained via objc.protocolNamed) that you define in Objective-C, then compile this python file into a loadable bundle via py2app (which uses a standard setup.py). Unfortunately, py2app hasn't had much love, especially the plugin (loadable bundle) target, and a serious memory leak was introduced sometime around 10.5 such that any data passed from python to Objective-C from a py2app-compiled bundle leaks. Yuck.
PyObjC manipulates the Objective-C runtime in accordance with the ObjC-related code executed in Python, thus to be able to call python code from Objective-C, the general outline goes like
Write PyObjC wrapper around python code
Execute code declaring PyObjC wrapper to add these definitions to the ObjC runtime
Call PyObjC wrapper from Objective-C. Because it's declared at runtime, the symbols aren't available at compile time, so you'll have to use NSClassFromString et al. to instantiate the class. It's helpful to declare a #protocol with the appropriate methods so that the Objective-C compiler doesn't complain about missing methods.
If you have flexibility, the best option is to use the Cocoa-Python app templates (i.e. create a Python app), and then load your Objective-C code as a loadable bundle from within Python. This takes care of managing the Python interpreter for you.
Otherwise, with the code in main.m of the Cocoa-Python app template, you should be able to create a Python interpreter, execute your PyObjC code and then continue on. Obviously, the interpreter needs to be kept running so that your python code can execute, so you'll likely have to do this from a separate thread. As you can see this can get a little hairy. Better to go with the Python app, as described above.
Keep in mind that PyObjC is not guaranteed to play well with the Objective-C garbage collector, so all of these options require that your Objective-C code not use GC.
Google is your friend. Performing a search on the string "Cocoa Python" quickly turned up PyObjc.