I am working on an application that uses C++ code and python code.
Data is shared between the two scripts, C++ <---> python. Right now, I achieve this via SQLite, which very roughly takes care of concurrency , and it works correctly. The problem is that it's very slow for my purpose.
My question is: how can I share data among python and C++ in a fast way?
I read that Boost can be used to implement python code in C++, but since in my python code I work with keras/tensorflow, I don't think that solution works for me.
I read that sockets are supposed to be slower than a local database, but I haven't tested this yet. Thanks!
Related
I have been searching for an answer for quite some time and I cannot figure out a robust solution: I have a C# application (on Unity) which needs to call some python functions and libraries. In particular, the python code needs to read a very large tabular file from C# -the equivalent of a large pandas dataframe-, avoiding writing and reading through a file if possible, because given the size it would reduce the performance.
I am a total beginner to C#/Unity, and I have given a look at the following workarounds:
IronPython: it is not an option because the last stable version is for python 2.7. My requirements are python 3.6 and higher, moving towards 3.8.
PyInstaller: it creates .exe files and not .dll. Moreover, it is not clear to me how you could call the executable from C# and pass a dataframe/2D array without writing it on disk. I am also not sure whether it can be called passing an entire input file as argument, if you really want to communicate via files.
Communication via socket: I am a bit lost about it. If this is really feasible on both python and Unity sides, I am happy to see your tutorials -especially for Unity!
Unity Python script editor: it works fine in dev mode. However, the python scripts cannot be included in the build. This: https://forum.unity.com/threads/python-for-unity-editor-only.914843/#post-8330904 seems very suboptimal for a stable build.
Thank you for any hint!
I need to perform some background advanced calculations on my data after it is collected in InfluxDb which is stored on the edge server, which means I have limited resources for the calculations.
Also I cannot block the data collection while I do calculations.
I am weighing using Kapacitor UDF streams vs custom Python scripts.
Please note I need to make the scripts configurable so that I can easily deply them to different environments with different sensors
It probably makes little difference, in general, especially for 'simple' usecases, though I lean towards standalone python scripts. (It may be better to use Kapacitor if you can cover your usecase using the kapacitor language for this instead of python based UDFs, but I found it insufficient since I needed to retrieve additional data from other databases)
Standalone python scripts could be a bit lighter, since you don't need to run the Kapacitor service.
Standalone python scripts could be a bit more configurable. Kapacitor is also pretty configurable but you'd have to spend a bit of time learning how to use it.
Standalone python scripts could be a bit more stable. I've experimented with python UDF's a couple of years ago and found them unstable and buggy. While this may have improved by now, you'd still be relying on this being supported and if go is not your language you might have trouble debugging and fixing issues yourself.
I have a legacy (but still internally maintained) application written in C++ which handles some hardware, interacts with databases, receives commands via serial line or socket... in short it does a non-trivial amount of work.
This application runs under Linux (ARM/Buildroot).
Now the need is to revamp the control interface adding a RESTful API.
I am exploring the possibility to do so via a Python extension.
Note I am a C++/java programmer and I'm not really proficient in Python, but I know the basics.
General idea would be:
Start a Python interpreter as a thread in the C++ application.
Use Flask/jinja2 (or just plain Bottle) to handle incoming RESTful requests.
Expose a few (perhaps just one) C++ class to Python.
Call from Python the appropriate C++ methods to perform the required actions.
I studied the official documentation (mainly pertaining plain C) and several alternatives, including:
Boost.Python (possibly too heavy for our limited hardware)
pybind11 (seems concerned only in extending Python, not embedding it).
http://www.codeproject.com/Articles/11805/Embedding-Python-in-C-C-Part-I (deals only with embedding, without giving Python access to C++ classes).
Question are:
does this make any sense?
what is the least intrusive way (if any) to achieve this?
which lib/framework is advised to use?
is there some tutorial project along these lines?
I know the question is quite broad, but I hope to narrow it as soon as the first comments will point me in the right direction.
Can you do it the other way around - embed your C++ code in Python program? This way it would be more prepared for moving existing functionality Python like you said.
Python makes a lot of things easier (faster to develop, easier to maintain) - communication with databases, libraries (if hey have Python wrappers), process/thread management... Keep in C++ just the stuff that needs to be in C++, like dealing with hardware, C/C++-only libraries, CPU-heavy code.
Have a look at Cython for embedding C++ in Python. Cython is basically a compiler from Python-like language to a .c/.cpp file that "just" calls the Python C API, so it can be used from "normal" Python code, but can also call other C/C++ code.
Alternative: Implement an API in the C++ application and create a separate Python application that uses this API. By API I don't mean REST API, but something more low-level - RPC, ZeroMQ, plain sockets...
I have a library written in Python complete with unit tests. I'm about to start porting of the functionality to Scala to run on Spark and as part of an Android application. But I'm loath to reproduce the unit tests in scala.
Is there a method for exposing the to-be-written Scala library to external interrogation from Python? I can rewrite the tests to use a command line interface, but I wondered if there were other ways.
I have ruled out Jython because it is not compatible with my existing Python 3 library and unit tests.
You might be able to do this by using Py4J to call your Scala code from Python. Py4J works in a regular cPython interpreter and is compatible with Python 3.
You will essentially be calling your Scala code from Java, so you'll have to be somewhat aware of how your Scala constructs map onto Java (here's a decent summary of some of the basic conventions). Depending on how your code is written, you might find it easier to implement a Java helper class that exposes a Java-friendly interface to your Scala class / application and call that new Java class from Py4J; this is the approach that we took for PySpark, Spark's Python API (more details here).
Short answer: Maybe, but don't.
Eventually common sense prevailed. I now aim to either use pyspark or port the unit tests across to Scala along with the rest of the library.
Exposing Scala to Python was achievable by generating Scala code that prints specifically what I wanted. Then from Python calling SBT to compile that code and run it and capture the stdout.
I started mocking the scala version of the same api in Python to create these custom scala scripts and by adding in a compilation step for each query it was getting very slow. As I started considering a command line interface or socket based api for my Mocked Scala classes the reality sunk in.
To answer the actual question of running my existing Python unittests, while I think it could still be possible it is not a very good idea.
Although I found many answers and discussions about this question, I am unable to find a solution particular to my situation. Here it is:
I have a main program written in FORTRAN.
I have been given a set of python scripts that are very useful.
My goal is to access these python scripts from my main FORTRAN program. Currently, I simply call the scripts from FORTRAN as such:
CALL SYSTEM ('python pyexample.py')
Data is read from .dat files and written to .dat files. This is how the python scripts and the main FORTRAN program communicate to each other.
I am currently running my code on my local machine. I have python installed with numpy, scipy, etc.
My problem:
The code needs to run on a remote server. For strictly FORTRAN code, I compile the code locally and send the executable to the server where it waits in a queue. However, the server does not have python installed. The server is being used as a number crunching station between universities and industry. Installing python along with the necessary modules on the server is not an option. This means that my “CALL SYSTEM ('python pyexample.py')” strategy no longer works.
Solution?:
I found some information on a couple of things in thread Is it feasible to compile Python to machine code?
Shedskin, Psyco, Cython, Pypy, Cpython API
These “modules”(? Not sure if that's what to call them) seem to compile python script to C code or C++. Apparently not all python features can be translated to C. As well, some of these appear to be experimental. Is it possible to compile my python scripts with my FORTRAN code? There exists f2py which converts FORTRAN code to python, but it doesn't work the other way around.
Any help would be greatly appreciated. Thank you for your time.
Vincent
PS: I'm using python 2.6 on Ubuntu
One way or another, you'll need to get the Python runtime on your server, otherwise it won't be possible to execute Python bytecode. Ignacio is on the right track with suggesting invoking libpython directly, but due to Fortran's parameter-passing conventions, it will be a lot easier for you to write a C wrapper to handle the interface between Fortran and the CPython embedding API.
Unfortunately, you're doing this the hard way -- it's a lot easier to write a Python program that can call Fortran subroutines than the other way around.
You don't want any of those. What you should do is use FORTRAN's FFI to talk with libpython and frob its API.