I have a use case where I want to deploy an application on end-user machines that don't necessarily have Python installed. Most of the application is written in Python, but the entry point is a legacy C++ application. This is why using tools like PyInstaller are not an option (although I'm certainly not an expert on this, so I could be wrong).
I am using Cython to embed the Python interpreter into the application. Everything is going more or less fine, but my Python code uses NumPy, which means there is still an external dependency. Ideally, I'd like to have a single binary that includes NumPy, as well as the Python interpreter, and my application code. Is it possible, using Cython or some other tool, to compile NumPy (or some portions of it that I am using) to a library that I can then link to my final executable binary? Or is it possible to somehow link to a wheel (or egg) containing the NumPy binary???
Related
I have a software that is mostly written in Python and, for now, I'm using PyInstaller to bundle and distribute the software in a user-friendly way (it's part of my CI pipeline, for Linux and Windows).
However, my performance is terrible and I want to rewrite some heavy parts in Julia while keeping the front-end in Python. I can use PyJulia to do this, but it means that the user has to install Julia manually in order to use my program.
Julia does have the equivalent of PyInstaller, which is PackageCompiler.jl, but I don't know how to call something compiled with PackageCompiler.jl from the Python side.
How can I make this work, so I can bundle and distribute an executable that has Python, Julia and everything it needs to run?
A little more details
My end user is someone (chemists and pharmacists) that have no idea what programming is. They don't have Python, Julia or Docker (and they don't even want to install it).
In my current approach, the software bundled with PyInstaller consists of a single executable with everything inside it (Python and everything it needs). What I really want is to keep the same user experience, but also with Julia running on the backstage.
I'll implement several functions in the Julia side, and I want (almost) the same level of integration as I get with PyJulia.
Maybe I'll go to Rust and just use the C interface, but I really would like to use Julia.
Thank y'all for your time.
Per here: https://julialang.github.io/PackageCompiler.jl/stable/apps.html#Creating-an-app-1 you can get what is basically an executable file and then you can just follow this post: Python Script execute commands in Terminal to execute the file you create.
You could try using JuliaWin. This provides a standalone Julia runtime that works fully self-contained.
We deployed a tool using PyInstaller with the JuliaWin included as one of the datas as PyInstaller calls them. Iirc it was not PyJulia but JuliaCall+PythonCall doing the interfacing in our case.
Unfortunately the startup was ridiculous (order of minutes). This is in part due to Julia's startup time, but largely due to the generated .exe first having to unpack the JuliaWin. So we are currently investigating using PackageCompiler, possibly also switching to providing the user with 2 files instead of 1.
I have a C++ application that uses machine learning from Python and my current approach is making a single file executable with pyInstaller and then just running it from C++. This have obvious drawbacks, notably interapplication communication. At the moment I'm using an intermediate JSON file to talk to each other but this is massively suboptimal for my future requirements. What's beautiful about this, is that is working on all major platforms without too much hassle.
Section 1.6. from Python's manual reads "Compiling and Linking under Unix-like systems"
Does this mean that Python interpreter will be inside my application binary and target system doesn't need to have Python installed as the program will always use embedded interpreter ? If so, whats with python libraries ? Can I embed a whole conda enviroment ?
Also, whats with:
"(...) under Unix-like systems"
Does this means this approach is not multiplatform ?
Thanks in advance.
Embedding the Python interpreter is possible on all platforms. However it will only be the interpreter. Embedding any libraries will be a lot harder or even impossible.
But since you seem to deploy the Python libs already, you can use them just fine from the embedded interpreter. And then you could bridge C++ and Python without IPC, since they are both running in the same process.
pybind11 is very nice for embedding and generating C++ <-> Python interfaces.
A possible alternative, depending on the libraries in use, may be to export the model and use a C++ library to load and use it (for instance Tensorflow -> ONNX -> ONNX runtime).
It means that cpython (Python interpreter) will be inside your application. You will be able to run Python code and observe and manipulate virtual machine state directly from C++ code (good entry point C API reference is here). Your application might have some additional dynamic library dependencies (which ones depends on compilation options of embedded Python). Also interpreter isn't completely self contained and depends on some external .py modules normally shipped with Python distribution (as standard library). If you plan to import external modules that expect standard library, you will have to ship it with your application. There are ways how to build modules into binary too (freeze) but you might run into issues specially with modules that rely on filesystem.
As far as I tried, this procedure works on UNIX like systems and Windows (where easiest way is to link against DLL which you then ship with your application). On Windows you also need to make sure that you compile with same compiler as was used to compile DLL (or you compile Python DLL from source). Here is additional information about embedding on Windows: https://docs.python.org/3/faq/windows.html#how-can-i-embed-python-into-a-windows-application
Just note that embedding Python and shipping 3rd party modules with your application might have some licensing consequences.
I have working Python script using scipy and numpy functions and I need to run it on the computer with installed Python but without modules scipy and numpy. How should I do that? Is .pyc the answer or should I do something more complex?
Notes:
I don't want to use py2exe. I am aware of it but it doesn't fit to the problem.
I have read, these questions (What is the difference between .py and .pyc files?, Python pyc files (main file not compiled?)) with obvious connection to this problem but since I am a physicist, not a programmer, I got totally lost.
It is not possible.
A pyc-file is nothing more than a python file compiled into byte-code. It does not contain any modules that this file imports!
Additionally, the numpy module is an extension written in C (and some Python). A substantial piece of it are shared libraries that are loaded into Python at runtime. You need those for numpy to work!
Python first "compiles" a program into bytecode, and then throws this bytecode through an interpreter.
So if your code is all Python code, you would be able to one-time generate the bytecode and then have the Python runtime use this. In fact I've seen projects such as this, where the developer has just looked through the bytecode spec, and implemented a bytecode parsing engine. It's very lightweight, so it's useful for e.g. "Python on a chip" etc.
Problem comes with external libraries not entirely written in Python, (e.g. numpy, scipy).
Python provides a C-API, allowing you to create (using C/C++ code) objects that appear to it as Python objects. This is useful for speeding things up, interacting with hardware, making use of C/C++ libs.
Take a look at Nuitka. If you'll be able to compile your code (not necessarily a possible or easy task), you'll get what you want.
I'm using Python to develop a commercial software.
But I don't want to offer source to my clients.
I knew that Cython can complie my code to binary.
But could I use Cython with 3rdparty pure python library(I mean those libraries written in pure Python)?
Cython compiles to .pyd (not pyc), which is basically a dll. In theory it should work with any python code, including 3rd party libraries, but there's no guarantee (see the first comment below). You'd also better check the licenses of the libraries you intend to use though, to make sure they don't prohibit distributing as closed source.
Your other option is something like cx_Freeze or py2exe, both of which I think compile to binary. I've never really inspected the files they produce though so you'll need to check up on that.
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Is it feasible to compile Python to machine code?
Is it possible to compile Python code (plus its dependencies, plus the interpreter library) into a single, native Windows executable (with nothing else bundled along with it) from a Python file? (Kind of like how the GNU compiler for Java compiles Java into a native (humongous) executable, which contains everything in true machine code.)
If so, how would I go about doing this?
(Specifically, py2exe does not do what I want -- it includes the libraries inside a separate ZIP file, and it includes the interpreter as a separate DLL.)
Note 1:
To emphasize, I'm not asking for a "self-extracting archive", an "executable packer", or some other way of 'cheating' by bundling the files inside an exe -- I'm looking for something that genuinely converts Python into a native executable, like what GCJ does for Java.
Note 2:
Only if the above isn't possible:
Is it possible to at least generate a single executable from a Python code containing the interpreter bundled along with all the library dependencies, such that the resulting executable does not need to self-extract onto the target disk before running?
In this scenario, the 'compilation' requirement is relaxed: it doesn't matter if the code is actually compiled into machine code (it could simply be embedded as a text resource into the target executable), but the result must nevertheless be a single exe file [and nothing else] that can run standalone, specifically without needing to unpack/install anything onto the target disk before running.
Shed Skin can compile Python to C++, but only a restricted subset of it. Some aspects of Python are very difficult to compile to native code.
The short answer is no, and that is going to go for almost any language: any program you write is going to depend on some external libraries even if just the Windows system DLLs.
If you wrote a C program and compiled it with Microsoft's compiler you would still need the C runtime libraries to be installed. Chances are they already will be on most systems but it isn't guaranteed. Likewise even if you managed to compile a C Python interpreter statically linked to its libraries you still have to get the C runtime from somewhere.
What I suspect you are really asking is whether you can compile to a single .exe that depends only on libraries which you have a reasonable expectation of already being installed. So it all depends on what you are willing to consider part of the base system? Can you assume .Net framework 4 or Silverlight are installed? If so you might want to look at IronPython.
Likewise pypy can be built with either the Visual Studio toolchain or MinGW but I'm pretty sure in both cases you'll still need some external libraries at runtime.