Develop Reference

How to convert a python project into an executable file with all additional scripts?

I know there have been a bunch of questions already asked regarding this but none of them really helped me. Let me explain the whole project scenario so that I provide a better clarity to my problem. The directory structure is somewhat like this shown below:
Project Directory Layout
I need to convert this whole GUI based project (The main file is using Tkinter module to create GUI) into main.exe which I can share with others while making sure that all the additional files work exactly the same way it is working now when I run this main.py via Command Prompt. When I use this command with pyinstaller -
"pyinstaller --onefile --noconsole main.py"
It creates main.exe which shows "Failed to execute script" on running. Please provide me a detailed explanation on what should I do to achieve what I have stated above. Thank you in advance.

pyinstaller uses a few dirty tricks to compress a bunch of files into one
I recommend using cx_Freeze instead along with inno setup installer maker
do pip install cx_Freeze to install that and go here for inno setup
then copy the following into a file named setup.py in the same folder as your project
from cx_Freeze import setup, Executable
setup(name = "YOUR APP NAME" ,
version = "1.0.0" ,
description = "DESCRIPTION" ,
executables = [Executable("PYTHON FILE", base = "Win32GUI")]
)
lastly run python setup.py build
if you want as onefile download this file here
just edit the file a bit and use inno compiler to make into installer

Suppose our project has the following structure.
MyApp
|-models
| |-login.kv
|-data
| |-words.json
| |-audio.tar.gz
|-fonts
| |-FredokaOne.ttf
|-images
| |-gb.pngsound.png
| |-icon.ico
|-main.py
|-main.kv
|-draw.py
|-image.py
and depends on the following packages:
- kivy
- kivymd
- ffpyplayer
- gtts
First things first is to install cx_Freeze.
pip install cx_Freeze
Copy the following into a file named setup.py in the same folder as your project.
# https://cx-freeze.readthedocs.io/en/latest/distutils.html
import sys
from cx_Freeze import setup, Executable
includes = []
# Include your files and folders
includefiles = ['models/','data/','fonts/','images/','main.kv','draw.py','image.py']
# Exclude unnecessary packages
excludes = ['cx_Freeze','pydoc_data','setuptools','distutils','tkinter']
# Dependencies are automatically detected, but some modules need help.
packages = ['kivy','kivymd', 'ffpyplayer','gtts']
base = None
shortcutName = None
shortcutDir = None
if sys.platform == "win32":
base = "Win32GUI"
shortcutName='My App'
shortcutDir="DesktopFolder"
setup(
name = 'MyApp',
version = '0.1',
description = 'Sample python app',
author = 'your name',
author_email = '',
options = {'build_exe': {
'includes': includes,
'excludes': excludes,
'packages': packages,
'include_files': includefiles}
},
executables = [Executable('main.py',
base = base, # "Console", base, # None
icon='images/icon.ico',
shortcutName = shortcutName,
shortcutDir = shortcutDir)]
)
Lastly run.
python setup.py build
This command will create a subdirectory called build with a further subdirectory starting with the letters exe. and ending with the typical identifier for the platform that distutils uses. This allows for multiple platforms to be built without conflicts.
On Windows, you can build a simple installer containing all the files cx_Freeze includes for your application, by running the setup script as:
python setup.py bdist_msi
Cx_freeze references
Doc
Git Hub

cython setuptools change output filename

I am using cython to cross-compile external python module. I am using python3.6 on the host and python3.5 on the target. Also I am compiling on x86_64 for target aarch64.
My setup.py looks like:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
import builder_config
import os
os.environ["PATH"] = builder_config.PATH
os.environ["CC"] = builder_config.COMPILER
os.environ["LDSHARED"] = builder_config.COMPILER + " -lpython3.5m -shared"
os.environ["CFLAGS"] = builder_config.CFLAGS
os.environ["LDFLAGS"] = builder_config.LDFLAGS
os.environ["ARCH"] = "aarch64"
setup(
ext_modules = cythonize((Extension("my_ext", ["file1.pyx", "file2.pyx", "file3.pyx", "file4.pyx", "file5.pyx"]))),
)
When I run python3.6 setup.py build_ext -i I get a file named: my_ext.cpython-36m-x86_64-linux-gnu.so
My problem is that on the target the library will not be loaded unless the name is changed to:
my_ext.cpython-35m-aarch64-linux-gnu.so
How can I change the generated filename?

As stated in the comments, what you are trying to achieve is unsafe.
You can work around the architecture tag with the environment variable _PYTHON_HOST_PLATFORM (e.g. you can change it in your sitecustomize.py). But, if the modules are actually incompatible (and they most likely are), you will only get core dumps later on.
I don't think you can work around the major Python version.
In order to come back to safer grounds, I would try to rely on portable solutions. For example, it doesn't look official, but we can find some articles on the web about Conda and aarch64 (e.g. you can look for 'Archiconda'). One more time, you wouldn't be able to simply copy the conda environments from one machine to another, but, you can freeze these environments (via a 'conda export') and build similar ones on the target machine.

An option is to upgrade the target interpreter to v3.6 if that's possible for you.
Another option is to install v3.5 on the machine you're using to build with that interpreter. It's pretty uncomplicated to get several different versions of the python interpreter installed on the same machine. I don't know your specifics so I can't provide any links but I'm sure a quick search will get you what you need.

Cython unable to find shared object file

I am trying to link to my own C library from Cython, following the directions I've found on the web, including this answer:
Using Cython To Link Python To A Shared Library
I am running IPython through Spyder.
My setup.py looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy as np
setup(
ext_modules = cythonize(
[Extension("*",["*.pyx"],
libraries =["MyLib"],
extra_compile_args = ["-fopenmp","-O3"],
extra_link_args=["-L/path/to/lib"])
]),
include_dirs = [np.get_include()],
)
The file libMyLib.so is in /path/to/lib and it compiles fine.
I have a Python script in my IPython profile startup folder that does this
try:
os.environ["LD_LIBRARY_PATH"] += ":/path/to/lib"
except KeyError:
os.environ["LD_LIBRARY_PATH"] = "/path/to/lib"
I can confirm that this is running, because if I type os.environ["LD_LIBRARY_PATH"] into the IPython interpreter, it returns /path/to/lib
But when I try to load the Cython module (i.e. import mycythonmodule) I get:
ImportError: libMyLib.so: cannot open shared object file: No such file or directory
I've also tried putting libMyLib.so in other places to see if cython would find it:
In the directory where Python is running
On the Python path
In the same folder as the cython module
But it still doesn't find the shared library. The only way I can get it to find the library is by dropping it in /usr/lib, but I don't want it there, I want to be able to set the library path.
Am I missing something?

I'm self-answering, in case anyone else runs into the same problem. Looks like the answers are here:
Set LD_LIBRARY_PATH before importing in python
Changing LD_LIBRARY_PATH at runtime for ctypes
According to these answers (and my experience), the linker reads LD_LIBRARY_PATH when python is launched, so changing it from within python doesn't have any useful effect, at least not the effect I was hoping for. The only solution is to either wrap python in a shell script that sets LD_LIBRARY_PATH, or else drop the shared object somewhere on the linker search path.
Kind of a pain, but it is what it is.

I have fixed it by change setup.py.
I have a C++ dynamic shared library called "libtmsmdreader.so". and a header file named "TmsMdReader.hpp"
I wrapper C++ shared library to cython library called "tmsmdreader-pythonxxxxxx.so"
from setuptools import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
name="tmsmdreader",
ext_modules=cythonize([
Extension(
name="tmsmdreader",
language="c++",
sources=["TmsMdReaderApi.pyx"],
libraries=["tmsmdreader"],
library_dirs=["."],
include_dirs=["."],
extra_compile_args=["-std=c++14"],
compiler_directives={'language_level': 3},
runtime_library_dirs=["."])
]))
library_dirs=["."] and runtime_library_dirs=["."] can fixed LD_LIBRARY_PATH if libtmsmdreader.so in python scripy directory

Compile Python C Extensions with cx_Freeze

I am trying to create an exe on Windows from a Python 3 package with a C extension module. In distutils, you can create an extension like this:
from distutils.core import setup, Extension
module1 = Extension('demo',
sources = ['demo.c'])
setup (name = 'PackageName',
version = '1.0',
description = 'This is a demo package',
ext_modules = [module1])
Than, the extension will be compiled with the appropriate compiler and placed alongside your other modules with the command:
python setup.py build_ext --inplace
cx_Freeze is a module that can package your code into an exe file along with a Python interpreter and the relevant packages. Then, an end user can use your program without having a Python installation. Unfortunately, cx_Freeze doesn't have an Extension class, and I cannot find a way to handle compilation with cx_Freeze.
One solution I am unsure about is to first build the extensions in place with distutils/setuptools, and then use cx_Freeze to create the executable. I don't want to reinvent the wheel though, so I wonder if someone else with more experience in this area has a solution.

I found a working solution. I can import Extension from distutils, and pass it into the setup from cx_Freeze:
from cx_Freeze import setup
from distutils.core import Extension
...
setup=(...
ext_modules=Extension(...))
This makes sense, since cx_Freeze is built on top of distutils. Originally, I was trying to use setuptools.setup, but that doesn't work.

Using Cython To Link Python To A Shared Library

I am trying to integrate a third party library written in C with my python application using Cython. I have all of the python code written for a test. I am having trouble finding an example for setting this up.
I have a pyd/pyx file I created manually. The third party has given me a header file (*.h) and a shared library (*.so). As far as I can tell, there are no other dependencies. Can someone provide an example of how to set this up using Cython and disutils?
Thanks

Sure !
(In the following, I assume that you already know how to deal with cimport and the interactions between .pxd and .pyx. If this is not completely the case, just ask and I will develop that part as well)
The sample (grabbed from a C++ project of mine, but a C project would work pretty much the same) :
1. The Distutils setup file :
Assuming that the extension to be created will be called myext and the 3rd party shared library is libexternlib.so (note the lib* prefix, here)...
# setup.py file
import sys
import os
import shutil
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("myext") and not(name.endswith(".pyx") or name.endswith(".pxd"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
# build "myext.so" python extension to be added to "PYTHONPATH" afterwards...
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("myext",
sources=["myext.pyx",
"SomeAdditionalCppClass1.cpp",
"SomeAdditionalCppClass2.cpp"
],
libraries=["externlib"], # refers to "libexternlib.so"
language="c++", # remove this if C and not C++
extra_compile_args=["-fopenmp", "-O3"],
extra_link_args=["-DSOME_DEFINE_OPT",
"-L./some/extra/dependency/dir/"]
)
]
)
Note : Your external .so file is linked via the libraries option :
libraries=["externlib"] # Without the 'lib' prefix and the '.so' extension...
Note : the sources option can be used to get some additional source files compiled.
Important : myext.pxd (do not confound with .pyd - Windows stuff) and myext.pyx should be in the same directory. At compile time the definition file, if it exists, is processed first (more).
2. Then run it as follows :
After having changed directory to the one containing your myext.pxd, your myext.pyx, as well as the above setup.py script :
# setup.sh
# Make the "myext" Python Module ("myext.so")
CC="gcc" \
CXX="g++" \
CFLAGS="-I./some/path/to/includes/ -I../../../DEPENDENCIES/python2.7/inc -I../../../DEPENDENCIES/gsl-1.15" \
LDFLAGS="-L./some/path/to/externlib/" \
python setup.py build_ext --inplace
Where :
libexternlib.so is assumed to be located at ./some/path/to/externlib/
yourheader.h is assumed to be located at ./some/path/to/includes/
Note : CFLAGS could also have been setup using the extra_compile_args option :
extra_compile_args=["-I./some/path/to/includes/", "-fopenmp", "-O3"]
Note : LDFLAGS could also have been setup using the extra_link_args option :
extra_link_args=["-L./some/path/to/externlib/", "-DSOME_DEFINE_OPT", "-L./some/extra/dependency/dir/"]
Once distutils is done with the build, you get some new files, specially the myext.cpp, myext.h and most importantly, the myext.so.
3. After that, you're good to go :
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./some/path/to/externlib/
export PYTHONPATH=$PYTHONPATH:./some/path/to/myext/
# Run some script requiring "myext.so"
python somescript.py
Where your freshly created Python extension can be imported by its name :
# somescript.py
import myext
from myext import PySomeFeature
...
Note about Optimization : By default -O2 is used for compiling the extension, but this can be overloaded (see above setup where -O3 is specified).
Note about Cython paths : If Cython was installed in a custom directory, you might want to add it to your environment, before all :
PYTHONPATH=$PYTHONPATH:../../../DEPENDENCIES/Cython-0.18 export PYTHONPATH;
PATH=$PATH:../../../DEPENDENCIES/Cython-0.18/bin; export PATH;
Well, hope I covered the main points...