Develop Reference

How do I resolve a missing utility module preventing my executable from running when I already installed the parent module?

I've written a few Python scripts to create a tkinter GUI for a machine learning algorithm process. I originally coded everything in PyCharm, but I'd really like to put everything together into a stand-alone executable. I've moved my main script and its .py dependencies into their own directory and tested it out using the Command Prompt, and it works great. However, when I run pyinstaller, the executable is created but fails on startup.
The program is made up of three files, with GUI.py being the main script. As mentioned above, I moved the dependent files into a new directory and tested GUI.py in the Command Prompt, and it worked great. Executable is created (albeit with a lot of warnings about missing 'api-ms-win-crt' files) but can't be run.
I created the executable using the command:
pyinstaller --onefile GUI.py
When the executable is run from the command line after creation, I get a big long traceback ending in the following:
File "site-packages\sklearn\metrics\pairwise.py", line 32, in <module>
File "sklearn\metrics\pairwise_fast.pyx", line 1, in init
sklearn.metrics.pairwise_fast
ModuleNotFoundError: No module named 'sklearn.utils._cython_blas'
[3372] Failed to execute script GUI
I know I've already explicitly imported sklearn through the command prompt, but from the traceback, it seems I'm missing a utility module somewhere. I tried to import the missing module specifically, but I got an error that no distributed module was available.
I don't have much experience with pyinstaller, and I have no idea where to go from here. I'm using Windows 10 and Python 3.7.3.

It seems that Pyinstaller can't resolve sklearn import. So one easy way is to just bring the whole module directory which located in <path_to_python>/Lib/site-packages/sklearn/ with executable output. So use below spec file to generate your executable:
# -*- mode: python -*-
block_cipher = None
a = Analysis(['test.py'],
pathex=['<path to root of your project>'],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
a.datas += Tree('<path_to_sklearn_in_python_dir>', prefix='sklearn')
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='test',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=False,
runtime_tmpdir=None,
console=True )
Finally generate your executable with
pyinstaller test.spec
This should resolve import errors for sklearn but if you face other NotFound imports add them like above to spec file.

Building up on M.R.'s answer, you can directly include the path to sklearn in your original pyinstaller command:
pyinstaller --onefile GUI.py --add-data "<path-to-python>\Lib\site-packages\sklearn;sklearn"
which results in the following line of code being added inside a = Analysis() in the automatically-generated GUI.spec file:
datas=[('<path-to-python>\\Lib\\site-packages\\sklearn', 'sklearn')]
Note that the --onefile option will result in an executable that is slower to start up than the default one-folder bundle (based on both the pyinstaller documentation and my own experience bundling up sklearn):
pyinstaller GUI.py --add-data "<path-to-python>\Lib\site-packages\sklearn;sklearn"

Build Cython-compiled modules and python code into executable binary using PyInstaller

I am trying to package my project code into a an executable binary using Cython and PyInstaller libraries.
My code directory looks like this:
The main.py is the main code which imports the logic from program_a.py and program_b.py.
I am successfully able to convert my program_a and program_b files into .so files which can be imported by any python code. I did this by executing the following script.
from distutils.core import setup
from Cython.Build import cythonize
sourcefiles = ['program_a.py', 'program_b.py']
setup(
name = "Hello World",
ext_modules = cythonize(sourcefiles),
)
By executing >python setup.py build_ext --inplace I get .so files as shown below
When I run python main.py it runs perfectly with .so files. Which shows that I can import them as a module.
Now, I want to package binaries (.so) files and main.py into single binary file. For that I used the following command provided by pyInstaller
pyinstaller "main.py" --onefile
It actually gives a binary in dist/ folder but I cannot able to import some modules and getting the following error:
Traceback (most recent call last):
File "main.py", line 1, in <module>
import program_a as lisence_checker
File "program_a.py", line 1, in init program_a
ModuleNotFoundError: No module named 'licensing'
[18032] Failed to execute script main
How can I link libraries with the pyinstaller or embed library information into my binaries?
What I found yet:
Building Cython-compiled python code with PyInstaller
https://riptutorial.com/cython/example/21982/bundling-a-cython-program-using-pyinstaller
But all of these above links are not using any external package inside there python code examples. I am able to compile the code without external modules

After getting familiar with PyInstaller package I am able to figure out the issue. I followed the following steps to make it work for me at the end.
Now, posting my answer to help others :)
## Build *.so files from python modules
1. Execute "setup.py" file
> python setup.py build
2. It will generate "*.so" modules inside "build/lib.linux-x86_64-3.6" dir.
## Created binary from cython modules
1. Copy the binaries (i.e. *.so) files into binary folder
2. Get inside the binary folder 'cd binary'
3. Run Pyinstaller command inside binary directory: `python -O -m PyInstaller --clean --onefile idps.spec`
4. Your binary will be inside dist folder 'binary/dist/'
5. Execute the binary in linux using './dist/sample_app'
6. Your app is ready :)
Here is spec file to make it work for me:
# -*- mode: python -*-
block_cipher = None
a = Analysis(['main.py'],
pathex=['cython_pyinstaller_sample/binary'],
binaries=[('program_a.cpython-36m-x86_64-linux-gnu.so', '.'),('program_b.cpython-36m-x86_64-linux-gnu.so', '.')],
datas=[('config_file.txt', '.')],
hiddenimports=['licensing', 'licensing.methods', 'pandas'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False) pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher) exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='sample_app',
debug=True,
bootloader_ignore_signals=False,
strip=False,
upx=True,
runtime_tmpdir=None,
console=True )

Just in case someone's looking for a quick fix.
I ran into the same situation and found a quick/dirty way to do the job. The issue is that pyinstaller is not adding the necessary libraries in the .exe file that are needed to run your program.
All you need to do is import all the libraries (and the .so files) needed into your main.py file (the file which calls program_a.py and program_b.py). For example, assume that program_a.py uses opencv library (cv2) and program_b.py uses matplotlib library. Now in your main.py file you need to import cv2 and matplotlib as well. Basically, whatever you import in program_a.py and program_b.py, you have to import that in main.py as well. This tells pyinstaller that the program needed these libraries and it includes those libraries in the exe file.

How to compile python script to binary executable

I need to convert a Python script to a Windows executable.
I have Python 2.6 installed to python26.
I have created one script and kept it in C:\pythonscript. Inside this folder there are two files
Setup.py and oldlogs.py (this file need coversion)
setup.py code is
from distutils.core import setup
import py2exe
setup(console=['oldlogs.py'])
How can I convert oldlogs.py to an exe file?

Or use PyInstaller as an alternative to py2exe. Here is a good starting point. PyInstaller also lets you create executables for linux and mac...
Here is how one could fairly easily use PyInstaller to solve the issue at hand:
pyinstaller oldlogs.py
From the tool's documentation:
PyInstaller analyzes myscript.py and:
Writes myscript.spec in the same folder as the script.
Creates a folder build in the same folder as the script if it does not exist.
Writes some log files and working files in the build folder.
Creates a folder dist in the same folder as the script if it does not exist.
Writes the myscript executable folder in the dist folder.
In the dist folder you find the bundled app you distribute to your users.

I recommend PyInstaller, a simple python script can be converted to an exe with the following commands:
utils/Makespec.py [--onefile] oldlogs.py
which creates a yourprogram.spec file which is a configuration for building the final exe. Next command builds the exe from the configuration file:
utils/Build.py oldlogs.spec
More can be found here

Since other SO answers link to this question it's worth noting that there is another option now in PyOxidizer.
It's a rust utility which works in some of the same ways as pyinstaller, however has some additional features detailed here, to summarize the key ones:
Single binary of all packages by default with the ability to do a zero-copy load of modules into memory, vs pyinstaller extracting them to a temporary directory when using onefile mode
Ability to produce a static linked binary
(One other advantage of pyoxidizer is that it does not seem to suffer from the GLIBC_X.XX not found problem that can crop up with pyinstaller if you've created your binary on a system that has a glibc version newer than the target system).
Overall pyinstaller is much simpler to use than PyOxidizer, which often requires some complexity in the configuration file, and it's less Pythony since it's written in Rust and uses a configuration file format not very familiar in the Python world, but PyOxidizer does some more advanced stuff, especially if you are looking to produce single binaries (which is not pyinstaller's default).

# -*- mode: python -*-
block_cipher = None
a = Analysis(['SCRIPT.py'],
pathex=[
'folder path',
'C:\\Windows\\WinSxS\\x86_microsoft-windows-m..namespace-downlevel_31bf3856ad364e35_10.0.17134.1_none_50c6cb8431e7428f',
'C:\\Windows\\WinSxS\\x86_microsoft-windows-m..namespace-downlevel_31bf3856ad364e35_10.0.17134.1_none_c4f50889467f081d'
],
binaries=[(''C:\\Users\\chromedriver.exe'')],
datas=[],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='NAME OF YOUR EXE',
debug=False,
strip=False,
upx=True,
runtime_tmpdir=None,
console=True )

How to compile all resources into one executable file?

I've wrote GTK application with python.
All graphical user interface is in glade file, and there are some images used. I wish to compile my application into EXEcutable file. For that I'm using PyInstaller compiler and UPX packer.
I've done as manual says:
python Configure.py
python Makespec.py --onefile --windowed --upx /path/to/yourscript.py
python Build.py /path/to/yourscript.spec
PyInstaller works perfectly and create one exe file. But to make my application work correctly i have to copy my glade and image files into exe's folder.
Is there any way to compile those files into executable?
I've edited my spec file in various ways but i can not achieve what i want. Spec file below only copies file to directory, but does not compile into executable file
# -*- mode: python -*-
a = Analysis([os.path.join(HOMEPATH,'support\\_mountzlib.py'), os.path.join(HOMEPATH,'support\\useUnicode.py'), 'r:\\connection\\main.py'],
pathex=['C:\\Documents and Settings\\Lixas\\Desktop\\pyinstaller-1.5-rc1'])
pyz = PYZ(a.pure)
exe = EXE( pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name=os.path.join('dist', 'NetworkChecker.exe'),
debug=False,
strip=False,
upx=True,
console=False,
icon='r:\\connection\\ikona.ico' )
coll = COLLECT(
exe,
[('gui.glade', 'r:\\connection\\gui.glade', 'DATA')],
[('question16.png', 'r:\\connection\\question16.png', 'DATA')],
# a.binaries,
# strip=False,
upx=True,
name='distFinal')
I wish to have only one executable file with everything included into

PyInstaller does allow you to bundle all your resources into the exe, without the trickyness of turning your data files into .py files -- your COLLECT object seems to be correct, the tricky step is accessing these files at runtime. PyInstaller will unpack them into a temporary directory and tell you where they are with the _MEIPASS2 variable. To get the file paths in both development and packed mode, I use this:
def resource_path(relative):
return os.path.join(
os.environ.get(
"_MEIPASS2",
os.path.abspath(".")
),
relative
)
# in development
>>> resource_path("gui.glade")
"/home/shish/src/my_app/gui.glade"
# in deployment
>>> resource_path("gui.glade")
"/tmp/_MEI34121/gui.glade"

With a few changes, you can incorporate everything into your source code and thus into your executable file.
If you run gdk-pixbuf-csource on your image files, you can convert them into strings, which you can then load using gtk.gdk.pixbuf_new_from_inline().
You can also include your Glade file as a string in the program and then load it using gtk.Builder.add_from_string().

Is there any way to compile those files into executable?
Strictly speaking: no, because you compile source code, while the glade file is XML and the images are binary data. What you would normally do is to create an installer (i.e. a self-unpacking archive that will place different files in the correct directories when the installer is ran).
EDIT: If your concern is simply to have a one-file executable (so it's not about "compiling" but really about the number of files that are permanently written on the filesystem) you could try to use this script based on py2exe. What it does, is to create temporary files each time the program is ran, removing them when execution is completed.
EDIT2: Apparently what you are asking for is also possible under PyInstaller. From the documentation:
By default, pyinstaller.py creates a distribution directory containing the main executable and the dynamic libraries. The option --onefile specifies that you want PyInstaller to build a single file with everything inside.

PyInstaller but keeping .py files upgradeable

I've managed to package my PyQt4 app as a "standalone" application on windows, it works.
However this application can upgrade itself, which is done by replacing the actual code written by me (.py files) with new versions, downloaded via the internet.
How can I tell PyInstaller do its job (putting together the DLLs, generating the launcher with the shiny icon, etc), BUT let the .py files untouched?
I need those files directly on disk, in order for the auto-update to work.

You can change the spec file to specifically not include files by name (when building lists), then make sure these files are included - I'd have to check whether there's an option to include but not compile.
I've not tried this myself (I use pyInstaller at work but don't have it set up on my home PC) but this is the sort of thing I think should be ok:
a = Analysis(['main.py'])
excluded = ['myfile0.py', 'myfile1.py', 'myfile2.py']
a.scripts = [script from script in a.scripts if script not in excluded]
pyz = PYZ(a.pure)
exe = EXE(a.scripts, pyz, name="main.exe", exclude_binaries=1)
dist = COLLECT(exe, a.binaries, excluded, name="dist")

Actually it's more like this :
a = Analysis(['main.py'])
excluded = ['pathto\\myfile0.py', 'pathto\\myfile1.py', 'pathto\\myfile2.py']
a.scripts = [script from script in a.scripts if script[1] not in excluded]
pyz = PYZ(a.pure)
excluded_files_collect = [(f.split('\\')[-1],f,'DATA') for f in excluded]
exe = EXE(a.scripts, pyz, name="main.exe", exclude_binaries=1)
dist = COLLECT(exe, a.binaries, excluded_files_collect , name="dist")
As script is actually a tuple with the form :
('myfile0.py', 'pathto\\myfile0.py', 'PYSOURCE')
You may also have to prevent files from being included in PYZ, refer to the pyz toc to see if they get included, I managed to exlude them using excludes=[myfile0] in Analysis().

I think the embedded interpreter in the executable will still search for .py files in the same directory and/or PYTHONPATH, py2exe uses a zip file for native python components, iirc pyinstaller embeds all of them in the executable, maybe there is an option to keep a zip like in py2exe (or not add them in the spec), then try to run the application without the files and monitor file accesses with procmon.

pyinstaller provides the --exclude option for your use case , and it is also possible to set the module or package you want to ignore using the excludes parameter of Analysis() in the .spec file .