Develop Reference

Error when including git.exe in a Pyinstaller app to be used by clients

I'm designing a GUI application that will be distributed to others, written with Python, that includes the use of Kivy and GitPython.
I'm running into major issues when trying to package the program with Pyinstaller.
This is a sample of my python script where I'm defining the path to the the git exe in the 'dist' folder created by Pyinstaller.
base_path = os.getcwd()
splitString = base_path.split('\\')
if 'dist' in splitString:
end_index = splitString.index('dist')
splitString = splitString[:end_index]
save_path = ''
for string in splitString:
save_path += string + '\\'
save_path = save_path[:-1]
# Establish path to git.exe
git_path = base_path + '\\git\\'
cmd = f'SET PATH=%PATH%;{git_path}'
os.system(cmd)
import git
And this is a sample of my spec file generated by Pyinstaller.
a = Analysis(['main.py'],
pathex=[path\to\main.py],
binaries=[('C:\\Program Files\\Git\\cmd\\git.exe', '.\\git')],
datas=[],
hiddenimports=['win32timezone'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
[],
exclude_binaries=True,
name='sample_app',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True)
I'm getting this error over and over again when trying to import git:
Traceback (most recent call last):
File "main.py", line 162, in <module>
File "C:\Users\<name>\anaconda3\lib\site-packages\Pyinstaller\loader\pyimod03_importers.py", line 623, in exec_module
File "site-packages\git\__init__.py", line 89, in <module>
manage the module state. Use the 'ctx' variable dictionary. This
ImportError: Failed to initialize: Bad git executable.
The git executable must be specified in one of the following ways:
- be included in your $PATH
- be set via $GIT_PYTHON_GIT_EXECUTABLE
- explicitly set via git.refresh()
All git commands will error until this is rectified.
This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
- quiet|q|silence|s|none|n|0: for no warning or exception
- warn|w|warning|1: for a printed warning
- error|e|raise|r|2: for a raised exception
Example:
export GIT_PYTHON_REFRESH=quiet
When I run the exe on my personal computer (which has git installed here C:\\Program Files\\Git\\cmd\\git.exe ) everything runs totally normally. It's just went the exe is run on a computer that DOES NOT have git installed where it throws the error shown above.
It should be able to work since I'm including git as a binary file to Pyinstaller but it's just not working for me right now.
I've tried all of the suggestions made in the error message and nothing is working. Not sure if I'm missing a hidden import or something but I'm completely stumped so any assistance would be appreciated!!

Pyinstaller's .exe file closes immediately

I have a working python package that's a CLI tool and I wanted to convert it into a single .exe file to upload it to other package managers so I used Pyinstaller. After building the .exe file with this command:
pyinstaller -c --log-level=DEBUG main.py 2> build.txt --onefile --exclude-module=pytest --add-data "src;src"
I double-clicked the .exe file but it closed immediately but in that split second, I saw the expected output which is supposed to be the command-line interface so the .exe does work but not entirely.
main.py
from src.Categorize_CLI.__main__ import main
if __name__ == "__main__":
main()
.spec file
# -*- mode: python ; coding: utf-8 -*-
block_cipher = None
a = Analysis(
['main.py'],
pathex=[],
binaries=[],
datas=[('src', 'src')],
hiddenimports=[],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=['pytest'],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='main',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
upx_exclude=[],
runtime_tmpdir=None,
console=True,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
Update
I got it working by dragging the main.exe file to a open command prompt and then pressed enter and it worked, and I got an error:
RuntimeError: 'package' is not installed. Try passing 'package_name' instead.
[15592] Failed to execute script 'main' due to unhandled exception!

It sounds like the script ran to end of file to fast for you to see. You can confirm this by opening your terminal (cmd/poweshell in windows) and running your program like any other CLI tool.
cd path\to\exe
./exe -arguments
Since you launched it from an allready opened terminal it won't close when the script ends.
If this is the problem you can solve it by adding
input("Continue...") # wait for user
Update
As #BokiX says pyinstaller can cause false positives with anti virus software. Try a diffrent one e.g. nuikta:
pip install Nuikta
python -m nuikta main.py
installing python programs vs tradtitional programs
A tradtitional program installer is usualy a fancy zip file with some additianal feautures to setup a program for the relevant system it's on (e.g. make registry changes, download additianal files).
A python program is just a python script that was "frozen" in state so it can run independently on a system without python or it's dependencies. once you have an exe it should just run without needing to be "installed".
using console programs
A console program is a program that is made to be exicuted from a terminal. In modern use these are usualy so they can be run by a script or someone who finds typing is faster than using a GUI.

Run the code in the cmd and not by clicking the exe
A pyinstaller exe "usually" closes when 1 - Theres an error in the code , 2 - The code execution finished , closes after displaying the output
Also as BokiX stated , pyinstaller exe's often get false flagged as malicious , so maybe add an exception to your anti virus.

PyInstaller. Run > pyinstaller myapp.spec from script works diffrent than from command line

I've created a script to automatize creating new executable versions of my app. The thing is that when I run command pyinstaller myapp.spec from commandline on windows platform it creates different file than when I run in from my script (which doesn't work BTW). Below is a snippet of code I use to create .exe.
SNIPPET
SPEC_PATH = 'venv_python37'
file = 'myapp.spec'
os.chdir(SPEC_PATH)
command = r'{} {}'.format('pyinstaller', file)
os.system(command)
When I run that I see all prints of pyinstaller and at the end there is a:
55830 INFO: Building COLLECT COLLECT-00.toc completed successfully.
Which looks exacly like when I run this command directly from commandline.
DIFFERENCES
myapp.exe size created from my script is 27MB but from commandline it is 40MB
When I run the .exe created from myscript there is an error that says:
The 'gcloud' distribution was not found and it is required by the application
That looks like when I run it from myscrip it user another dependencies than from commandline and I think command from myscript doesn't use dependencies from my virtual enviroment. A'm I right?
MYAPP.SPEC
# -*- mode: python -*-
from kivy.deps import sdl2, glew
block_cipher = None
_excludes=("'tcl'", "'tk'", "'FixTk'", "'_tkinter'", "'tkinter'", "'Tkinter'")
a = Analysis(['C:\\Users\\Patryk\\PycharmProjects\\myapp\\myapp.py'],
pathex=['venv_python37', 'C:\\Users\\Patryk\\PycharmProjects\\myapp'],
binaries=None,
datas=[],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=_excludes,
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
*[Tree(p) for p in (sdl2.dep_bins + glew.dep_bins)],
name='BajotWorkSpace',
debug=False,
strip=False,
upx=True,
console=True,
runtime_tmpdir=None,
icon='C:\\Users\\Patryk\\PycharmProjects\\myapp\\main_logo.ico' )
coll = COLLECT(exe,
Tree('C:\\Users\\Patryk\\PycharmProjects\\myapp\\Added_files'),
name='myapp')

I've found the reason. When I run .spec from my script it uses packages from my virtual enviroment but the same command run from commandline uses global packages. The issue with gcloud module was lack of hook file in Lib\site-packages\PyInstaller\hooks. Adding file named hook-gcloud.py with below code made my .exe work.
from PyInstaller.utils.hooks import copy_metadata
datas = copy_metadata('gcloud')

How do I resolve a missing utility module preventing my executable from running when I already installed the parent module?

I've written a few Python scripts to create a tkinter GUI for a machine learning algorithm process. I originally coded everything in PyCharm, but I'd really like to put everything together into a stand-alone executable. I've moved my main script and its .py dependencies into their own directory and tested it out using the Command Prompt, and it works great. However, when I run pyinstaller, the executable is created but fails on startup.
The program is made up of three files, with GUI.py being the main script. As mentioned above, I moved the dependent files into a new directory and tested GUI.py in the Command Prompt, and it worked great. Executable is created (albeit with a lot of warnings about missing 'api-ms-win-crt' files) but can't be run.
I created the executable using the command:
pyinstaller --onefile GUI.py
When the executable is run from the command line after creation, I get a big long traceback ending in the following:
File "site-packages\sklearn\metrics\pairwise.py", line 32, in <module>
File "sklearn\metrics\pairwise_fast.pyx", line 1, in init
sklearn.metrics.pairwise_fast
ModuleNotFoundError: No module named 'sklearn.utils._cython_blas'
[3372] Failed to execute script GUI
I know I've already explicitly imported sklearn through the command prompt, but from the traceback, it seems I'm missing a utility module somewhere. I tried to import the missing module specifically, but I got an error that no distributed module was available.
I don't have much experience with pyinstaller, and I have no idea where to go from here. I'm using Windows 10 and Python 3.7.3.

It seems that Pyinstaller can't resolve sklearn import. So one easy way is to just bring the whole module directory which located in <path_to_python>/Lib/site-packages/sklearn/ with executable output. So use below spec file to generate your executable:
# -*- mode: python -*-
block_cipher = None
a = Analysis(['test.py'],
pathex=['<path to root of your project>'],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
a.datas += Tree('<path_to_sklearn_in_python_dir>', prefix='sklearn')
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='test',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=False,
runtime_tmpdir=None,
console=True )
Finally generate your executable with
pyinstaller test.spec
This should resolve import errors for sklearn but if you face other NotFound imports add them like above to spec file.

Building up on M.R.'s answer, you can directly include the path to sklearn in your original pyinstaller command:
pyinstaller --onefile GUI.py --add-data "<path-to-python>\Lib\site-packages\sklearn;sklearn"
which results in the following line of code being added inside a = Analysis() in the automatically-generated GUI.spec file:
datas=[('<path-to-python>\\Lib\\site-packages\\sklearn', 'sklearn')]
Note that the --onefile option will result in an executable that is slower to start up than the default one-folder bundle (based on both the pyinstaller documentation and my own experience bundling up sklearn):
pyinstaller GUI.py --add-data "<path-to-python>\Lib\site-packages\sklearn;sklearn"

Pyinstaller exe conversion fail - using lightgbm and sklearn

I am trying to convert my code into an exe using pyinstaller spec. Pyinstaller initially failed with the following reason:
main__.PyInstallerImportError: Failed to load dynlib/dll
'C:\\Users\\...\\lightgbm\\../compile\\lib_lightgbm.dll'. Most probably this
dynlib/dll was not found when the application was frozen.
I tried to correct it by adding the following line to my pathex list in the spec:
'myenv\\lib\\site-packages\\lightgbm'
Note: myenv is my virtualenv created for this project.
Which led to ImportError for sklearn. I added sklearn to hidden imports. This is my final spec fie:
# -*- mode: python -*-
import sys
sys.setrecursionlimit(5000)
block_cipher = None
a = Analysis(['myscript.py'],
pathex=['C:\\project_folder', 'myenv\\lib\\site-packages\\lightgbm'],
binaries=[],
datas=[('lib_lightgbm.dll', '.')],
hiddenimports=['cython', 'sklearn', 'sklearn.feature_extraction','sklearn.pipeline', 'sklearn.ensemble', 'sklearn.neighbors.typedefs', 'sklearn.neighbors.quad_tree', 'sklearn.tree._utils'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
[],
exclude_binaries=True,
name='myscript',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
name='myscript')
This leads to an error I find very weird:
File "myenv\lib\site-packages\lightgbm\sklearn.py", line 9, in <module>
ImportError: attempted relative import with no known parent package
[12692] Failed to execute script myscript
I have no idea why it would be searching for sklearn.py inside lightgbm?
There is sklearn.py inside lightgbm, but I am not sure how to fix this error. Also, there is no sklearn folder in the dist/myscript folder. Ideally it should have been present. I tried to manually copy the sklearn folder, although, as suspected, it made no difference. Could someone please let me know where the spec file is wrong?

Let's go back to the first error you had. That error suggests lib_lightgbm.dll was not seen when your application was frozen.
Assuming you run pyinstaller from Windows cmd.exe, you can fix this by passing a hook for lightgbm to pyinstaller, so that it knows where to get it from, e.g.
pyinstaller --additional-hooks-dir dir_with_lightgbm_hook --hidden-import lightgbm myscript.py
The name of the hook should be hook-lightgbm.py and for its contents, you can look here: https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-numpy.py , it should be something similar. Just make sure that the library is added to the binaries list, i.e.
binaries.append((os.path.join(os.path.dirname(
get_module_file_attribute('lightgbm')), "lib_lightgbm.dll"), "lightgbm"))
The second "lightgbm" argument is the location where the .dll will be copied, relative to your distribution directory, so in your case it should end up as dist\myscript\lightgbm\lib_lightgbm.dll.
Note: Adding lib_lightgbm.dll through a hook will make pyinstaller add the dependencies for it to the distribution, so it is better to do it like this than just copying it manually.
Second note: When copying your distribution package to another machine, you might need to install the appropriate Visual Studio redistributable used by the library (e.g. VS 2015 redistributable).
EDIT: Forgot to mention that if you do it like this, you can remove the path you added to your pathex.