I try to create an exe file using PyInstaller 3.2.1, for test purpose I tried to make an exe for following code:
import pandas as pd
print('hello world')
After considerable amount of time (15mins +) I finished with dist folder as big as 620 MB and build - 150 MB. I work on Windows using Python 3.5.2 |Anaconda custom (64-bit). Might be worth noting that in dist folder mkl files are responsible for almost 300 MB.
I run pyinstaller using 'pyinstaller.exe foo.py'. I tried using --exclude-module to exclude some dependencies, still ended up with huge files. Whether I use onefile or onedir doesn't make any difference.
I am aware that exe must contain some important files but is it normal to be as big as almost 1 GB? I can provide warning log if necessary or anything that could be helpful to solve the matter.
P.S. In parallel my coworker created an exe from same sample script and ended up with less than 100 MB, difference is he is not using anaconda. Could that be the matter?
Any help will be appreciated.
PyInstaller creates a big executable from conda packages and a small executable from pip packages. From this simple python code:
from pandas import DataFrame as df
print('h')
I obtain a 203MB executable using conda packages and a 30MB executable using pip packages. But conda is a nice replacement for pure virtualenv. I can develop with conda and Jupyter, create some script 'mycode.py' (I can download Jupyter notebook as py-file in myfolder).
But my final solution is next: If you do not have it, install Miniconda and from the Windows Start Menu open Anaconda Prompt;
cd myfolder
conda create -n exe python=3
activate exe
pip install pandas pyinstaller pypiwin32
echo hiddenimports = ['pandas._libs.tslibs.timedeltas'] > %CONDA_PREFIX%\Lib\site-packages\PyInstaller\hooks\hook-pandas.py
pyinstaller -F mycode.py
Where I create a new environment 'exe', pypiwin32 is needed for pyinstaller but is not installed automaticaly, and hook-pandas.py is needed to compile with pandas. Also, importing submodules does not help me optimize the size of the executable file. So I do not need this thing:
from pandas import DataFrame as df
but I can just use the usual code:
import pandas as pd
Also, some errors are possible along using the national letters in paths, so it is nice the english user account for development tools.
This is probably because the Anaconda version of numpy is built using mkl.
If you want to reduce the size of the distributable, you could work with a seperate building virtual environment with the packages installed through pip instead of conda
Here's a way to still be using conda and avoid mkl. Install numpy before installing pandas with this alternate command:
conda install -c conda-forge numpy
Avoids mkl, uses an OpenBLAS package in its place.
Full explanation in this issue at conda/conda-forge/numpy-feedstock github repo.
A simple solution while working with Anaconda:
-Make a new environment inside Anaconda Navigator. (The new environment is free from the large amounts of packages that are causing the problem.)
-Open a terminal and use pipinstall to include the packages you need. ( Make sure it is in the new environment)
-Run pyinstaller.
I reduced my .exe from 300 MB to 30 MB.
I have the Anaconda 3.5.5 build for Python on Windows 10 and was also getting excessively large executables using the Anaconda distribution.
I was able to correct this by doing the following:
First create a virtual environment (forums suggest virtualenv, but this gave me problems so instead I used venv)
python -m venv C:/Python/NewEnv
This creates a virtual environment inside C:/Python/NewEnv with base python, pip and setuptools
Next switch to the newly created environment
C:/Python/NewEnv/Scripts/activate
You'll know that the environment is different as your command prompt will be prefaced with your new environment name (NewEnv)
Install numpy first, then scipy, then pandas
pip install numpy==1.13.3
pip install scipy==1.1.0
pip install pandas==0.18.1
pip install pypiwin32==223
pip install pyinstaller==3.2
I had to use these versions as I've tried different ones, but any later version of pandas were giving me further issues.
Once these have been installed you can compile your program
C:/Python/NewEnv/Scripts/pyinstaller --onefile program.py
This will create a .spec file, which you'll need to modify with this version of pandas and pyinstaller to add hidden imports otherwise loading pandas from the executable will fail (Not sure if there's a pyinstaller command to just create the spec file, but if there is then rather do that - see ammendment#1)
There will be a hidden imports line inside the newly created .spec file:
hiddenimports=[],
Change this to add pandas._libs.tslibs.timedeltas
hiddenimports=['pandas._libs.tslibs.timedeltas'],
Then you can compile your program again against the .spec file
C:/Python/NewEnv/Scripts/pyinstaller --onefile program.spec
Note that this will install the program in whichever directory you are in so change directories before executing pyinstaller.
Ammendmend#1: I see that it's possible to add the hook-pandas.py to the Pyinstaller hooks.
So after you install pyinstaller in the new environment, run
echo hiddenimports = ['pandas._libs.tslibs.timedeltas'] > C:\Python\NewEnv\Lib\site-packages\PyInstaller\hooks\hook-pandas.py
I had a similar problem and found a solution.
I used Windows terminal preview. This program allows creation of various virtual environments like Windows Power Shell (btw. Linux Ubuntu too. Also, worth noting: you can have many terminals in this program installed and, even, open a few at once. Very cool stuff).
Inside Windows Power Shell in Windows terminal preview I installed all the necessary libraries (like for example re, pandas, numpy, etc), then I opened the path to my file and tried to use this command:
pyinstaller --onefile -w 'filename.py'
...but, the output exe didn't work. For some reason, the console said that there is a lack of one library (which I had installed earlier). I've found the solution in mimic the auto-py-to-exe library. The command used by this GUI is:
pyinstaller --noconfirm --onedir --console "C:/Users/something/filename.py"
And this one works well. I reduced the size of my output exe program from 911MB to 82,9MB !!!
BTW. 911MB was the size of output made by auto-py-to-exe.
I wonder how is it possible that no one yet has created a compressor that reads the code, checks what libraries are part of the code, then putting only them inside the compression. In my case, auto-py-to-exe probably loaded all libraries that I ever installed. That would explain the size of this compressed folder.
Some suggest using https://virtualenv.pypa.io/en/stable/ but in my opinion, this library is very difficult, at least for me.
I created an executable file within a virtual environment. It did not help to reduce the app size. According to the closed issue QST: Pandas without MKL?, 'pandas does not use mkl directly, your issue is with pyinstaller.'
Then I tried to make a standalone application using py2app (py2exe for Windows). As a result, the app takes 156 MB in contrast to 923 MB when using pyinstaller.
You need pure python environment, No Anaconda.
Because, it has too many useless packages.
Install new python environment on another PC with as few package as possible!
Then try to use pyinstaller again. With this method, pyinstaller reduced the file from 200M to 8M.
PS: If you lack of some packages, you can pip install ...
Related
I created an application using PyQt5, and I'm willing to convert it to a 32bit executable file (.exe) using auto-py-to-exe! I searched a lot about this and figured out that I should use a 32bit version of Python for this purpose(examples:[1],[2],[3],[4]). Since I'm comfortable with using Conda environments, I tried to make a clone from my preferred Conda environment(that contains PyQt5 and auto-py-to-exe) in this way:
set CONDA_SUBDIR=win-32
conda create --name py32 --clone python3.10
conda activate py32
conda update --all
# Then I tried to run auto-py-to-exe
auto-py-to-exe
After this, I did a transformation using auto-py-to-exe successfully. But still, I get this error on 32bit windows when I try to execute the .exe file:
Now I'm somewhat disappointed about how I should achieve my goal.
Important Question: Why did I use set CONDA_SUBDIR=win-32? Because I think that command helps me clone everything with 32bit format and converts my cloned Python to a 32bit version, this helps me run auto-py-to-exe and convert my .py file to a .exe 32bit file. But it seems I'm wrong about this since I can't run the .exe file in 32bit OS.
Can you please help me how I can create a 32bit version of Python in a Conda environment and then use auto-py-to-exe to create the 32bit .exe file? (I assume that auto-py-to-exe also uses Python for running, and the 32bit version of Python influence on auto-py-to-exe result.)
Additional details:
My OS: 64bit Windows 10
But I want to run the .exe file on another machine that has 32bit Windows 10
Update:
Since I didn't get an answer about Conda environments, I tried installing 32bit Python. I achieved a 32bit .exe file with these steps:
Installing Python 3.10.1 32bit using this link.
Adding the Python path to the User variables and System variables:
Then I opened cmd and installed the required packages like auto-py-to-exe (also those used in .py) using pip.
run auto-py-to-exe in cmd and start converting.
The result is a 32bit .exe file that a 32bit OS can execute. But This isn't exactly what I looked for(it works, but it made me install a 32bit Python and add it to the path, which isn't what I looked for). So I write this here and hope for someone to help me do these in a Conda environment.
Don't use Conda environment use venv or pipenv. you can't convert conda environment .py file to exe because of its dependency.While using venv use pyinstaller. install comand -
pip install pyinstaller
convert command -
pyinstaller --onefile -w 'filename.py'
When using pipenv use auto-py-to-exe. yeah auto-py-to-exe is old, but there are no other tools.as for conda you can't do anything.
I have been trying to build .exe file using pyinstaller in windows 10. It worked, but the size of the exe file is ~212 MB, even by using a venv (as in here). I thought it might be because I am using python by anaconda!
Then I installed a separate version of Python so not to use anaconda! But it did not work (still large file).
Then I uninstalled anaconda to test it. Pyinstaller is still trying to access Python in 'C:\Program Files\anaconda3\python.exe' (this error: No Python at 'C:\Program Files\anaconda3\python.exe'). However I have removed all path to anaconda. Probably it has always tried to reach anaconda, and this is why I haven't been successful to build a small size .exe file.
How can I clearly indicate paths for pyinstaller and python?
Finally, after a lot of researching, could solve my problem:
Uninstalled all pythons and anaconda from my PC
Removed all Path from the system variables
Restarted the windows
Installed a fresh Python from its website
Installed Pyinstaller using pip install pyinstaller
Tested my .py code in cmd. It showed me all the packages that are missing.
Installed all required packages by using pip install name-of-package
Ran final command by pyinstaller -F -w --clean file.py
(Optional) Install Anaconda if you need (don't add Anaconda Python as the default python. Also don't add its path to the system variables).
Note: You can build virtualenv and do pyinstaller in them.
My previous tries which used anaconda resulted in file of 212 MB in size. This process generated a .exe file of size 27 MB (Importing only pandas module).
I ran into a similar problem and found PyCharms virtualenv manager very helpful. https://www.jetbrains.com/help/pycharm/creating-virtual-environment.html
This just necessitated downloading python from python.org and linking the virtual environment to this interpreter, rather than the conda interpreter (otherwise it will throw strange SSL errors).
This seems to allow neat parallel use of conda and virtualenv.
I have a Jupyter notebook script that will be used to teach others how to use python.
Instead of asking each participant to install the required packages, I would like to provide a folder with the environment ready from the start.
How can I do this?
What is the easiest way to teach python without running into technical problems with packages/environments etc.?
If you just need to install Python dependencies, you can use #Aero Blue solution. However, the users would need probably to make a virtual environment, so they don't mess with other environments and versions, etc.
However, if they should need some Linux packages, this would not be enough. Therefore, I would suggest using Docker. You would need to provide them with a Dockerfile, that you should set to install any dependencies (whether is for Python or Linux), and they would just need to use docker build and docker run commands.
The easiest way I have found to package python files is to use pyinstaller which packages your python file into an executable file.
If it's a single file I usually run pyinstaller main.py --onefile
Another option is to have a requirements file
This reduces installing all packages to one command pip install -r requirements.txt
You would need to use a program such as py2exe, pyinstaller, or cx_freeze to package each the file, the modules, and a lightweight interpreter. The result will be an executable which does not require the user to have any modules or even python installed to access it; however, because of the built-in interpreter, it can get quite large (which is why Python is not commonly used to make executables).
Have you considered using Azure notebooks or another Jupyter hosting service ? Most of these have a special syntax you can use to perform pip installs. For Azure it is !pip install
https://notebooks.azure.com
I am trying to convert a .py to .exe using pyinstaller. When I type pyinstaller my_code.py everything seems to be working and after a couple of minutes the process stops and I get the recursion error. I have tried to create a my_code.spec file in the same folder, edit it and change the number of recursions but when I run pyinstaller apparently a new .spec is created since I can't find the sys.setrecursionlimit() command that I had previously added to the my_code.spec file.
I am running all the above from the anaconda command prompt and not from the command line but I think that this is not a problem since I have tried to convert to .exe a simple "hello world" script and it works perfectly.
I have python 3.6.3 installed.
Please see this link:https://github.com/pyinstaller/pyinstaller/issues/2919
The issue is with python 3.6, and most issues can be resolved by downgrading to python 3.5 to use pyinstaller.
If you are using anaconda3 this can be done by opening a command prompt and running:
conda update conda
And then running:
conda install python=3.5
It's better to build a different environment for when wanting to make an executable python file. This should work to the python version your executable compiler works better. Usually, the older the python version the easiest is to compile.
The way suggested by alphabet5 took too long for me. I wanted to find another way.
What I did was to create another environment for Python3.5 in Anaconda Navigator. Then, I activated the Python3.5 environment (for example py35):
conda activate py35
Then, I installed necessary libraries such as pyqt5, pyqtgraph, pyinstaller etc. since py35 is a newly created environment.
When I ran pyinstaller and didn't get any error.
I would like to use PyInstaller to create a Windows executable. I installed pyinstaller using pip, as well as the correct version of pywin32.
When I attempt to verify the pyinstaller installation by typing pyinstaller --version, I get the message failed to create a process.
What am I doing wrong? Thanks in advance for your help.
There could be two reasons:
1) The python install location has spaces. See the answer at https://stackoverflow.com/a/34546220/3559967
2) You renamed the python install location. See the answer at https://stackoverflow.com/a/17560177/3559967
You can always reinstall pyinstaller:
python -m pip uninstall pyinstaller
[output of uninstall]
python -m pip install pyinstaller
where python is the command you use to run the python version of your choice (py, python, python3, etc.).
Notes:
Moving or changing exe names (i.e. the python.exe filename), folders, etc. can cause issues, per the other answer. If you want to make changes to the python exe or it's location, you typically need to do that immediately upon install, before installing any other packages.
There are ways of recording all your existing packages (See PIP Freeze) to a requirements.txt file, and then reinstalling them later, for another version of Python or to a new virtual environment.
If you are not familiar with virtual environments, you can start learning in the Python venv package docs. Note that you should never rename or move a venv folder after creating it (but it's easy to create a new one and reinstall the packages).