When you build an exe file using --onefile option with Pyinstaller, you can specify datas (like picture or whatever...)
During the runtime, a tmp directory is created (MEI*) which contains python interpreter, your data in original format ...
How does Pyinstaller embed all theses datas?
.exe files are only machine code instructions right? They are not supposed to be "container file"...
Thanks !
Have you read the pyinstaller manual?
https://pythonhosted.org/PyInstaller/#id22
How the One-File Program Works The bootloader is the heart of the
one-file bundle also. When started it creates a temporary folder in
the appropriate temp-folder location for this OS. The folder is named
_MEIxxxxxx, where xxxxxx is a random number.
The one executable file contains an embedded archive of all the Python
modules used by your script, as well as compressed copies of any
non-Python support files (e.g. .so files). The bootloader uncompresses
the support files and writes copies into the the temporary folder.
This can take a little time. That is why a one-file app is a little
slower to start than a one-folder app.
After creating the temporary folder, the bootloader proceeds exactly
as for the one-folder bundle, in the context of the temporary folder.
When the bundled code terminates, the bootloader deletes the temporary
folder.
(In Linux and related systems, it is possible to mount the /tmp folder
with a "no-execution" option. That option is not compatible with a
PyInstaller one-file bundle. It needs to execute code out of /tmp.)
Because the program makes a temporary folder with a unique name, you
can run multiple copies of the app; they won't interfere with each
other. However, running multiple copies is expensive in disk space
because nothing is shared.
The _MEIxxxxxx folder is not removed if the program crashes or is
killed (kill -9 on Unix, killed by the Task Manager on Windows, "Force
Quit" on Mac OS). Thus if your app crashes frequently, your users will
lose disk space to multiple _MEIxxxxxx temporary folders.
Alright, I found this : https://en.wikipedia.org/wiki/Executable_compression
Compressed data can be along with a decompression code into a single excutable.
So, Pyinstaller may include a runtime packer to do that.
Related
I have a big project, and I update it very often.
Later I used py2exe one time to create the main .exe file, and just copied only 1-2 fresh .py files after every update to the directory with .exe file (no need to recreate .exe file, really quick update).
But I can't force pyinstaller to use my raw .py files this way. It packs all .py files to .exe and on running it extract and use that files. So I should recreate .exe even after the smallest changes of code (and it takes too long time...).
How to force pyinstaller to use my .py files like I did with py2exe?
I developed my code using PyCharm and am using PyInstaller to create a desktop .exe application. I am able to create the application, however, my current method requires navigating multiple directories, and to copy/paste some file dependencies. The whole reason to use PyInstaller was to make it more user-friendly, and easy to access the file dependencies. My question is, how should my code be organized so that a general user can easily access the file dependencies, and developer not need to copy/paste the dependency?
Below is my generalized current approach and the result.
To develop using PyCharm, edit:
/projectFolder/main.py,
/projectFolder/helper.py,
/projectFolder/data.xlsx
To create application using PyInstaller:
in command prompt, /projectFolder/venv/Scripts, execute pyinstaller ../../main.py
This creates projectFolder/venv/Scripts/dist/main/main.exe, among many more files (generated by PyInstaller) that the user shouldn't interact with or even see.
At this point, I need to copy/paste /projectFolder/data.xlsx into /projectFolder/venv/Scripts/dist/main for the .exe application to function.
The executable is now ready to be used.
I am looking for a better approach, where the user will see only the relevant files, main.exe and data.xlsx (since this will be modified, periodically). Also, I'd like data.xlsx need not be copy/pasted.
Again, how should I organize my code / package the distribution in order to simplify things for the user?
You can add files or even a folder to install with PyInstaller by using the command --add-data. The documentation explains the format for the file / folder to add but in your case it should be:
pyinstaller --add-data "data.xlsx;." main.py
Now your 'data.xlsx' will automatically copied and pasted into the folder with your .exe file.
PyInstaller also has a feature to compress all your files into one single .exe (including added data) and then when you run your program, it creates a temporary folder in your OS temp folder, usually C:\Users\USERNAME\AppData\Local\Temp\ that starts with _MEIxxxxx where the xxxx is a randomly generated number so your exe programs wont interfer with each other if more than one is running at a time. The folder is automatically deleted upon successful exit of the program.
The code to have only one file instead of a folder:
pyinstaller --onefile --add-data "data.xlsx;." main.py
There are 2 issues with having one file instead of a folder. The first isn't too big of a problem, but everytime your program crashes or doesn't close properly, the temp folder _MEIxxxxx where all your program was unloaded to wont get deleted, potentially clogging up their system. A simple fix would be to add some extra code into your program that checks if there is already a _MEIxxxxx folder with an older date/time in their temp folder and delete it.
The second is that if your program requires to read / write to a file, the code will check in the current directory of the exe, not the temp folder that is created. A workaround to this would be to write some extra code that looks for the a folder that starts with _MEI in their temp and uses that as the path. If it finds more than one folder with _MEI it should take the most recent (and hopefully delete the older ones)
Another cool feature is adding the --runtime-tmpdir command to your pyinstaller which allows you to specify the where you want your _MEIxxxxx to unload, potentially making it easier to check where the data and files you need to run the program.
pyinstaller --onefile --runtime-tmpdir "C:\TemporaryFolder" --add-data "data.xlsx;." main.py
Unfortunately, if you specify a folder or a path that doesn't exit your program wont work at all, so I would recommend creating a separate setup.exe that creates that folder for them and that should be run before running the main.exe file. Afterwards they can delete the setup file and their program should be running.
I am creating a python script that should modify itself and be portable.
I can achieve each one of those goals separately, but not together.
I use cx_freeze or pyinstaller to pack my .py to exe, so it's portable; but then I have a lot of .pyc compiled files and I can't edit my .py file from the software itself.
Is there a way to keep a script portable and lightweight (so a 70mb portable python environment is not an option) but still editable?
The idea is to have a sort of exe "interpreter" like python.exe but with all the libraries linked, as pyinstaller allows, that runs the .py file, so the .py script can edit itself or be edited by other scripts and still be executed with the interpreter.
First define your main script (cannot be changed) main_script.py. In a subfolder (e.g. named data) create patch_script.py
main_script.py:
import sys
sys.path.append('./data')
import patch_script
inside the subfolder:
data\patch_script.py:
print('This is the original file')
In the root folder create a spec file e.g. by running pyinstaller main_script.py.
Inside the spec file, add the patch script as a data resource:
...
datas=[('./data/patch_script.py', 'data' ) ],
...
Run pyinstaller main_sript.spec. Execute the exe file, it should print
This is the original file
Edit the patch script to e.g. say:
print('This is the patched file')
Rerun the exe file, it should print
This is the patched file
Note: As this is a PoC, this works but is prone to security issues, as the python file inside the data directory can be used for injection of arbitrary code (which you don't have any control of). You might want to consider using proper packages and update scripts as used by PIP etc.
Recently I was experimenting with pyinstaller to create an executable file from my Python script. Everything works as expected.
I tested two options: --onefile, which takes quite a long time (like 20-30sec) to start because it depacks everything into a temporary directory.
The --onedir option is much faster (4sec) to start but it's not very comfortable to use. When I move exe file outside this directory program no longer works.
My question is: is there a possibility to make the exe file point to this directory location? I want to keep all the pyinstaller files in one place and allow users to have the exe file in any location they want.
Thanks for help.
Let's just see a real-life production case. Whenever you download say a pirated game, or and original copy of software, generally they are compressed together. When you unzip them, a new folder is extracted and inside that folder there are a lot of other folders. What you do to run the software is you simply double click the .exe file.
Your situation is the same. If you move the exe file outside the original extracted folder then it simply doesn't work. So, the work around way is to create a shortcut to the exe file.
Hope this clarifies your doubt :)
Shortcut. Create .exe shortcut. This way original .exe will be still in parent directory but shortcut can be placed anywhere
Most of the answers are about creating shortcuts, but its not the true solution. What we want is a clean folder having one dir and the exe outside that dir.
Unfortunately this is not possible at the moment. This issue is there since 2010 and was not fixed till date. Here is the link to that issue:
https://github.com/pyinstaller/pyinstaller/issues/1048
All they say is to create your own bootloader.
Nobody was able to give the PR for that.
I also found a blog on separating the exe from onedir with a hook file. I tried this but was unsuccessful with the latest version of Pyinstaller.
At last you have to do something hacky.
I found a way to do this:
Make a folder Modules in the same directory where executable is present.
Copy and paste all the heavy modules inside that folder.
Add the search path for that module folder inside your program:
# add this code in the top (before imports)
if getattr(sys, 'frozen', False):
app_path = os.path.join(os.path.dirname(sys.executable),"Modules")
sys.path.append(app_path) # search this path for modules
Under pyinstaller option --exclude-module, write the names of all those modules you excluded.
Use the one-file option but don't pack any other assets like images. Add all those external assets/folders outside.
Add this option: --runtime-tmpdir "Temp". With this, the executable will unpack the required files in the same directory under a new folder "Temp".
That's all, now you will get a very small sized exe file with mainly two required folders "modules" and the "temp". The booting time will also increase, and it will look a lot cleaner.
I think you should have some other files which is being required by that exe file & hence when you move exe file outside of directory it's giving you error. One of the example can be that the exe program require chrome driver & you have placed it within that directory. If you move exe program outside then you need to place the chrom driver also in the new position. I hope it will help you to detect , otherwise we can use exe program anywhere if it does not require any dependency of other files.
A workaround allowing a clean folder structure for the user that only contains the main exe and the libs folder is to create a second Python program containing only one instruction : call the executable of the main Python program within the over-crowed folder with libs. The main program being distributed without the --onefile option, the execution remains faster.
The principle is simple : create a new Python project with a single script your_program_launcher.py with this content :
import os
if __name__ == '__main__':
os.chdir(".{0}your_program_folder".format(os.sep))
os.system("your_program.exe")
The script is very simple and limit itself to move on the main program folder (to avoid resource access problems) and call the main program executable.
You just have to distribute this launch program with --onefile and possibly an icon (pyinstaller -i icon.png --onefile your_program_launcher.py) but this script doesn't use any libs (except os) so this executable will be very light and its execution immediate. Then you will have to put this program in the parent folder so you get a clean folder with this launch executable and the folder containing libraries and main program exe, without using a .bat file which is less natural.
dist
| your_program_launcher.exe
|
|____your_program_folder
| |_____lib1
| |_____lib2
| |_____libn
| | your_program.exe
Since the file is still an executable that needs the files in the subfolder, the user won't be able to move this executable anywhere he wants, but at least the user won't be lost in the folder containing all the libraries.
Is it possible to do a two steps decompression with PyInstaller?
e.g. it can decompress archived files from itself as needed, like InnoSetup, Nullsoft Installer (NSIS).
For --onefile exe generated with PyInstaller, everything is decompressed at invocation runtime, and it takes a lot of time, if there's a lot of bundled datafile.
What I trying to do is to replicate InnoSetup with PyInstaller+PyQt. Any ideas?
Yes you can. You can bundle installer files by appending to a.data in your spec file. Then at runtime your data files will be in the MEIPASS folder and you can copy them wherever you want. https://stackoverflow.com/a/20088482/259538