Develop Reference

PyCharm/Pyinstaller - How to optimize distribution

I developed my code using PyCharm and am using PyInstaller to create a desktop .exe application. I am able to create the application, however, my current method requires navigating multiple directories, and to copy/paste some file dependencies. The whole reason to use PyInstaller was to make it more user-friendly, and easy to access the file dependencies. My question is, how should my code be organized so that a general user can easily access the file dependencies, and developer not need to copy/paste the dependency?
Below is my generalized current approach and the result.
To develop using PyCharm, edit:
/projectFolder/main.py,
/projectFolder/helper.py,
/projectFolder/data.xlsx
To create application using PyInstaller:
in command prompt, /projectFolder/venv/Scripts, execute pyinstaller ../../main.py
This creates projectFolder/venv/Scripts/dist/main/main.exe, among many more files (generated by PyInstaller) that the user shouldn't interact with or even see.
At this point, I need to copy/paste /projectFolder/data.xlsx into /projectFolder/venv/Scripts/dist/main for the .exe application to function.
The executable is now ready to be used.
I am looking for a better approach, where the user will see only the relevant files, main.exe and data.xlsx (since this will be modified, periodically). Also, I'd like data.xlsx need not be copy/pasted.
Again, how should I organize my code / package the distribution in order to simplify things for the user?

You can add files or even a folder to install with PyInstaller by using the command --add-data. The documentation explains the format for the file / folder to add but in your case it should be:
pyinstaller --add-data "data.xlsx;." main.py
Now your 'data.xlsx' will automatically copied and pasted into the folder with your .exe file.
PyInstaller also has a feature to compress all your files into one single .exe (including added data) and then when you run your program, it creates a temporary folder in your OS temp folder, usually C:\Users\USERNAME\AppData\Local\Temp\ that starts with _MEIxxxxx where the xxxx is a randomly generated number so your exe programs wont interfer with each other if more than one is running at a time. The folder is automatically deleted upon successful exit of the program.
The code to have only one file instead of a folder:
pyinstaller --onefile --add-data "data.xlsx;." main.py
There are 2 issues with having one file instead of a folder. The first isn't too big of a problem, but everytime your program crashes or doesn't close properly, the temp folder _MEIxxxxx where all your program was unloaded to wont get deleted, potentially clogging up their system. A simple fix would be to add some extra code into your program that checks if there is already a _MEIxxxxx folder with an older date/time in their temp folder and delete it.
The second is that if your program requires to read / write to a file, the code will check in the current directory of the exe, not the temp folder that is created. A workaround to this would be to write some extra code that looks for the a folder that starts with _MEI in their temp and uses that as the path. If it finds more than one folder with _MEI it should take the most recent (and hopefully delete the older ones)
Another cool feature is adding the --runtime-tmpdir command to your pyinstaller which allows you to specify the where you want your _MEIxxxxx to unload, potentially making it easier to check where the data and files you need to run the program.
pyinstaller --onefile --runtime-tmpdir "C:\TemporaryFolder" --add-data "data.xlsx;." main.py
Unfortunately, if you specify a folder or a path that doesn't exit your program wont work at all, so I would recommend creating a separate setup.exe that creates that folder for them and that should be run before running the main.exe file. Afterwards they can delete the setup file and their program should be running.

Visual Studio Code - setting it up for multiple languages

I've just downloaded VSC, because I wanted to edit all my code in one editor (I'm writing in C, Python and HTML). On my computer I have a folder (Programming), in which I have 3 sub-folders (C, Python, HTML). So when I started up VSC I opened this Programming folder, and also downloaded the necessary extensions for python and C in VSC.
Here comes my problem: if I run for example a .py which generates a .txt file, it's not generated in the same location as the .py file is, however in the main folder (programming). Also this issue occurs, when I build an .exe from a .c, it also builds in the main folder and not where my .c is.
Thank you for your help!

The default cwd(current working directory) is the folder you open, like you can see:
Even the script_2.py is under VSCODE\Python\Too\b_folder, its cwd is still under the folder VSCODE and not the specific project.
We can change cwd by the method os.chdir() in the file:
OR the second way, add "python.terminal.executeInFileDir": true, in settings.json
These are some workarounds, and it's still recommended that you turn to the specific project and go on development.

Pycharm mistakes folder for module

Recently I switched computers and re-downloaded my python and pycharm, I tried activating my saved projects on the new computer, but I ran into problem, pycharm doesn't recognize the parent folder of some of the files, and mistakes it for a module.
The folder is called: "Enrichment_extra_stuff", and when I try to import file in that folder, from another file in that folder, it seems like pycharm recognize it, because it shows me all of the files inside it, but when I run the code, I get the error ModuleNotFoundError: No module named 'Enrichment_extra_stuff'.
Also weirdly, when I try doing explict import and just write import fr to import the file fr, then pycharm shows an error but when I run it, it works like it should.
I tried digging a bit on pycharm, but got confused and didn't found anything, my python interpreter version is 3.8 and I program on windows if that helps.

A folder (or better a directory) is not seen as a module, unless you put __init__.py file in it. This could be empty, or it should be the module content. This is Python: a module is either a file, or a directory with __init__.py
The second part is only for Pycharm: PyCharm is created to handle large projects, and often your program is not in the base (root) directory of your project, but in one (or more) subdirectories (e.g. src). So you should explicitly tell PyCharm which are the root directories. So, go to the project structure panel (the panel with the files, usually on left side), go to your "root" directory, and set with right mouse click on your base source directory: on the pop-up menu select Mark directory as, and then select Source Root.

Why can python files be modified while open in IDLE editor?

When a .py file is being edited in IDLE, it can be renamed or deleted while still being able to run, but in other file types such as word files this is not allowed. You can even create a script using the os module to delete itself and then do an action, and that action still works. How is this possible?

When you run python scripts you scripts were loaded into memory. So you can edit your script files.

Pyinstaller onedir option - exe file outside the directory

Recently I was experimenting with pyinstaller to create an executable file from my Python script. Everything works as expected.
I tested two options: --onefile, which takes quite a long time (like 20-30sec) to start because it depacks everything into a temporary directory.
The --onedir option is much faster (4sec) to start but it's not very comfortable to use. When I move exe file outside this directory program no longer works.
My question is: is there a possibility to make the exe file point to this directory location? I want to keep all the pyinstaller files in one place and allow users to have the exe file in any location they want.
Thanks for help.

Let's just see a real-life production case. Whenever you download say a pirated game, or and original copy of software, generally they are compressed together. When you unzip them, a new folder is extracted and inside that folder there are a lot of other folders. What you do to run the software is you simply double click the .exe file.
Your situation is the same. If you move the exe file outside the original extracted folder then it simply doesn't work. So, the work around way is to create a shortcut to the exe file.
Hope this clarifies your doubt :)

Shortcut. Create .exe shortcut. This way original .exe will be still in parent directory but shortcut can be placed anywhere

Most of the answers are about creating shortcuts, but its not the true solution. What we want is a clean folder having one dir and the exe outside that dir.
Unfortunately this is not possible at the moment. This issue is there since 2010 and was not fixed till date. Here is the link to that issue:
https://github.com/pyinstaller/pyinstaller/issues/1048
All they say is to create your own bootloader.
Nobody was able to give the PR for that.
I also found a blog on separating the exe from onedir with a hook file. I tried this but was unsuccessful with the latest version of Pyinstaller.
At last you have to do something hacky.
I found a way to do this:
Make a folder Modules in the same directory where executable is present.
Copy and paste all the heavy modules inside that folder.
Add the search path for that module folder inside your program:
# add this code in the top (before imports)
if getattr(sys, 'frozen', False):
app_path = os.path.join(os.path.dirname(sys.executable),"Modules")
sys.path.append(app_path) # search this path for modules
Under pyinstaller option --exclude-module, write the names of all those modules you excluded.
Use the one-file option but don't pack any other assets like images. Add all those external assets/folders outside.
Add this option: --runtime-tmpdir "Temp". With this, the executable will unpack the required files in the same directory under a new folder "Temp".
That's all, now you will get a very small sized exe file with mainly two required folders "modules" and the "temp". The booting time will also increase, and it will look a lot cleaner.

I think you should have some other files which is being required by that exe file & hence when you move exe file outside of directory it's giving you error. One of the example can be that the exe program require chrome driver & you have placed it within that directory. If you move exe program outside then you need to place the chrom driver also in the new position. I hope it will help you to detect , otherwise we can use exe program anywhere if it does not require any dependency of other files.

A workaround allowing a clean folder structure for the user that only contains the main exe and the libs folder is to create a second Python program containing only one instruction : call the executable of the main Python program within the over-crowed folder with libs. The main program being distributed without the --onefile option, the execution remains faster.
The principle is simple : create a new Python project with a single script your_program_launcher.py with this content :
import os
if __name__ == '__main__':
os.chdir(".{0}your_program_folder".format(os.sep))
os.system("your_program.exe")
The script is very simple and limit itself to move on the main program folder (to avoid resource access problems) and call the main program executable.
You just have to distribute this launch program with --onefile and possibly an icon (pyinstaller -i icon.png --onefile your_program_launcher.py) but this script doesn't use any libs (except os) so this executable will be very light and its execution immediate. Then you will have to put this program in the parent folder so you get a clean folder with this launch executable and the folder containing libraries and main program exe, without using a .bat file which is less natural.
dist
| your_program_launcher.exe
|
|____your_program_folder
| |_____lib1
| |_____lib2
| |_____libn
| | your_program.exe
Since the file is still an executable that needs the files in the subfolder, the user won't be able to move this executable anywhere he wants, but at least the user won't be lost in the folder containing all the libraries.