My question is essentially, "How should I structure the files and folders of my frozen, deployed Python-based Windows application." To understand my situation, here's some background:
I'm building a desktop application with Python 2.7 for my workplace. It is a GUI-based application built on PyQt. I am building the application with Esky which is a cross-platform freezing and updating framework. Esky basically wraps/calls py2exe, py2app, bb_freeze, or whatever tool you have installed that is appropriate for the current platform. Esky creates a zipped package that looks like this:
prog.exe - esky bootstrapping executable
appdata/ - container for all the esky magic
appname-X.Y.platform/ - specific version of the application
prog.exe - executable(s) as produced by freezer module
library.zip - pure-python frozen modules
pythonXY.dll - python DLL
esky-files/ - esky control files
bootstrap/ - files not yet moved into bootstrapping env
bootstrap-manifest.txt - list of files expected in bootstrap env
lockfile.txt - lockfile to block removal of in-use versions
...other deps...
updates/ - work area for fetching/unpacking updates
These zipped packages can then be placed on a file server which Esky looks to for updates. A number of methods are provided for managing updates including a very simple auto_update(). When an update occurs, the appname-X.Y.platform folder is essentially replaced with the next version folder... so the myApp.0.1.win32 folder is replaced by a myApp.0.2.win32 folder.
The other aspect of background you should know is that I am distributing the application to my coworkers, who do not have Python installed. I'm not distributing a Python package or library, I'm deploying a desktop application (my coworkers don't particularly care what it's written in, just that it works). I've built an Inno installer which installs the application, provides an uninstaller, and various shortcuts. Because everyone on the team has essentially the same Windows 7 64-bit environment, I'm pretty safe building for just that platform.
So, back to the issue of structure. I've read guides that recommend a certain format for a project skeleton, such as Learn Python the Hard Way, Exercise 46 or the Hitchhiker's Guide to Packaging. However these guides are oriented toward Python package developers, not compiled application developers.
I've also run into problems with Esky's appname-X.Y.platform folder, since it changes names every time the program is updated (to reflect the version number). Because I want some shortcuts in the Start Menu to always refer to documentation, changelog, etc, I have the installer place some of those files under the appdata folder. When the program updates, I have some code to check for newer versions of those files I want to be externally "visible" and copy the newer versions out of the appname-X.Y.platform folder and overwrite the copies in the appdata folder. I then also needed a means of storing persistent user settings, so the program generates and uses an appdata\settings folder (otherwise the settings would be wiped with each update).
Should I continue the process of having the application push new files out to the appdata folder post-update? Should I build my own structure of Docs, Examples, Settings, etc. and let the program populate those folders with newer files whenever necessary? Should I attempt to alter or take better advantage of Esky's behavior to better fit my usage? Perhaps I should rework my application to be destributable as both a Python package and an end-user application?
This question relates to this one about static files with Esky, this one about Python deployed application structure, and numerous generic questions about Python project structure which don't specifically address using Esky. Some videos discussing Esky are also available here and here.
I'm seeking recommendations for "best practice" methods to handle these challenges. If this doesn't fit the StackOverflow Question format, I'll gladly attempt to reword or narrow the focus of my question.
So here's my solution to the problem mentioned about about making files available to shortcuts at a static location despite the fact that Esky's auto-updating changes the name of my application folder every update. The function below I have within a class definition for a QMainWindow.
Logging statements could be replaced with print statements if your application doesn't use the logging module, though I highly recommend logging, especially if deploying a standalone application like this.
import os
import shutil
import logging
def push_updated_files(self):
"""
Manually push auto-updated files from the application folder up to the appdata folder
This enables shortcuts and other features on the computer to point to these files since the application
directory name changes with each update.
"""
logger = logging.getLogger(__name__)
#Verify whether running script or frozen/deployed application
if getattr(sys, 'frozen', False):
logger.info("Verifying Application Integrity...")
#Files which should by copied to appdata directory to be easily referenced by shortcuts, etc.
data_files = ['main.ico',
'uninstall.ico',
'ReadMe.txt',
'changelog.txt',
'WhatsNew.txt',
'copyright.txt',
'Documentation.pdf']
logger.debug(" App Path: {0}".format(self._app_path))
#Get application top directory
logger.debug(" AppData Directory: {0}".format(self._appdata_path))
#Get application internal file path
for f in data_files:
a_file = f
int_path = os.path.join(self._app_path, a_file)
logger.debug(" Internal File Path: {0}".format(int_path))
#Get file's creation time
mtime_int = os.stat(int_path).st_mtime
logger.debug(" Internal File Modified Time: {0}".format(time.ctime(mtime_int)))
#Get external file path
ext_path = os.path.join(self._appdata_path, a_file)
if os.path.exists(ext_path):
mtime_ext = os.stat(ext_path).st_mtime
logger.debug(" External File Modified Time: {0}".format(time.ctime(mtime_ext)))
if mtime_int > mtime_ext:
logger.debug(" Replacing external file with new file...")
try:
os.remove(ext_path)
shutil.copy(int_path, ext_path)
except Exception, e:
logger.error(" Failed to replace the external file...", exc_info=True)
else:
logger.debug(" External file is newer than internal file - all is well.")
else:
logger.debug(" Copying file to appdata to be externally accessible")
shutil.copy(int_path, ext_path)
Also related to this, when dealing with user settings (which currently is only a history.txt file used to populate a recent files list) I have a settings folder under appdata but outside the application folder so that settings aren't lost each update. I may make similar folders for documentation and icons.
Related
I have a Tkinter app that uses images included in the same folder as the .py file. pyinstaller script.py produces an executable that runs but does not open any windows. This is because it is looking for images that don't exist in the same subdirectory. When I copy the important images to the dist folder Pyinstaller creates, the application runs correctly.
However, I would like to have a single executable that I can share with other users that doesn't also require them to have the images stored. The images should be bundled with the software somehow, like how commercial software (usually) doesn't require you to download assets separately from the program itself.
Is there a way to bundle Python programs and the assets they use into single-click applications?
Note that I am using Python 3 on Linux Mint. I am also something of a novice, so don't be surprised if I'm missing something obvious here.
It appears I've solved my own problem.
Instead of having the images included in the same folder as main.py and using the resulting short relative filepath to reach them, install the images in an appropriate space in the system directory tree (I used /home/$USER$/.$PROGRAMNAME$/) and have the program access the files using the absolute path to that directory. This will allow you to copy the program anywhere on your computer you want and have it run without a problem.
However, if you want to share it with someone else, you'll need to also include an installation script that places the assets in the correct directory on their computer.
I've built a project with Python in which a module of functions can be changed by the user. More specifically, functions can be added or deleted inside this module by other processes in the application. Now I have just converted the whole project into an executable file using auto-py-to-exe to run it through a console window, instead of running it through VS Code for instance. I can't change the module if it was not added as an additional file in auto-py-to-exe rather can the application use this module if I do add it as an additional file.
My question is: how can I turn this project into an executable with the possibility of changing this module by the program itself?
This may be tricky, but it is possible with tools like pyinstaller when you bundle your app as a directory, rather than a single file executable. The source files (albeit compiled) will be present in the directory.
In principle, you could edit files in a single-file executable, but it's probably more trouble than it's worth and may be blocked by permissions and other issues.
You could also design your software to read a file from a predefined location (or the bundle directory -- anywhere accessible by the user) and simply exec the string of the code to 'load' it. It could look more or less like an extension/scripting system for your program. One example comes to mind: the iterm2 software Python API.
It should go without saying that you must trust your users/inputs if you give them the ability to arbitrarily change the software code.
Is it possible to tell the Django development server to listen to changes on some files (not the regular Python files in the project, they already creates a reload) and reload if these files have changed?
Why do I want this? When my gulp-script builds the JS and CSS, it takes all files and appends a hash to them. My working file frontend/css/abc.css will be built to build/css/abc-hash123.css. This so that browsers are forced to reload the content if anything has changed since last deploy. When this builds, a manifest file is created that contains the mappings. In the settings in Django, I read this manifest file at start and get all these mappings (from this example here). If I change a file, it gets a new hash, and a new manifest is written. The mappings currently loaded into the memory are then wrong, and thus I need to force a reload of the development server to get the new mappings.
Clarification: I got comments on using a VCS. So just for clarification: I use a VCS and check in my abc.css. abc-hash123.css is automatically built as a part of the build system.
As far as doc goes this is not supported.
This is a bit hacky, but just touching (or changing the timestamp in any other way) some .py file or some other file that development server observes (e.g. settings.py) in your build will do the job.
Also as comments suggest, versioning is better left to a VCS.
A bit of a hacky way, but I found a way to make force the Django server to reload on changes to the build directory. After some research on the Django autoreload module, I found that it only listen to changes on Python files, from modules that are loaded into sys.modules. So, I only needed to make the build directory a Python module:
In my build script (gulp), after I've built my files, I added a step to create the __init__.py file, to make the build directory a module. Then I also wrote a comment in this file that contains a time stamp, making it unique for each build:
fs.writeFile('./build/__init__.py', '#' + Date.now())
And then in my local.py (the settings to use locally), I simply import this new module, so it appears in sys.modules:
try:
import build
except ImportError:
print("Could not import from build folder. Will not auto reload when building new files.")
I recently came across an interesting way to deploy a Python application to users on a local Windows network:
Install Python properly on user machines (the same minor version on all machines)
Create a shared network folder
Copy Python application files into the folder
Create a .bat script that tweaks the PYTHONPATH and invokes the main .py file
Copy shortcut onto each Windows desktop
User invokes the application by double-clicking the shortcut
This deployment option isn't listed as a shipping option in the Python Guide. Are there any obvious gotchas to having multiple users run the same .py files from a shared location?
(Assume that any other resource sharing is handled correctly by the application(s), e.g. they don't try to write to a shared file.)
(I'm open to suggestions on making this a more valid and answerable question.)
Your creative approach requires your PC to always be on, and the network working, and what about performance, when is it safe to update python, what about someone who wants to use your app away from your network, ...
Why not package the script into an exe? I've used py2exe http://www.py2exe.org/
This has the added advantage that it's not so easy for people to play with your script and 'adapt' it in ways that might break it. The exe is a lot bigger than the script, but we're not talking hundreds of megs. It doesn't need an installer or anything like that.
I am making an app in python, which is able to open different file types. This code in running fine on eclipse while passing filename which I want to open and configuration file as arguments respectively. Now I converted this into application by using py2app. So, the issue is how to deal with arguments, as different file types need to be open through app and this app also needs configuration file while processing. Is there any different methods available for making app which can be installed on mac osx.
Have you had a look at py2app? Its pretty much py2exe for Mac OSX and creates a standalone app. You can handle the logger and configuration files pretty easily and keep your app nice and simple to distribute.
I usually store logger files in the users home directory as a hidden file - a lot of Mac OSX applications do this - i'm not sure if its officially the way to go but it works. The home directory is guaranteed to exist and you won't get permission errors like you would when you write to other random folders or to location inside the app bundle.
You can do this and keep your code cross platform buy opening files like this
import platform
import os
if platform.system() == 'Darwin': # We're on a mac
# Store the saved rates in the users home dir - note the dot at the start
# of the filename to keep the file hidden
home = os.path.expanduser("~")
filename = os.path.join(home, '.myHiddenLogger')
else: # We're on another platform, create whatever filename you were using before
filename = 'myLogger'
As for the configuration file, this answer here tells you how to bundle a resource file in with you're app. If you want to write stuff to this configuration file and save it for next time, you'll have to store it as a hidden file like the logger - this will stop the app from crashing due to permission errors when it tried to write to a file inside the app bundle.