Is it possible to tell the Django development server to listen to changes on some files (not the regular Python files in the project, they already creates a reload) and reload if these files have changed?
Why do I want this? When my gulp-script builds the JS and CSS, it takes all files and appends a hash to them. My working file frontend/css/abc.css will be built to build/css/abc-hash123.css. This so that browsers are forced to reload the content if anything has changed since last deploy. When this builds, a manifest file is created that contains the mappings. In the settings in Django, I read this manifest file at start and get all these mappings (from this example here). If I change a file, it gets a new hash, and a new manifest is written. The mappings currently loaded into the memory are then wrong, and thus I need to force a reload of the development server to get the new mappings.
Clarification: I got comments on using a VCS. So just for clarification: I use a VCS and check in my abc.css. abc-hash123.css is automatically built as a part of the build system.
As far as doc goes this is not supported.
This is a bit hacky, but just touching (or changing the timestamp in any other way) some .py file or some other file that development server observes (e.g. settings.py) in your build will do the job.
Also as comments suggest, versioning is better left to a VCS.
A bit of a hacky way, but I found a way to make force the Django server to reload on changes to the build directory. After some research on the Django autoreload module, I found that it only listen to changes on Python files, from modules that are loaded into sys.modules. So, I only needed to make the build directory a Python module:
In my build script (gulp), after I've built my files, I added a step to create the __init__.py file, to make the build directory a module. Then I also wrote a comment in this file that contains a time stamp, making it unique for each build:
fs.writeFile('./build/__init__.py', '#' + Date.now())
And then in my local.py (the settings to use locally), I simply import this new module, so it appears in sys.modules:
try:
import build
except ImportError:
print("Could not import from build folder. Will not auto reload when building new files.")
Related
I am using uWSGI to deploy my WSGI application. Are the Python file compiled for every request, or are they precompiled once? I don't see any .pyc files.
Python caches modules bytecode - directly in the same location for python2.x, under a __pycache__ subfolder for python3 - but scripts (the difference is in usage - if you import it it's a module, if you execute it it's a script) are always recompiled (which is why main scripts are usually very very short an simple).
IOW, your main wsgi script will be recompiled once for each new server process. Typically a wsgi app is served as a long running process which will handle much more than one single request, so even then the initial compilation overhead is really not an issue (short script + only compiled once per process)...
Also, once a Python process is started, imported modules are cached in memory so they are only really imported (loaded) once per process.
Just note that the user under which the process is running must have write permissions on your app's directory in order to create the .pyc files... and of course read permissions on the .pyc files too.
If you're using CPython then by default, no it does not compile every time a request is received unless you manually configure it to do this.
By the first request, the uWSGI will load the python scripts bytecode and will reload it in 2 different scenarios I know of:
There is a new import that has not been loaded before (only the new module will be parsed and converted to bytecode).
You explicitly ran service uwsgi restart
Yet, there is still a way to keep reloading python scripts in every request by suppressing it by the interpreter, example: python -B my_amazing_view.py
For more details check here:
What is pycache
Should I generate .pyc
Description
Normally if you change your python code means, you need to restart the server in order to apply the new changes.
If the --auto-reload parameter is enabled means, you don't need to restart the server. It enables auto-reloading of python files and xml files without having to restart the server. It requires pyinotify. It is a Python module for monitoring filesystems changes.
Previous Problem
I got the error:
ERROR ? pyinotify: add_watch: cannot watch /home/user/.local/share/Odoo/addons/8.0 WD=-1, Errno=No space left on device (ENOSPC)
But I followed the advice of this link and now I don't get that error anymore:
sysctl -n -w fs.inotify.max_user_watches=16384
After this I got this in the server log:
openerp.service.server: Watching addons folder /opt/odoo_8/src/linked-addons
openerp.service.server: AutoReload watcher running
That means that's working properly. And in fact I tested it with a physical addon path and it worked.
Current Problem
I have all my modules en several folders but I only use one addons path: /opt/odoo_8/src/linked-addons. This folder contains all the links of the modules that I'm using. All modules are working well when I run Odoo.
But the problem is that pyinotify is not able to check the files beyond the links and it doesn't reload the files well. What I should do to fix this? Is there a way that pyinotify can recognise the content of the links?
PD: I don't want to change my way of managing the modules folders in Odoo.
The only solution I found is to isolate the custom modules in a different folder. I have added the new folder to the addons path in the configuration file and now everything works fine
addons_path = /opt/odoo_8/src/linked-addons,/opt/odoo_8/src/custom
PD: I saved the module folders directly in the custom folder instead of saving the links to the folders.
My question is essentially, "How should I structure the files and folders of my frozen, deployed Python-based Windows application." To understand my situation, here's some background:
I'm building a desktop application with Python 2.7 for my workplace. It is a GUI-based application built on PyQt. I am building the application with Esky which is a cross-platform freezing and updating framework. Esky basically wraps/calls py2exe, py2app, bb_freeze, or whatever tool you have installed that is appropriate for the current platform. Esky creates a zipped package that looks like this:
prog.exe - esky bootstrapping executable
appdata/ - container for all the esky magic
appname-X.Y.platform/ - specific version of the application
prog.exe - executable(s) as produced by freezer module
library.zip - pure-python frozen modules
pythonXY.dll - python DLL
esky-files/ - esky control files
bootstrap/ - files not yet moved into bootstrapping env
bootstrap-manifest.txt - list of files expected in bootstrap env
lockfile.txt - lockfile to block removal of in-use versions
...other deps...
updates/ - work area for fetching/unpacking updates
These zipped packages can then be placed on a file server which Esky looks to for updates. A number of methods are provided for managing updates including a very simple auto_update(). When an update occurs, the appname-X.Y.platform folder is essentially replaced with the next version folder... so the myApp.0.1.win32 folder is replaced by a myApp.0.2.win32 folder.
The other aspect of background you should know is that I am distributing the application to my coworkers, who do not have Python installed. I'm not distributing a Python package or library, I'm deploying a desktop application (my coworkers don't particularly care what it's written in, just that it works). I've built an Inno installer which installs the application, provides an uninstaller, and various shortcuts. Because everyone on the team has essentially the same Windows 7 64-bit environment, I'm pretty safe building for just that platform.
So, back to the issue of structure. I've read guides that recommend a certain format for a project skeleton, such as Learn Python the Hard Way, Exercise 46 or the Hitchhiker's Guide to Packaging. However these guides are oriented toward Python package developers, not compiled application developers.
I've also run into problems with Esky's appname-X.Y.platform folder, since it changes names every time the program is updated (to reflect the version number). Because I want some shortcuts in the Start Menu to always refer to documentation, changelog, etc, I have the installer place some of those files under the appdata folder. When the program updates, I have some code to check for newer versions of those files I want to be externally "visible" and copy the newer versions out of the appname-X.Y.platform folder and overwrite the copies in the appdata folder. I then also needed a means of storing persistent user settings, so the program generates and uses an appdata\settings folder (otherwise the settings would be wiped with each update).
Should I continue the process of having the application push new files out to the appdata folder post-update? Should I build my own structure of Docs, Examples, Settings, etc. and let the program populate those folders with newer files whenever necessary? Should I attempt to alter or take better advantage of Esky's behavior to better fit my usage? Perhaps I should rework my application to be destributable as both a Python package and an end-user application?
This question relates to this one about static files with Esky, this one about Python deployed application structure, and numerous generic questions about Python project structure which don't specifically address using Esky. Some videos discussing Esky are also available here and here.
I'm seeking recommendations for "best practice" methods to handle these challenges. If this doesn't fit the StackOverflow Question format, I'll gladly attempt to reword or narrow the focus of my question.
So here's my solution to the problem mentioned about about making files available to shortcuts at a static location despite the fact that Esky's auto-updating changes the name of my application folder every update. The function below I have within a class definition for a QMainWindow.
Logging statements could be replaced with print statements if your application doesn't use the logging module, though I highly recommend logging, especially if deploying a standalone application like this.
import os
import shutil
import logging
def push_updated_files(self):
"""
Manually push auto-updated files from the application folder up to the appdata folder
This enables shortcuts and other features on the computer to point to these files since the application
directory name changes with each update.
"""
logger = logging.getLogger(__name__)
#Verify whether running script or frozen/deployed application
if getattr(sys, 'frozen', False):
logger.info("Verifying Application Integrity...")
#Files which should by copied to appdata directory to be easily referenced by shortcuts, etc.
data_files = ['main.ico',
'uninstall.ico',
'ReadMe.txt',
'changelog.txt',
'WhatsNew.txt',
'copyright.txt',
'Documentation.pdf']
logger.debug(" App Path: {0}".format(self._app_path))
#Get application top directory
logger.debug(" AppData Directory: {0}".format(self._appdata_path))
#Get application internal file path
for f in data_files:
a_file = f
int_path = os.path.join(self._app_path, a_file)
logger.debug(" Internal File Path: {0}".format(int_path))
#Get file's creation time
mtime_int = os.stat(int_path).st_mtime
logger.debug(" Internal File Modified Time: {0}".format(time.ctime(mtime_int)))
#Get external file path
ext_path = os.path.join(self._appdata_path, a_file)
if os.path.exists(ext_path):
mtime_ext = os.stat(ext_path).st_mtime
logger.debug(" External File Modified Time: {0}".format(time.ctime(mtime_ext)))
if mtime_int > mtime_ext:
logger.debug(" Replacing external file with new file...")
try:
os.remove(ext_path)
shutil.copy(int_path, ext_path)
except Exception, e:
logger.error(" Failed to replace the external file...", exc_info=True)
else:
logger.debug(" External file is newer than internal file - all is well.")
else:
logger.debug(" Copying file to appdata to be externally accessible")
shutil.copy(int_path, ext_path)
Also related to this, when dealing with user settings (which currently is only a history.txt file used to populate a recent files list) I have a settings folder under appdata but outside the application folder so that settings aren't lost each update. I may make similar folders for documentation and icons.
I'm working on a really simple python package for our internal use, and want to package it as a .egg file, and when it's installed/used I want it to access a text file that is placed in an appropriate place on the computer.
So where is the best place to put application data in python? (that is meant to be edited by users) How do I get my python package to automatically install a default file there?
note: I know about the Windows application data directory, but would like to avoid it, as it's nonportable and kind of cumbersome both for users to access and for my application to access.
os.path.expanduser('~')
Each OS will have it's own directory where application data is expected to exist. There does not appear to be a method that provides this path in a platform-independent manner. You can write your own function to do this for you by checking os.name and then returning the appropriate value depending on the result.
i have a following solution structure in python:
main_app
main_app/template_processor/
main_app/template_processor/models
main_app/template_processor/views
everything works just fine on my local machine. as soon as code gets to server (it stopped working after i removed all .pyc files from svn), it doesn't see the assembly (if it could be called like that in terms of python) with models. during syncdb command, it creates no tables except for admin ones. but in runtime it finds models themselves, but does not find tables (since they were not created by syncdb)
i've added application to settings.py as installed app, everything seems to be fine.
the only difference that i can see for now is that on my local machine i have main_app/template_processor/models/models.pyc file, but it doesn't precompile it on server for some reason (might be a hint??)
__init__.py files are existing in every folder/subfolder.
have anyone faced an issue like that?
Hm, looking at your directory structure, I think that you need to import all your models in template_processor/models/__init__.py, because Django looks only in <app_name>.models module when loading models automatically (i.e. for syncdb).
My guess would be that you renamed a file, and didn't delete the oldname.pyc file.
So if you try to
import oldname
then you rename oldname to rename, but don't update your import statement, the code will work on systems where oldname.pyc exists, however python won't be able to recreate oldname.pyc if oldname.py doesn't exist.
try
find . | grep py | xargs grep import | grep -v django | sort -u
that should give you a list of all imports in your project, see if one of those imports is pointing at a module for which you have a pyc file but not a .py file.
In general, python quickly compiles .py files into .pyc files, and it does this once. I wouldn't worry about the time it takes to generate new .pyc files.
Sounds like Django isn't seeing that module (folder, in this case) for some reason. Make sure all the folders have a file called __init__.py (notice the two underscores before and after). Once that's done, make sure it's listed in your installed apps.
Maybe you made some change to it that's causing it to stop loading. You can also try moving the .pyc files out of the directory on your local machine and see whether they're regenerated or not when you runserver.
Maybe the most helpful thing: ./manage.py shell to bring up an interactive shell and then 'import app_name' (without quotes) to see whether django is having trouble finding the module.
Ok, if anyone's interested in what really was happening, i've got a story to tell ya:
http://code.djangoproject.com/ticket/4470
that's basically what i was going to implement.
in order to really get this thing work i still should have a file models.py, which will have a proper list of classess, with Pass inside of it. Then i should have taken all the files (my models) and changed their meta for syncdb to understand they are from the certain "assembly".
source code is available (pls see url above).
thx for helping out!