I'm trying to watch for changes in .py files, in a directory.
I went through existing solution.
I'm curious on how django library solves this problem. The development server is restarted on file changes.
The code can be found in django.utils.autoreload. The autoreloader uses a separate thread that watches any python module that has been imported, and any translation file.
If inotify is available, Django uses that to listen to change events. Otherwise, it checks the timestamps of every file every second. If there are any changes, the process is restarted.
Django's autoreloader may not be the best source of inspiration. Better options may be Watchman (with the appropriate python bindings) or the pure-python alternative Watchdog.
Fast forward to April 2019:
With django 2.2 pywatchman as part of Watchman will be supported and pyinotify (being unmaintained since mid 2015) is dropped:
If you're using Linux or MacOS and install both pywatchman and the
Watchman service, kernel signals will be used to autoreload the server
(rather than polling file modification timestamps each second). This offers
better performance on large projects, reduced response time after code changes,
more robust change detection, and a reduction in power usage.
source: django-admin
When using Watchman with a project that includes large non-Python
directories like node_modules, it's advisable to ignore this directory
for optimal performance.
See the watchman documentation for information on how to do this.
Related
The Current State:
I have some non-negligeble amount of microservices written in python.
Each such microservice has its own yaml configuration file that is located in the git repo. We use dynaconf to read the configuraion.
The Problem:
At first it was fine, the configurations were relatively small and it was easy to maintain them. Time went by, and the configurations went larger. It became annoying to change the configurations and it is bad that they are scattered between different git repos, i.e. not centralized.
I want to use "Externalized Configurations" in order to maintain all the configurations in a single repo and that each microservice will read its portion on startup. I have heard about Spring Boot, but it seems to be way too much and apart from it, it seems that the pip libraries seems to be at beta stage, new and unriliable...
Is there another reccomendation in this particular use case? Or should I proceed with Spring Boot?
You can use Microconfig.IO to manage your configuration with powerful templating and distribution. It's language agnostic as long as your configs are YAML or properties format.
I think you could create a default config file for each service and use the external storage idea of the dynaconf.
One possible solution would be to create a simple system to manage these variables with Redis.
And the DynaConf CLI allows you to make changes on the fly.
We have few Zope&Plone projects in our company and until today I was only one single developer developing all changes throught ZMI or ZopeEdit. Our company is growing so I need to start cooperating with others developers which can help me with developing features and solving bugs in projects. This means that is no more possible to use ZMI but every developer needs to make and test own changes without affecting others work and paste own changes to production enviroment using git merge in git repo.
I need to move development to git - this means I need to start tracking all portals files and settings in git.
I think I need to move whole projects from ZODB/ZMI (including templates, scripts, sql methods, properties as portal_properties or portal_javascripts etc.) to filesystem and run git on this file system. In the next step every developer can install own pure Plone instance, pull source code and settings from git, create own branch, make changes, test, commit, push, code review ...
My question is: Is there any way to do this and start well-known rapid development process using git? Supports ZODB something like "live migration" of content/settings to/from filesystem? Is there any way to tel Zope to load some folder with content/settings from filesystem instead only from ZODB?
I know there is something called eggs, but is possible to move all types of files mentioned above to separated egg?
Thank you for your help.
The way your company was following until now was the "Old Way Way" of Plone development, but this was a deprecated and discouraged way to do.
Nowadays ZMI can still be used for "quick and dirty" fixes, but commonly this changes stored in DB must be removed (and moved to real code) as soon as possible. This was already possible on Plone 2.0!
More important: every new Plone release tend to reduce the ZMI powers (for example: until Plone 2.1 you were able to do lot of stuff from ZMI, starting from Plone 2.5 some UI elements where impossible to be modified TTW).
So: the answer to your question is "yes". Plone can (must) read code from filesystem, and this code can be stored on VCS (it can be git svn, ...).
All of those information can be found in the Plone Developer Manual.
Creating a new Plone package for modern Plone? Use mr.bob.
Automatically integrate VCS in your buildout? Use mr.developer
If you are starting today from a project you where developing through ZMI you must probably mode code from ZMI to filesystem.
This can be done manually; it's simpler as you are using Zope External Editor.
There is also a very old add-on (Plone Skin Dump) for flushing skin content to filesystem, but I fear it won't work on recent Plone, thus it was not supporting some stuff like SQL methods (if you are using them).
You can have a look at http://docs.plone.org/develop/
There you can find how to create a package (egg). Its source code can be added to git. You can checkout your git repository using mr.developer during buildout. https://pypi.python.org/pypi/mr.developer/
You can use mr.bob to create a PloneAddon. For MySQL you can use MySQL Python. Add this Packages to your Buildout in the eggs section.
Then you can write your own MySQL Statements in the Addon. I know reimplementation is expensive, but you have more control in the future.
One of the things I really like about django is the way the server automatically resets when you edit the project. I've recently started doing some development in twisted/cyclone.
Is there a similar way to make twistd reset automatically when a program file is changed?
You can use inotify - it enables monitoring filesystem events, such as file modification. There are also python bindings: pyinotify
There are many tutorials there, my suggestion is that you implement a wrapper that starts twisted monitors you source path. When a file modification occurs you restart twisted.
I use my own Pyquitter, which polls the source files for every module that the process has imported. Check out the 'Sample use' in the README, which covers how to use it with Twisted.
I'm trying to incorporate server-based code diff and highlighting in my GWT (Java) project. I managed to incorporate Pygments and difflib into my code using Jython. The basic idea is to generate complete markup on the server and then simply inject code into the page as innerHTML.
I found Jython completely inadequate as even for relatively small files (2K-3K lines) it takes Pygments or difflib forever (minutes not seconds) to process these files. Difflib actually reliably causes OOM errors in the process with dedicated 500M of memory
So I'm wondering if my current setup is wrong or Jython is simply unsuitable for this purpose?
If so, what's next? I discover Jepp but then I would have to build my project for each platform and it has little documentation and don't seem very stable. Another possibility would be to run Pylons as a separate webservice on the same host and get the markup directly to client or channel it through server. And yet another way is to use Java System to execute python script as a process and capture the output.
I would be very interested to hear solid suggestion on the matter.
Having a separate service sounds like the best way to go. For Pygments, there is already a service available (on Google App Engine). The source for the app is BSD open source and on GitHub here. You could adapt this to add difflib functionality too, of course.
I'm going to accept answer above since it coincides with my findings but just to let anyone who reads this know - running separate webservice for Pygments using Python-native solution such as Bottle performs many times better than embedded Jython. Especially on Linux
We currently run a small shared hosting service for a couple of hundred small PHP sites on our servers. We'd like to offer Python support too, but from our initial research at least, a server restart seems to be required after each source code change.
Is this really the case? If so, we're just not going to be able to offer Python hosting support. Giving our clients the ability to upload files is easy, but we can't have them restart the (shared) server process!
PHP is easy -- you upload a new version of a file, the new version is run.
I've a lot of respect for the Python language and community, so find it hard to believe that it really requires such a crazy process to update a site's code. Please tell me I'm wrong! :-)
Python is a compiled language; the compiled byte code is cached by the Python process for later use, to improve performance. PHP, by default, is interpreted. It's a tradeoff between usability and speed.
If you're using a standard WSGI module, such as Apache's mod_wsgi, then you don't have to restart the server -- just touch the .wsgi file and the code will be reloaded. If you're using some weird server which doesn't support WSGI, you're sort of on your own usability-wise.
Depends on how you deploy the Python application. If it is as a pure Python CGI script, no restarts are necessary (not advised at all though, because it will be super slow). If you are using modwsgi in Apache, there are valid ways of reloading the source. modpython apparently has some support and accompanying issues for module reloading.
There are ways other than Apache to host Python application, including the CherryPy server, Paste Server, Zope, Twisted, and Tornado.
However, unless you have a specific reason not to use it (an since you are coming from presumably an Apache/PHP shop), I would highly recommed mod_wsgi on Apache. I know that Django recommends modwsgi on Apache and most of the other major Python frameworks will work on modwsgi.
Is this really the case?
It Depends. Code reloading is highly specific to the hosting solution. Most servers provide some way to automatically reload the WSGI script itself, but there's no standardisation; indeed, the question of how a WSGI Application object is connected to a web server at all differs widely across varying hosting environments. (You can just about make a single script file that works as deployment glue for CGI, mod_wsgi, passenger and ISAPI_WSGI, but it's not wholly trivial.)
What Python really struggles with, though, is module reloading. Which is problematic for WSGI applications because any non-trivial webapp will be encapsulating its functionality into modules and packages rather than simple standalone scripts. It turns out reloading modules is quite tricky, because if you reload() them one by one they can easily end up with bad references to old versions. Ideally the way forward would be to reload the whole Python interpreter when any file is updated, but in practice it seems some C extensions seem not to like this so it isn't generally done.
There are workarounds to reload a group of modules at once which can reliably update an application when one of its modules is touched. I use a deployment module that does this (which I haven't got around to publishing, but can chuck you a copy if you're interested) and it works great for my own webapps. But you do need a little discipline to make sure you don't accidentally start leaving references to your old modules' objects in other modules you aren't reloading; if you're talking loads of sites written by third parties whose code may be leaky, this might not be ideal.
In that case you might want to look at something like running mod_wsgi in daemon mode with an application group for each party and process-level reloading, and touch the WSGI script file when you've updated any of the modules.
You're right to complain; this (and many other WSGI deployment issues) could do with some standardisation help.