One of the things I really like about django is the way the server automatically resets when you edit the project. I've recently started doing some development in twisted/cyclone.
Is there a similar way to make twistd reset automatically when a program file is changed?
You can use inotify - it enables monitoring filesystem events, such as file modification. There are also python bindings: pyinotify
There are many tutorials there, my suggestion is that you implement a wrapper that starts twisted monitors you source path. When a file modification occurs you restart twisted.
I use my own Pyquitter, which polls the source files for every module that the process has imported. Check out the 'Sample use' in the README, which covers how to use it with Twisted.
Related
I'm trying to watch for changes in .py files, in a directory.
I went through existing solution.
I'm curious on how django library solves this problem. The development server is restarted on file changes.
The code can be found in django.utils.autoreload. The autoreloader uses a separate thread that watches any python module that has been imported, and any translation file.
If inotify is available, Django uses that to listen to change events. Otherwise, it checks the timestamps of every file every second. If there are any changes, the process is restarted.
Django's autoreloader may not be the best source of inspiration. Better options may be Watchman (with the appropriate python bindings) or the pure-python alternative Watchdog.
Fast forward to April 2019:
With django 2.2 pywatchman as part of Watchman will be supported and pyinotify (being unmaintained since mid 2015) is dropped:
If you're using Linux or MacOS and install both pywatchman and the
Watchman service, kernel signals will be used to autoreload the server
(rather than polling file modification timestamps each second). This offers
better performance on large projects, reduced response time after code changes,
more robust change detection, and a reduction in power usage.
source: django-admin
When using Watchman with a project that includes large non-Python
directories like node_modules, it's advisable to ignore this directory
for optimal performance.
See the watchman documentation for information on how to do this.
I have a python server that I need to run in both a Linux and Windows environment, and my question is about deployment. What is the best method for deploying the solution instead of just double clicking on the file and running it?
Since I use the server_forever() on the server, I can just run the script from command line, but this keeps the python window open. If I log off the machine, naturally the process will stop. So what is the best method for deploying a python script that needs to keep running if the user is logged in or off a machine.
Since I am going to be using multiple environment, Linux and Windows, can you please be specific in what OS you are talking about?
For windows, I was thinking of running the script 'At Startup' using the Windows scheduler. But I wanted to see if anyone had a better option. For linux, I really don't know what to create. I am assuming a CRON job?
Deployment does refer to coding, so using serve_forever() on a multiprocessing job manager keeps the python window open upon execution. Is there a way to hide this window through code? Would you recommend using a conversion tool like py2exe instead?
This is the subject matter of a whole library of books, so I will just give an introduction here :-)
You can basically start scripts directly and then have multiple options to do this in a way that they keep running in the background.
If you have certain functionality that needs to run on regular moments, you would do this by scheduling it:
Windows: Windows Scheduler or specific scheduling tools
Linux: Cron
If your problem is that you want to start a script without it closing on you while SSH'ing into Linux, you want to look into the "screen" or "tmux" tools.
If you want to have it started automatically this could be done by using the "At Startup" as you point out and Linux has similar functionalities, but the preferred and more robust way would be to set up a service that is better integrated with the OS.
Windows: Windows Service
Linux: Daemon
Even more capabilities can be yielded by using an application server such a Django
Tomcat (see comment) is an option, but definitely not the standard one; you'll have a hard time finding support both from Tomcat people running Python or Python people running their stuff on Tomcat. That being said, I imagine you could enable CGI and have it run a Python command with your script.
Yet, instead of just starting a Python script I would strongly encourage you to have a look at different Python options that are probably available for your specific use case. From lightweight web solutions like Flask over a versatile networking engine like Twisted to a full blown web framework like Django.
They all have rather well-thought-out deployment solutions available. Look up WSGI for more background.
There is a quite similar question here: What's the difference between a twistd plugin and a twistd service?
And the answer is:
A plugin is nicer in that you can have command-line options
When i started learning twisted i was working with an existing application, and i was confused where is the entry point, which was somewhere in twisted/plugins folder.
From then my preferred way is using tac files in the top folder, but now i got to a point when i need to pass some command line arguments to my script. And i am told to use twisted application plugins.
I am confused by the term plugin in twisted. For me it means somethings that changes an application in a seamless way - you don't really have to know they exist - they just 'plug in' into your application changing its behavior.
But i cannot understand the conceptual difference between twisted applications and twisted application plugins. For me - they serve the same purpose, but are given different features - why?
When should i use twisted applications and when plugins?
Plugins in twisted only adds commands and/or options to twistd script. They don't mean anything more.
So well yes, there are two ways to write a startup script for your application, one is using .tac file and one is to add a command (through plugin) to twistd.
I think .tac file is easier to write.
I think it is not a wrong thing to use both: plugins and .tac files.
There is also a third way: write your own startup script instead of twistd.
But i cannot understand the conceptual difference between twisted
applications and twisted application plugins. For me - they serve the
same purpose, but are given different features - why?
Well no. They don't serve same purpose. Twisted application is just a .tac file that can be started with twistd script. It is more like a config file. Config file with python syntax. It's purpose is that you don't need to write your own start-up script. But if twistd does not provide enough options for you, you can write a plugin for it. So the purpose for plugins is to extend twistd.
And if I had to publish a stand alone application I would write my own startup script and would not use twistd script. twistd is ok if users of the application are familiar with twisted and or has more twisted apps. I think it is only a burden to user to see all the different options of twistd and it is very frustrating not to be able to start application without referring to documentation.
For example scrapy does that: it provides a script scrapyd for users who are not familiar with twisted.
We currently run a small shared hosting service for a couple of hundred small PHP sites on our servers. We'd like to offer Python support too, but from our initial research at least, a server restart seems to be required after each source code change.
Is this really the case? If so, we're just not going to be able to offer Python hosting support. Giving our clients the ability to upload files is easy, but we can't have them restart the (shared) server process!
PHP is easy -- you upload a new version of a file, the new version is run.
I've a lot of respect for the Python language and community, so find it hard to believe that it really requires such a crazy process to update a site's code. Please tell me I'm wrong! :-)
Python is a compiled language; the compiled byte code is cached by the Python process for later use, to improve performance. PHP, by default, is interpreted. It's a tradeoff between usability and speed.
If you're using a standard WSGI module, such as Apache's mod_wsgi, then you don't have to restart the server -- just touch the .wsgi file and the code will be reloaded. If you're using some weird server which doesn't support WSGI, you're sort of on your own usability-wise.
Depends on how you deploy the Python application. If it is as a pure Python CGI script, no restarts are necessary (not advised at all though, because it will be super slow). If you are using modwsgi in Apache, there are valid ways of reloading the source. modpython apparently has some support and accompanying issues for module reloading.
There are ways other than Apache to host Python application, including the CherryPy server, Paste Server, Zope, Twisted, and Tornado.
However, unless you have a specific reason not to use it (an since you are coming from presumably an Apache/PHP shop), I would highly recommed mod_wsgi on Apache. I know that Django recommends modwsgi on Apache and most of the other major Python frameworks will work on modwsgi.
Is this really the case?
It Depends. Code reloading is highly specific to the hosting solution. Most servers provide some way to automatically reload the WSGI script itself, but there's no standardisation; indeed, the question of how a WSGI Application object is connected to a web server at all differs widely across varying hosting environments. (You can just about make a single script file that works as deployment glue for CGI, mod_wsgi, passenger and ISAPI_WSGI, but it's not wholly trivial.)
What Python really struggles with, though, is module reloading. Which is problematic for WSGI applications because any non-trivial webapp will be encapsulating its functionality into modules and packages rather than simple standalone scripts. It turns out reloading modules is quite tricky, because if you reload() them one by one they can easily end up with bad references to old versions. Ideally the way forward would be to reload the whole Python interpreter when any file is updated, but in practice it seems some C extensions seem not to like this so it isn't generally done.
There are workarounds to reload a group of modules at once which can reliably update an application when one of its modules is touched. I use a deployment module that does this (which I haven't got around to publishing, but can chuck you a copy if you're interested) and it works great for my own webapps. But you do need a little discipline to make sure you don't accidentally start leaving references to your old modules' objects in other modules you aren't reloading; if you're talking loads of sites written by third parties whose code may be leaky, this might not be ideal.
In that case you might want to look at something like running mod_wsgi in daemon mode with an application group for each party and process-level reloading, and touch the WSGI script file when you've updated any of the modules.
You're right to complain; this (and many other WSGI deployment issues) could do with some standardisation help.
Well I want to use WEb2Py because it's pretty nice..
I just need to change the working directory to the directory where all my modules/libraries/apps are so i can use them. I want to be able to import my real program when I use the web2py interface/applications. I need to do this instead of putting all my apps and stuff inside the Web2Py folder... I'm trying to give my program a web frontend without putting the program in the Web2Py folder.. Sorry if this is hard to understand .
In any multi-threaded Python program (and not only Python) you should not use os.chdir and you should not change sys.path when you have more than one thread running. It is not safe because it affects other threads. Moreover you should not sys.path.append() in a loop because it may explode.
All web frameworks are multi-threaded and requests are executed in a loop. Some web frameworks do not allow you to install/un-install applications without restarting the web server and therefore IF os.chdir/sys.path.append are only executed at startup then there is no problem.
In web2py we want to be able to install/uninstall applications without restarting the web server. We want apps to be very dynamical (for example define models based on information provided with the http request). We want each app to have its own models folder and we want complete separation between apps so that if two apps need to different versions of the same module, they do not conflict with each other, so we provide APIs to do so (request.folder, local_import).
You can still use the normal os.chdir and sys.path.append but you should do it outside threads (and this is not a web2py specific issue). You can use import anywhere you like as you would in any other Python program.
I strongly suggest moving this discussion to the web2py mailing list.
os.chdir lets you change the OS's working directory, but for your purposes (enabling imports of a bunch of modules &c which are constrained to live in some strange place) it seems better to add the needed directories to sys.path instead.
I had to do this very thing. I have few modules that I wanted to use from my controllers. If you want to be able to use the code that resides in the modules directory in the controller, you can use:
# use this in your controller code
impname = local_import('module_in_modules', reload=True)
# reload true will ensure that it will re load whenever
# there are changes to the module
Jay