There is a quite similar question here: What's the difference between a twistd plugin and a twistd service?
And the answer is:
A plugin is nicer in that you can have command-line options
When i started learning twisted i was working with an existing application, and i was confused where is the entry point, which was somewhere in twisted/plugins folder.
From then my preferred way is using tac files in the top folder, but now i got to a point when i need to pass some command line arguments to my script. And i am told to use twisted application plugins.
I am confused by the term plugin in twisted. For me it means somethings that changes an application in a seamless way - you don't really have to know they exist - they just 'plug in' into your application changing its behavior.
But i cannot understand the conceptual difference between twisted applications and twisted application plugins. For me - they serve the same purpose, but are given different features - why?
When should i use twisted applications and when plugins?
Plugins in twisted only adds commands and/or options to twistd script. They don't mean anything more.
So well yes, there are two ways to write a startup script for your application, one is using .tac file and one is to add a command (through plugin) to twistd.
I think .tac file is easier to write.
I think it is not a wrong thing to use both: plugins and .tac files.
There is also a third way: write your own startup script instead of twistd.
But i cannot understand the conceptual difference between twisted
applications and twisted application plugins. For me - they serve the
same purpose, but are given different features - why?
Well no. They don't serve same purpose. Twisted application is just a .tac file that can be started with twistd script. It is more like a config file. Config file with python syntax. It's purpose is that you don't need to write your own start-up script. But if twistd does not provide enough options for you, you can write a plugin for it. So the purpose for plugins is to extend twistd.
And if I had to publish a stand alone application I would write my own startup script and would not use twistd script. twistd is ok if users of the application are familiar with twisted and or has more twisted apps. I think it is only a burden to user to see all the different options of twistd and it is very frustrating not to be able to start application without referring to documentation.
For example scrapy does that: it provides a script scrapyd for users who are not familiar with twisted.
Related
I have a python server that I need to run in both a Linux and Windows environment, and my question is about deployment. What is the best method for deploying the solution instead of just double clicking on the file and running it?
Since I use the server_forever() on the server, I can just run the script from command line, but this keeps the python window open. If I log off the machine, naturally the process will stop. So what is the best method for deploying a python script that needs to keep running if the user is logged in or off a machine.
Since I am going to be using multiple environment, Linux and Windows, can you please be specific in what OS you are talking about?
For windows, I was thinking of running the script 'At Startup' using the Windows scheduler. But I wanted to see if anyone had a better option. For linux, I really don't know what to create. I am assuming a CRON job?
Deployment does refer to coding, so using serve_forever() on a multiprocessing job manager keeps the python window open upon execution. Is there a way to hide this window through code? Would you recommend using a conversion tool like py2exe instead?
This is the subject matter of a whole library of books, so I will just give an introduction here :-)
You can basically start scripts directly and then have multiple options to do this in a way that they keep running in the background.
If you have certain functionality that needs to run on regular moments, you would do this by scheduling it:
Windows: Windows Scheduler or specific scheduling tools
Linux: Cron
If your problem is that you want to start a script without it closing on you while SSH'ing into Linux, you want to look into the "screen" or "tmux" tools.
If you want to have it started automatically this could be done by using the "At Startup" as you point out and Linux has similar functionalities, but the preferred and more robust way would be to set up a service that is better integrated with the OS.
Windows: Windows Service
Linux: Daemon
Even more capabilities can be yielded by using an application server such a Django
Tomcat (see comment) is an option, but definitely not the standard one; you'll have a hard time finding support both from Tomcat people running Python or Python people running their stuff on Tomcat. That being said, I imagine you could enable CGI and have it run a Python command with your script.
Yet, instead of just starting a Python script I would strongly encourage you to have a look at different Python options that are probably available for your specific use case. From lightweight web solutions like Flask over a versatile networking engine like Twisted to a full blown web framework like Django.
They all have rather well-thought-out deployment solutions available. Look up WSGI for more background.
One of the things I really like about django is the way the server automatically resets when you edit the project. I've recently started doing some development in twisted/cyclone.
Is there a similar way to make twistd reset automatically when a program file is changed?
You can use inotify - it enables monitoring filesystem events, such as file modification. There are also python bindings: pyinotify
There are many tutorials there, my suggestion is that you implement a wrapper that starts twisted monitors you source path. When a file modification occurs you restart twisted.
I use my own Pyquitter, which polls the source files for every module that the process has imported. Check out the 'Sample use' in the README, which covers how to use it with Twisted.
For very simple, internal web-apps using ASP I was able to just switch IIS 'on' and then write some ASP scripts in the www directory that would start working immediately.
Is there an equivalent webserver app for Python scripts that I can run that will automatically start serving dynamic pages (python scripts) in a certain folder (with virtually no configuration)?
Solutions I've already found are either too limited (e.g. SimpleHTTPRequestHandler doesn't serve dynamic content) or require configuring the script that does the serving.
There's always CGI. Add a script mapping of .py to "C:\Python27\python.exe" -u "%s" then drop .py files in a folder and IIS will execute them.
I'd not generally recommend it for real work—in the longer term you would definitely want to write apps to WSGI, and then deploy them through any number of interfaces including CGI—but it can be handy for quick prototyping.
For development or just to play around, here's an example using the standard Python library that I have used to help friend who wanted to get a basic CGI server up and running. It will serve python scripts from cgi-bin and files from the root folder. I'm not near a Windows computer at the moment to make sure that this still works. This also assumes Python2.x. Python 3.x has this, it's just not named the same.
Make a directory on your harddrive with a cgi-bin folder in it (Ex. "C:\server\cgi-bin")
In a command window, navigate to "C:\server" directory
Type the following assuming you've installed python 2.7 in C:\Python27:
"c:\python27\python.exe -m CGIHTTPServer"
You should get a message like "Serving HTTP on 0.0.0.0 port 8000"
Linux is the same - "python -m CGIHTTPServer" in a directory with a cgi-bin/ in it.
WSGI setups are fairly easy to get started, but in no anyway turn key. django MVC has a simple built in development server if you plan on using a more comprehensive framework.
My limited experience with Python web frameworks has taught me that most go to one extreme or the other: Django on one end is a full-stack MVC framework, that will do pretty much everything for you. On the other end, there are Flask, web.py, CherryPy, etc., which do much less, but stay out of your way.
CherryPy, for example, not only comes with no ORM, and doesn't require MVC, but it doesn't even have a templating engine. So unless you use it with something like Cheetah, you can't write what would look like .asp at all.
I received a project recently and I am wondering how to do something in a correct and secure manner.
The situation is the following:
There are classes to manage linux users, mysql users and databases and apache virtual hosts. They're used to automate the addition of users in a small shared-hosting environnement. These classes are then used in command-line scripts to offer a nice interface for the system administrator.
I am now asked to build a simple web interface to offer a GUI to the administrator and then offer some features directly to the users (change their unix password and other daily procedures).
I don't know how to implement the web application. It will run in Apache (with the apache user) but the classes need to access files and commands that are only usable by the root user to do the necessary changes (e.g useradd and virtual hosts configuration files). When using the command-line scripts, it is not a problem as they are run under the correct user. Giving permissions to the apache user would probably be dangerous.
What would be the best technique to allow this through the web application ? I would like to use the classes directly if possible (it would be handier than calling the command line scripts like external processes and parsing output) but I can't see how to do this in a secure manner.
I saw existing products doing similar things (webmin, eBox, ...) but I don't know how it works.
PS: The classes I received are simple but really badly programmed and barely commented. They are actually in PHP but I'm planning to port them to python. Then I'd like to use the Django framework to build the web admin interface.
Thanks and sorry if the question is not clear enough.
EDIT: I read a little bit about webmin and saw that it uses its own mini web server (called miniserv.pl). It seems like a good solution. The user running this server should then have permissions to modify the files and use the commands. How could I do something similar with Django? Use the development server? Would it be better to use something like CherryPy?
Hello
You can easily create web applications in Python using WSGI-compliant web frameworks such as CherryPy2 and templating engines such as Genshi. You can use the 'subprocess' module to manadge external commands...
You can use sudo to give the apache user root permission for only the commands/scripts you need for your web app.
We currently run a small shared hosting service for a couple of hundred small PHP sites on our servers. We'd like to offer Python support too, but from our initial research at least, a server restart seems to be required after each source code change.
Is this really the case? If so, we're just not going to be able to offer Python hosting support. Giving our clients the ability to upload files is easy, but we can't have them restart the (shared) server process!
PHP is easy -- you upload a new version of a file, the new version is run.
I've a lot of respect for the Python language and community, so find it hard to believe that it really requires such a crazy process to update a site's code. Please tell me I'm wrong! :-)
Python is a compiled language; the compiled byte code is cached by the Python process for later use, to improve performance. PHP, by default, is interpreted. It's a tradeoff between usability and speed.
If you're using a standard WSGI module, such as Apache's mod_wsgi, then you don't have to restart the server -- just touch the .wsgi file and the code will be reloaded. If you're using some weird server which doesn't support WSGI, you're sort of on your own usability-wise.
Depends on how you deploy the Python application. If it is as a pure Python CGI script, no restarts are necessary (not advised at all though, because it will be super slow). If you are using modwsgi in Apache, there are valid ways of reloading the source. modpython apparently has some support and accompanying issues for module reloading.
There are ways other than Apache to host Python application, including the CherryPy server, Paste Server, Zope, Twisted, and Tornado.
However, unless you have a specific reason not to use it (an since you are coming from presumably an Apache/PHP shop), I would highly recommed mod_wsgi on Apache. I know that Django recommends modwsgi on Apache and most of the other major Python frameworks will work on modwsgi.
Is this really the case?
It Depends. Code reloading is highly specific to the hosting solution. Most servers provide some way to automatically reload the WSGI script itself, but there's no standardisation; indeed, the question of how a WSGI Application object is connected to a web server at all differs widely across varying hosting environments. (You can just about make a single script file that works as deployment glue for CGI, mod_wsgi, passenger and ISAPI_WSGI, but it's not wholly trivial.)
What Python really struggles with, though, is module reloading. Which is problematic for WSGI applications because any non-trivial webapp will be encapsulating its functionality into modules and packages rather than simple standalone scripts. It turns out reloading modules is quite tricky, because if you reload() them one by one they can easily end up with bad references to old versions. Ideally the way forward would be to reload the whole Python interpreter when any file is updated, but in practice it seems some C extensions seem not to like this so it isn't generally done.
There are workarounds to reload a group of modules at once which can reliably update an application when one of its modules is touched. I use a deployment module that does this (which I haven't got around to publishing, but can chuck you a copy if you're interested) and it works great for my own webapps. But you do need a little discipline to make sure you don't accidentally start leaving references to your old modules' objects in other modules you aren't reloading; if you're talking loads of sites written by third parties whose code may be leaky, this might not be ideal.
In that case you might want to look at something like running mod_wsgi in daemon mode with an application group for each party and process-level reloading, and touch the WSGI script file when you've updated any of the modules.
You're right to complain; this (and many other WSGI deployment issues) could do with some standardisation help.