We currently run a small shared hosting service for a couple of hundred small PHP sites on our servers. We'd like to offer Python support too, but from our initial research at least, a server restart seems to be required after each source code change.
Is this really the case? If so, we're just not going to be able to offer Python hosting support. Giving our clients the ability to upload files is easy, but we can't have them restart the (shared) server process!
PHP is easy -- you upload a new version of a file, the new version is run.
I've a lot of respect for the Python language and community, so find it hard to believe that it really requires such a crazy process to update a site's code. Please tell me I'm wrong! :-)
Python is a compiled language; the compiled byte code is cached by the Python process for later use, to improve performance. PHP, by default, is interpreted. It's a tradeoff between usability and speed.
If you're using a standard WSGI module, such as Apache's mod_wsgi, then you don't have to restart the server -- just touch the .wsgi file and the code will be reloaded. If you're using some weird server which doesn't support WSGI, you're sort of on your own usability-wise.
Depends on how you deploy the Python application. If it is as a pure Python CGI script, no restarts are necessary (not advised at all though, because it will be super slow). If you are using modwsgi in Apache, there are valid ways of reloading the source. modpython apparently has some support and accompanying issues for module reloading.
There are ways other than Apache to host Python application, including the CherryPy server, Paste Server, Zope, Twisted, and Tornado.
However, unless you have a specific reason not to use it (an since you are coming from presumably an Apache/PHP shop), I would highly recommed mod_wsgi on Apache. I know that Django recommends modwsgi on Apache and most of the other major Python frameworks will work on modwsgi.
Is this really the case?
It Depends. Code reloading is highly specific to the hosting solution. Most servers provide some way to automatically reload the WSGI script itself, but there's no standardisation; indeed, the question of how a WSGI Application object is connected to a web server at all differs widely across varying hosting environments. (You can just about make a single script file that works as deployment glue for CGI, mod_wsgi, passenger and ISAPI_WSGI, but it's not wholly trivial.)
What Python really struggles with, though, is module reloading. Which is problematic for WSGI applications because any non-trivial webapp will be encapsulating its functionality into modules and packages rather than simple standalone scripts. It turns out reloading modules is quite tricky, because if you reload() them one by one they can easily end up with bad references to old versions. Ideally the way forward would be to reload the whole Python interpreter when any file is updated, but in practice it seems some C extensions seem not to like this so it isn't generally done.
There are workarounds to reload a group of modules at once which can reliably update an application when one of its modules is touched. I use a deployment module that does this (which I haven't got around to publishing, but can chuck you a copy if you're interested) and it works great for my own webapps. But you do need a little discipline to make sure you don't accidentally start leaving references to your old modules' objects in other modules you aren't reloading; if you're talking loads of sites written by third parties whose code may be leaky, this might not be ideal.
In that case you might want to look at something like running mod_wsgi in daemon mode with an application group for each party and process-level reloading, and touch the WSGI script file when you've updated any of the modules.
You're right to complain; this (and many other WSGI deployment issues) could do with some standardisation help.
Related
I'm trying to build a desktop application using Python. To make it able to be used on as many platforms as possible, I think web UI may be a good choice. This boils down to the problem of making a local HTTP server first. I did some survey and found that people are mainly talking about BaseHTTPServer and SimpleHTTPServer. For prototyping, subclassing them may suffice.
Besides pure prototyping, I also want to leave some room for extension to real service. That is, once mature, I'd like to move the codes to a real dedicated HTTP server, so that end users only need a browser to use it.
I say "extensible" in the following sense:
The code modification is as minimum as possible in the migration process.
I will focus on algorithm in the prototyping stage. I also want to leave some room for future front end designer.
It looks WSGI + Django is a widely mentioned combination. After some search, what I found is using WSGI in apache or nginx. Is it possible to use self-contained modules? i.e. wsgiref + Django, so that I can start everything just from one entry script. I don't want to bother potential first adopters by asking them install apache and configure it. It will be very good if you have sample codes or pointers for further reading.
I'm new to Python and web programming in Python. Thanks for your help. I just try to make sure I'm on the right track. My underlying algorithms is implemented in Python 2.7. So the UI solution had better also be in Python 2.7.
I think what you may want is Bottle. It is a web framework that only needs the standard library to be installed. It also has compatibility with many other production servers, as well as shipping with it's own development server. And if that isn't good enough, it is all in a single file, and has support with many different templating languages, as well as it's own built in templating language.
Check it out here: http://bottlepy.org/docs/dev/
As mentioned bottle is a good choice, I personally like Flask, which if I recall correctly is what bottle is based off of. Anyways there are three things that really make Flask a joy to use.
Blueprints - essentially an application architecture
Flask-Sijax - allows for comet technology
Celery - an asynchronous task queue/job queue based on distributed message passing
there are a lot of other plugins, including one for an admin interface that I haven't tried out yet but it looks promising, and it works with Python 2.7
I am used to PHP having applications. For example,
c:\xampp\htdocs\app1
c:\xampp\htdocs\app2
can be accessed as
localhost://app1/page.php
localhost://app2/page.php
Things to be noticed:
a directory placed inside the www-root directory maps directly with the URL
when a file/directory is added/removed/changed, the worker processes seamlessly reflect that change (new files are hot-deployed).
I am on the lookout for a mature python web framework. Its for a web API which will be deployed for multiple clients, and each copy will diverge on customization. And our workflow has frequent interaction/revision cycles between us and our clients. Hence the "drag and drop" deployment is a must.
Which python framework enables this? I prefer a lightweight solution (which doesnt impose MVC, ORMs etc)
Related
How to build Twisted servers which are able to do hot code swap in Python?
Python web hosting: Why are server restarts necessary?
fastcgi, cherrypy, and python
No mature python framework that I'm aware of allows you to map urls to python modules, and frankly, for good reason. You can do this with CGI, but it's definitely not the recommended way to deploy python apps. Setting that requirement aside, flask and bottle are both lightweight micro web-frameworks with similar approaches, both allow you to reload automatically when changes are detected (this is only wise during development).
There is no web framework in Python that I know of that lets you do that out of the box, but if you need it it's not to hard to add with a bit of convention over configuration.
Simply pick your web framework of choice in Python and then write a wrapper to the main application that walks a directory or set of directory and auto-registers routes from the modules inside of them. Have your modules do the same thing in their __init__.py files to the other files located with them. Then just set up your WSGI code to autoreload when the WSGI script is updated and your deployment during development simply becomes a two step process - add file then touch dev_app.wsgi. You could even add a real deployment option to this wrapper that walks a set up dev environment like this and generates hard-coded URL-to-function mappings for deployment.
However, all of this work isn't really necessary. Python is not PHP and the way you develop in one doesn't necessarily translate to the other well. If the client wants variable routes, use dynamic routes and give them (or you) an admin interface to control the mapping of content to URL. Use flat files, SQLite, a NoSQL datastore, or the ether to store these mappings and the content. Use a template engine like Jinja2, Mako, Cheetah or Genshi to maintain your general layout. Wrap this all up with an object oriented structure to make extending it easy (or use a functional paradigm if that comes more naturally to you). Or, drop the whole dynamic in production portion and generate flat HTML files a la Jekyll.
CherryPy is a mature web framework that redeploys automatically when changes are detected. The file structure - URL isn't there, but it is a lightweight framework that doesn't impose ORM, MVC, or even a templating engine.
If you are used to PHP, you might want to take a look at the Apache modules mod_python or mod_wsgi (and WSGI in general if you do web development -- which is the Pythonic way).
With those two modules, the Python interpreter gets started every time a request comes in (similar to PHP). Needless to say, this slows things down but you'll always get the result based on your newest code. Depending on your expected traffic numbers, this might or might not be okay for you.
BUT: If you decide to write your own framework, you most probably do not want to write a system that supports "hot-deploying". Even though the reload() command is built-in, it takes more than just that and will get you into a world full of pain.
I've finally concluded that I can no longer afford to just hope the ongoing Py3k/WSGI disasterissues will be resolved anytime soon, so I need to get ready to move on.
Unfortunately, my available options don't seem a whole lot better:
While I find a few different Python modules for FastCGI scattered around the web, none of them seem to be getting much (if any) attention and/or maintenance, particularly with regard to Python 3.x, and it's difficult to distinguish which, if any, are really viable.
Falling all the way back to the built-in CGI module is hardly better than building something myself from scratch (worse, there's an important bug or two in there that may not get attention until Python 3.3).
There is no higher sin than handling HTTP directly in a production webapp. And anyway, that's still reinventing the wheel.
Surely somebody out there is deploying webapps on 3.x in production. What gateway interface are you using, with which module/libraries, and why?
CherryPy 3.2 release candidates support Python 3.X. Because it only supports WSGI at the web server interface layer and not through the whole stack, then you are isolated from issues as to whether WSGI will change. CherryPy has its own internal WSGI server, but also can run under Apache/mod_wsgi with Python 3.1+. See:
http://www.cherrypy.org/wiki/WhatsNewIn32
http://code.google.com/p/modwsgi/wiki/SupportForPython3X
bottle supports Python 3, but it suffers from the broken stdlib. However, multipart reimplements cgi.FieldStorage and can be used with bottle to build a Python 3 WSGI web app. I just published a demo. For the moment it is just a test, but as far as I can tell it works well.
Given that I know no web frameworks in Python and would like to keep it Very Simple at the moment (as I am Very Stupid), for what is a prototype of sketchy longevity, are there any streamlined, simple, "batteries-included" modules for this? (It is also too early in my Python career to evaluate frameworks, select one, and learn it.) I see a module named "Cookie," which could serve as a foundation, but nothing session-specific.
I'm familiar with the basic session concepts, having used them in classic ASP and gotten into the nuts-and-bolts of them in Perl, but I am not seeing a lot for Python. Beaker looks interesting, but then the documentation seems to require middleware with WSGI and I'm back to the frameworks problem.
I've found an old recipe on ActiveState for sessions, which could obviously use some buffing up. The information being held is not anything anyone would mind having been grabbed, so while I am normally quite security conscious, I would be willing to be a little bit more lax with this prototype.
Or is this a "roll-your-own" problem?
I will be using Python 2.6 on IIS 7.0.
I think the web2py (web framework) is easy enough for you. I think it is the simplest approach of making a website or webservice. It will be also easier, than to understand Cookie or the other modules of python related to web-things.
You can start a session, by just typing:
session.your_session_name = "blabla" # or whatever you want to store
To make a cookie, just look here.
In web2py you don't have to configure anything. Just download it and start web2py.py. (you must have python 2.6 < installed.) You can also find some examples and a web-slide.
The Python Cookie module does nothing more than to hold some values in a dictonary-like object, but I think you have to store it yourself on your harddisk.
CherryPy is worth looking into. Yes it is a framework, and yes it requires WSGI, but it is extremely lightweight compared to other more robust alternatives.
There is another question that was answered on SO that gives a brief example on how to manage sessions with CherryPy. As you can see it makes it very easy to get up and running quickly.
Lastly, here is a little document about setting up IIS for use with CherryPy.
WSGI is not a framework, nor does it require that you choose one -- it's THE standard way to run any Python web app framework on any Python-supporting web server, including a CGI one. If you have a WSGI application named app, and want to run it on CGI, see the docs and use wsgiref.handlers.CGIHandler().run(app), as the docs say.
So, you can perfectly well use Beaker via WSGI (on top of CGI) -- e.g., take the example in Beaker's docs and just add (the needed imports and) the run call above (using the wsgi_app object that example constructs, plus of course a session.save and as well needed as, again, the Beaker docs explain right afterwards).
Rich or heavy frameworks have their place but so do lightweight, flexible components like Beaker -- and WSGI middleware is a great way to leverage such components without requiring any "framework-y" arrangements, just good old WSGI (on top of CGI or anything else).
BTW, the best way to run WSGI on IIS might be isapi-wsgi (I can only say "might" because I have no IIS installation on which to test it;-). But as long as you code to WSGI (with any framework or with none at all), that will only be an optimization -- your application won't change (net of what handler's run or equivalent method you need to call;-) whether it's running on CGI, IIS via ISAPI, Google App Engine, or any other server-and-interface-thereto combination
which would you recommend?
which is faster, reliable?
apache mod_python or nginx/lighttpd FastCGI?
I've done both, and Apache/mod_python tended to be easier to work with and more stable. But these days I've jumped over to Apache/mod_wsgi, which is everything I've ever wanted and more:
Easy management of daemon processes.
As a result, much better process isolation (running multiple sites in the same Apache config with mod_python almost always ends in trouble -- environment variables and C extensions leak across sites when you do that).
Easy code reloads (set it up right and you can just touch the .wsgi file to reload instead of restarting Apache).
More predictable resource usage. With mod_python, a given Apache child process' memory use can jump around a lot. With mod_wsgi it's pretty stable: once everything's loaded, you know that's how much memory it'll use.
lighttpd with FastCGI will be nominally faster, but really the time it takes to run your python code and any database hits it does is going to absolutely dwarf any performance benefit you get between web servers.
mod_python and apache will give you a bit more flexibility feature-wise if you want to write code outside of django that does stuff like digest auth, or any fancy HTTP header getting/setting. Perhaps you want to use other builtin features of apache such as mod_rewrite.
If memory is a concern, staying away form apache/mod_python will help a lot. Apache tends to use a lot of RAM, and the mod_python code that glues into all of the apache functionality occupies a lot of memory-space as well. Not to mention the multiprocess nature of apache tends to eat up more RAM, as each process grows to the size of it's most intensive request.
Nginx with mod_wsgi
I'm using it with nginx. not sure if it's really faster, but certainly less RAM/CPU load. Also it's easier to run several Django processes and have nginx map each URL prefix to a different socket. still not taking full advantage of nginx's memcached module, but first tests show huge speed advantage.
There's also mod_wsgi, it seems to be faster than mod_python and the daemon mode operates similar to FastCGI
Personally I've had it working with FastCGI for some time now (6 months or so) and the response times 'seem' quicker when loading a page that way vs mod___python. The critical reason for me though is that I couldn't see an obvious way to do multiple sites from the same apache / mod_python install whereas FastCGI was a relative no-brainer.
I've not conducted any particularly thorough experiments though :-)
[Edit] Speaking from experience though, setting up FastCGI can be a bit of a pain the first time around. I keep meaning to write a guide..!
I'd recommend WSGI configurations; I keep meaning to ditch apache, but there is always some legacy app on the server that seems to require it. Additionally, the WSGI app ecology is very diverse, and it allows neat tricks such as daisy-chaining WSGI "middleware" between the server and the app.
However, there are currently known issues with some apps and apache mod_wsgi, particularly some ctypes apps, so be wary if you are trying to run, say, geodjango which uses ctypes extensively. I'm currently working around those issues by going back to fastcgi myself.