Develop Reference

Which way to go with twisted and web-programming?

So, I programmed this twisted application a few months ago, which I now would like to extend with a web-based user interface for configuration.
The Twisted website recommends Nevow, but I am not really sure if this is a good choice. Their website is down for a while it seems and their launchpad page hadn't seen any update in half a year. Is this project dead?
Additionally I have seen discussion of moving parts of Nevow into twisted.web on the twisted-web mailinglist. So, is it still recommended for new developments?
Another idea was using Django. I would need user authentication and permissions anyway in the config-interface, and I am quite familiar with it. (I have never worked with Nevow or twisted.web)
But it seems quite difficult to interface both worlds, all I could find were examples of running Django with WSGI in Twisted.
Are there any other possibilities to have a slick looking user interface on top of twisted?

First, let me address the perception that Nevow is dead. The launchpad project containing the code for Nevow (and the rest of the Divmod projects) is divmod.org on launchpad. A hardware failure has badly impacted the project's public presence, but it's still there, and other things (like the wiki and the tickets) are in the process of being recovered. There isn't a lot of active maintenance work going on right now, but that's mostly because it's good enough for most of its users; there are lots of people who depend on Nevow and would be very upset if it stopped working. Those people have the skills and experience necessary to continue maintaining it. So, while it's not being actively promoted right now, I think it's unlikely that it's going to go away.
My long-term hope for Nevow would be as follows. (I'd say "plan", but since I haven't been actively involved with its maintenance lately, this is really up to those who are.) First, I'd like to extract its templating facilities and move them into twisted.web. The clean, non-deprecated API for Nevow is mostly covered by nevow.page.Element and the various loaders. Twisted itself wants to generate HTML in a few places and these facilities could be useful. Then we should throw out the "appserver" and resource-model parts of Nevow. Those are mostly just a random collection of bugfixes or alterations for twisted.web, most of which were present in some form in twisted.web2 and will therefore either be rolled back into twisted.web anyway, or have already been applied there. Finally there's the question of Athena. While two-way communication is one of Twisted's strengths, Athena is itself a gigantic, sprawling JavaScript codebase and should probably remain its own project.
Third, on to the main question, given this information, what should you do now?
Generally speaking, I'd say, "use nevow". The project has some warts, it needs more documentation and its API needs to be trimmed to eliminate some old and broken stuff, but it's still quite useful and very much alive. To make up for the slightly sparse documentation, you can join the #divmod or #twisted.web channels on Freenode to get help with it. If you help out by contributing patches where you can, you will find that you'll get a lot of enthusiastic help there. When you ignore the deprecated parts Nevow has a pretty small, sane, twisted friendly API. The consequence of the plan for Nevow's evolution that I outlined above are actually pretty minimal. If it even happens at all, what it means for you is, in 1-5 years, when you upgrade to a new version of Twisted, you'll get a couple of deprecation warnings, change some import lines in your code from from nevow.page import ...; from nevow.loaders import ... to some hypothetical new thing like from twisted.web.page.element import ...; from twisted.web.page.templates import ..., or somesuch. Most of the API past that point should remain the same, and definitely the high-level concepts shouldn't change much.
The main advantage that you get from using Nevow is that it's async-friendly and can render pages in your main thread without blocking things. Plus, you can then get really easy COMET for free with Athena.
You can also use Django. This is not quite as async-friendly but it obviously does have a broader base of support. However, "not as async friendly" doesn't mean "hard to use". You can run it in twisted.web via WSGIResource, and simply use blockingCallFromThread in your Django application to invoke any Twisted API that returns a Deferred, which should be powerful enough to do just about anything you want. If you have a more specific question about how to instantiate Twisted web resources to combine Twisted Web and Django, you should probably ask it in its own Stack Overflow question.

Nevow is still a good choice if you want support for Deferreds in the templating system you use (it's not dead). It also has a few advantages over plain Twisted Web when it comes to complicated URL dispatch. However, it is basically just a templating system. Twisted Web is the real web server. So either way, you're going to use Twisted Web. In fact, even if you use Django in Twisted Web's WSGI container, you're still going to use Twisted Web. So learning things about Twisted Web isn't going to hurt you.
If you're going to be generating any amount of HTML, then you very much want to use an HTML templating library. By this point no one should be constructing HTML using primitive string operations. So if you want to use one of the other Python HTML templating libraries out there - Cheetah, Quixote, etc - instead of Nevow, that's great! You're just going to use the templating library to get a string to write out in response to an HTTP request. Twisted Web doesn't care where the string came from.
And if you do want to do something with Django (or another WSGI-based system), then you can certainly deploy this in your Twisted process using Twisted Web's WSGI support. And you can still interact between the WSGI applications and the rest of your Twisted code, as long as you exercise a little care - WSGI applications run in a thread pool, and Twisted APIs are not thread-safe, you have to invoke them with reactor.callFromThread or one of the small number of similar APIs (in particular, blockingCallFromThread is sometimes a useful higher-level tool to use).

At this point Nevow is definitively dead. As illustration of how dead it is, there is a bug that prevents installation of Nevow using pip, which was fixed on trunk in 2009, but it isn't in any release because there has been no release since then.
twisted.web and in particular twisted.web.template cover pretty much all of what was useful in Nevow, and should be used for any new project that was considering using Nevow.

What's the next steps for moving from appengine to full django?

I'm super new to programming and I've been using appengine to help me learn python and general coding. I'm getting better quickly and I'm loving it all the way :)
Appengine was awesome for allowing me to just dive into writing my app and getting something live that works (see http://www.7bks.com/). But I'm realising that the longer I continue to learn on appengine the more I'm constraining myself and locking myself into a single system.
I'd like to move to developing on full django (since django looks super cool!). What are my next steps? To give you a feel for my level of knowledge:
I'm not a unix user
I'm not familiar with command line controls (I still use appengine/python completely via the appengine SDK)
I've never programmed in anything other than python, anywhere other than appengine
I know the word SQL, but don't know what MySQL is really or how to use it.
So, specifically:
What are the skills I need to learn to get up and running with full django/python?
If I'm going to host somewhere else I suppose I'll need to learn some sysadmin type skills (maybe even unix?). Is there anywhere that offers easy hosting (like appengine) but that supports django?
I hear such great things about heroku I'm considering switching to RoR and going there
I appreciate that I'm likely not quite ready to move away from appengine just yet but I'm a fiercely passionate learner (http://www.7bks.com/blog/179001) and would love it if I knew all the steps I needed to learn so I could set about learning them. At the moment, I don't even know what the steps are I need to learn!
Thank you very much. Sorry this isn't a specific programming question but I've looked around and haven't found a good how-to for someone of my level of experience and I think others would appreciate a good roadmap for the things we need to learn to get up and running.
Thanks,
Tom
PS - if anyone is in London and fancies showing me the ropes in person that would be super awesome :)

First up, you can benefit by doing some RoR work by learning a new language. However, I don't know if that'll be entirely beneficial to you right now since you still are learning. I'd stick with Python and Django (or AppEngine) for the moment, until you can grasp some of the more advance concepts. Then, by all means, learning new languages will be fantastic.
As for moving to Django from AppEngine. There isn't a whole lot that's different. The way you define models is similar, but has different types for the definition. As you mentioned, hosting is another consideration.
There should be plenty of hosting options (mod_wsgi is what you're after) based on Apache. Django in particular has seen quite a bit of popularity, and hosting usually springs up for popular frameworks.
I don't think you'll need to know too much sysadmin stuff. This will all depend on the kind of hosting you can find. Same goes for the database. Hosting providers usually offer databases preconfigured so you shouldn't need to worry about that too much.
Django, along with many other frameworks, provide an ORM (Object Relational Mapper) which abstracts away having to write SQL, by calling methods on objects and accessing their properties. I'd advise learning a little bit of SQL to understand it at a bare minimum though.
The Django tutorial is excellent! If you decide to go the Django route, I'd highly recommend working through the entire thing. A development server comes bundled, so you can try out your work instantly without worrying about a provider. Once you have something you want to share with the world, you can worry about hosting then.
I started off using Windows for Django development and it was quite easy. The amount of command line work you need to do is minimal. Really. Not something you need to worry about, as the tutorial covers all 4 or so commands you need to know.
Django hosting provides links to hosting providers, though I'm not sure how up to date that list is.
Getting started in Django is pretty simple. Once you want to host it, there's a bit more work involved - but that can come later. The friction is minimal. Follow the tutorial, it will take you through running the server, setting up the database (a free one comes bundled), and coding your first app.

What makes you think you're being locked into a single platform? Did you know that Google's App Engine SDK is open source? Also, there are universities and other organizations who are working on building platforms that will use the App Engine SDK outside of the context of Google? Amazon EC2 is also capable of running App Engine's SDK in a limited capacity. I'd say lock-in is perhaps not the right word to use.
Additionally, I believe AppEngine is going to continue to improve as time goes on. Google is the leader of the Internet; they've done great things and will continue to do so. I believe that anyone who sticks with their platform as a service will experience these great benefits in the years to come.
If your reasoning for moving is purely academic, I'd suggest starting a new project. Moving off of AppEngine's SDK is similar to switching from one framework to another on an already-built application. Like with any framework or platform, there are dependencies that must be dealt with in order to successfully migrate the app from platform A to platform B.

django-nonrel makes it possible to run Django on Google AppEngine: http://www.allbuttonspressed.com/projects/django-nonrel
Beside that there exists a couple of cloud offers like djangy https://www.djangy.com/
With both options you can focus on Django and Python programming and don't have to care about the sysadmin stuff.
On the django homepage there is a very good tutorial to get started with django development: http://docs.djangoproject.com/en/1.2/intro/tutorial01/

What are the skills I need to learn to get up and running with full django/python?
The question can't easily be answered because you haven't described the app. You have to actually write down the technology stack -- in detail -- or you'll never know what skills you need.
The skill list mostly comes from your technology choices. So write down your technology choices. (That's part of configuration management, an important skill you'll need if you move away from GAE.)
Since you've chosen to talk about yourself and not your technology choices, I can only guess what technologies you're using and what skills you'll need.
Here's a common technology stack.
Technically, the OS doesn't matter. Most hosting environments use open source GNU/Linux because the licensing is inexpensive. You, too, can do this. You can start with VMWare and download a nice Linux distro. Or, you can buy a very cheap PC and install Linux directly from a DVD image that you can download and burn.
My company demands that I use the Windows PC they give me. So I develop in Windows and test in VMWare Linux (Fedora 14, actually)
To learn Linux, start with a download and install. Then find a tutorial. Then stop using Windows and learn by doing. Flipping back and forth between Windows and Linux is difficult. I can do it because I don't know Windows very well. I treat Windows as a hyper-complex IDE with all kinds of non-standard, non-POSIX quirks that I try to ignore.
RDBMS. Python comes with SQLite. For a lot of applications, it works fine. It works because web sites tend to be heavy on queries and light on updates/inserts, so SQLite works well. MySQL is nice. It's easy to install and runs on Windows as well as other OS's.
The good thing about Django is you need to know very little SQL. Very little.
However, you do need to know a tiny bit about the "Data Control Language" (Grant, Revoke and Create User) to work with MySQL. You won't create a lot of users. But you do need to create a few to get things running. Also, as your database matures, you'll often need to know a little bit about the "Data Definition Language" (Drop Table).
You will need to know how to backup and restore your database. That's absolutely critical.
So find Database Administrator tutorials to get started.
If your application really uses a lot of sophisticated data, you'll need to buy real books on database design so you can understand how the Django models really work. You don't need to become a SQL guru, but it does help to know what's really going on inside the database.
Application Server. We use Apache with the mod_wsgi module. There are numerous choices. Hosting services vary in what they require and what they permit. Some have Apache, mod_wsgi and Django pre-configured. Some don't. Some do not permit tinkering with the Apache configuration. Some do. You probably don't need to know much about this, because you can probably find a hosting service that will handle the details.
Apache tutorials are all over the place. mod_wsgi is very simple; once you understand how Apache works, mod_wsgi is obvious.
Since you have stuff working, presumably you know about HTML and CSS. Those are important skills, but you probably already have them.
Since you have stuff working, you also might know a lot about Configuration Management and how to control change. This isn't obvious and many people do it wrong. If you don't know about CM, you should find some books or articles on configuration management and change control.
Since you have stuff working, you also might know a lot about quality assurance, unit testing and related topics. If you don't have a complete suite of unit tests, you should probably start learning about unit testing before you start any serious coding for the next release of your product.
Bug Tracking, Problem Reporting, Feature Requests and other management skills are also essential. I can't tell if you have them or don't have them. Or what tools you're using for this. If you're working by yourself, you don't need a lot of formality. However, these are critical skills even if you're a one-person developer. Sticky-Notes on your workstation can work. What's important is the skills to manage bugs and features.

Hey Tom,
I suggest the reasonable evaluation you can make is carefully list the advantages and disadvantages of the choices.
The way I don't regret taking was physical rack server (2006-07), moved to virtual hosting (2008) and now moved to GAE (2009-current). Seeing the rate new features get added to GAE and the costeffectiveness are more reasons to stay. I agree more stuff django can are needed like in my case form preview and form validation with GAE are difficult or too difficult to set up.
I tried RoR and soon thought RoR requires more code to do the same GAE can with less code.
Also with GAE you have absolutely no hardware that can break. If you move to a rack server or a virtual hosting where there are places you can get 5 GB hosting for free but you don't have a plan when you run out of 5 GB and may need to migrate again which you don't want.
MySQL has been around for over 10 years and is quite a different kind of system. It's possible to save blobs in MySQL but don't you think the blobstore GAE has is much better?
If you choose to migrate to a solution with MySQL you can export you data from GAE and import it to MySQL with a tool such as approcket.
Kind regards/Niklas R

Please point to the right webdev tools

I'm making a simple web app where I have some simple python scripts that do the text crunchin g I need - but I'm not quite sure how to interface it with a client who'd only want to see some HTML forms.
There's so many different server side frameworks out there - but I don't think I need anything too heavy duty - just a mechanism to accept data from the forms that the user fills in, and feed it to my Python code and back.
Could someone suggest what tools I should look into and what design paradigms I should follow? Simple pointers to different references around the web with a single line about their significance would also help.
Best.

At the moment, the best lightweight and yet very powerful framework for python IMO is Flask. If you want form abstraction there is a WTFlask plugin for it which is WTForms adapted for flask - http://flask.pocoo.org/.
Web2py is also a very good framework for starters because it has helpers and wizards for creating/running an application, and also has something like an admin interface, offering an On-line IDE functionality. - http://www.web2py.com/
I proposed these two frameworks because you can start fast with them, they both have good documentation, both are powerful yet easy and they are pretty friendly for beginners.

Not normally supported with cheap hosting, but mod_python on apache might be right up your alley.
You could also have html talk to python through cgi (Which has been the way of doing it for years before php came along).
The standard Design pattern commonly seen in web apps is Model-View-Controller, so google around.

I've used Django (heavy and something of a non-trivial initial learning curve), and web.py (very small & simple), and over the past year, I've been pretty happy using tornado.
People can argue about the async details, but that's not even why I use it. It's pretty fast, it's super quick to get started with, it comes with examples that get you doing things quickly, and -- most important to me -- It's a web server and framework in one, which means you don't have to muck with mod_python vs mod_wsgi vs cgi vs fcgi vs whatever. Deployment is really easy.

What are the benefits of using Python for web programming?

What makes Python stand out for use in web development? What are some examples of highly successful uses of Python on the web?

Django is, IMHO, one of the major benefits of using Python. Model your domain, code your classes, and voila, your ORM is done, and you can focus on the UI. Add in the ease of templating with the built-in templating language (or one of many others you can use as well), and it becomes very easy to whip up effective web applications in no time. Throw in the built-in admin interface, and it's a no-brainer.

Certainly one successful use of Python on the web is Google App Engine. Site authors write code in (a slightly restricted subset of) Python, which is then executed by the App Engine servers in a distributed and scalable manner.

Quotes about Python:
"Python is fast enough for our site
and allows us to produce maintainable
features in record times, with a
minimum of developers," said Cuong Do,
Software Architect, YouTube.com.

YouTube uses a lot of Python and is probably the best example of a Python success story.
A great example of a Django success story is the Washington Post, who recently shared a big list of applications they have developed:
http://push.cx/2009/washington-post-update
www.lawrence.com and www.ljworld.com are two of the first sites to use Django (before it was even open source).
djangositeoftheweek.com has a bunch of good case studies.
www.everyblock.com is another great example.
Finally, http://www.djangosites.org/ links to nearly 2,000 other Django powered sites.

Short anwser: the diversity of tools readily available and freedom of choice.
This sounds like a simple question but which it really isn't. While Python is very good for web development and this has been shown by the, oh so famous, Google App Engine, Plone and Django. One has to point out that the development way in Python requires a lot more from the developer than PHP but it gives a lot more to the mix as well.
The entry level on actually producing something is higher. This is because there are bunch of different tools for doing web development with Python. Choosing the web development framework can be a hard decision for an inexperienced developer.
Having a lot of different tools is a two edged sword. To some extent it brings you the freedom of choice to pick the one you might want but then again how do you really know which one is good for what you're doing. This brings me to my point. Python stands out from the mass by not having a standard or de facto web development library. While this is pretty much against the principle of having only one simple way of doing on thing it also brings us a wide variety of different tools with different kind of design choices. At first this might feel very frustrating because it would be so much easier if somebody had made the choice for you but now that you're left to make the choice you actually might have to think about what you're doing and what would fit. ...or you might just end up picking one and blowing your head off after you've realized that you made the wrong choice. Anyway you end up, you've made the choice and no one else.

Furthermore,
Python is both strong in web and in data analytics and machine learning. For example scikit, sci-py and numpy are very strong. In some cases, it can be very interesting to have the both elements on the same server.
For example http://rankmytweet.com uses this a lot.

trac(bug tracker) and moinmoin(wiki) are too web based python tools that I find invaluable.

GNU Mailman is another project written in python that is widely successful.

As many have pointed out, Django is a great reason to use Python...so in order to figure out why Python is great for web development, the best bet is to look at why it is a good language to build a framework like Django.
IMHO Python combines the cleanest, or at least one of the cleanest, metaprogramming models of any language with a very pure object orientation. This not only makes it possible to write extremely general abstractions that are easy to use, but also allows the abstractions to combine relatively cleanly with others. This is harder to do in languages that take a code-generation based approach to metaprogramming (e.g. Ruby).

Dynamic languages are in general good for web apps because the speed of development. Python in particular has two advantages over most of them:
"batteries included" means lots of available libraries
Django. For me this is the only reason why i use Python instead of Lua (which i like a lot more).

Besides the frameworks...
Python's pervasive support for Unicode should make i18n much smoother.
A sane namespace system makes debugging much nicer, because it's typically easier to find where things are defined.
Python's inability to function as a standalone templating language should discourage the mixture of HTML with model code
Great standard library

Other examples of Python sites are Reddit and YouTube.

Using python to build web applications

This is a follow-up to two questions I asked a week or so back. The upshot of those was that I was building a prototype of an AI-based application for the web, and I wondered what language(s) to use. The conclusion seemed to be that I should go for something like python and then convert any critical bits into something faster like Java or C/C++.
That sounds fine to me, but I'm wondering now whether python is really the right language to use for building a web application. Most web applications I've worked on in the past were C/C++ CGI and then php. The php I found much easier to work with as it made linking the user interface to the back-end so much easier, and also it made more logical sense to me.
I've not used python before, but what I'm basically wondering is how easy is CGI programming in python? Will I have to go back to the tedious way of doing it in C/C++ where you have to store HTML code in templates and have the CGI read them in and replace special codes with appropriate values or is it possible to have the templates be the code as with php?
I'm probably asking a deeply ignorant question here, for which I apologise, but hopefully someone will know what I'm getting at! My overall question is: is writing web applications in python a good idea, and is it as easy as it is with php?

Python is a good choice.
I would avoid the CGI model though - you'll pay a large penalty for the interpreter launch on each request. Most Python web frameworks support the WSGI standard and can be hooked up to servers in a myriad of ways, but most live in some sort of long-running process that the web server communicates with (via proxying, FastCGI, SCGI, etc).
Speaking of frameworks, the Python landscape is ripe with them. This is both good and bad. There are many fine options but it can be daunting to a newcomer.
If you are looking for something that comes prepackaged with web/DB/templating integration I'd suggest looking at Django, TurboGears or Pylons. If you want to have more control over the individual components, look at CherryPy, Colubrid or web.py.
As for whether or not it is as "easy as PHP", that is subjective. Usually it is encouraged to keep your templates and application logic separate in the Python web programming world, which can make your life easier. On the other hand, being able to write all of the code for a page in a PHP file is another definition of "easy".
Good luck.

"how easy is CGI programming in python?" Easier than C, that's for sure. Python is easier because -- simply -- it's an easier language to work with than C. First and foremost: no memory allocation-deallocation. Beyond that, the OO programming model is excellent.
Beyond the essential language simplicity, the Python WSGI standard is much easier to cope with than the CGI standard.
However, raw CGI is a huge pain when compared with the greatly simplified world of an all-Python framework (TurboGears, CherryPy, Django, whatever.)
The frameworks impose a lot of (necessary) structure. The out-of-the-box experience for a CGI programmer is that it's too much to learn. True. All new things are too much to learn. However, the value far exceeds the investment.
With Django, you're up and running within minutes. Seriously. django-admin.py startproject and you have something you can run almost immediately. You do have to design your URL's, write view functions and design page templates. All of which is work. But it's less work than CGI in C.
Django has a better architecture than PHP because the presentation templates are completely separated from the processing. This leads to some confusion (see Syntax error whenever I put python code inside a django template) when you want to use the free-and-unconstrained PHP style on the Django framework.
linking the user interface to the back-end
Python front-end (Django, for example) uses Python view functions. Those view functions can contain any Python code at all. That includes, if necessary, modules written in C and callable from Python.
That means you can compile a CLIPS module with a Python-friendly interface. It becomes something available to your Python code with the import statement.
Sometimes, however, that's ineffective because your Django pages are waiting for the CLIPS engine to finish. An alternative is to use something like a named pipe.
You have your CLIPS-based app, written entirely in C, reading from a named pipe. Your Django application, written entirely in Python, writes to that named pipe. Since you've got two independent processes, you'll max out all of your cores pretty quickly like this.

I would suggest Django, but given that you ask for something "as easy as it is with php" then you must take a look at PSP (Python Server Pages).
While Django is a complete framework for doing websites, PSP can be used in the same way than PHP, without any framework.

It is easier to write web-apps in python than it's in php. Particularly because python is not a broken language.
Pick up some web framework that supports mod_wsgi or roll out your own. WSGI apps are really easy to deploy after you get a hold from doing it.
If you want templates then genshi is about the best templating engine I've found for python and you can use it however you like.