Develop Reference

Django alternatives for real time web applications [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I've been developing a web server application with Django for the last year. The stack is Django + Apache + MySql. This stack was perfectly suited to our needs. Recently came the need of some real-time capabilities and the opportunity to change/rewrite a lot of stuff in the server application. To my surprise, Django is not the best option here.
I've been reading a lot (push, WebSockets, gunicorn... lots of stuff.. http://curella.org/blog/django-push-using-server-sent-events-and-websocket/) but I haven't managed to decide if I want to go with Django or if I should purpose a new stack of software which is more suitable for the job. Going with Django seems a little bit unnatural. Can any experienced developers point me in the right direction?
So, the basic question is: what are some alternatives to the current software stack I have for building a real time web application?
Thanks
PS: Not a native English speaker. :)
EDIT: The alternatives need to allow secure connections.
EDIT 2: The web applications we develop are games.

Look into http://www.tornadoweb.org/ + http://www.mongodb.org/ + https://github.com/bitly/asyncmongo + http://socket.io/
I think it's a good idea to use that to create real-time application.

You need to weigh your decisions against your goals.
You want a product
Then write your code in the language framework that you are most familiar with, only when you have a product and it has limitations consider switching tools.
You want to learn something new
Try out the new language, framework - but it may take a lot longer to produce a product and you may find that the new tool isn't any better than the old one.
Half and Half
Try starting two projects, and building the same thing in each project, just using the other tool. This will take even longer, but you should be able to see which tool you prefer / is best suited to your task very quickly.
Alternatives
I'm not quite sure what you mean by django is 'unnatural'. Django can do server / client communication - just use some AJAX to talk back to the django server, then call another pythonic library to process the request. Python alternatives to django include the wonderful flask and web.py, though neither will do client / server communication unless you program them too (like django). If you're not into python you might try ruby on rails. For the client side you will need to know something about JavaScript so go and learn up on that (coffescript and jQuery can ease JS pain, but do learn JavaScript first).
Of course you could bite the bullet and go with node.js as the server base, apparently it's pretty good (I've not tried it yet) - and written in JS.
EDIT:
In light of your comments, take a closer look at
django-websockets,
node.js + push server
websockets
Noting also that this question is a potential duplicate.

For the real time web applications I suggest that you go with the websockets, they can be secure and response times are very fast as when connection is made there are no overheads in communication anymore. If you are proficient in python you can make server in python twisted and clients in python and/or javascript using autobahn. P.S. here is a really great tutorial for twisted.

Which way to go with twisted and web-programming?

So, I programmed this twisted application a few months ago, which I now would like to extend with a web-based user interface for configuration.
The Twisted website recommends Nevow, but I am not really sure if this is a good choice. Their website is down for a while it seems and their launchpad page hadn't seen any update in half a year. Is this project dead?
Additionally I have seen discussion of moving parts of Nevow into twisted.web on the twisted-web mailinglist. So, is it still recommended for new developments?
Another idea was using Django. I would need user authentication and permissions anyway in the config-interface, and I am quite familiar with it. (I have never worked with Nevow or twisted.web)
But it seems quite difficult to interface both worlds, all I could find were examples of running Django with WSGI in Twisted.
Are there any other possibilities to have a slick looking user interface on top of twisted?

First, let me address the perception that Nevow is dead. The launchpad project containing the code for Nevow (and the rest of the Divmod projects) is divmod.org on launchpad. A hardware failure has badly impacted the project's public presence, but it's still there, and other things (like the wiki and the tickets) are in the process of being recovered. There isn't a lot of active maintenance work going on right now, but that's mostly because it's good enough for most of its users; there are lots of people who depend on Nevow and would be very upset if it stopped working. Those people have the skills and experience necessary to continue maintaining it. So, while it's not being actively promoted right now, I think it's unlikely that it's going to go away.
My long-term hope for Nevow would be as follows. (I'd say "plan", but since I haven't been actively involved with its maintenance lately, this is really up to those who are.) First, I'd like to extract its templating facilities and move them into twisted.web. The clean, non-deprecated API for Nevow is mostly covered by nevow.page.Element and the various loaders. Twisted itself wants to generate HTML in a few places and these facilities could be useful. Then we should throw out the "appserver" and resource-model parts of Nevow. Those are mostly just a random collection of bugfixes or alterations for twisted.web, most of which were present in some form in twisted.web2 and will therefore either be rolled back into twisted.web anyway, or have already been applied there. Finally there's the question of Athena. While two-way communication is one of Twisted's strengths, Athena is itself a gigantic, sprawling JavaScript codebase and should probably remain its own project.
Third, on to the main question, given this information, what should you do now?
Generally speaking, I'd say, "use nevow". The project has some warts, it needs more documentation and its API needs to be trimmed to eliminate some old and broken stuff, but it's still quite useful and very much alive. To make up for the slightly sparse documentation, you can join the #divmod or #twisted.web channels on Freenode to get help with it. If you help out by contributing patches where you can, you will find that you'll get a lot of enthusiastic help there. When you ignore the deprecated parts Nevow has a pretty small, sane, twisted friendly API. The consequence of the plan for Nevow's evolution that I outlined above are actually pretty minimal. If it even happens at all, what it means for you is, in 1-5 years, when you upgrade to a new version of Twisted, you'll get a couple of deprecation warnings, change some import lines in your code from from nevow.page import ...; from nevow.loaders import ... to some hypothetical new thing like from twisted.web.page.element import ...; from twisted.web.page.templates import ..., or somesuch. Most of the API past that point should remain the same, and definitely the high-level concepts shouldn't change much.
The main advantage that you get from using Nevow is that it's async-friendly and can render pages in your main thread without blocking things. Plus, you can then get really easy COMET for free with Athena.
You can also use Django. This is not quite as async-friendly but it obviously does have a broader base of support. However, "not as async friendly" doesn't mean "hard to use". You can run it in twisted.web via WSGIResource, and simply use blockingCallFromThread in your Django application to invoke any Twisted API that returns a Deferred, which should be powerful enough to do just about anything you want. If you have a more specific question about how to instantiate Twisted web resources to combine Twisted Web and Django, you should probably ask it in its own Stack Overflow question.

Nevow is still a good choice if you want support for Deferreds in the templating system you use (it's not dead). It also has a few advantages over plain Twisted Web when it comes to complicated URL dispatch. However, it is basically just a templating system. Twisted Web is the real web server. So either way, you're going to use Twisted Web. In fact, even if you use Django in Twisted Web's WSGI container, you're still going to use Twisted Web. So learning things about Twisted Web isn't going to hurt you.
If you're going to be generating any amount of HTML, then you very much want to use an HTML templating library. By this point no one should be constructing HTML using primitive string operations. So if you want to use one of the other Python HTML templating libraries out there - Cheetah, Quixote, etc - instead of Nevow, that's great! You're just going to use the templating library to get a string to write out in response to an HTTP request. Twisted Web doesn't care where the string came from.
And if you do want to do something with Django (or another WSGI-based system), then you can certainly deploy this in your Twisted process using Twisted Web's WSGI support. And you can still interact between the WSGI applications and the rest of your Twisted code, as long as you exercise a little care - WSGI applications run in a thread pool, and Twisted APIs are not thread-safe, you have to invoke them with reactor.callFromThread or one of the small number of similar APIs (in particular, blockingCallFromThread is sometimes a useful higher-level tool to use).

At this point Nevow is definitively dead. As illustration of how dead it is, there is a bug that prevents installation of Nevow using pip, which was fixed on trunk in 2009, but it isn't in any release because there has been no release since then.
twisted.web and in particular twisted.web.template cover pretty much all of what was useful in Nevow, and should be used for any new project that was considering using Nevow.

Learning to write reusable libraries [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
We need to write simple scripts to manipulate the configuration of our load balancers (ie, drain nodes from pools, enabled or disable traffic rules). The load balancers have a SOAP API (defined through a bunch of WSDL files) which is very comprehensive but using it is quite low-level with a lot of manual error checking and list manipulation. It doesn't tend to produce reusable, robust code.
I'd like to write a Python library to handle the nitty-gritty of interacting with the SOAP interface but I don't really know where to start; all of my coding experience is with writing one-off monolithic programs for specific jobs. This is fine for small jobs but it's not helping me or my coworkers -- we're reinventing the wheel with a different number of spokes each time :~)
The API already provides methods like getPoolNames() and getDrainingNodes() but they're a bit awkward to use. Most take a list of nodes and return another list, so (say) working out which virtual servers are enabled involves this sort of thing:
names = conn.getVirtualServerNames()
enabled = conn.getEnabled(names)
for i in range(0, len(names)):
if (enabled[i]):
print names[i]
conn.setEnabled(['www.example.com'], [0])
Whereas something like this:
lb = LoadBalancer('hostname')
for name in [vs.name for vs in lb.virtualServers() if vs.isEnabled()]:
print name
www = lb.virtualServer('www.example.com').disable()
is more Pythonic and (IMHO) easier.
There are a lot of things I'm not sure about: how to handle errors, how to deal with 20-odd WSDL files (a SOAPpy/suds instance for each?) and how much boilerplate translation from the API methods to my methods I'll need to do.
This is more an example of a wider problem (how to learn to write libraries instead of one-off scripts) so I don't want answers to these specific questions -- they're there to demonstrate my thinking and illustrate my problem. I recognise a code smell in the way I do things at the moment (one-off, non-reusable code) but I don't know how to fix it. How does one get into the mindset for tackling problems at a more abstract level? How do you 'learn' software design?

"I don't really know where to start"
Clearly false. You provided an excellent example. Just do more of that. It's that simple.
"There are a lot of things I'm not sure about: how to handle errors, how to deal with 20-odd WSDL files (a SOAPpy/suds instance for each?) and how much boilerplate translation from the API methods to my methods I'll need to do."
Handle errors by raising an exception. That's enough. Remember, you're still going to have high-level scripts using your API library.
20-odd WSDL files? Just pick something for now. Don't overengineer this. Design the API -- as you did with your example -- for the things you want to do. The WSDL's and the number of instances will become clear as you go. One, Ten, Twenty doesn't really matter to users of your API library. It only matters to you, the maintainer. Focus on the users.
Boilerplate translation? As little as possible. Focus on what parts of these interfaces you use with your actual scripts. Translate just what you need and nothing more.
An API is not fixed, cast in concrete, a thing of beauty and a joy forever. It's just a module (in your case a package might be better) that does some useful stuff.
It will undergo constant change and evolution.
Don't overengineer the first release. Build something useful that works for one use case. Then add use cases to it.
"But what if I realize I did something wrong?" That's inevitable, you'll always reach this point. Don't worry about it now.
The most important thing about writing an API library is writing the unit tests that (a) demonstrate how it works and (b) prove that it actually works.

There's an excellent presentation by Joshua Bloch on API design (and thus leading to library design). It's well worth watching. IIRC it's Java-focused, but the principles will apply to any language.

If you are not afraid of C++, there is an excellent book on the subject called "Large-scale C++ Software Design".
This book will guide you through the steps of designing a library by introducing "physical" and "logical" design.
For instance, you'll learn to flatten your components' hierarchy, to restrict dependency between components, to create levels of abstraction.
The is really "the" book on software design IMHO.

Are there any cpython libraries that work with jsr168 and/or jsr286?

On a Java portal you can have portlets that include data provided by other applications. We want to replace our existing Java portal with a Django application, which means duplicating the Java portal's ability to display portlets. The two Sun specifications in question that we want to duplicate are JSR168 and JSR286.
I need a cPython solution. Not Jython or Java. Nothing against those tools, we just don't use them. For the record, the Jython based Portletpy does the opposite of what we are aiming to do.
Also, I suspect this question has been caused by a misunderstanding on our part of how the JSR168/JSR286 specification works. I think that JSR168/JSR286 is an arcane protocol for communicating some sort of content between separate applications, but in the Java world that tends to be done by other methods such as SOAP. Instead, the issue might be that these protocols are simply definitions of how to display content objects in views. If all we have to do is handle SOAP calls and display data, then this whole question is moot.
Simple architecture image below of what we think we want to do:

I'm not sure you can do this. From JSR 168:
If I understand correctly, you want the Django application to take the place of the existing "Java Portal/Portlet Container" in the diagram. Unfortunately, the interface between the portlet container and the individual portlets is using in-memory API calls, not as a Web service. There's no easy URL-like interface where you can call into the Java piece to get a chunk of HTML which you then incorporate into a Django-served page.
JSR 286 is an update and while it refines the mechanisms for communicating between portlets, as well as serving resources from portlets, it doesn't really change the above model radically.
I'm not saying it couldn't be done - just that there's no easy, standard way to do it.

One way to get around this could be using a WSRP (Web Services for Remote Portlets, see Wikipedia) producer, that converts a JSR 168/286 into web services and consume them from django. But it seems that WSRP has not been very popular and I couldn't find any Python platform implementations (although partial works could exist). Beside this, I'm also interested in this topic.

Why is the PyObjC documentation so bad? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
For example, http://developer.apple.com/cocoa/pyobjc.html is still for OS X 10.4 Tiger, not 10.5 Leopard.. And that's the official Apple documentation for it..
The official PyObjC page is equally bad, http://pyobjc.sourceforge.net/
It's so bad it's baffling.. I'm considering learning Ruby primarily because the RubyCocoa stuff is so much better documented, and there's lots of decent tutorials (http://www.rubycocoa.com/ for example), and because of the Shoes GUI toolkit..
Even this badly-auto-translated Japanese tutorial is more useful than the rest of the documentation I could find..
All I want to do is create fairly simple Python applications with Cocoa GUI's..
Can anyone shed light on the horrible documentation, or point me at some tutorials that don't just give you huge blocks of code and assume you know what NSThread.detachNewThreadSelector_toTarget_withObject_("queryController", self, None) does..?

The main reason for the lack of documentation for PyObjC is that there is one developer (me), and as most developers I don't particularly like writing documentation. Because PyObjC is a side project for me I tend to focus on working on features and bugfixes, because that's more interesting for me.
The best way to improve the documentation is to volunteer to help on the pyobjc-dev mailing list.
As an aside: the pythonmac-sig mailinglist (see google) is an excelent resource for getting help on Python on MacOSX (not just PyObjC).

I agree that that tutorial is flawed, throwing random, unexplained code right in front of your eyes. It introduces concepts such as the autorelease pool and user defaults without explaining why you would want them ("Autorelease pool for memory management" is hardly an explanation).
That said…
basically all I want to do is write Cocoa applications without having to learn ObjC.
I'm afraid that for the time being, you will need a basic grasp of ObjC in order to benefit from any language that uses Cocoa. PyObjC, RubyCocoa, Nu and others are niches at best, and all of them were developed by people intimately familiar with the ins and outs of ObjC and Cocoa.
For now, you will benefit the most if you realistically see those bridges as useful where scripting languages truly shine, rather than trying to build a whole application with them. While this has been done (with LimeChat, I'm using a RubyCocoa-written app right now), it is rare and likely will be for a while.

To be blunt:
If you want to be an effective Cocoa programmer, you must learn Objective-C. End of story.
Neither Python or Ruby are a substitute for Objective-C via their respective bridges. You still have to understand the Objective-C APIs, the behaviors inherent to NSObject derived classes, and many other details of Cocoa.
PyObjC and RubyCocoa are a great way to access Python or Ruby functionality from a Cocoa application, including building a Cocoa application mostly -- if not entirely -- in Python or Ruby. But success therein is founded upon a thorough understanding of Cocoa and the Objective-C APIs it is composed of.

Tom's and Martin's response are definitely true (in just about any open source project, you'll find that most contributors are particularly interested in, well, developing; not so much in semi-related matters such as documentation), but I don't think your particular question at the end would fit well inside PyObjC documentation.
NSThread.detachNewThreadSelector_toTarget_withObject_("queryController", self, None)
NSThread is part of the Cocoa API, and as such documented over at Apple, including the particular method + detachNewThreadSelector:toTarget:withObject: (I'd link there, but apparently stackoverflow has bugs with parsing it). The CocoaDev wiki also has an article.
I don't think it would be a good idea for PyObjC to attempt to document Cocoa, other than a few basic examples of how to use it from within Python. Explaining selectors is also likely outside the scope of PyObjC, as those, too, are a feature of Objective-C, not PyObjC specifically.

I stumbled across a good tutorial on PyObjC/Cocoa:
http://lethain.com/entry/2008/aug/22/an-epic-introduction-to-pyobjc-and-cocoa/

All I want to do is create fairly simple Python applications with Cocoa GUI's.. Can anyone shed light on the horrible documentation, or point me at some tutorials that don't just give you huge blocks of code and assume you know what NSThread.detachNewThreadSelector_toTarget_withObject_("queryController", self, None) does..?
[...]
basically all I want to do is write Cocoa applications without having to learn ObjC.
Although I basically agree with Soeren's response, I'd take it even further:
It will be a long time, if ever, before you can use Cocoa without some understanding of Objective C. Cocoa isn't an abstraction built independently from Objective C, it is explicitly tied to it. You can see this in the example line of code you quoted above:
NSThread.detachNewThreadSelector_toTarget_withObject_("queryController", self, None)
This is the Python way of writing the Objective C line:
[NSThread detachNewThreadSelector:#selector(queryController:) toTarget:self withObject:nil];
Now, it's important to notice here that this line can be seen in two ways: (1) as a line of Objective C, or (2) as an invocation of the Cocoa frameworks. We see it as (1) by the syntax. We see it as (2) by recognizing that NSThread is a Cocoa framework which provides a set of handy features. In this case, this particular Cocoa framework is making it easy for us to have an object start doing something on a new thread.
But the kicker is this: The Cocoa framework here (NSThread) is providing us this handy service in a way that is explicitly tied to the language the framework has been written in. Namely, NSThread gave us a feature that explicitly refers to "selectors". Selectors are, in point of fact, the name for something fundamental about how Objective C works.
So there's the rub. Cocoa is fundamentally an Objective-C creation, and its creators have built it with Objective C in mind. I'm not claiming that it's impossible to translate the interface to the Cocoa features into a form more natural for other languages. It's just that as soon as you change the Cocoa framework to stop referring to "selectors", it's not really the Cocoa framework any more. It's a translated version. And once you start going down that road, I'm guessing things get really messy. You're trying to keep up with Apple as they update Cocoa, maybe you hit some parts of Cocoa that just don't translate well into the new language, whatever. So instead, things like PyObjC opt to expose Cocoa directly, in a way that has a very clear and simple correlation. As they say in the documentation:
In order to have a lossless and unambiguous translation between Objective-C messages and Python methods, the Python method name equivalent is simply the selector with colons replaced by underscores.
Sure, it's a bit ugly, and it does mean you need to know something about Objective-C, but that's because the alternative, if one truly exists, is not necessarily better.

I didn't know anything at all about Objective C or Cocoa (but plenty about Python), but I am now writing a rather complex application in PyObjc. How did I learn? I picked up Cocoa Programming for OSX and went through the whole book (a pretty quick process) using PyObjC. Just ignore anything about memory management and you'll pretty much be fine. The only caveat is that very occasionally you have to use a decorator like endSheetMethod (actually I think that's the only one I've hit):
#PyObjcTools.AppHelper.endSheetMethod
def alertEnded_code_context_(self, alert, choice, context):
pass

This answer isn't going to be very helpful but, as a developer I hate doing documentation. This being a opensource project, it's hard to find people to do documentation.

Tom says it all really. Lots of open source projects have dedicated developers and few who are interested in documenting. It isn't helped by the fact that goalposts can shift on a daily basis which means documentation not only has to be created, but maintained.