I'm writing an application that sends files over network, I want to develop a custom protocol to not limit myself in term on feature richness (http wouldn't be appropriate, the nearest thing is the bittorrent protocol maybe).
I've tried with twisted, I've built a good app but there's a bug in twisted that makes my GUI blocking, so I've to switch to another framework/strategy.
What do you suggest? Using raw sockets and using gtk mainloop (there are select-like functions in the toolkit) is too much difficult?
It's viable running two mainloops in different threads?
Asking for suggestions
Disclaimer: I have little experience with network applications.
That being said, the raw sockets isn't terribly difficult to wrap your head around/use, especially if you're not too worried about optimization. That takes more thought, of course. But using GTK and raw sockets should be fairly straightforward. Especially since you've used the twisted framework, which IIRC, just abstracts some of the more nitty-gritty details of socket managing.
Two threads: one for the GUI, one for sending/receiving data. Tkinter would be a perfectly fine toolkit for this. You don't need twisted or any other external libraries or toolkits -- what comes out of the box is sufficient to get the job done.
If your application is somewhat similar to bittorrent, why not check the source code of Deluge http://deluge-torrent.org/ and build from it? It is written in Python, it does use the bittorrent protocol and it does have a GTK user interface.
As an alternative to twisted and whatever GUI library you seem to be using, how about trying PyQt? It provides a GUI and non-blocking sockets all in the same event loop. That way you don't have to worry about interoperability issues, which seem to be the issue you are facing.
Hope this helps!
Related
I am trying to decide on a technology for developing a desktop application that can interface with the serial port. I have looked into python and it looks like a console based app would not be difficult - http://pyserial.sourceforge.net/ , and I have also looked at PyQt, which would fulfill the GUI portion of my project.
But is it possible to include 3rd party modules like PySerial in PyQt?
PyQt and pyserial play nice together (in fact, I'm using them in one of my apps.) The examples on the pyserial website are mostly simple console-like examples, but there's no reason you cannot take data from any of the pyserial objects and use them in PyQt. You could, for instance, take data that you received over a serial port and push it into a QByteArray or NumPy array or anything similar as fits your fancy.
A couple of caveats: if you use Serial.readline(), it is blocking until it gets a newline (\n). This could be bad for your GUI. If you must use readline() instead of read(), I recommend putting your pyserial related activities in a separate thread. How you do this is up to you, but I'd recommend using Qt's built in threading. You can then do data conversions to Qt types and such inside the thread. More info on threading in pyqt.
Edit: almost forgot. If anyone cares, you can use PyQt to write console apps too. Just use QCoreApplication instead of QApplication. Not quite sure why one would do that in python, however, unless you were really fond of Qt's data types... in this case you could use PyQt and pyserial to write a pure console app :D
I have been playing around with the twisted framework for about a week now(more because of curiosity rather than having to use it) and its been a lot of fun doing event driven asynchronous network programming.
However, there is something that I fail to understand. The twisted documentation starts off with
Twisted is a framework designed to be very flexible and let you write powerful servers.
My doubt is :- Why do we need such an event-driven library to write powerful servers when there are already very efficient implementations of various servers out there?
Surely, there must have been more than a couple of concrete implementations which the twisted developers had in mind while writing this event-driven I\O library. What are those? Why exactly was twisted made?
In a comment on another answer, you say "Every library is supposed to have ...". "Supposed" by whom? Having use-cases is certainly a nice way to nail down your requirements, but it's not the only way. It also doesn't make sense to talk about the use-cases for all of Twisted at once. There is no use case that justifies every single API in Twisted. There are hundreds or thousands of different use cases, each which justifies a lesser or greater subdivision of Twisted. These came and went over the years of Twisted's development, and no attempt has been made to keep a list of them. I can say that I worked on part of Twisted Names so that I would have a topic for a paper I was presenting at the time. I implemented the vt102 parser in Twisted Conch because I am obsessed with terminals and wanted a fun project involving them. And I implemented the IMAP4 support in Twisted Mail because I worked at a company developing a mail server which required tighter control over the mail store than any other IMAP4 server at the time offered.
So, as you can see, different parts of Twisted were written for widely differing reasons (and I've only given examples of my own reasons, not the reasons of any other developers).
The initial reason for a program being written often doesn't matter much in the long run though. Now the code is written: Twisted Names now runs the DNS for many domain names on the internet, the vt102 parser helped me get a job, and the company that drove the IMAP4 development is out of business. What really matters is what useful things you can do with the code now. As MattH points out, the resulting plethora of functionality has resulted in a library that (perhaps uniquely) addresses a wide array of interesting problems.
Why do we need such an event-driven library to write powerful servers when there are already very efficient implementations of various servers out there?
So paraphrasing: you can't imagine why anyone would need a toolkit when dyecast products already exist?
I'm guessing you've never needed to knock up a protocol gateway, e.g.
- write a daemon to md5 local files on demand over a unix socket
- interrogate a piece of software using udp and expose statistics over http.
I wrote a little proof-of-concept for the second example for a question here on SO in a handful of minutes. I couldn't do that without twisted.
Have you looked at: ProjectsUsingTwisted?
More on 'why': (disclaimer: I'm not a developer of Twisted proper), it's necessary to consider Twisted's high age (relative to Python's). When Twisted was written there was no sufficiently powerful non-blocking network/event driven library written around the reactor pattern (almost everyone was using threads back then). Twisted's initial use case was a large multiplayer game, although the specifics of this game seems to be somewhat lost in time.
Since the origins, as #MattH's link suggest, a very large amount of various network servers written in Python is based on Twisted.
This PyCon talk by the creator of Twisted should give you answers.
It has changed my opinion of Twisted. Before I viewed it as a massive piece of software with interfaces and weird names, two things that many developers dislike but that are actually just superficial things, and now that I’ve seen the history behind and the amazing number of use cases I respect it a lot. Life is short, you need Twisted :)
I was wondering what good networking libraries/frameworks there are for Python.
Please provide a link to the standard API documentation for the library, and perhaps a link to a decent tutorial to get started with it.
A comment or two about its advantages/disadvantages would be nice as well.
The standard library has asyncore which is good for very simple stuff as well as the SocketServer stuff if you'd prefer something that does threads. There's also Twisted but the barrier of entry to that is a bit high if you're not used to event-driven IO. If you're after web frameworks, CherryPy is a good start or there's Django and TurboGears if you're looking for something more full-featured.
Consider the Twisted framework. The advantage:
solid reactor implementation
support for almost all network protocols found in the wild
well documented
Disadvantages:
it's huge
the asynchronous APIs need some time to get used to (but once you are familiar, things are actually pretty usable)
CPython itself ships with a tiny reactor/socket package. Never used it myself, though.
Twisted is the most complete, and complex, of all Python networking frameworks.
It's well-established and very complete, but it has a steep learning curve.
Documentation here; FAQ here.
In case you want to build/manipulate your own packets there is Scapy too :)
The usage is pretty straight forward, it lets you do whatever you want with the packets
and it's multi-Platform.
Project Page: http://www.secdev.org/projects/scapy/
Docs: http://www.secdev.org/projects/scapy/doc/
Example: http://www.secdev.org/projects/scapy/demo.html
I am making a simple multiplayer economic game in pygame. It consists of turns of a certain length, at the end of which, data is sent to the central server. A few quick calculations are done on the data and the results are sent back to the players. My question is how I should implement the network support. I was looking at Twisted and at Pyro and any suggestions or advice would be appreciated.
Twisted would certainly be a good idea. Here is example code that integrates twisted and pygame.
I've nothing against Twisted and PyRo, but the sort of simple messages you're going to be sending don't require anything like that and might be overcomplicated by using some sort of framework. Pickling an object and sending it over a socket is actually a very easy operation and well worth trying, even if you do eventually go with a more heavyweight framework. Don't fear the network!
There are a number of plug-and-play libraries tailored specifically to work nicely with PyGame on the pygame.org website.
These include PodSixNet, PygLibs.net, and my own Mastermind (which is, at the risk of self-aggrandizement, lightweight, easy to use, and comes with a simple tutorial).
Many python libraries, even recently written ones, use httplib2 or the socket interface to perform networking tasks.
Those are obviously easier to code on than Twisted due to their blocking nature, but I think this is a drawback when integrating them with other code, especially GUI one. If you want scalability, concurrency or GUI integration while avoiding multithreading, Twisted is then a natural choice.
So I would be interested in opinions in those matters:
Should new networking code (with the exception of small command line tools) be written with Twisted?
Would you mix Twisted, http2lib or socket code in the same project?
Is Twisted pythonic for most libraries (it is more complex than alternatives, introduce a dependency to a non-standard package...)?
Edit: please let me phrase this in another way. Do you feel writing new library code with Twisted may add a barrier to its adoption? Twisted has obvious benefits (especially portability and scalability as stated by gimel), but the fact that it is not a core python library may be considered by some as a drawback.
See asychronous-programming-in-python-twisted, you'll have to decide if depending on a non-standard (external) library fits your needs. Note the answer by #Glyph, he is the founder of the Twisted project, and can authoritatively answer any Twisted related question.
At the core of libraries like Twisted, the function in the main loop is not sleep, but an operating system call like select() or poll(), as exposed by a module like the Python select module. I say "like" select, because this is an API that varies a lot between platforms, and almost every GUI toolkit has its own version. Twisted currently provides an abstract interface to 14 different variations on this theme. The common thing that such an API provides is provide a way to say "Here are a list of events that I'm waiting for. Go to sleep until one of them happens, then wake up and tell me which one of them it was."
Should new networking code (with the exception of small command line tools) be written with Twisted?
Maybe. It really depends. Sometimes its just easy enough to wrap the blocking calls in their own thread. Twisted is good for large scale network code.
Would you mix Twisted, http2lib or socket code in the same project?
Sure. But just remember that Twisted is single threaded, and that any blocking call in Twisted will block the entire engine.
Is Twisted pythonic for most libraries (it is more complex than alternatives, introduce a dependency to a non-standard package...)?
There are many Twisted zealots that will say it belongs in the Python standard library. But many people can implement decent networking code with asyncore/asynchat.