Well this is my last ditch effort of finding help/guidance. We've only just started learning Python at university and i really want to get a jump on everyone else and i've really been focusing on networking with Python. I've had a look at the twisted framework and it's massive ! even the krondo.com site is.
I've seen Concurrence for python on LinuxQuestions a couple of times, which seems easier (however it looks like Concurrence has been dis-continued?), and was wondering if im better off knuckling down and really getting into twisted or maybe trying something else.... Its not that im afraid of hard work, but i dont want to go down a path that when im finished, i find out that know one use's it out in the real world, or knows of it.
Im sorry its not a technical question, but this is really the only place where i know i can get proper guidance. If you think putting my head down and learning twisted will be worth my while, then ill do that. Thanks in advance for your time.
Honestly, I would take a step back on frameworks and work with the Python standard library on a more basic level. The python docs are a very good and authoritative source of information on Python. If you're looking into networking, here's a path I'd recommend looking over:
Sockets
Learn how to work with basic socket connections, making clients, and making servers.
SSL
A lot of web services use this to communicate data securely for APIs. It may be difficult to wrap your head around, but it's a highly valuable piece of knowledge to have.
Select
One way of handling multiple clients connecting to a service. This gets into the concurrency realm which is a fairly sought after skill.
Threads
Another way of handling multiple connections, networking tasks.
Protocols
Now that you've wrapped your head around basic networking, it's time to get into protocols. These are sending specifically formatted data over the network to a server that can handle such data. Yes, I'd recommend going through all the different protocols.
Data Parsing
Data that is sent and received during different protocol communication can be encoded in various ways to make it easier to deal with data.
Pack and Unpack
Sometimes network communication isn't done with a standard format. These functions are helpful in dealing with binary protocols if there's no library available.
Pickle
This is some advanced stuff and python specific. You can use it to send python objects over the network. Just be careful to do this on trusted networks only for security purposes.
With that in mind these are general recommendations. Check the docs out, write some code, and ask questions if you get stuck.
Related
I was looking at a book on python network programming and i wanted to know what would be the benefits to learning python network programming comprehensively? This would be in the context of being able to develop some really cool, ground breaking web apps. I am a python newbe so all opinions woul be appreciated.
Kind Regards
4 Years later:
This was 4yrs ago, its crazy how much I've grown as a developer. Regarding how it has helped, I've developed an email application, a chat application using Objective C, python Twisted on the server side, it also helped with developing my apns push notification pipeline.
If you want to develop wed apps, than you should rather focus on web frameworks like Django or Pylons.
"Network programming" isn't about "cool web apps". It's more about creating servers, and creating clients that talk to servers. It's about sockets, tcp/ip communication, XMLRPC, http protocols and all the technologies that let two computers talk to each other.
If all you're interested in is web apps, learning network programming won't benefit you a whole lot.
"python network programming" isn't any special kind of network programming. It sounds like if you had a better grasp on network programming you would be able to see where python would fit in to your overall design. And instead of reading a generic book about it, you would dig through the python API's and go from there.
The cool thing about python is that it's a huge collection of libraries which are optimized to each task. At work we use python to do all our server-side heavy lifting. We then use jquery and the Objective-J based Cappuccino to present an interface to the user.
The network is and always will be the sexiest arena for a hacker. An attacker can do almost anything with simple network access, such as scan for hosts, inject packets, sniff data, remotely exploit hosts, and much more. But if you are an attacker who has worked your way into the deepest depths of an enterprise target, you may find yourself in a bit of a conundrum: you have no tools to execute network attacks. No netcat. No Wireshark. No compiler and no means to install one. However, you might be surprised to find that in many cases, you’ll find a Python install.
Hence, the one benefit of python network programming I see is that one has a chance of becoming a penetration tester or an Offensive security guy.
Edit (10-06-2020):
I have no knowledge of what I was thinking when I was writing this answer. It's clearly not helpful.
Regarding the question, making web apps is not equivalent to network programming. In network programming with python, you can start with lower level of the OSI model by using libraries like scapy. Here you can make raw packets and understand the various protocols with it. And then maybe move to application level with libraries like Scrapy which involves web scraping, making http requests, etc. But develping python web apps would use tools like flask, django, jinja, etc and the development of a web app can take the form totally different than that of a regular scripting tool.
Python's stronger web tool is definitely Django.
Said that the biggest benefit of learning it is to achieve a pretty robust backend on your web project.
So, I programmed this twisted application a few months ago, which I now would like to extend with a web-based user interface for configuration.
The Twisted website recommends Nevow, but I am not really sure if this is a good choice. Their website is down for a while it seems and their launchpad page hadn't seen any update in half a year. Is this project dead?
Additionally I have seen discussion of moving parts of Nevow into twisted.web on the twisted-web mailinglist. So, is it still recommended for new developments?
Another idea was using Django. I would need user authentication and permissions anyway in the config-interface, and I am quite familiar with it. (I have never worked with Nevow or twisted.web)
But it seems quite difficult to interface both worlds, all I could find were examples of running Django with WSGI in Twisted.
Are there any other possibilities to have a slick looking user interface on top of twisted?
First, let me address the perception that Nevow is dead. The launchpad project containing the code for Nevow (and the rest of the Divmod projects) is divmod.org on launchpad. A hardware failure has badly impacted the project's public presence, but it's still there, and other things (like the wiki and the tickets) are in the process of being recovered. There isn't a lot of active maintenance work going on right now, but that's mostly because it's good enough for most of its users; there are lots of people who depend on Nevow and would be very upset if it stopped working. Those people have the skills and experience necessary to continue maintaining it. So, while it's not being actively promoted right now, I think it's unlikely that it's going to go away.
My long-term hope for Nevow would be as follows. (I'd say "plan", but since I haven't been actively involved with its maintenance lately, this is really up to those who are.) First, I'd like to extract its templating facilities and move them into twisted.web. The clean, non-deprecated API for Nevow is mostly covered by nevow.page.Element and the various loaders. Twisted itself wants to generate HTML in a few places and these facilities could be useful. Then we should throw out the "appserver" and resource-model parts of Nevow. Those are mostly just a random collection of bugfixes or alterations for twisted.web, most of which were present in some form in twisted.web2 and will therefore either be rolled back into twisted.web anyway, or have already been applied there. Finally there's the question of Athena. While two-way communication is one of Twisted's strengths, Athena is itself a gigantic, sprawling JavaScript codebase and should probably remain its own project.
Third, on to the main question, given this information, what should you do now?
Generally speaking, I'd say, "use nevow". The project has some warts, it needs more documentation and its API needs to be trimmed to eliminate some old and broken stuff, but it's still quite useful and very much alive. To make up for the slightly sparse documentation, you can join the #divmod or #twisted.web channels on Freenode to get help with it. If you help out by contributing patches where you can, you will find that you'll get a lot of enthusiastic help there. When you ignore the deprecated parts Nevow has a pretty small, sane, twisted friendly API. The consequence of the plan for Nevow's evolution that I outlined above are actually pretty minimal. If it even happens at all, what it means for you is, in 1-5 years, when you upgrade to a new version of Twisted, you'll get a couple of deprecation warnings, change some import lines in your code from from nevow.page import ...; from nevow.loaders import ... to some hypothetical new thing like from twisted.web.page.element import ...; from twisted.web.page.templates import ..., or somesuch. Most of the API past that point should remain the same, and definitely the high-level concepts shouldn't change much.
The main advantage that you get from using Nevow is that it's async-friendly and can render pages in your main thread without blocking things. Plus, you can then get really easy COMET for free with Athena.
You can also use Django. This is not quite as async-friendly but it obviously does have a broader base of support. However, "not as async friendly" doesn't mean "hard to use". You can run it in twisted.web via WSGIResource, and simply use blockingCallFromThread in your Django application to invoke any Twisted API that returns a Deferred, which should be powerful enough to do just about anything you want. If you have a more specific question about how to instantiate Twisted web resources to combine Twisted Web and Django, you should probably ask it in its own Stack Overflow question.
Nevow is still a good choice if you want support for Deferreds in the templating system you use (it's not dead). It also has a few advantages over plain Twisted Web when it comes to complicated URL dispatch. However, it is basically just a templating system. Twisted Web is the real web server. So either way, you're going to use Twisted Web. In fact, even if you use Django in Twisted Web's WSGI container, you're still going to use Twisted Web. So learning things about Twisted Web isn't going to hurt you.
If you're going to be generating any amount of HTML, then you very much want to use an HTML templating library. By this point no one should be constructing HTML using primitive string operations. So if you want to use one of the other Python HTML templating libraries out there - Cheetah, Quixote, etc - instead of Nevow, that's great! You're just going to use the templating library to get a string to write out in response to an HTTP request. Twisted Web doesn't care where the string came from.
And if you do want to do something with Django (or another WSGI-based system), then you can certainly deploy this in your Twisted process using Twisted Web's WSGI support. And you can still interact between the WSGI applications and the rest of your Twisted code, as long as you exercise a little care - WSGI applications run in a thread pool, and Twisted APIs are not thread-safe, you have to invoke them with reactor.callFromThread or one of the small number of similar APIs (in particular, blockingCallFromThread is sometimes a useful higher-level tool to use).
At this point Nevow is definitively dead. As illustration of how dead it is, there is a bug that prevents installation of Nevow using pip, which was fixed on trunk in 2009, but it isn't in any release because there has been no release since then.
twisted.web and in particular twisted.web.template cover pretty much all of what was useful in Nevow, and should be used for any new project that was considering using Nevow.
I'm building a little MMORPG and im trying to use asyncore rather then threading.
1) How would i send data to certain clients, because in threading a saved each clients socket and current in a dictionary a with its unique id as a key. So how could i subjective send data to all the clients.
Thankyou, please say if i didnt give enough information
I find it great that you want to move to asynchronous programming instead of threading, since it is so much reliable and easier to debug.
However, asyncore is a bad library to do so. I don't advise you use it at all, since it requires a significant rewrite to do simple things like read standard io.
I suggest you move to twisted - it is a great asynchronous framework, well tested and developed, with good api documentation and good community support.
Regardless of your decision on which library to use, I find this series of blog posts by Dave Peticolas to be a great source of beginner information on asynchronous programming. Please read it.
There are some games under development using twisted. One example is Minions of Mirth - I never played it but it seems cool.
There's also divmod's imaginary - it's a simulationist's take on the realm of role playing, interactive fiction, and multiplayer dungeons. It incorporates gameplay features from each area while attempting to provide a richer environment than is generally available from existing systems.
Hope I helped.
I have been playing around with the twisted framework for about a week now(more because of curiosity rather than having to use it) and its been a lot of fun doing event driven asynchronous network programming.
However, there is something that I fail to understand. The twisted documentation starts off with
Twisted is a framework designed to be very flexible and let you write powerful servers.
My doubt is :- Why do we need such an event-driven library to write powerful servers when there are already very efficient implementations of various servers out there?
Surely, there must have been more than a couple of concrete implementations which the twisted developers had in mind while writing this event-driven I\O library. What are those? Why exactly was twisted made?
In a comment on another answer, you say "Every library is supposed to have ...". "Supposed" by whom? Having use-cases is certainly a nice way to nail down your requirements, but it's not the only way. It also doesn't make sense to talk about the use-cases for all of Twisted at once. There is no use case that justifies every single API in Twisted. There are hundreds or thousands of different use cases, each which justifies a lesser or greater subdivision of Twisted. These came and went over the years of Twisted's development, and no attempt has been made to keep a list of them. I can say that I worked on part of Twisted Names so that I would have a topic for a paper I was presenting at the time. I implemented the vt102 parser in Twisted Conch because I am obsessed with terminals and wanted a fun project involving them. And I implemented the IMAP4 support in Twisted Mail because I worked at a company developing a mail server which required tighter control over the mail store than any other IMAP4 server at the time offered.
So, as you can see, different parts of Twisted were written for widely differing reasons (and I've only given examples of my own reasons, not the reasons of any other developers).
The initial reason for a program being written often doesn't matter much in the long run though. Now the code is written: Twisted Names now runs the DNS for many domain names on the internet, the vt102 parser helped me get a job, and the company that drove the IMAP4 development is out of business. What really matters is what useful things you can do with the code now. As MattH points out, the resulting plethora of functionality has resulted in a library that (perhaps uniquely) addresses a wide array of interesting problems.
Why do we need such an event-driven library to write powerful servers when there are already very efficient implementations of various servers out there?
So paraphrasing: you can't imagine why anyone would need a toolkit when dyecast products already exist?
I'm guessing you've never needed to knock up a protocol gateway, e.g.
- write a daemon to md5 local files on demand over a unix socket
- interrogate a piece of software using udp and expose statistics over http.
I wrote a little proof-of-concept for the second example for a question here on SO in a handful of minutes. I couldn't do that without twisted.
Have you looked at: ProjectsUsingTwisted?
More on 'why': (disclaimer: I'm not a developer of Twisted proper), it's necessary to consider Twisted's high age (relative to Python's). When Twisted was written there was no sufficiently powerful non-blocking network/event driven library written around the reactor pattern (almost everyone was using threads back then). Twisted's initial use case was a large multiplayer game, although the specifics of this game seems to be somewhat lost in time.
Since the origins, as #MattH's link suggest, a very large amount of various network servers written in Python is based on Twisted.
This PyCon talk by the creator of Twisted should give you answers.
It has changed my opinion of Twisted. Before I viewed it as a massive piece of software with interfaces and weird names, two things that many developers dislike but that are actually just superficial things, and now that I’ve seen the history behind and the amazing number of use cases I respect it a lot. Life is short, you need Twisted :)
I am making a simple multiplayer economic game in pygame. It consists of turns of a certain length, at the end of which, data is sent to the central server. A few quick calculations are done on the data and the results are sent back to the players. My question is how I should implement the network support. I was looking at Twisted and at Pyro and any suggestions or advice would be appreciated.
Twisted would certainly be a good idea. Here is example code that integrates twisted and pygame.
I've nothing against Twisted and PyRo, but the sort of simple messages you're going to be sending don't require anything like that and might be overcomplicated by using some sort of framework. Pickling an object and sending it over a socket is actually a very easy operation and well worth trying, even if you do eventually go with a more heavyweight framework. Don't fear the network!
There are a number of plug-and-play libraries tailored specifically to work nicely with PyGame on the pygame.org website.
These include PodSixNet, PygLibs.net, and my own Mastermind (which is, at the risk of self-aggrandizement, lightweight, easy to use, and comes with a simple tutorial).