Can someone please tell how to write a Non-Blocking server code using the socket library alone.Thanks
Frankly, just don't (unless it's for an exercise). The Twisted Framework will do everything network-related for you, so you have to write only your protocol without caring about the transport layer. Writing socket code is not easy, so why not use code somebody else wrote and tested.
Why socket alone? It's so much simpler to use another standard library module, asyncore -- and if you can't, at the very least select!
If you're constrained by your homework's condition to only use socket, then I hope you can at least add threading (or multiprocessing), otherwise you're seriously out of luck -- you can make sockets with timeout, but juggling timing-out sockets without the needed help from any of the other obvious standard library modules (to support either async or threaded serving) is a serious mess indeed-y...;-).
Not sure what you mean by "socket library alone" - you surely will need other modules from the standard Python library.
The lowest level of non-blocking code is the select module. This allows you to have many simultaneous client connections, and reports which of them have input pending to process. So you select both the server (accept) socket, plus any client connections that you have already accepted. A thin layer on top of that is the asyncore module.
Use eventlets or gevent. It monkey patches existing libraries. socket module can be used without any changes. Though code appears synchronous, it executes asynchronously.
Example:
http://eventlet.net/doc/examples.html#socket-connect
Related
I am wondering how I can make a simple socket server in Python 2.7 which can handle and add/accept multiple clients at a time. I do not want to use Twisted nor threading, nor any libraries; just Python, and sockets. I have looked around SoF (stackoverflow- is that a thing?) and found people asking the same question but not getting a clear answer.
If you are wondering why I need to do this, It's because I want to create a simple data forwarder which forwards client data to another server. I think a very simple example showing me Server.py, Client1.py, and Client2.py is just what I need. Again, just a very simple example with no threading, no twisted, no libraries.
I hope you can help me, I'm fairly new to Python and I feel like this project will help get me on my feet, and I learn great from examples.
Consider using asyncio (available for python 3.3 and later).
Asyncio is the new python standard for single-threaded concurrent programming:
This module provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives.
The documentation provides a few examples:
TCP echo client
TCP echo server
If you're not ready to migrate to python 3, you can use trollius, the portage of asyncio for python 2. There is a few differences between the two modules, as listed in the documentation:
replace asyncio with trollius (or use import trollius as asyncio)
replace yield from ... with yield From(...)
replace yield from [] with yield From(None)
in coroutines, replace return res with raise Return(res)
Other solutions for single-threaded concurrent programming on python 2.7:
gevent: a coroutine-based Python networking library that uses greenlet.
asyncore: built-in asynchronous socket library (echo server example).
I am trying to develop a script in Python which would function like the NetDisturb utility. Some of you might ask why am I doing this if there is a ready made utility, but the thing is I want to integrate this in a web page. Using this web page I can access a particular set of interfaces which I already know or are present in the back end script and eventually degrade the performance or simply block packets.
I have succeeded a little but now I want to implement a socket connection between the two interfaces which would be connected. I am unable to have a full duplex communication using a socket connection. I am unable to decide which interface should act as master and which interface should act as slave. Because when I make one of the interface as master the listen and accept statements block further execution of the code.
Would using SOCK_DGRAM sockets instead of SOCK_STREAM help me?
You have to use non-blocking sockets. Here is an explanation how to use select to handle non-blocking sockets (for beginners I could really recommend the complete article, it is a good start). Alternatives would be a multi-threaded architecture or asynchore. If you want additionally a GUI, I can recommend pygtk for the interface and glib.io_add_watch to handle the sockets.
But in general I would recommend some high level framework like zeromq. A second high level alternative would be Twisted, but it has a non-pythonic Java-like API and is (IMO) badly documented.
thanks for the interesting responses thus far. In light of said responses I have changed my question a bit.
guess what I really need to know is, is socketserver as opposed to the straight-up socket library designed to handle both periods of latency and stress, i.e. does it have additional mechanisms or features that justify its implicitly advertised status as a "server," or is it just slightly easier to use?
everyone seems to be recommending socketserver but I'm still not entirely clear why, as opposed to socket.
thanks!!!
I've built some server programs in
python based on the standard socket
library
http://docs.python.org/library/socket.html
I've noticed that they seem to work
just fine except that without load
they have a tendency to go to sleep
after a while. I guess this may not
be an issue in production (no doubt
there will be plenty of other issues)
but I would like to know if I am
using the right code for the job here.
Looking around I saw that python also
provides a socketserver library -
http://docs.python.org/library/socketserver.html
The socket library provides the
ability to listen for multiple
connections, typically up to 5.
According to the socketserver page,
its services are synchronous, i.e.
blocking, but one may support
asynchronous behavior via threading.
I did notice it has the ability to
maintain a request queue, with a
default value of up to 5 requests...so
maybe not much difference there.
I have also read that Twisted runs
socketserver under the hood. Though I
would rather not get into a beast the
size of Twisted unless it's going to
be worthwhile.
so my question is, is socketserver
more robust than socket? If so, why?
(And how do you know?)
incidentally, is socketserver built on
top of python's socket or is it
entirely separate?
finally, as a bonus if anyone knows
what one could do wrong such that
standard sockets 'fall asleep' please
feel free to chime in on that too.
Oh, and I'm talking python 2.x rather
than 3.x here if that makes a
difference.
thanks folks!
jsh
Well, I don't have a technical answer but I've implemented SocketServer per folks' recommendations and it IS definitely more reliable. If anyone ever comes up with the low-level explanation please let me know...thanks!
The socket module is a very low-level module for sending and receiving packets. As said in the documentation, it "provides access to the BSD socket interface".
If you want something more elaborate, there is "socketserver" that takes care of the gory details for you, but it is still relatively low level.
On top of that you can find an HTTP server, with or without CGI, an XML-RPC server, and so on. These are frameworks, which usually means that their code calls your code. It makes things simpler because you just have to fill some "gaps" to have a fully working server, but it also means you have a little bit less control over what it does.
If you only need features of socketserver, I would probably go with it, unless you want to reinvent the wheel for some reason (and there are always good reasons to design new wheels, for example to understand how it works).
I'm looking for some general information on how I should approach a problem that I think Twisted is a great fit for. (I'm new to Twisted but not Python)
I have a home automation controller that can support a single TCP socket connection, sending and receiving binary data. I'd like to use XMPP as a bridge to the socket so a user can send commands and receive events.
I got a rudimentary socket connection working with Twisted that was able to send and receive commands from one of the examples in the O'Reilly book. I also have a fully working Python XMPP bot written with the SleekXMPP library that I'm happy with. I'm just not sure how to bring these together.
The basic scenario is:
User sends message to XMPP bot, which figures out what command to send to the socket
ASCII Socket command is converted to binary and sent to socket
Socket receives command and sends binary response
Binary response converted to ASCII
XMPP bot sends response back to user.
Network events (independent from user action) can also be received by network socket and should be sent to user
It's #6 that is presenting the challenge, otherwise I'd just open/close the socket on demand when in need to write something.
The part that I'm having trouble wrapping my head around with Twisted is the best approach to make these two event loops communicate. I've seen lots of info on using Queues, deferred, threads, select, etc. I have a feeling that Twisted can handle much of the complexity if I just learn to use the tool properly.
If someone can point me in the right direction, I'll take the ball and run with it. As I mentioned, I'm happy with my XMPP bot and I'd like to use the existing code. I think my problem now comes down to creating the socket in the background, then sending and receiving data from that socket in the foreground.
By the way, I'm very happy to share back my code once it's working so someone else can benefit from the help I'm asking for.
-- Scott
One of the problems with a non-blocking IO engine is that its pretty much all-or-nothing. As soon as you introduce blocking code, you can quickly lose most of the benefits of the event-driven asynch approach. Wherever possible (as a rule of thumb), its best to have the entire app running off the same reactor.
As i see it, you have two options:
Twisted is not thread safe. That said, you can use mechanisms like deferToThread and callFromThread to interact with other threads. This is by far the most confusing and needlessly complex approach for your application design. It's particularly painful if you're new to twisted.
Use twisted.words.protocols.jabber, and implement your XMPP stuff in a non-blocking manner using the twisted reactor. That way it will happily exist alongside all your other twisted code. and allow you to cleanly interact between protocols. It will result in less code, and a robust implementation that is easy to extend, maintain, and test.
What library should I use for network programming? Is sockets the best, or is there a higher level interface, that is standard?
I need something that will be pretty cross platform (ie. Linux, Windows, Mac OS X), and it only needs to be able to connect to other Python programs using the same library.
You just want to send python data between nodes (possibly on separate computers)? You might want to look at SimpleXMLRPCServer. It's based on the inbuilt HTTP server, which is based on the inbuilt Socket server, neither of which are the most industrial-strength servers around, but it's easy to set up in a hurry:
from SimpleXMLRPCServer import SimpleXMLRPCServer
server = SimpleXMLRPCServer(("localhost", 9876))
def my_func(a,b):
return a + b
server.register_function(my_func)
server.serve_forever()
And easy to connect to:
import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:9876')
print s.my_func(2,3)
>>> 5
print type(s.my_func(2,3))
>>> <type 'int'>
print s.my_func(2,3.0):
>>> 7.0
Twisted is popular for industrial applications, but it's got a brutal learning curve.
There is a framework that you may be interested in: Twisted
the answer depends on what you are trying to do.
"What library should I use for network programming?" is pretty vague.
for example, if you want to do HTTP, you might look at such standard libraries as urllib, urllib2, httplib, sockets. It all depends on which protocol you are looking to use, and which network layer you want to work at.
there are libraries in python for various network tasks... email, web, rpc, etc etc...
for starters, look over the standard library reference manual and see which tasks you want to do, then go from there: http://docs.python.org/library/index.html
As previously mentioned, Twisted is the most popular (by far). However, there are a lot of other alternative worth exploring. Tornado and Diesel are probably the top two contenders. A more complete comparison is found here.
Personally I just use asyncore from the standard library, which is a bit like a very cut-down version of Twisted, but this is because I prefer a simple and low level interface. If you want a higher level interface, especially just to communicate with another instance of your own program, you don't necessarily have to worry about the networking layer, and can consider something higher level like RPyC or pyro instead. The network then becomes an implementation detail and you can concentrate on just sending the information.
A lot of people like Twisted. I was a huge fan for awhile, but on working with it a bit and thinking about it more I've become disenchanted. It's complex, and last I looked, a lot of it had the assumption that your program would always be able to send data leading to possible situations in which your program grows memory usage endlessly buffering data to send that isn't being picked up by the remote side or isn't being picked up fast enough.
In my opinion, it depends a lot on what kind of network programming you want to do. A lot of times you don't really care about getting stuff done while you're waiting for IO. HTTP, for example, is very request-response oriented, and if you're only talking to a single server there is little reason to need something like Twisted and plain sockets or Python's built-in HTTP libraries will work fine.
If you're writing a server of any kind, you almost certainly need to be event-driven. Twisted has a slight edge there, but it still seems overly complex to me. Bittorrent, for example, was written in Python and doesn't use Twisted at all.
Another factor favoring Twisted is that there is code for a lot of protocols already written for it. So if you want to speak an existing protocol a lot of hard work may already have been done for you.
The socket module in the standard lib is in my opinion a good choice if you don't need high performance.
It is a very famous API that is known by almost every developpers of almost every languages. It's quite sipmple and there is a lot of information available on the internet. Moreover, it will be easier for other people to understand your code.
I guess that an event-driven framework like Twisted has better performance but in basic cases standard sockets are enough.
Of course, if you use a higher-level protocol (http, ftp...), you should use the corresponding implementation in the python standard library.
Socket is low level api, it is mapped directly to operating system interface.
Twisted, Tornado ... are high level framework (of course they are built on socket because socket is low level).
When it come to TCP/IP programming, you should have some basic knowledge to make a decision about what you shoud use:
Will you use well-known protocol like HTTP, FTP or create your own protocol?
Blocking or non-blocking? Twisted, Tornado are non-blocking framework (basically like nodejs).
Of course, socket can do everything because every other framework is base on its ;)