I am wondering how I can make a simple socket server in Python 2.7 which can handle and add/accept multiple clients at a time. I do not want to use Twisted nor threading, nor any libraries; just Python, and sockets. I have looked around SoF (stackoverflow- is that a thing?) and found people asking the same question but not getting a clear answer.
If you are wondering why I need to do this, It's because I want to create a simple data forwarder which forwards client data to another server. I think a very simple example showing me Server.py, Client1.py, and Client2.py is just what I need. Again, just a very simple example with no threading, no twisted, no libraries.
I hope you can help me, I'm fairly new to Python and I feel like this project will help get me on my feet, and I learn great from examples.
Consider using asyncio (available for python 3.3 and later).
Asyncio is the new python standard for single-threaded concurrent programming:
This module provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives.
The documentation provides a few examples:
TCP echo client
TCP echo server
If you're not ready to migrate to python 3, you can use trollius, the portage of asyncio for python 2. There is a few differences between the two modules, as listed in the documentation:
replace asyncio with trollius (or use import trollius as asyncio)
replace yield from ... with yield From(...)
replace yield from [] with yield From(None)
in coroutines, replace return res with raise Return(res)
Other solutions for single-threaded concurrent programming on python 2.7:
gevent: a coroutine-based Python networking library that uses greenlet.
asyncore: built-in asynchronous socket library (echo server example).
Related
I'm getting started with websockets. Trying to write a python server for a browser based (javascript) client.
I have also never really done asynchronous programming before (except "events"). I was trying to avoid it - I have searched and searched for an example of websocket use that did not involve importing tornado or asyncio. But I've found nothing, even the "most basic examples" do it.
So now I'm internalising it, but clear it up for me - is "full duplex" server code necessarily asynchronous?
Full-duplex servers are necessarily concurrent. In Tornado and asyncio, concurrency is based on the asynchronous programming model, so if you use a websocket library based on one of those packages, your code will need to be asynchronous.
But that's not the only way: full-duplex websockets could be implemented in a synchronous way by dedicating a thread to reading from the connection (in addition to whatever other threads you're using). I don't know if there are any python websocket implementations that support this kind of multithreading model for full-duplex, but that's how Go's websocket implementation works for example.
That said, the asynchronous/event-driven model is a natural fit for websockets (it's how everything works on the javascript side), so I would encourage you to get comfortable with that model instead of trying to find a way to work with websockets synchronously.
I am trying to develop a script in Python which would function like the NetDisturb utility. Some of you might ask why am I doing this if there is a ready made utility, but the thing is I want to integrate this in a web page. Using this web page I can access a particular set of interfaces which I already know or are present in the back end script and eventually degrade the performance or simply block packets.
I have succeeded a little but now I want to implement a socket connection between the two interfaces which would be connected. I am unable to have a full duplex communication using a socket connection. I am unable to decide which interface should act as master and which interface should act as slave. Because when I make one of the interface as master the listen and accept statements block further execution of the code.
Would using SOCK_DGRAM sockets instead of SOCK_STREAM help me?
You have to use non-blocking sockets. Here is an explanation how to use select to handle non-blocking sockets (for beginners I could really recommend the complete article, it is a good start). Alternatives would be a multi-threaded architecture or asynchore. If you want additionally a GUI, I can recommend pygtk for the interface and glib.io_add_watch to handle the sockets.
But in general I would recommend some high level framework like zeromq. A second high level alternative would be Twisted, but it has a non-pythonic Java-like API and is (IMO) badly documented.
I'm trying to work out how to approach building a "machine" to send and receive messages to WebSphere MQ, via Twisted. I want it to be as generic as possible, so I can reuse it for many different situations that interface with MQ.
I've used Twisted before, but many years ago now and I'm trying to resurrect the knowledge I once had...
The specific problem I'm having is how to implement the MQ IO using Twisted. There's a pymqi Python library that interfaces with MQ, and it provides all the interfaces I need. The MQ calls I need to implement are:
initiate a connection to a specific MQ server/port/channel/queue-manager/queue combination
take content and post it as a message to the desired queue
poll a queue and return the content of the next message in the queue
send a request to a queue manager to find the number of messages currently in a queue
All of these involve blocking calls to MQ.
As I'm intending to reuse the Twisted/MQ interface many times across a range of projects, should I be looking to implement the MQ IO as a Twisted protocol, as a Twisted transport, or just call the pymqi methods via deferToThread() calls? I realise this is a very broad question with possibly no definitive answer; I'm really after advice from those who may have encountered similar challenges before (i.e. working with queueing interfaces that will always block) and found a way that works well.
If you're going to use this functionality a lot, then having a native Twisted implementation is probably worth the effort. A wrapper based on deferToThread will be less work, but it will also be harder to test and debug, perform less well, and have problems on certain platforms where Python threads don't work extremely well (eg FreeBSD).
The approach to take for a native Twisted implementation is probably to implement a protocol that can speak to MQ servers and give it a rich API for interacting with channels, queues, queue managers, etc, and then build a layer on top of that which abstracts the actual network connection away from the application (as I believe mqi/pymqi largely do).
I'm trying to figure out how to make a server that can accept multiple clients at one time. While doing so, I need the client able to send and receive data from the server at the same time.
Would i have to make a threaded server? And have a thread for listening for data.
And then another thread for sending out information to the client?
Then for the client side, do i need use threads to send/get info?
Use async IO. There are dozen of async IO socket libs for python. Here is a brief benchmark.
I also tested gevent, eventlet, asyncore, twisted, pyev, pycurl, tornado.
Twsited
is stable but most slow and also not easy to start with.
gevent, eventlet (libevent)
easy to start and fast (code looks like blocking) but have some issues with forking.
pycurl (libcurl)
fast and easy (if you ok to do flags magic.. but there are example) but only http.
pyev (libev)
you must understand what you are doing almost like polling yourself.
tornado (polling in python)
fast enough and i think stable and also easy to start.
asyncore
really fast.. but don't use it.. it is ugly-ugly.
Don't use threads in python unless you are really know what you are doing.
Python and threads not really big friends (unless version <3.2 in 3.2 there must be a new gil).
On server-side you clearly need a Socket Server. This server creates a new thread for every incoming client connection.
Once a connection is established, both the client and the thread that was instantiated for the communication require an additional thread if they have to do other business in parallel than listening to the socket if the communication is synchronous. In case an asynchronous communication is what you need, then Python provides an excellent Asynchronous Socket Handler.
Use a asynchronous socket. Example server could be found here and the client code here. No direct hassle with threads. Depending on your needs, you probably don't need the asynchronous client.
You don't need threads for either client or server; you can instead select() to multiplex all the I/O inside a single thread.
Can someone please tell how to write a Non-Blocking server code using the socket library alone.Thanks
Frankly, just don't (unless it's for an exercise). The Twisted Framework will do everything network-related for you, so you have to write only your protocol without caring about the transport layer. Writing socket code is not easy, so why not use code somebody else wrote and tested.
Why socket alone? It's so much simpler to use another standard library module, asyncore -- and if you can't, at the very least select!
If you're constrained by your homework's condition to only use socket, then I hope you can at least add threading (or multiprocessing), otherwise you're seriously out of luck -- you can make sockets with timeout, but juggling timing-out sockets without the needed help from any of the other obvious standard library modules (to support either async or threaded serving) is a serious mess indeed-y...;-).
Not sure what you mean by "socket library alone" - you surely will need other modules from the standard Python library.
The lowest level of non-blocking code is the select module. This allows you to have many simultaneous client connections, and reports which of them have input pending to process. So you select both the server (accept) socket, plus any client connections that you have already accepted. A thin layer on top of that is the asyncore module.
Use eventlets or gevent. It monkey patches existing libraries. socket module can be used without any changes. Though code appears synchronous, it executes asynchronously.
Example:
http://eventlet.net/doc/examples.html#socket-connect