I'm getting started with websockets. Trying to write a python server for a browser based (javascript) client.
I have also never really done asynchronous programming before (except "events"). I was trying to avoid it - I have searched and searched for an example of websocket use that did not involve importing tornado or asyncio. But I've found nothing, even the "most basic examples" do it.
So now I'm internalising it, but clear it up for me - is "full duplex" server code necessarily asynchronous?
Full-duplex servers are necessarily concurrent. In Tornado and asyncio, concurrency is based on the asynchronous programming model, so if you use a websocket library based on one of those packages, your code will need to be asynchronous.
But that's not the only way: full-duplex websockets could be implemented in a synchronous way by dedicating a thread to reading from the connection (in addition to whatever other threads you're using). I don't know if there are any python websocket implementations that support this kind of multithreading model for full-duplex, but that's how Go's websocket implementation works for example.
That said, the asynchronous/event-driven model is a natural fit for websockets (it's how everything works on the javascript side), so I would encourage you to get comfortable with that model instead of trying to find a way to work with websockets synchronously.
Related
I am wondering how I can make a simple socket server in Python 2.7 which can handle and add/accept multiple clients at a time. I do not want to use Twisted nor threading, nor any libraries; just Python, and sockets. I have looked around SoF (stackoverflow- is that a thing?) and found people asking the same question but not getting a clear answer.
If you are wondering why I need to do this, It's because I want to create a simple data forwarder which forwards client data to another server. I think a very simple example showing me Server.py, Client1.py, and Client2.py is just what I need. Again, just a very simple example with no threading, no twisted, no libraries.
I hope you can help me, I'm fairly new to Python and I feel like this project will help get me on my feet, and I learn great from examples.
Consider using asyncio (available for python 3.3 and later).
Asyncio is the new python standard for single-threaded concurrent programming:
This module provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives.
The documentation provides a few examples:
TCP echo client
TCP echo server
If you're not ready to migrate to python 3, you can use trollius, the portage of asyncio for python 2. There is a few differences between the two modules, as listed in the documentation:
replace asyncio with trollius (or use import trollius as asyncio)
replace yield from ... with yield From(...)
replace yield from [] with yield From(None)
in coroutines, replace return res with raise Return(res)
Other solutions for single-threaded concurrent programming on python 2.7:
gevent: a coroutine-based Python networking library that uses greenlet.
asyncore: built-in asynchronous socket library (echo server example).
I need to develop an application in Python handling a few thousand of persistent TCP connection in parallel. Clients connected to the server at bootstrap and send some message (in binary format) from time to time. The server also send both in reply to clients' message and asynchronously some other binary messages. Basically it is a persistent connection initiated by the client because I have no way to reach clients that are behind a NAT.
The question is: which is the libraries/framework i shall consider for this task. Spawning a thread for each client is not an option. I'm not aware of thread pool library for python. I also recently discovered gevent. Which other options do I have?
This link is an excellent read. It lists all the available event driven and asynchronous network frameworks within Python and also has good analysis of the performance for each framework.
It appears that the Tornado framework is one of the most-performant when developing such applications.
Hope this helps
'greenlets' is a leighweight concurrency package. See http://greenlet.readthedocs.org/en/latest/.
Besides greenlets, you might also want to consider multiprocessing. See http://docs.python.org/2/library/multiprocessing.html.
I'm trying to work out how to approach building a "machine" to send and receive messages to WebSphere MQ, via Twisted. I want it to be as generic as possible, so I can reuse it for many different situations that interface with MQ.
I've used Twisted before, but many years ago now and I'm trying to resurrect the knowledge I once had...
The specific problem I'm having is how to implement the MQ IO using Twisted. There's a pymqi Python library that interfaces with MQ, and it provides all the interfaces I need. The MQ calls I need to implement are:
initiate a connection to a specific MQ server/port/channel/queue-manager/queue combination
take content and post it as a message to the desired queue
poll a queue and return the content of the next message in the queue
send a request to a queue manager to find the number of messages currently in a queue
All of these involve blocking calls to MQ.
As I'm intending to reuse the Twisted/MQ interface many times across a range of projects, should I be looking to implement the MQ IO as a Twisted protocol, as a Twisted transport, or just call the pymqi methods via deferToThread() calls? I realise this is a very broad question with possibly no definitive answer; I'm really after advice from those who may have encountered similar challenges before (i.e. working with queueing interfaces that will always block) and found a way that works well.
If you're going to use this functionality a lot, then having a native Twisted implementation is probably worth the effort. A wrapper based on deferToThread will be less work, but it will also be harder to test and debug, perform less well, and have problems on certain platforms where Python threads don't work extremely well (eg FreeBSD).
The approach to take for a native Twisted implementation is probably to implement a protocol that can speak to MQ servers and give it a rich API for interacting with channels, queues, queue managers, etc, and then build a layer on top of that which abstracts the actual network connection away from the application (as I believe mqi/pymqi largely do).
I use Tornado as the web server. I write some daemons with Python, which run in the server hardware. Sometimes the web server needs to send some data to the daemon and receives some computed results. There are two working:
1. Asynchronous mode: the server sends some data to the daemons, and it doesn't need the results soon. Can I use message queue to do it perfectly?
2. Synchronous mode: the server sends data to the daemons, and it will wait until it get the results. Should Iuse sockets?
So what's the best way of communication between tornado and Python based daemon?
ZeroMQ can be used for this purpose. It has various sockets for different purposes and it's fast enough to never be your bottleneck. For asynchronous you can use DEALER/ROUTER sockets and for strict synchronous mode you can use REQ/REP sockets.
You can use the python binding for this --> http://www.zeromq.org/bindings:python.
For the async mode you can try something like this from the zguide chapter 3 Router-to-dealer async routing :
In your case, the "client" in the diagram will be your web server and your daemon will be the "worker".
For synchronous you can try a simple request-reply broker or some variant to suit your need.
The diagram above shows a strictly synchronous cycle of send/recv at the REQ/REP sockets. Read through the zguide link to understand how it works. They also have a python code snippet on the page.
Depending on the scale - the simple thing is to just use HTTP and the AsyncHTTPClient in Tornado. For the request<->response case in our application we're going 300 connections/second with such an approach.
For the first case Fire and forget, you could also use AsyncHTTP and just have the server close out the connection and continue working...
I'm looking for some general information on how I should approach a problem that I think Twisted is a great fit for. (I'm new to Twisted but not Python)
I have a home automation controller that can support a single TCP socket connection, sending and receiving binary data. I'd like to use XMPP as a bridge to the socket so a user can send commands and receive events.
I got a rudimentary socket connection working with Twisted that was able to send and receive commands from one of the examples in the O'Reilly book. I also have a fully working Python XMPP bot written with the SleekXMPP library that I'm happy with. I'm just not sure how to bring these together.
The basic scenario is:
User sends message to XMPP bot, which figures out what command to send to the socket
ASCII Socket command is converted to binary and sent to socket
Socket receives command and sends binary response
Binary response converted to ASCII
XMPP bot sends response back to user.
Network events (independent from user action) can also be received by network socket and should be sent to user
It's #6 that is presenting the challenge, otherwise I'd just open/close the socket on demand when in need to write something.
The part that I'm having trouble wrapping my head around with Twisted is the best approach to make these two event loops communicate. I've seen lots of info on using Queues, deferred, threads, select, etc. I have a feeling that Twisted can handle much of the complexity if I just learn to use the tool properly.
If someone can point me in the right direction, I'll take the ball and run with it. As I mentioned, I'm happy with my XMPP bot and I'd like to use the existing code. I think my problem now comes down to creating the socket in the background, then sending and receiving data from that socket in the foreground.
By the way, I'm very happy to share back my code once it's working so someone else can benefit from the help I'm asking for.
-- Scott
One of the problems with a non-blocking IO engine is that its pretty much all-or-nothing. As soon as you introduce blocking code, you can quickly lose most of the benefits of the event-driven asynch approach. Wherever possible (as a rule of thumb), its best to have the entire app running off the same reactor.
As i see it, you have two options:
Twisted is not thread safe. That said, you can use mechanisms like deferToThread and callFromThread to interact with other threads. This is by far the most confusing and needlessly complex approach for your application design. It's particularly painful if you're new to twisted.
Use twisted.words.protocols.jabber, and implement your XMPP stuff in a non-blocking manner using the twisted reactor. That way it will happily exist alongside all your other twisted code. and allow you to cleanly interact between protocols. It will result in less code, and a robust implementation that is easy to extend, maintain, and test.