Should I call the send function from WinAPI (https://msdn.microsoft.com/ru-ru/library/windows/desktop/ms740149(v=vs.85).aspx) in a loop to guarantee that all data has been sent (like the similar send function from Python standard library) or does it work like the sendall function?
Python and the Winsock API both implement the BSD socket api, so the send function works pretty much identically. So no, it is not a convenient sendall and far FAR too much code assumes it is.
You could write a simple sendall function on windows by looping on send until all the bytes have been successfully sent, but this is a bad idea in GUI applications as it will prevent the application from responding to the user, and in more complex console applications it might prevent other sockets communicating.
If either of these is a concern, you should investigate WSAAsyncSelect for GUI applications, where your window proc will receive window messages each time the socket is ready to be read or written to, or select which allows a console program / non gui worker thread to loop on a collection of up to 32 sockets (windows limitation) to test which ones are readable or writable.
Related
Im trying to make a tcp communication, where the server sends a message every x seconds through a socket, and should stop sending those messages on a certain condition where the client isnt sending any message for 5 seconds.
To be more detailed, the client also sends constant messages which are all ignored by the server on the same socket as above, and can stop sending them at any unknown time. The messages are, for simplicity, used as alive messages to inform the server that the communication is still relevant.
The problem is that if i want to send repeated messages from the server, i cannot allow it to "get busy" and start receiving messages instead, thus i cannot detect when a new messages arrives from the other side and act accordingly.
The problem is independent of the programming language, but to be more specific im using python, and cannot access the code of the client.
Is there any option of receiving and sending messages on a single socket simultaneously?
Thanks!
Option 1
Use two threads, one will write to the socket and the second will read from it.
This works since sockets are full-duplex (allow bi-directional simultaneous access).
Option 2
Use a single thread that manages all keep alives using select.epoll. This way one thread can handle multiple clients. Remember though, that if this isn't the only thread that uses the sockets, you might need to handle thread safety on your own
As discussed in another answer, threads are one common approach. The other approach is to use an event loop and nonblocking I/O. Recent versions of Python (I think starting at 3.4) include a package called asyncio that supports this.
You can call the create_connection method on an event_loop to create an asyncio connection. See this example for a simple server that reads and writes over TCP.
In many cases an event loop can permit higher performance than threads, but it has the disadvantage of requiring most or all of your code to be aware of the event model.
I have the backend to a toy video game engine written in Python. It's running on my server in its continuous game loop. I want it to be able to send messages to web browser clients over websockets.
However, it looks to me like websockets are universally limited to sending information on callbacks alone. I tried using the Autobahn websockets library for Python, but when the server is run, it runs in a blocking loop, so you can't even interact with it -- you can only define its behavior ahead of time in callbacks.
I just want to be able to instantiate a type of MyWebsocketNetwork, which will run its server in the background, and be able to call myWebsocketNetwork.sendToAll("my message") anywhere in my code to send my messages on demand. NOT in callbacks, but on demand. Again, I can't find a way to do this with Autobahn (or any other library) since they all run in blocking lops.
Is this in general not possible due to the nature of websockets? Or is there some way I can send websocket messages to my clients on demand in Python (on demand = dynamically and conditionally based upon what happens in my game's loop).
Not sure I understand what you mean. Callbacks are on-demand. It sounds like you need to create your WebSocket in a separate thread.
Short version of my question:
How do I design a single Python script that can listen and respond to inputs received via HTTP or a serial port, and also initiate communications via these channels on its own? My problem is that I don't understand how to design a single script that both (i) uses a web framework to listen on some port for HTTP inputs, and (ii) also does other work that's independent of incoming HTTP requests.
Long version:
I want to use Python to design a system that does the following:
Listens to a serial port for occasional reports. Specifically, I have a network of JeeNode sensors (wireless Arduino-compatible modules) that talk to a central JeeLink, which connects to my computer via USB and talks to my Python script via pySerial.
Listens to a web URL for occasional inputs. Specifically, users send commands to the system via SMS to a Twilio number. Twilio intercepts the SMS messages and posts them to a URL I designate, and I use the Bottle micro web-framework to listen for new HTTP requests.
Responds to both types (serial and HTTP) of inputs. For example, if a user texts the command "Sleep", I want to (i) tell the sensors to go to sleep via the serial port -> JeeLink (which will then forward the command onto the remotes); and (ii) reply to the sender -- and maybe other users -- that the command has been received and is being executed.
Occasionally initiates its own communications to users (via HTTP -> Twilio -> SMS) or remote sensors (via serial -> JeeLink) without any precipitating input event. Two examples: (1) I want to report out to users or remote sensors every N minutes even if I haven't received any new inputs. (2) I want to tell users remotes have actually entered Sleep mode. Because the remotes are battery-powered, they spend most of the time in an inaccessible low-power mode. They can only receive new commands from the JeeLink when they initiate a wireless "check-in" every 5 min. So while technically remotes go to sleep (or wake up, etc.) in response to a user command, commands and responses are effectively independent.
My problem is that all of usage examples of web frameworks I've seen seem to assume that all precipitating events occur via HTTP requests. I can create a Bottle object, and use decorators to bind code to that object that get executed whenever it sees an HTTP request that matches some specified URL path. But I don't know how to do that while simultaneously doing other work that's independent of HTTP events, for example, listening to the serial port.
After struggling a lot, the potential solutions I'm considering now are:
Splitting the functionality into separate scripts. A.py listens for text messages via HTTP and writes the relevant information to some database; B.py continuously reads the database for new records and reacts accordingly, as well as listening to the serial monitor and doing other work. This seems like it would work fine, but it feels inelegant, and I suspect there's a simpler solution I'm unaware of.
Maybe the answer is related to Python decorators? I use various decorators to specify the URL paths that, when a matching HTTP request comes in, execute the code bound to the decorator. So I'm guessing that maybe there's a way to specify some other kind of decorator that, rather than listening for HTTP requests, gets executed when my "main" Python code tells it to? But I don't know enough about decorators to know if this is true.
It seems like you are trying to write an asynchronous application to manage your network of nodes via HTTP. You want to respond to incoming communications on multiple channels as they occur, you want to initiate communications on a schedule, on multiple channels, and you want those two forms of communication to interact. All of these communications are with an outside world that is slow, so it behooves you not to block if you don't need to.
It will probably be easiest to maintain your system if you organize your code into several Python modules, split by their area of concern - serial interface code, HTTP interface code, common processing code-paths, etc. Weave those components together in a central control module, which imports your libraries, and knows how to start and stop cleanly. Then you can test the serial interface independent of the web interface, and potentially reuse some of those Python modules in other projects.
Should sockets be set to non-blocking when used with select.select in Python?
What difference does it make if they are or aren't?
Occasionally I find that calling send on a socket that returns sendable will block. Furthermore I find that blocking sockets will generally send the whole buffer given (128 KiB). In non-blocking mode, sending will accept far fewer bytes (20-40 KiB compared with the example given earlier) and return quicker. I'm using Python 3.1 on Lucid.
The answer might be OS dependent unfortunately. I'm replying only regarding Linux.
I'm not aware of differences regarding blocking/non-blocking sockets in select, but on linux, the select system call man page has this in it 'BUGS' section:
Under Linux, select() may report a
socket file descriptor as "ready for
reading", while nevertheless a
subsequent read blocks. This could
for example happen when data has
arrived but upon examination has
wrong checksum and is discarded. There may be other
circumstances in which a file
descriptor is spuriously reported as
ready. Thus it may be safer to use
O_NONBLOCK on sockets that should not
block.
I doubt a python abstraction above that could "hide" this issue without side-effects.
As for the blocking write sending more data, that's expected. send will block until there is enough buffer space to pass your whole request down if the socket is blocking. If the socket is non-blocking, it only sends as much as can currently fit in the socket's send buffer.
I'm looking for some general information on how I should approach a problem that I think Twisted is a great fit for. (I'm new to Twisted but not Python)
I have a home automation controller that can support a single TCP socket connection, sending and receiving binary data. I'd like to use XMPP as a bridge to the socket so a user can send commands and receive events.
I got a rudimentary socket connection working with Twisted that was able to send and receive commands from one of the examples in the O'Reilly book. I also have a fully working Python XMPP bot written with the SleekXMPP library that I'm happy with. I'm just not sure how to bring these together.
The basic scenario is:
User sends message to XMPP bot, which figures out what command to send to the socket
ASCII Socket command is converted to binary and sent to socket
Socket receives command and sends binary response
Binary response converted to ASCII
XMPP bot sends response back to user.
Network events (independent from user action) can also be received by network socket and should be sent to user
It's #6 that is presenting the challenge, otherwise I'd just open/close the socket on demand when in need to write something.
The part that I'm having trouble wrapping my head around with Twisted is the best approach to make these two event loops communicate. I've seen lots of info on using Queues, deferred, threads, select, etc. I have a feeling that Twisted can handle much of the complexity if I just learn to use the tool properly.
If someone can point me in the right direction, I'll take the ball and run with it. As I mentioned, I'm happy with my XMPP bot and I'd like to use the existing code. I think my problem now comes down to creating the socket in the background, then sending and receiving data from that socket in the foreground.
By the way, I'm very happy to share back my code once it's working so someone else can benefit from the help I'm asking for.
-- Scott
One of the problems with a non-blocking IO engine is that its pretty much all-or-nothing. As soon as you introduce blocking code, you can quickly lose most of the benefits of the event-driven asynch approach. Wherever possible (as a rule of thumb), its best to have the entire app running off the same reactor.
As i see it, you have two options:
Twisted is not thread safe. That said, you can use mechanisms like deferToThread and callFromThread to interact with other threads. This is by far the most confusing and needlessly complex approach for your application design. It's particularly painful if you're new to twisted.
Use twisted.words.protocols.jabber, and implement your XMPP stuff in a non-blocking manner using the twisted reactor. That way it will happily exist alongside all your other twisted code. and allow you to cleanly interact between protocols. It will result in less code, and a robust implementation that is easy to extend, maintain, and test.