I'm starting to learn more about TCP protocols in Python and I've been having some trouble with blocking threads inside clients.
Ideally, my application would work like this: I have different clients with thread functions, each one of them containing an input function in order to receive a specific command to send to the server (for example 'X'). When the 'X' is tapped in ONE client, the server receives it and sends a message to all the other clients informing that the program will continue and releasing them from their input functions - almost like cancelling them.
The problem lies on the fact that the input functions are blocking the clients from leaving the loop. I've tried setting the input thread functions as daemon but it blocks until you tap something anyway - which is unfortunately the only workaround that I've found so far.
I would like to use socket and the select module for connection, without being attached to any particular OS (so no msvcrt that works on Windows or the select module to monitor the stdin, which is only available in UNIX based OS).
Any help would be greatly appreciated!
Related
Im trying to make a tcp communication, where the server sends a message every x seconds through a socket, and should stop sending those messages on a certain condition where the client isnt sending any message for 5 seconds.
To be more detailed, the client also sends constant messages which are all ignored by the server on the same socket as above, and can stop sending them at any unknown time. The messages are, for simplicity, used as alive messages to inform the server that the communication is still relevant.
The problem is that if i want to send repeated messages from the server, i cannot allow it to "get busy" and start receiving messages instead, thus i cannot detect when a new messages arrives from the other side and act accordingly.
The problem is independent of the programming language, but to be more specific im using python, and cannot access the code of the client.
Is there any option of receiving and sending messages on a single socket simultaneously?
Thanks!
Option 1
Use two threads, one will write to the socket and the second will read from it.
This works since sockets are full-duplex (allow bi-directional simultaneous access).
Option 2
Use a single thread that manages all keep alives using select.epoll. This way one thread can handle multiple clients. Remember though, that if this isn't the only thread that uses the sockets, you might need to handle thread safety on your own
As discussed in another answer, threads are one common approach. The other approach is to use an event loop and nonblocking I/O. Recent versions of Python (I think starting at 3.4) include a package called asyncio that supports this.
You can call the create_connection method on an event_loop to create an asyncio connection. See this example for a simple server that reads and writes over TCP.
In many cases an event loop can permit higher performance than threads, but it has the disadvantage of requiring most or all of your code to be aware of the event model.
I have a Python program which spawns several other Python programs as subprocesses. One of these subprocesses is supposed to open and bind a ZMQ publisher socket, such that other subprocesses can subscribe to it.
I cannot give guarantees about which tcp ports will be available, so when I bind to a random port in the subprocess, my main program will not know what to tell the other subprocesses.
Is there a way to bind the socket in the main process and then somehow pass the socket to my subprocess? Or either some other way to preregister the socket or a standard way to pass the port information from the subprocess back to my main process (stdout and stderr are already used by other data)?
Just checking for a free port in the main process and passing that to the subprocess is not really optimal, because this could still fail if the socket is being assigned in the meantime. Also, since my program should work on Unix and Windows, I cannot really use ipc sockets, which would otherwise solve my problem.
The simplest is to create a logic for a pool-of-ports manager ( rather avoid attempts to share / pass ZeroMQ sockets to / among other processes )
One may create a persistent, a-priori known, tcp://A.B.C.D:8765-transport-class based .bind() access-point, exposed to all client processes as a port-assignment service, to which client processes .connect(), handshake in whatever manner is needed to proof an identity/credentials/purpose/etc and .recv() in a coordinated manner one actually free messaging/signalling-service port number, that is system-wide guaranteed to not be used at the very moment / until returned to the port-manager ( a rotating pool of ports is centrally managed, under your code-control, whereas all the sockets, created locally in the distributed process(es)/thread(s) .connect() / .bind()-ing to the pool-manager announced port#, and thus will still remain, and ought remain, consistently within ZeroMQ advice, not to be shared per-se ).
Let say we have a server application written in Python.
Let also say that this main server process forked two more processes at the startup.
Server awaits its clients, and when one comes decides to which of two forked processes should pass the client's socket.
I do not want to fork a process each time a client comes; I want to have fixed number of servers, but one main server that receives a connection, then pass it to a server that deals with a specific work client asked for.
This should be a DOS attack protection, job separation, etc. etc.
Is there any trick to pass a Python object between started Python programs.
Some shared memory or something like that?
Would pickling the socket object and pushing it through IPC work?
Would pickling the socket object and pushing it through IPC work?
No. Inside that object is a file descriptor or handle to the kernel socket. It's just a number that the process uses to identify the socket when making system calls.
If you pickle that Python socket object and send it to another process, that process will be using a handle for a socket it didn't open. Or worse, that handle may refer to a different open file.
The most efficient way to handle this (on Linux) is like this:
Master process opens listening socket (e.g. TCP port 80)
Master process forks N children who all inherit that open socket
They all call accept() and block, waiting for a new connection
When a new client connects, the kernel will select one of the processes with a handle to that socket to accept the connection; the others will continue to wait
This way, you let the kernel handle the load balancing.
If you don't want this behavior, there is a way (in UNIX) to pass an open socket to another process. Again, this is more than just the handle; the kernel effectively copies the open socket to your processs's open file list. This mechanism is known as SCM_RIGHTS, and you can see an example (in C) here:
http://man7.org/tlpi/code/online/dist/sockets/scm_rights_send.c.html
Otherwise, your master process will need to effectively proxy the connection to the child processes, reducing thr efficiency of the system.
I was looking at the socket programming module of the python standard library and I noticed a fucntion socket.setblocking. The documentation mentioned that setting a socket to non blocking mode would mean that an error would be raised if the data was not sent out through the socket immediately or if data was not available upon trying to read from the socket.
I'm having trouble understanding usecases in which this function might be useful. I'm working on a Linux machine(just in case the answer to this would be OS dependent).
Thanks!
When you set the socket to blocking, the socket waits for the specified time on that socket. While it is waiting on the socket, your program cannot do anything. At the end of the wait time it raises an error. Sometimes you dont want blocking to occur.
A good use case for this might be when you are sending an receiving message on a single threaded program using multiple sockets. You don't want to block on a socket while waiting to send or receive messages rather you may want to check if there are messages to send or receive for each of the sockets hence you would want no blocking time or limited blocking time while you loop through the sockets. This will provide a more in depth discussion of python sockets.
I am developing python service for xbmc and I am hopelessly stuck. XBMC has TCP API that communicates by JSON-RPC. XBMC has server TCP socket that is mainly design to recieve commands and respond, but if something happens in system it sends "Notification" to TCP. The problem is that I need to create TCP client that behaves like server therefore it is able to recieve this "Notification". Wherever I run socket.recv(4096) it waits for data and stucks my code, because I need to loop my code. Structure of code is basically like this:
import xbmc, xbmcgui, xbmcaddon
class XPlayer(xbmc.Player) :
def __init__ (self):
xbmc.Player.__init__(self)
def onPlayBackStarted(self):
if xbmc.Player().isPlayingVideo():
doPlayBackStartedStuff()
player=XPlayer()
doStartupStuff()
while (not xbmc.abortRequested):
if xbmc.Player().isPlayingVideo():
doPlayingVideoStuff()
else:
doPlayingEverythingElseStuff()
xbmc.sleep(500)
# This loop is the most essential part of code
if (xbmc.abortRequested):
closeEverything()
xbmc.log('Aborting...')
I tried everything threading, multiprocessing, blocking, non-blocking and nothing helped.
Thanks,
You likely want select(), poll() or epoll():
http://docs.python.org/library/select.html
This Python pipe-progress-meter application uses select, as an example:
http://stromberg.dnsalias.org/~strombrg/reblock.html
If you know what sort of delimiters are separating the various portions of the protocol, you may also find this useful, without a need for select or similar:
http://stromberg.dnsalias.org/~strombrg/bufsock.html
It deals pretty gracefully with "read to the next null byte", "read a maximum of 100 bytes", etc.