Develop Reference

Raise TimeOutError after given time for multiprocessing.connection.Listener.accept()

I'm trying to interrupt multiprocessing.connection.Listener.accept(), but have thus far been unsuccessful. Since it doesn't provide a timeout parameter, I thought perhaps I could use socket.setdefaulttimeout() to interrupt it, as suggested in post I cannot find anymore, here on SO.
This didnt work. I then tried calling close() on the Listener() object. according to this post's answer, this should have worked.
It appears, however, that these objects to not play along with the usual socket-related solutions.
I can confirm that that the Listener is closed by the Timer object as expected, but the accept() call isn't interrupted.
The Code:
import logging
import socket
import os
from multiprocessing.connection import Listener
from queue import Queue, Empty
from threading import Thread, Event, Timer
class Node(Thread):
"""Base Class providing a AF_INET, AF_UNIX or AF_PIPE connection to its
data queue. It offers put() and get() method wrappers, and therefore
behaves like a Queue as well as a Thread.
Data from the internal queue is automatically fed to any connecting client.
"""
def __init__(self, sock_name, max_q_size=None, timeout=None,
*thread_args, **thread_kwargs):
"""Initialize class.
:param sock_name: UDS, TCP socket or pipe name
:param max_q_size: maximum queue size for self.q, default infinite
"""
self._sock_name = sock_name
self.connector = Listener(sock_name)
max_q_size = max_q_size if max_q_size else 0
self.q = Queue(maxsize=max_q_size)
self._running = Event()
self.connection_timer = Timer(timeout, self.connection_timed_out)
super(Node, self).__init__(*thread_args, **thread_kwargs)
def connection_timed_out(self):
"""Closes the Listener and shuts down Node if no Client connected.
:return:
"""
self.connector.close()
self.join()
def _start_connection_timer(self):
self.connection_timer.start()
def start(self):
self._running.set()
super(Node, self).start()
def join(self, timeout=None):
print("clearing..")
self._running.clear()
print("internal join")
super(Node, self).join(timeout=timeout)
print("Done")
def run(self):
while self._running.is_set():
print("Accepting connections..")
self._start_connection_timer()
try:
client = self.connector.accept()
self.connection_timer.cancel()
self.feed_data(client)
except (TimeoutError, socket.timeout):
continue
except Exception as e:
raise
print("Run() Terminated!")
def feed_data(self, client):
try:
while self._running.is_set():
try:
client.send(self.q.get())
except Empty:
continue
except EOFError:
return
if __name__ == '__main__':
import time
n = Node('/home/nils/git/spab2/test.uds', timeout=10)
n.start()
print("Sleeping")
time.sleep(15)
print("Manual join")
n.join()
I realize my question is a duplicate of this question - however, it is almost one year old and has not even received a comment. In addition, I'm using Unix Domain Sockets, as opposed to the linked post's TCP connection.

I managed to set the timeout in the following way in Python 2.7:
self.listener = mpc.Listener((address, port))
self.listener._listener._socket.settimeout(3)
With this, the accept() call is interrupted.
Result:
conn = self.listener.accept()
File "/usr/lib/python2.7/multiprocessing/connection.py", line 145, in accept
c = self._listener.accept()
File "/usr/lib/python2.7/multiprocessing/connection.py", line 275, in accept
s, self._last_accepted = self._socket.accept()
File "/usr/lib/python2.7/socket.py", line 202, in accept
sock, addr = self._sock.accept()
timeout: timed out
Regards,
Henri

python : threads do not work properly in daemon

I'm writing simple script that will check for SSH connection, and I cannot understand, why it hangs on one thread.
class myThread(threading.Thread):
def __init__(self, hostname ):
threading.Thread.__init__(self)
self.hostname = hostname
def run(self):
return self.doSSH(self.hostname)
def doSSH(self,hostname):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((hostname, 22))
result = s.recv(1024)
if re.findall(r'^SSH.+?SSH.+',result) :
return "Up"
else :
return "Down"
def main():
q = Queue.Queue()
completeHostlist = ["host1","host2","google.com","host3"]
for hostname in completeHostlist:
thread = myThread(hostname)
thread.daemon = True
q.put_nowait(thread.run())
q.get_nowait()
I don't understand why this script hangs at google.com? I would be expecting it to spawn daemon thread and continue with host3. As soon it finish host3 it has to kill thread with Google and return results. What I did wrong ?
I already figured out about run() and start(). Anyway this is not working as expected, after all host[1-3] threads was started, script stuck at thread with google , waiting it to end. Should it be kill it at the end of the script ?
Should i be using multiprocessing instead of multithreading , to spawn separate process for each host?

In your code you do q.put_nowait(thread.run()). That immediately runs the ssh thing on the current thread. You need to call thread specific method to start the thread. You need to call thread.start().
Not sure what you're doing with the Queue.

Don't call .run() method directly for any thread. As #Sorin said call thread.start() instead.
You don't need to define a new thread class, a function is enough in this case:
from Queue import Queue
from threading import Thread
def is_ssh_up(result_queue, hostname, port=22):
# try to connect here
# ...
# write results
result_queue.put((hostname, True)) # Up
def main():
q = Queue()
hosts = ["host1", "host2", "google.com", "host3"]
for hostname in hosts: # start worker threads
t = Thread(target=is_ssh_up, args=[q, hostname])
t.daemon = True
t.start()
for _ in hosts: # collect results
hostname, is_up = q.get()
print("%s is %s" % (hostname, "Up" if is_up else "Down"))
Or you could use a thread pool:
from multiprocessing.pool import ThreadPool
def is_ssh_up(hostname, port=22):
# try to connect here
# ...
# return results
return hostname, is_up
hosts = ["host1", "host2", "google.com", "host3"]
pool = ThreadPool(20) # limit number of concurrent connections to 20
for hostname, is_up in pool.imap_unordered(is_ssh_up, hosts):
status = "Up" if is_up else "Down" if is_up is not None else "Unknown"
print("%s status is %s" % (hostname, status))

How to implement multi-threading into a Python WSGI server

Just to clarify up front that this is just a learning project and I have no intention of using this in production. There are several very good Python application servers out there already. But I am trying to learn more about concurrency, so I set out to write one of the things (I thought) I knew.
Also, because I wanted to "closer to the metal" so I started out with just Socket and want to keep it that way.
Below is the important parts of what I have so far. self.iq is a Queue object (inbound_queue) which then does nothing really, but puts the request (which includes the socket object) into the outbound_queue and then a Consumer object takes the request from the outbound_queue and passes it to the ResponseHandler. This seems to work fine with just me hitting it but I am concerned that I am opening myself up to a race condition with a naive implementation. Specifically assigning things to the ServerClass object that are request specific.
So the question is: Is there a better way to do this, or does my Queue usage prevent two threads from picking up and operating on the same object? Should I be encapsulating things like the WSGI environment stuff into a separate object that can be also passed into the queue? Doing stuff like that gets tricky in trying to write a server that is WSGI compliant because of the need to pass in callback functions.
class Consumer(threading.Thread):
def __init__(self, out_queue, server):
threading.Thread.__init__(self)
self.out_queue = out_queue
self.server = server
def run(self):
while True:
item = self.out_queue.get()
self.server.ResponseHandler(self.server, item)
self.out_queue.task_done()
class QueueConsumerServer(object):
methods_allowed = ['get', 'post', 'put', 'patch', 'delete', 'options', 'upgrade']
def __init__(self, host, port, application):
self.host = host
self.port = port
self.application = application
self.iq = Queue.Queue()
self.oq = Queue.Queue()
self.socket = socket.socket()
self.socket.bind((self.host, self.port))
#<snip of lots of assigning stuff to environ>
self.environ = environ
headers_set = []
headers_sent = []
for i in xrange(3):
thr = Producer(self.iq, self.oq)
thr.daemon = True
thr.start()
for i in xrange(3):
thr = Consumer(self.oq, self)
thr.daemon = True
thr.start()
def handle_request(self):
self.socket.listen(1)
try:
while True:
cli, addr = self.socket.accept()
data = cli.recv(1024)
request_data = self.parse_request_data(data)
req.path = request_data[1]
req.cli = cli
self.iq.put(req)
return
except Exception, ex:
print 'e', ex,
sys.exit(1)
finally:
sys.stdout.flush()
self.socket.close()

Python queues are thread safe, so there is no race in your code as written.
Regarding a better way to approach this, your code will probably hit the GIL before too long. I would suggest looking to into multiprocessing.

python socket + multiprocessing

I'm currently working on a websocket implementation that allows multiprocessing over the same listening socket.
I'm able to achieve an amazing performance with 4 processes on a quad core machine.
When I go upper, like 8 processes, after 4 request, the epoll.poll don't fire any event anymore. Interestingly, I tried running the same program , with 2 listener on 2 different ports. With 4 processes per listener, it blocks after 2 requests per socket. With 2 processes per listener, il all go fine through it.
Any thought?
main.py (extract)
#create the WSServer
wsserver = WSServer(s.bind_ip, s.bind_port, s.max_connections)
# specify on how many process we'll run
wsserver.num_process = s.num_process
Process(target=wsserver.run,args=()).start()
wsserver.py (extract)
def serve_forever_epoll(wsserver):
log(current_process())
epoll = select.epoll()
epoll.register(wsserver.socket.fileno(), select.EPOLLIN)
try:
client_map = {}
while wsserver.run:
events = epoll.poll(1)
for fileno, event in events:
if fileno == wsserver.socket.fileno():
channel, details = wsserver.socket.accept()
channel.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
aclient = wsclient.WSClient(channel, wsserver, process_server.client_manager)
client_map[channel.fileno()] = aclient
epoll.register(channel.fileno(), select.EPOLLIN )
log('Accepting client on %s' % current_process())
aclient.do_handshake()
elif event & select.EPOLLIN:
aclient = client_map[fileno]
threading.Thread(target=aclient.interact).start()
except Exception, e:
log(e)
finally:
epoll.unregister(wsserver.socket.fileno())
epoll.close()
wsserver.socket.close()
class WSServer():
def __init__(self, address, port, connections):
self.address = address
self.port = port
self.connections = connections
self.onopen = onopen
self.onclose = onclose
log('server init')
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
#self.socket.setblocking(0)
self.socket.bind((self.address, int(self.port)))
self.socket.listen(self.connections)
def run(self, *args):
multiprocessing.log_to_stderr(logging.DEBUG)
log("Run server")
try:
log("Starting Server")
self.run = True
serve_forever = serve_forever_epoll
for i in range(self.num_process-1):
log('Starting Process')
Process(target=serve_forever,args=(self,)).start()
serve_forever(self)
except Exception as e:
log("Exception-- %s " % e)
pass

OK so finally this weird case was caused by another module I was using.
I am using Pyro4 as a manager for keeping track of which process holds what client. This simplifies greately the IPC and also permits me for some client filtering based on some user_data.
The problem was the Pyro4 daemon was running on the MainProcess but not on the Main Thread!...
As long as I had less that 4 processes, all was OK (don't ask me why).
Moving Pyro in the main-process + thread event loop, it was working perfectly!
So now, i'm able to achieve 8, 16 or 32 processes for the same listening port, as well as spawning new configuration to replicate it or expose a new endpoint for the websocket server!
Thanks for your contributions, and sorry for your time...

How do I abort a socket.recvfrom() from another thread in python?

This looks like a duplicate of How do I abort a socket.recv() from another thread in Python, but it's not, since I want to abort recvfrom() in a thread, which is UDP, not TCP.
Can this be solved by poll() or select.select() ?

If you want to unblock a UDP read from another thread, send it a datagram!
Rgds,
Martin

A good way to handle this kind of asynchronous interruption is the old C pipe trick. You can create a pipe and use select/poll on both socket and pipe: Now when you want interrupt receiver you can just send a char to the pipe.
pros:
Can work both for UDP and TCP
Is protocol agnostic
cons:
select/poll on pipes are not available on Windows, in this case you should replace it by another UDP socket that use as notification pipe
Starting point
interruptable_socket.py
import os
import socket
import select
class InterruptableUdpSocketReceiver(object):
def __init__(self, host, port):
self._host = host
self._port = port
self._socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._r_pipe, self._w_pipe = os.pipe()
self._interrupted = False
def bind(self):
self._socket.bind((self._host, self._port))
def recv(self, buffersize, flags=0):
if self._interrupted:
raise RuntimeError("Cannot be reused")
read, _w, errors = select.select([self._r_pipe, self._socket], [], [self._socket])
if self._socket in read:
return self._socket.recv(buffersize, flags)
return ""
def interrupt(self):
self._interrupted = True
os.write(self._w_pipe, "I".encode())
A test suite:
test_interruptable_socket.py
import socket
from threading import Timer
import time
from interruptable_socket import InterruptableUdpSocketReceiver
import unittest
class Sender(object):
def __init__(self, destination_host, destination_port):
self._socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
self._dest = (destination_host, destination_port)
def send(self, message):
self._socket.sendto(message, self._dest)
class Test(unittest.TestCase):
def create_receiver(self, host="127.0.0.1", port=3010):
receiver = InterruptableUdpSocketReceiver(host, port)
receiver.bind()
return receiver
def create_sender(self, host="127.0.0.1", port=3010):
return Sender(host, port)
def create_sender_receiver(self, host="127.0.0.1", port=3010):
return self.create_sender(host, port), self.create_receiver(host, port)
def test_create(self):
self.create_receiver()
def test_recv_async(self):
sender, receiver = self.create_sender_receiver()
start = time.time()
send_message = "TEST".encode('UTF-8')
Timer(0.1, sender.send, (send_message, )).start()
message = receiver.recv(128)
elapsed = time.time()-start
self.assertGreaterEqual(elapsed, 0.095)
self.assertLess(elapsed, 0.11)
self.assertEqual(message, send_message)
def test_interrupt_async(self):
receiver = self.create_receiver()
start = time.time()
Timer(0.1, receiver.interrupt).start()
message = receiver.recv(128)
elapsed = time.time()-start
self.assertGreaterEqual(elapsed, 0.095)
self.assertLess(elapsed, 0.11)
self.assertEqual(0, len(message))
def test_exception_after_interrupt(self):
sender, receiver = self.create_sender_receiver()
receiver.interrupt()
with self.assertRaises(RuntimeError):
receiver.recv(128)
if __name__ == '__main__':
unittest.main()
Evolution
Now this code is just a starting point. To make it more generic I see we should fix follow issues:
Interface: return empty message in interrupt case is not a good deal, is better to use an exception to handle it
Generalization: we should have just a function to call before socket.recv(), extend interrupt to others recv methods become very simple
Portability: to make simple port it to windows we should isolate the async notification in a object to choose the right implementation for our operating system
First of all we change test_interrupt_async() to check exception instead empty message:
from interruptable_socket import InterruptException
def test_interrupt_async(self):
receiver = self.create_receiver()
start = time.time()
with self.assertRaises(InterruptException):
Timer(0.1, receiver.interrupt).start()
receiver.recv(128)
elapsed = time.time()-start
self.assertGreaterEqual(elapsed, 0.095)
self.assertLess(elapsed, 0.11)
After this we can replace return '' by raise InterruptException and the tests pass again.
The ready to extend version can be :
interruptable_socket.py
import os
import socket
import select
class InterruptException(Exception):
pass
class InterruptableUdpSocketReceiver(object):
def __init__(self, host, port):
self._host = host
self._port = port
self._socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._async_interrupt = AsycInterrupt(self._socket)
def bind(self):
self._socket.bind((self._host, self._port))
def recv(self, buffersize, flags=0):
self._async_interrupt.wait_for_receive()
return self._socket.recv(buffersize, flags)
def interrupt(self):
self._async_interrupt.interrupt()
class AsycInterrupt(object):
def __init__(self, descriptor):
self._read, self._write = os.pipe()
self._interrupted = False
self._descriptor = descriptor
def interrupt(self):
self._interrupted = True
self._notify()
def wait_for_receive(self):
if self._interrupted:
raise RuntimeError("Cannot be reused")
read, _w, errors = select.select([self._read, self._descriptor], [], [self._descriptor])
if self._descriptor not in read:
raise InterruptException
def _notify(self):
os.write(self._write, "I".encode())
Now wraps more recv function, implement a windows version or take care of socket timeouts become really simple.

The solution here is to forcibly close the socket. The problem is that the method for doing this is OS-specific and Python does not do a good job of abstracting the way to do it or the consequences. Basically, you need to do a shutdown() followed by a close() on the socket. On POSIX systems such as Linux, the shutdown is the key element in forcing recvfrom to stop (a call to close() alone won't do it). On Windows, shutdown() does not affect the recvfrom and the close() is the key element. This is exactly the behavior that you would see if you were implementing this code in C and using either native POSIX sockets or Winsock sockets, so Python is providing a very thin layer on top of those calls.
On both POSIX and Windows systems, this sequence of calls results in an OSError being raised. However, the location of the exception and the details of it are OS-specific. On POSIX systems, the exception is raised on the call to shutdown() and the errno value of the exception is set to 107 (Transport endpoint is not connected). On Windows systems, the exception is raised on the call to recvfrom() and the winerror value of the exception is set to 10038 (An operation was attempted on something that is not a socket). This means that there's no way to do this in an OS-agnositc way, the code has to account for both Windows and POSIX behavior and errors. Here's a simple example I wrote up:
import socket
import threading
import time
class MyServer(object):
def __init__(self, port:int=0):
if port == 0:
raise AttributeError('Invalid port supplied.')
self.port = port
self.socket = socket.socket(family=socket.AF_INET,
type=socket.SOCK_DGRAM)
self.socket.bind(('0.0.0.0', port))
self.exit_now = False
print('Starting server.')
self.thread = threading.Thread(target=self.run_server,
args=[self.socket])
self.thread.start()
def run_server(self, socket:socket.socket=None):
if socket is None:
raise AttributeError('No socket provided.')
buffer_size = 4096
while self.exit_now == False:
data = b''
try:
data, address = socket.recvfrom(buffer_size)
except OSError as e:
if e.winerror == 10038:
# Error is, "An operation was attempted on something that
# is not a socket". We don't care.
pass
else:
raise e
if len(data) > 0:
print(f'Received {len(data)} bytes from {address}.')
def stop(self):
self.exit_now = True
try:
self.socket.shutdown(socket.SHUT_RDWR)
except OSError as e:
if e.errno == 107:
# Error is, "Transport endpoint is not connected".
# We don't care.
pass
else:
raise e
self.socket.close()
self.thread.join()
print('Server stopped.')
if __name__ == '__main__':
server = MyServer(5555)
time.sleep(2)
server.stop()
exit(0)

Implement a quit command on the server and client sockets. Should work something like this:
Thread1:
status: listening
handler: quit
Thread2: client
exec: socket.send "quit" ---> Thread1.socket # host:port
Thread1:
status: socket closed()

To properly close a tcp socket in python, you have to call socket.shutdown(arg) before calling socket.close(). See the python socket documentation, the part about shutdown.
If the socket is UDP, you can't call socket.shutdown(...), it would raise an exception. And calling socket.close() alone would, like for tcp, keep the blocked operations blocking. close() alone won't interrupt them.
Many suggested solutions (not all), don't work or are seen as cumbersome as they involve 3rd party libraries. I haven't tested poll() or select(). What does definately work, is the following:
firstly, create an official Thread object for whatever thread is running socket.recv(), and save the handle to it. Secondly, import signal. Signal is an official library, which enables sending/recieving linux/posix signals to processes (read its documentation). Thirdly, to interrupt, assuming that handle to your thread is called udpThreadHandle:
signal.pthread_kill(udpthreadHandle.ident, signal.SIGINT)
and ofcourse, in the actual thread/loop doing the recieving:
try:
while True:
myUdpSocket.recv(...)
except KeyboardInterrupt:
pass
Notice, the exception handler for KeyboardInterrupt (generated by SIGINT), is OUTSIDE the recieve loop. This silently terminates the recieve loop and its thread.