I've read quite a few things and this still escapes me. I know how to do it when using raw sockets. The following works just fine, times out after 1 second if no data is received:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((HOST, PORT))
sock.listen(1)
while 1:
conn, addr = sock.accept()
data = ''
conn.settimeout(1)
try:
while 1:
chunk = conn.recv(1024)
data += chunk
if not chunk:
break
print 'Read: %s' % data
conn.send(data.upper())
except (socket.timeout, socket.error, Exception) as e:
print(str(e))
finally:
conn.close()
print 'Done'
But when trying something similar when using SocketServer.TCPServer with SocketServer.BaseRequestHandler (not with SocketServer.StreamRequestHandler where I know how to set a timeout) it seems not as trivial. I didn't find a way to set a timeout for receiving the data. Consider this snippet (not complete code):
class MyTCPHandler(SocketServer.BaseRequestHandler):
def handle(self):
data = ''
while 1:
chunk = self.request.recv(1024)
data += chunk
if not chunk:
break
if __name__ == "__main__":
HOST, PORT = "0.0.0.0", 9987
SocketServer.TCPServer.allow_reuse_address = True
server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
server.serve_forever()
Suppose the client sends only 10 bytes. The while loop runs once, chunk is not empty, so then executes self.request.recv() again but the client has no more data to send and recv() blocks indefinitely ...
I know I can implement a small protocol, check for terminating strings/chars, check message length etc., but I really want to implement a timeout as well for unforeseen circumstances (client "disappears" for example).
I'd like to set and also update a timeout, i.e. reset the timeout after every chunk, needed for slow clients (though that's a secondary issue at the moment).
Thanks in advance
You can do the same thing with SocketServer.BaseRequestHandler.request.settimeout() as you did with the raw socket.
eg:
class MyTCPHandler(SocketServer.BaseRequestHandler):
def handle(self):
self.request.settimeout(1)
...
In this case self.request.recv() will terminate if it takes longer than 1 second to complete.
class MyTCPHandler(SocketServer.BaseRequestHandler):
timeout=5
...
... will raise an exception (which serve_forever() will catch) and shut down the connection if 5 seconds pass without receiving data after calling recv(). Be careful, though; it'll also shut down your connection if you're sending data for more than 5 seconds as well.
This may be Python 3 specific, mind, but it works for me.
Related
Good evening,
I am trying to make a basic implementation of the clients-to-server model using sockets.
Basically, I want the client to retrieve a small piece of information (IE: MESSAGE key) to the server. To accomplish this, I've used threading.
When the client sends the request to receive the info, there is no error on the client side.
However, the server sends an error
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Users\Stock\AppData\Local\Continuum\anaconda3\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:\Users\Stock\AppData\Local\Continuum\anaconda3\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "server.py", line 40, in listeningForRetriever
dataBytes = conn.recv(1024)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
After doing some research, this thread seemed pretty relevant, https://stackoverflow.com/a/49289545/6902431, but I had already implemented the suggested fixes.
server.py
import socket
import threading
from Utility import *
HOST = '127.0.0.1' # Standard loopback interface address (localhost)
PORT = 65432 # Port to listen on (non-privileged ports are > 1023)
PORT_A = 22221
PORT_B = 22222
PORT_C = 22223
bank = {'me':'you'}
def listeningForRetriever():
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind((HOST, PORT_B))
s.listen()
conn, addr = s.accept()
with conn:
while True:
dataBytes = conn.recv(1024)
dataString = bytesToString(dataBytes)
try:
retrievedValueBytes = bytesToString(bank[cleanGET(dataString)])
print(f"Value: {retrievedValueBytes} has been retrieved with Key {cleanGET(dataString)}")
conn.sendall(retrievedValueBytes)
except:
print("SERVER: Errored out")
conn.sendall(stringToBytes("ERROR! Value retrieval for command: " + dataString + " doesn't exist!"))
retrieverThread = threading.Thread(target=listeningForRetriever)
retrieverThread.start()
retriever.py
#Retrieves values from server using keys
#retriever.py --key=username
import sys
import socket
from Utility import *
# total arguments
n = len(sys.argv)
print("Total arguments passed:", n)
# Arguments passed
print("\nName of Python script:", sys.argv[0])
print("\nArguments passed:", end = " ")
key = ''
value = ''
for i in range(1, n):
print(sys.argv[i], end = " ")
if '--key=' in sys.argv[i]:
data = sys.argv[i].replace('--key=','')
key = data
print()
if key == '':
print("ERROR! Invalid syntax.")
print("Try something like: Bob.py --key=username")
else:
print(f"Key: {key} {type(key)}")
#TODO: Send key to storage of server
HOST = '127.0.0.1' # The server's hostname or IP address
PORT = 22222 # The port used by the server
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(stringToBytes("GET key"+"="+key))
print("DEBUG 1")
value = s.recv(1024)
print("DEBUG 2")
print(f"Retrieved Value: {value}")
print(f"Value Retrieved: {value}")
s.close()
What am I overlooking/misunderstanding?
I could (more or less...) reproduce and fix. But I may be wrong, because your code uses an Utility module that you failed to show so I could only guess from the names what bytesToString, stringToBytes and cleanGET were supposed to do.
First there is a misunderstanding about how TCP actually behaves. It is a streaming protocol, that guarantees that all sent bytes will be received an in same order. But there is no guarantee on the packets themselves: they may be splitted and/or reassembled by the network. So you must use a higher level way to delimit a message. It does not exhibit any problem here because short packets are often left unmodified, but the protocol does not guarantee that so you could experience weird errors later. Here as you send a single message, you could shut down the socket to signal the end of the message: the receiver will see a 0 size packet at that point.
Next (and the actual cause of your problem), you loop after accepting a connection. So you get a message, successfully process it, read again on a socket that will be closed by the peer and get a 0 size byte string. It probably causes a key error when looking in the bank dict, but you silently swallow that error message (which is BAD), so you try to write on a socket closed by the peer, which causes the connection to be aborted. And finally on next recv you get the error that you see.
What you should have learned from that:
never ever use a silent try: ... except: ... that swallows any exception without letting you know what has happened. Always limit the filtered exception to the smaller expected set, or (at least and at dev time) display the caught exception to make sure you have not inadvertently caught an unwanted one.
use a higher level protocol to delimit messages on TCP
Now for the fixes:
retriever.py
...
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(stringToBytes("GET key"+"="+key))
s.shutdown(socket.SHUT_WR) # signals the end of the message
print("DEBUG 1")
...
server.py
...
s.listen()
conn, addr = s.accept()
with conn:
# read until the end of a message signaled by a 0 size packet
dataBytes = b''
while True:
b = conn.recv(1024)
if len(b) == 0:
break
dataBytes += b
dataString = bytesToString(dataBytes)
try:
retrievedValueBytes = stringToBytes(bank[cleanGET(dataString)])
print(f"Value: {retrievedValueBytes} has been retrieved with Key {cleanGET(dataString)}")
conn.sendall(retrievedValueBytes)
except Exception as e:
print(e)
print("SERVER: Errored out")
conn.sendall(stringToBytes("ERROR! Value retrieval for command: " + dataString + " doesn't exist!"))
...
Here the thread stops after one single message. If you want the server to be able to accept many clients, you can loop over accept:
s.listen()
while True:
conn, addr = s.accept()
...
Your server is coded to listen in a loop (while True / conn.recv), but your
client only listens once, then exits, which closes the connection.
So the message on the server side is actually not vague at all, it's perfectly to
the point: it's telling you that the connection has been closed. It has indeed,
by the client.
(To be very specific, your retriever code uses a socket as a context manager,
meaning that when you exit the 'with' bock, the socket gets automatically
closed - you can remove the s.close() call, BTW)
Your code works fine, you should be getting the data back (I did, but your post
doesn't say anything about that). The only thing missing is the exception
handling on the server side to gracefully handle the client disconnection.
I started learning networking with python can somebody help me out with this code as I am unable to connect more than 5 clients at a time. Can someone please suggest me a solution for this?
def main():
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', listening_port))
s.listen(5)
print "[*] Initializing Sockets ... Done"
print "[*] Sockets Binded Successfully ..."
print("[*] Server Started Successfully [%d]\n" % (listening_port))
except Exception, e:
print e
sys.exit(2)
while 1:
try:
conn, addr = s.accept()
data = conn.recv(buffer_size)
start_new_thread(conn_string, (conn, data, addr))
except KeyboardInterrupt:
s.close()
print "\n[*] Proxy Server Shutting Down ..."
sys.exit(1)
s.close()
def conn_string(conn, data, addr):
print conn
print addr
print data
As described in the python socket API:
socket.listen(backlog)
Listen for connections made to the socket. The
backlog argument specifies the maximum number of queued connections
and should be at least 0; the maximum value is system-dependent
(usually 5), the minimum value is forced to 0.
Increase the number from 5 to the number of simultaneous connections you wish to have to the server.
The accepted answer is incorrect. I'll try to explain you why.
listen takes an argument of queued connections. That means, how many new connections can be put into the queue. That's why even in the documentation it's specified 5, while obviously servers serves more than 5 concurrent clients usually.
Once a socket is created (accepted) it is being forwarded to another thread, which performs the actual work. This allows to the listening thread moves back to listening mode and wait for more clients.
The problem with your code is, your main thread which is the listening thread actually reads data from buffer, where it shouldn't. The recv() function blocks (read more about I/O stream blocking)
You need to move the reading process from the listening thread to the worker; and make sure to close() the socket when it's done. The code would look something like this:
def main():
try:
...
s.listen(5)
except Exception, e:
...
while 1:
try:
conn, addr = s.accept()
start_new_thread(conn_string, (conn, addr))
except KeyboardInterrupt:
s.close()
print "\n[*] Proxy Server Shutting Down ..."
sys.exit(1)
s.close()
def conn_string(conn, addr):
data = conn.recv(buffer_size)
print conn
print addr
print data
conn.close()
I wrote a python client to communicate with server side. Each time when I finished sanding out data, I have to call sock.shutdown(socket.SHUT_WR), otherwise the server would not do any response. But after calling sock.shutdown(socket.SHUT_WR), I have to reconnect the connection as sock.connect((HOST, PORT)), other wise I can not send data to server. So how can I keep the connection alive without close it.
My sample code as following:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
sock.sendall(data)
sock.shutdown(socket.SHUT_WR)
received = sock.recv(1024)
while len(received)>0:
received = sock.recv(1024)
sock.sendall(newdata) # this would throw exception
The Server Side code as following:
def handle(self):
cur_thread = threading.current_thread()
while True:
self.data = self.rfile.read(bufsiz=100)
if not self.data:
print 'receive none!'
break
try:
data = self.data
print 'Received data, length: %d' % len(data)
self.wfile.write('get received data\n')
except Exception:
print 'exception!!'
You didn't show any server side code but I suspect it simply reads bytes until it gets none anymore.
You can't do this as you found out, because then the only way to tell the server the message is complete is by killing the connection.
Instead you'll have to add some form of framing in your protocol. Possible approaches include a designated stop character that the server recognises (such as a single newline character, or perhaps a 0-byte), sending frames in fixed sizes that your client and server agree upon, or send the frame size first as a network encoded integer followed by exactly the specified number of bytes. The server then first reads the integer and then exactly that same number of bytes from the socket.
That way you can leave the connection open and send multiple messages.
How can I set a limit on the number of connections that a server socket can accept at once? I want to be able to set a max number of connections, and then once that limit is reached, any further attempts from clients to connect will result in a timeout. So far, I have tried something like this for the server:
sock = socket.socket()
sock.setblocking(0)
sock.bind(address)
sock.listen(0)
connections = []
while True:
readable, writable, exceptional = select.select([sock], [], [])
if readable and len(connections) < MAX_CONNECTIONS:
connection, client_address = s.accept()
connections.append(connection)
# Process connection asynchronously
and for the client:
try:
sock = socket.create_connection(self.address, timeout=TIMEOUT)
sock.settimeout(None)
print "Established connection."
except socket.error as err:
print >> sys.stderr, "Socket connection error: " + str(err)
sys.exit(1)
# If connection successful, do stuff
Because of the structure of the rest of the program, I have chosen to use a non-blocking socket on the server and I do not wish to change this. Right now, the clients are able to connect to the server, even after the limit is reached and the server stops accepting them. How do I solve this? Thanks.
I believe there might be a slight misunderstanding of select() at play here. According to the manpage, select() returns file descriptors that are "ready for some class of IO operation", where ready means "it is possible to perform a corresponding IO operation without blocking".
The corresponding IO operation on a listening socket is accept(), which can only be performed without blocking if the OS already made the full TCP handshake for you, otherwise you might block waiting for the client's final ACK, for instance.
This means that as long as the listening socket is open, connections will be accepted by the OS, even if not being handled by the application.
If you want to reject connections after a set number, you have basically two options:
simply accept and close directly after accepting.
close the listening socket upon reaching the limit and reopen when done.
The second option is more convoluted and requires use of the SO_REUSEADDR option, which might not be the right thing in your case. It might also not work on all OSs, though it does seem to work reliably on Linux.
Here's a quick sketch of the second solution (since the first is pretty straightforward).
def get_listening_socket():
sock = socket.socket()
sock.setblocking(0)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('0.0.0.0', 5555))
sock.listen(0)
return sock
sock = get_listening_socket()
LIMIT = 1
conns = {sock}
while True:
readable, writable, exceptional = select.select(conns, [], [])
if sock in readable: # new connection on the listening socket
conn, caddr = sock.accept()
conns.add(conn)
if len(conns) > LIMIT: # ">" because "sock" is also in there
conns.remove(sock)
sock.close()
else: # reading from an established connection
for c in readable:
buf = c.recv(4096)
if not buf:
conns.remove(c)
sock = get_listening_socket()
conns.add(sock)
else:
print("received: %s" % buf)
You may, however, want to rethink why you'd want to do this in the first place. If it's only about saving some memory on the server, than you might be over-optimizing and should be looking into syn-cookies instead.
I'm writing a multithreaded distributed networking algorithm.
I've one thread that listens to new connections. Every time a new connection is established a separate thread is started for listening to messages from that connection.
My problem is that the socket I open works perfectly in both directions inside the connection listener. After I pass the socket object for that connection to the message listener I can read data from the socket, but sending data through it doesn't reach the remote host.
Here's the essential snip from my code:
def connection_listener(port, start_e, terminate_e):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.settimeout(1)
s.bind(('', port))
s.listen(1)
while (not start_e.isSet()):
try:
conn, addr = s.accept()
msg_in = conn.recv(1024).split(":")
if (msg_in[1]=="hello"):
# If addr sends us a 'id:hello', we reply with a 'my_id:welcome'
conn.send(str(my_id)+":welcome")
t = Thread(target=message_listener, args=(conn, addr[0], terminate_e, ))
t.start()
except:
pass # timeout
def message_listener(conn, address, terminate_e):
while (not terminate_e.isSet()):
try:
msg_in = conn.recv(1024)
# Here I can receive everything that I send from the other end of conn,
# but conn.send("any data") doesn't reach the remote host
What I'd like to do is send acknowledgement-like messages from the message listener thread using the conn. Is this possible somehow or am I thinking and doing it wrong?
I sorted this out myself, so I'll share my answer.
I made the protocol exchange fixed size messages by padding with zeroes up to the desired length. I used a length of 32 bytes, which might be quite tiny from the hardware's point of view. Nevertheless it seems to work as supposed.
Pragmatically my solution looks like:
def send_everyone(message):
for i in range(len(peers)):
chunk = (str(my_id)+":"+message).rjust(32, '0')
peers[i].send(chunk)
And on the receiving side we want only 32 bytes at a time:
def message_listener(conn, address, terminate_e):
while (not terminate_e.isSet()):
try:
msg_in = conn.recv(32)
...