Develop Reference

Python Socket How to get a continuous stream of data with recv in a thread?

So I'm trying to write a program that gets a continuous stream of data from a server, in a thread. The data being sent from the server is always very short, but does range in length
I'm currently trying to use the recv function in the socket library to constantly receive data, but for some reason I'm not getting anything at all. I know a common problem with the recv function is that it sometimes splits up the data, but for me it's not receiving anything. I know that the problem doesn't come from the server, because when the client connects it does receive confirmation from the server. Is this because I'm trying to do this in a different thread?
This is the socket part of my code:
client = socket(AF_INET, SOCK_STREAM)
def connectToServer(host, port):
global client
try:
client.connect((host, port))
connectionResult = client.recv(2048).decode()
if connectionResult == 'succesfull connection':
return connectionResult
else:
return 'connection failed'
except:
return 'connection failed'
# This function is ran in a thread (from another module)
def getUpdatesFromServer(updateQueue):
global client
while True:
receivedUpdate = client.recv(10000).decode()
if receivedUpdate == 'server went down':
updateQueue.put('server went down')
And this is the code that starts the Thread:
getUpdatesFromServerThread = Thread(target=ServerConnect.getUpdatesFromServer, args=(GameState.screenUpdateQueue,))
getUpdatesFromServerThread.daemon = True
getUpdatesFromServerThread.start()
The connectToServer function is called here (the first thing that happens when the client is ran):
connectionOutcome = ServerConnect.connectToServer(host, port)
if connectionOutcome == 'succesfull connection':
startGame()
else:
print('\n\nFailed to connect to the server. Please check your internet connection. If your internet connection is okay, it probably means the server is down and you should wait for a while.\n\n')
And here is the startGame function:
def startGame():
print('You have been succesfully connected to the server.')
getUpdatesFromServerThread = Thread(target=ServerConnect.getUpdatesFromServer, args=(GameState.screenUpdateQueue,))
getUpdatesFromServerThread.daemon = True
getUpdatesFromServerThread.start()
screenUpdateThread = Thread(target=doScreenUpdates)
screenUpdateThread.daemon = True
screenUpdateThread.start()
sleep(1)
MainGameScreen.main()
# When the eventloop stops, this is ran and sends a disconnect signal to the server, which closes the connection (which does work)
sleep(0.5)
ServerConnect.sendCommandToServer('disconnect')
Thank you in advance!
Edit: I've found the problem. I thought the server was sending data, but in fact it wasn't, it just didn't give any error message, so I assumed that was working.

Python: Multithreaded socket server runs endlessly when client stops unexpectedly

I have created a multithreaded socket server to connect many clients to the server using python. If a client stops unexpectedly due to an exception, server runs nonstop. Is there a way to kill that particular thread alone in the server and the rest running
Server:
class ClientThread(Thread):
def __init__(self,ip,port):
Thread.__init__(self)
self.ip = ip
self.port = port
print("New server socket thread started for " + ip + ":" + str(port))
def run(self):
while True :
try:
message = conn.recv(2048)
dataInfo = message.decode('ascii')
print("recv:::::"+str(dataInfo)+"::")
except:
print("Unexpected error:", sys.exc_info()[0])
Thread._stop(self)
tcpServer = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcpServer.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
tcpServer.bind((TCP_IP, 0))
tcpServer.listen(10)
print("Port:"+ str(tcpServer.getsockname()[1]))
threads = []
while True:
print( "Waiting for connections from clients..." )
(conn, (ip,port)) = tcpServer.accept()
newthread = ClientThread(ip,port)
newthread.start()
threads.append(newthread)
for t in threads:
t.join()
Client:
def Main():
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect((host,int(port)))
while True:
try:
message = input("Enter Command")
s.send(message.encode('ascii'))
except Exception as ex:
logging.exception("Unexpected error:")
break
s.close()

Sorry about a very, very long answer but here goes.
There are quite a many issues with your code. First of all, your client does not actually close the socket, as s.close() will never get executed. Your loop is interrupted at break and anything that follows it will be ignored. So change the order of these statements for the sake of good programming but it has nothing to do with your problem.
Your server code is wrong in quite a many ways. As it is currently written, it never exits. Your threads also do not work right. I have fixed your code so that it is a working, multithreaded server, but it still does not exit as I have no idea what would be the trigger to make it exit. But let us start from the main loop:
while True:
print( "Waiting for connections from clients..." )
(conn, (ip,port)) = tcpServer.accept()
newthread = ClientThread(conn, ip,port)
newthread.daemon = True
newthread.start()
threads.append(newthread) # Do we need this?
for t in threads:
t.join()
I have added passing of conn to your client thread, the reason of which becomes apparent in a moment. However, your while True loop never breaks, so you will never enter the for loop where you join your threads. If your server is meant to be run indefinitely, this is not a problem at all. Just remove the for loop and this part is fine. You do not need to join threads just for the sake of joining them. Joining threads only allows your program to block until a thread has finished executing.
Another addition is newthread.daemon = True. This sets your threads to daemonic, which means they will exit as soon as your main thread exits. Now your server responds to control + c even when there are active connections.
If your server is meant to be never ending, there is also no need to store threads in your main loop to threads list. This list just keeps growing as a new entry will be added every time a client connects and disconnects, and this leaks memory as you are not using the threads list for anything. I have kept it as it was there, but there still is no mechanism to exit the infinite loop.
Then let us move on to your thread. If you want to simplify the code, you can replace the run part with a function. There is no need to subclass Thread in this case, but this works so I have kept your structure:
class ClientThread(Thread):
def __init__(self,conn, ip,port):
Thread.__init__(self)
self.ip = ip
self.port = port
self.conn = conn
print("New server socket thread started for " + ip + ":" + str(port))
def run(self):
while True :
try:
message = self.conn.recv(2048)
if not message:
print("closed")
try:
self.conn.close()
except:
pass
return
try:
dataInfo = message.decode('ascii')
print("recv:::::"+str(dataInfo)+"::")
except UnicodeDecodeError:
print("non-ascii data")
continue
except socket.error:
print("Unexpected error:", sys.exc_info()[0])
try:
self.conn.close()
except:
pass
return
First of all, we store conn to self.conn. Your version used a global version of conn variable. This caused unexpected results when you had more than one connection to the server. conn is actually a new socket created for the client connection at accept, and this is unique to each thread. This is how servers differentiate between client connections. They listen to a known port, but when the server accepts the connection, accept creates another port for that particular connection and returns it. This is why we need to pass this to the thread and then read from self.conn instead of global conn.
Your server "hung" upon client connetion errors as there was no mechanism to detect this in your loop. If the client closes connection, socket.recv() does not raise an exception but returns nothing. This is the condition you need to detect. I am fairly sure you do not even need try/except here but it does not hurt - but you need to add the exception you are expecting here. In this case catching everything with undeclared except is just wrong. You have also another statement there potentially raising exceptions. If your client sends something that cannot be decoded with ascii codec, you would get UnicodeDecodeError (try this without error handling here, telnet to your server port and copypaste some Hebrew or Japanese into the connection and see what happens). If you just caught everything and treated as socket errors, you would now enter the thread ending part of the code just because you could not parse a message. Typically we just ignore "illegal" messages and carry on. I have added this. If you want to shut down the connection upon receiving a "bad" message, just add self.conn.close() and return to this exception handler as well.
Then when you really are encountering a socket error - or the client has closed the connection, you will need to close the socket and exit the thread. You will call close() on the socket - encapsulating it in try/except as you do not really care if it fails for not being there anymore.
And when you want to exit your thread, you just return from your run() loop. When you do this, your thread exits orderly. As simple as that.
Then there is yet another potential problem, if you are not only printing the messages but are parsing them and doing something with the data you receive. This I do not fix but leave this to you.
TCP sockets transmit data, not messages. When you build a communication protocol, you must not assume that when your recv returns, it will return a single message. When your recv() returns something, it can mean one of five things:
The client has closed the connection and nothing is returned
There is exactly one full message and you receive that
There is only a partial message. Either because you read the socket before the client had transmitted all data, or because the client sent more than 2048 bytes (even if your client never sends over 2048 bytes, a malicious client would definitely try this)
There are more than one messages waiting and you received them all
As 4, but the last message is partial.
Most socket programming mistakes are related to this. The programmer expects 2 to happen (as you do now) but they do not cater for 3-5. You should instead analyse what was received and act accordingly. If there seems to be less data than a full message, store it somewhere and wait for more data to appear. When more data appears, concatenate these and see if you now have a full message. And when you have parsed a full message from this buffer, inspect the buffer to see if there is more data there - the first part of the next message or even more full messages if your client is fast and server is slow. If you process a message and then wipe the buffer, you might have wiped also bytes from your next message.

How can I continually send data without shutdown socket connection in python

I wrote a python client to communicate with server side. Each time when I finished sanding out data, I have to call sock.shutdown(socket.SHUT_WR), otherwise the server would not do any response. But after calling sock.shutdown(socket.SHUT_WR), I have to reconnect the connection as sock.connect((HOST, PORT)), other wise I can not send data to server. So how can I keep the connection alive without close it.
My sample code as following:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
sock.sendall(data)
sock.shutdown(socket.SHUT_WR)
received = sock.recv(1024)
while len(received)>0:
received = sock.recv(1024)
sock.sendall(newdata) # this would throw exception
The Server Side code as following:
def handle(self):
cur_thread = threading.current_thread()
while True:
self.data = self.rfile.read(bufsiz=100)
if not self.data:
print 'receive none!'
break
try:
data = self.data
print 'Received data, length: %d' % len(data)
self.wfile.write('get received data\n')
except Exception:
print 'exception!!'

You didn't show any server side code but I suspect it simply reads bytes until it gets none anymore.
You can't do this as you found out, because then the only way to tell the server the message is complete is by killing the connection.
Instead you'll have to add some form of framing in your protocol. Possible approaches include a designated stop character that the server recognises (such as a single newline character, or perhaps a 0-byte), sending frames in fixed sizes that your client and server agree upon, or send the frame size first as a network encoded integer followed by exactly the specified number of bytes. The server then first reads the integer and then exactly that same number of bytes from the socket.
That way you can leave the connection open and send multiple messages.

How do I clear the buffer upon start/exit in ZMQ socket? (to prevent server from connecting with dead clients)

I am using a REQ/REP type socket for ZMQ communication in python. There are multiple clients that attempt to connect to one server. Timeouts have been added in the client script to prevent indefinite wait.
The problem is that when the server is not running, and a client attempts to establish connection, it's message gets added to the queue buffer, which should not even exist at this moment ideally. When the script starts running and a new client connects, the previous client's data is taken in first by the server. This should not happen.
When the server starts, it assumes a client is connected to it since it had tried to connect previously, and could not exit cleanly (since the server was down).
In the code below, when the client tries the first time, it gets ERR 03: Server down which is correct, followed by Error disconnecting. When server is up, I get ERR 02: Server Busy for the first client which connects. This should not occur. The client should be able to seamlessly connect with the server now that it's up and running.
Server Code:
import zmq
def server_fn():
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://192.168.1.14:5555")
one=1
while one == 1:
message = socket.recv()
#start process if valid new connection
if message == 'hello':
socket.send(message) #ACK
#keep session alive until application ends it.
while one == 1:
message = socket.recv()
print("Received request: ", message)
#exit connection
if message == 'bye':
socket.send(message)
break
#don't allow any client to connect if already busy
if message == 'hello':
socket.send ('ERR 00')
continue
#do all data communication here
else:
socket.send('ERR 01: Connection Error')
return
server_fn()
Client Code:
import zmq
class client:
def clientInit(self):
hello='hello'
#zmq connection
self.context = zmq.Context()
print("Connecting to hello world server...")
self.socket = self.context.socket(zmq.REQ)
self.socket.connect("tcp://192.168.1.14:5555")
#RCVTIMEO to prevent forever block
self.socket.setsockopt(zmq.RCVTIMEO, 5000)
#SNDTIME0 is needed since script may not up up yet
self.socket.setsockopt(zmq.SNDTIMEO, 5000)
try:
self.socket.send(hello)
except:
print "Sending hello failed."
try:
echo = self.socket.recv()
if hello == echo:
#connection established.
commStatus = 'SUCCESS'
elif echo == 'ERR 00':
#connection busy
commStatus = "ERR 00. Server busy."
else:
#connection failed
commStatus="ERR 02"
except:
commStatus = "ERR 03. Server down."
return commStatus
def clientQuit(self):
try:
self.socket.send('bye')
self.socket.recv()
except:
print "Error disconnecting."
cObj = client()
commStatus=cObj.clientInit()
print commStatus
cObj.clientQuit()
PS - I have a feeling the solution may lie in the correct usage of socket.bind and socket.connect.

Answering my own question-
The problem is that the first client sends a message which the server accepts when it starts running, regardless of the status of the client.
To prevent this, 2 things have to be done. The most important thing is to use socket.close() to close the client connection. Secondly, the LINGER parameter can be set to a low value or zero. This clears the buffer after the timeout value from the time the socket is closed.
class client:
def clientInit(self):
...
self.socket.setsockopt(zmq.LINGER, 100)
...
def clientQuit(self):
try:
self.socket.send('bye')
self.socket.recv()
except:
print "Error disconnecting."
self.socket.close()

How do multiple clients communicate using messaging system? (Python)

I have a server listening to all the clients that are connected to it, and each client should have this "username" they are identified by, but I'm stuck here... when a client wants to send a message to another client, they send the other client's username as the first word in the message sent to the server. Then I wouldn't know how to direct the message to the right socket (the receiving client). How should I link a client to its socket?? so I can call socket.send??
I'm currently modifying the code from this website: http://www.binarytides.com/code-chat-application-server-client-sockets-python/
Thanks!

def broadcast_data (sock, message):
#Do not send the message to master socket and the client who has send us the message
for socket in CONNECTION_LIST:
if socket != server_socket and socket != sock :
try :
socket.send(message)
except :
# broken socket connection may be, chat client pressed ctrl+c for example
socket.close()
CONNECTION_LIST.remove(socket)
This function could be used as a starting point for a send function
def send_data (sock, message):
#send message to only one client
#sock is the socket of the client to send to
try :
sock.send(message)
except :
# broken socket connection may be, chat client pressed ctrl+c for example
sock.close()
CONNECTION_LIST.remove(socket)
then you could have a dictionary of sockets associating them with a user name
usernameSockets={}
usernameSockets[username]=socket_for_user_name
then you could call the send_data function like
send_data(usernameSockets[username],message)
In the code you linked to there is no method for getting a username so you would have to have the clients send their username to the server when they are connecting to be able to populate a dictionary like the one I have mentioned above.