I have python script with only one socket object that is connect to a java server.
I started a thread for sending heart beat message to server per 5 secs.
And another thread for receiving message from server.
BTW, all the data send/recv is in protobuffer format.
# socket_client.py
def recv_handler():
global client_socket
while True:
try:
# read 4 bytes first
pack_len = client_socket.recv(4)
pack_len = struct.unpack('!i', pack_len)[0]
# read the rest
recv_data = client_socket.recv(pack_len)
# decode
decompressed_data = data_util.decompressMessage(recv_data)
sc_pb_message = data_util.decodePBMessage(decompressed_data)
sc_head = data_util.parseHead(sc_pb_message)
except:
print 'error'
def heart_handler():
global client_socket
while True:
if client_socket:
message = data_util.makeMessage('MSG_HEART_BEAT')
compressed_data = data_util.compressMessage(message)
send_data = data_util.makeSendData(compressed_data)
try:
client_socket.send(send_data)
except:
print 'except'
pass
time.sleep(5)
def connect(address, port):
global client_socket
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect((address, port))
# thread sending heart beat message
th = threading.Thread(target = heart_handler)
th.start()
# thread recving message
tr = threading.Thread(target = recv_handler)
tr.start()
The code above works just fine. The script will send a heart beat message per 5 secs and receive the message from server and the message can be decoded successfully.
And here comes the trigger part than I do not know how to implement.
My python script need to receive input from the browser at the same time, so I started a BaseHTTPServer, to handle the POST request from the browser.
When a request come, I would like to call the client_socket.send method to send a specific message to the server and of course I need to return the data from server back to the browser.
# http_server.py
def do_POST(self):
# ...
result = socket_client.request(message)
self.send_response(200)
self.end_headers()
self.wfile.write(...)
And here is what I tried to do in request:
def request(message):
global client_socket
client_socket.send(message)
pack_len = client_socket.recv(4)
pack_len = struct.unpack('!i', pack_len)[0]
recv_data = client_socket.recv(pack_len)
return recv_data
The problem I am having is the data I received in the request method after calling the send method seems to be disturbed by the data of heart beat in the thread.
If I comment out the heart beat thread and the receive thread, than the request method will work just fine. The data from server can decoded with no error and it can be sent back to the browser successfully.
My solution now might be wrong and I really do not know how to get this work.
Any advice will be appreciated, thanks :)
socket object in Python is not thread-safe, you need to access the shared resources (in this case the client_socket object) with the help of some synchronization primitives, such as threading.Lock in Python 2. Check here for a similar problem: Python: Socket and threads?
Related
I'm trying to create Python TCP client. It is fully functional, but I desire a functionality where if a "connected" message hasn't been recieved from the server for x seconds, then I can close my connection. I currently have a separate thread for recieving messages from the server. Basically, if I receive a message that says "connected" I want to reset a timer.
I've tried a few different approaches to solving it. I've tried to have a separate thread with a timer that keep track of when to disconnect using global variables which is a messy solution which didn't work regardless. I tried to make my own threading subclass but I still was not able to solve the problem and it introduced a lot more complexity than I really need.
a simple approach that I took was to try to have a timer in the while loop in my recieve function. This works for disconnecting from the server, but it will only timeout when a message is sent (due to the try block) and not actually the second the timer is up.
import threading, time, sys
def main():
host = sys.argv[1]
port = sys.argv[2]
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
address = (host, port)
sock.connect(address)
rec_thread = threading.Thread(target=rec, args=(sock,))
rec_thread.setDaemon(True)
rec_thread.start()
while True:
message = input()
sock.sendall(str.encode(message))
def rec(sock):
start_time = time.time()
while True:
elapsed_time = time.time() - start_time
if(elapsed_time > 10):
sock.close()
try:
msg = sock.recv(1024).decode("utf8")
if msg == 'connected':
start_time = time.time()
else:
print(msg)
except KeyboardInterrupt:
sock.close()
break
If I use this approach then I will only actually close the connection after I recieve a message from the server.
ex: I want it to disconnect immediately after 10s, but if I don't get any message from the server until 14s, then it won't disconnect until 14s.
So I'm trying to write a program that gets a continuous stream of data from a server, in a thread. The data being sent from the server is always very short, but does range in length
I'm currently trying to use the recv function in the socket library to constantly receive data, but for some reason I'm not getting anything at all. I know a common problem with the recv function is that it sometimes splits up the data, but for me it's not receiving anything. I know that the problem doesn't come from the server, because when the client connects it does receive confirmation from the server. Is this because I'm trying to do this in a different thread?
This is the socket part of my code:
client = socket(AF_INET, SOCK_STREAM)
def connectToServer(host, port):
global client
try:
client.connect((host, port))
connectionResult = client.recv(2048).decode()
if connectionResult == 'succesfull connection':
return connectionResult
else:
return 'connection failed'
except:
return 'connection failed'
# This function is ran in a thread (from another module)
def getUpdatesFromServer(updateQueue):
global client
while True:
receivedUpdate = client.recv(10000).decode()
if receivedUpdate == 'server went down':
updateQueue.put('server went down')
And this is the code that starts the Thread:
getUpdatesFromServerThread = Thread(target=ServerConnect.getUpdatesFromServer, args=(GameState.screenUpdateQueue,))
getUpdatesFromServerThread.daemon = True
getUpdatesFromServerThread.start()
The connectToServer function is called here (the first thing that happens when the client is ran):
connectionOutcome = ServerConnect.connectToServer(host, port)
if connectionOutcome == 'succesfull connection':
startGame()
else:
print('\n\nFailed to connect to the server. Please check your internet connection. If your internet connection is okay, it probably means the server is down and you should wait for a while.\n\n')
And here is the startGame function:
def startGame():
print('You have been succesfully connected to the server.')
getUpdatesFromServerThread = Thread(target=ServerConnect.getUpdatesFromServer, args=(GameState.screenUpdateQueue,))
getUpdatesFromServerThread.daemon = True
getUpdatesFromServerThread.start()
screenUpdateThread = Thread(target=doScreenUpdates)
screenUpdateThread.daemon = True
screenUpdateThread.start()
sleep(1)
MainGameScreen.main()
# When the eventloop stops, this is ran and sends a disconnect signal to the server, which closes the connection (which does work)
sleep(0.5)
ServerConnect.sendCommandToServer('disconnect')
Thank you in advance!
Edit: I've found the problem. I thought the server was sending data, but in fact it wasn't, it just didn't give any error message, so I assumed that was working.
I am creating a socket client and trying to obtain some data. In order to do so, I need to connect to a web server via socket and the server actually creates another socket which listens and awaits for the data after which sends back to the client.
The problem I have with the code below is that my socket client does not wait for the incoming data from the server and just accepts empty data.
How can I wait for a non-empty data from the server using Python sockets?
My code:
import sys
import json
import socketIO_client
import time
host = 'https://SOME_URL'
socketIO = socketIO_client.SocketIO(host, params={"email" : "edmund#gmail.com"})
def on_connect(*args):
print "socket.io connected"
def on_disconnect(*args):
print "socketIO diconnected"
socketIO.on('connect', on_connect)
socketIO.on('disconnect', on_disconnect)
def on_response_state(*args):
print args # Prints ()
socketIO.emit('receive_state',on_response_state)
socketIO.wait_for_callbacks(seconds=3)
Here's an example using socket. Using s.accept(), the client will wait till a client accepts the connection before starting the while loop to receive data. This should help with your problem.
def receiver():
PORT = 123
CHUNK_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('0.0.0.0', PORT))
s.listen(1)
conn,address=s.accept() # accept an incoming connection using accept() method which will block until a new client connects
while True:
datachunk = conn.recv(CHUNK_SIZE) # reads data chunk from the socket in batches using method recv() until it returns an empty string
if not datachunk:
break # no more data coming in, so break out of the while loop
data.append(datachunk) # add chunk to your already collected data
conn.close()
print(data)
return
receiver()
put the recv socket in a while thread.
like this:
def rec(self):
while 1:
sleep 0.01
rdata = self.clientsocket.recv(self.buffsize)
print("rec from server: ", rdata.decode('utf8'),'\n','press enter to continue')
....
t2 = threading.Thread(target=y.rec, name="rec")
t2.start()
Since you're using the SocketIO library to include parameters (achieved using requests), and want to emit a message, you can wait indefinitely for a response by not specifying a wait time.
with SocketIO(host, params={"email" : "edmund#gmail.com"}) as socketIO:
def on_response_state(*args):
print args # Prints ()
socketIO.emit('receive_state', on_response_state)
socketIO.wait()
Hello everyone!
I'm new to python networking programming.
My development environments are as below.
Windows 7
Python 3.4
I am studying with "Python Network Programming Cookbook". In this book, there's an example of ThreadingMixIn socket server application.
This book's code is written in Python 2.7. So I've modified for python 3.4.
The code is...
# coding: utf-8
import socket
import threading
import socketserver
SERVER_HOST = 'localhost'
SERVER_PORT = 0 # tells the kernel to pick up a port dynamically
BUF_SIZE = 1024
def client(ip, port, message):
""" A client to test threading mixin server"""
# Connect to the server
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip, port))
try:
message = bytes(message, encoding="utf-8")
sock.sendall(message)
response = sock.recv(BUF_SIZE)
print("Client received: {0}".format(response))
finally:
sock.close()
class ThreadedTCPRequestHandler(socketserver.BaseRequestHandler):
""" An example of threaded TCP request handler """
def handle(self):
data = self.request.recv(1024)
current_thread = threading.current_thread()
response = "{0}: {0}".format(current_thread.name, data)
response = bytes(response, encoding="utf-8")
self.request.sendall(response)
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
"""Nothing to add here, inherited everything necessary from parents"""
pass
if __name__ == "__main__":
# Run server
server = ThreadedTCPServer((SERVER_HOST, SERVER_PORT),
ThreadedTCPRequestHandler)
ip, port = server.server_address # retrieve ip address
# Start a thread with the server -- one thread per request
server_thread = threading.Thread(target=server.serve_forever)
# Exit the server thread when the main thread exits
server_thread.daemon = True
server_thread.start()
print("Server loop running on thread: {0}".format(server_thread))
# Run clients
client(ip, port, "Hello from client 1")
client(ip, port, "Hello from client 2")
client(ip, port, "Hello from client 3")
This code works perfect. Every client's request processed by new thread. And when the client's request is over, program ends.
I want to make server serves forever. So when the additional client's request has come, server send its response to that client.
What should I do?
Thank you for reading my question.
P.S: Oh, one more. I always write say hello in top of my post of stack overflow. In preview it shows normally. But when the post has saved, first line always gone. Please anyone help me XD
Your program exits because your server thread is a daemon:
# Exit the server thread when the main thread exits
server_thread.daemon = True
You can either remove that line or add server_thread.join() at the bottom of the code to prevent the main thread from exiting early.
You will have to run on an infinite loop and on each loop wait for some data to come from client. This way the connection will be kept alive.
Same infinite loop for the server to accept more clients.
However, you will have to somehow detect when a client closes the connection with the server because in most times the server won't be notified.
I am trying to send messages on TCP/IP all on host machine. This is working, although for some reason the socket needs to be re-instantiated for every new message on the client side only. For example here is a basic client that sends three separate messages:
import socket
host = '127.0.0.1'
class Client:
def __init__(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def connect(self):
self.sock.connect((host,12347))
def send(self,message):
self.sock.sendall(message)
def close(self):
self.sock.close()
if __name__ == "__main__":
message1 = "I am message 1"
message2 = "I am message 2"
message3 = "I am message 3"
#exp = Client()
#exp.connect()
for i in range(0,3):
try:
exp = Client()
exp.connect()
if i == 0:
txt = message1
elif i == 1:
txt = message2
elif i == 2:
txt = message3
exp.send(txt)
exp.close()
print i
exp.send(txt)
except:
pass
and the server that receives:
#!/usr/bin/env python
import socket
class communication:
def __init__(self):
try:
host = '127.0.0.1'
self.Server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.Server.bind((host,12347))
self.Server.listen(1)
finally:
print "setup finished"
def recieve(self):
(connection, client_address) = self.Server.accept()
data = connection.recv(128)
return data
def close(self):
self.server.close()
if __name__ == "__main__":
exp = communication()
while True:
try:
(connection,client_address) = exp.Server.accept()
message = connection.recv(128)
finally:
print message
if message == "I am message 3":
exp.close()
You see how I re-call the Client class in each iteration of the for loop. This seems to be necessary for sending messages 2 and 3. If the socket is instantiated only once at the start of the main code along with the connect() function, then the server hangs on the recv() after the first message has been sent.
I can't understand why this is happening and the socket only needs to be setup once on the server side. I am doing something wrong, or is this normal?
Thanks!
It's even worse than you think. Take a look at your server code. exp.Server.accept() accepts a connection from the client, but connection.receive() ignores that connection completely and does a second self.Server.accept(). You ignore half of your connections!
Next, your server only does a single receive.... Even if you tried to send more messages on the connection, the server would ignore them.
But you can't just add a recv loop. Your client and server need some way to mark message boundaries so the server knows how to pull them out. Some text based systems use a new line. Others send a message size or fixed size header that the server can read. HTTP for example uses a combination of new lines and data count.
If you want to learn sockets from the ground up just know that they are complicated and you'll need to study. There are lots of ways to build a server and you'll need to understand the trade-offs. Otherwise, there are many frameworks from XMLRPC to zeromq that do some of the heavy lifting for you.