I'm currently working with python's socket library for the first time and i'm not very experienced with computer networking.
I'm able to connect to the server and the tcp handshake has happened as viewed by wireshark. After establishing a connection to the server(I have no control over the server), the connection stays open for a while, but after a small amount of time, the server sends a "FIN, ACK" and the connection is terminated. I'm trying to understand how I can keep this connection alive while the client is capable of reaching the server.
Looking at a tcp connection, it seems a packet can be sent every so often. Maybe a sort of keep alive message. I had thought using socket.send('hello') every 5 seconds in another thread would keep the connection with the server open, but I still get the "FIN, ACK" after some time.
In the documentation I found a setsockopt() but using this made no noticeable difference. I've tried client.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) both before and after the connection is made. I don't completely understand how this method is supposed to work, so maybe I used it incorrectly. There isn't much mention of this. I read somewhere about it being broken on windows. I don't know the truth in that.
What am I missing? The documentation for sockets doesn't seem to have anything about this unless I may have missed something.
import socket
import time
import threading
SERVER_IP = 'THE SERVER'
SERVER_PORT = SERVER_PORT
SOURCE_IP = socket.gethostname()
SOURCE_PORT = 57004
KEEP_ALIVE_INTERVAL = 5
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def keep_alive(interval):
data = 'hello'
while True:
client.send(data)
time.sleep(interval)
client.connect((SERVER_IP, SERVER_PORT))
t = threading.Thread(target=keep_alive, args = (KEEP_ALIVE_INTERVAL,))
t.start()
while True:
data = client.recv(1024)
if not data:
break
print data
client.close()
For enabling keep alive there is a duplicate question at How to change tcp keepalive timer using python script?
Keep in mind some servers and intermediate proxies forcibly close long lived connections regardless of keep alives being used or not, in which case you will see a FIN,ACK after X amount of time no matter what.
Related
I have a fairly general question about best practice when using socket to communicate with remote hardware: should the socket be closed after each message is sent or left open?
To illustrate this question: I'm using python (and socket) to interface with a remote piece of hardware. Typically, I'll send a command to the device every 30 seconds or so, receive the reply and then wait ~ 30 seconds.
At present I'm doing:
# Open socket
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.settimeout(10)
self.sock.connect((self.host_ip_address, self.port))
# Send Message
self.sock.send(my_command)
# Receive Reply
data = self.sock.recv(1024)
# Close socket
self.sock.shutdown(socket.SHUT_RDWR)
self.sock.close()
I wonder if this advisable, or should I simply leave the socket open for the duration of my session with the device (say ~ 1hr). Would this be robust?
Any tips / pointers welcomed thanks!
This is robust as long as you exchange data from time to time over your socket. If not, a Firewall/NAT can decide that the TCP connection is broken and stop routing the TCP packet.
I'm going crazy writing a little socket server in python. Everything was working fine, but I noticed that in the case where the client just disappears, the server can't tell. I simulate this by pulling the ethernet cable between the client and server, close the client, then plug the cable back in. The server never hears that the client disconnected and will wait forever, never allowing more clients to connect.
I figured I'd solve this by adding a timeout to the read loop so that it would try and read every 10 seconds. I thought maybe if it tried to read from the socket it would notice the client was missing. But then I realized there really is no way for the server to know that.
So I added a heartbeat. If the server goes 10 seconds without reading, it will send data to the client. However, even this is successful (meaning doesn't throw any kind of exception). So I am able to both read and write to a client that isn't there any more. Is there any way to know that the client is gone without implementing some kind of challenge/response protocol between the client and server? That would be a breaking change in this case and I'd like to avoid it.
Here is the core of my code for this:
def _loop(self):
command = ""
while True:
socket, address = self._listen_socket.accept()
self._socket = socket
self._socket.settimeout(10)
socket.sendall("Welcome\r\n\r\n")
while True:
try:
data = socket.recv(1)
except timeout: # Went 10 seconds without data
pass
except Exception as e: # Likely the client closed the connection
break
if data:
command = command + data
if data == "\n" or data == "\r":
if len(command.strip()) > 0:
self._parse_command(command.strip(), socket)
command = ""
if data == '\x08':
command = command[:-2]
else: # Timeout on read
try:
self._socket.sendall("event,heartbeat\r\n") # Send heartbeat
except:
self._socket.close()
break
The sendall for the heartbeat never throws an exception and the recv only throws a timeout (or another exception if the client properly closes the connection under normal circumstances).
Any ideas? Am I wrong that sending to a client that doesn't ACK should generate an exception eventually (I've tested for several minutes).
The behavior you are observing is the expected behavior for a TCP socket connection. In particular, in general the TCP stack has no way of knowing that an ethernet cable has been pulled or that the (now physically disconnected) remote client program has shut down; all it knows is that it has stopped receiving acknowledgement packets from the remote peer, and for all it knows the packets could just be getting dropped by an overloaded router somewhere and the issue will resolve itself momentarily. Given that, it does what TCP always does when its packets don't get acknowledged: it reduces its transmission rate and its number-of-packets-in-flight limit, and retransmits the unacknowledged packets in the hope that they will get through this time.
Assuming the server's socket has outgoing data pending, the TCP stack will eventually (i.e. after a few minutes) decide that no data has gone through for a long-enough time, and unilaterally close the connection. So if you're okay with a problem-detection time of a few minutes, the easiest way to avoid the zombie-connection problem is simply to be sure to periodically send a bit of heartbeat data over the TCP connection, as you described. When the TCP stack tries (and repeatedly fails) to get the outgoing data sent-and-acknowledged, that is what eventually will trigger it to close the connection.
If you want something quicker than that, you'll need to implement your own challenge/response system with timeouts (either over the TCP socket, or over a separate TCP socket, or over UDP), but note that in doing so you are likely to suffer from false positives yourself (e.g. you might end up severing a TCP connection that was not actually dead but only suffering from a temporary condition of lost packets due to congestion). Whether or not that's a worthwhile tradeoff depends on what sort of program you are writing. (Note also that UDP has its own issues, particularly if you want your system to work across firewalls, etc)
So I'm messing around with a very simple echo server/client kind of deal. I'm eventually going to be making a game server, but for now I'm just trying to setup the basic parts.
Here's the server: http://pastebin.com/qtfrMGur
Here's the client: http://pastebin.com/3VK3VxPX
So, my problem is that when the client disconnects, I get socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host. I was under the impression that select.select() returned false(or an empty list, whatever) when the readable socket list had no pending information. This seems to be the cause of the error, but then why is select.select() returning true(on line 26) if the client is no longer there? The actual error occurs when I try to retrieve some info in line 27. I'm really confused by this, assuming the client-side of the socket is closed, shouldn't it just return [], and then go to the else on line 31 and destroy that client?
I don't really know what else to post, this seems like a very elementary problem, usually I use a more abstract library like twisted that makes it easy, but for this project I must use traditional BSD sockets. If you guys have any other info you need just ask.
Thanks.
EDIT: So I put a print statement in there to print the value of select.select(...)[0], like so:
for cl in clients:
ready = select.select([cl], [], [], .1)[0]
print ready
if ready:
data = cl.recv(size)
if data:
print data
#cl.send(data)
else:
print "Client Quit\n"
clients.remove(cl)
cl.shutdown(socket.SHUT_RDWR)
cl.close()
print len(clients)
As you can see I do a proper shutdown now as well. The problem is, that right as the client quits, select.select(...) returns the client socket as a readable source, so cl.recv gets called on it, which then throws an error because cl is closed on the other end. At least this is my suspicion.
So, could somebody please explain what I may be doing wrong here? Why does the client send a last dying message, is it some kind of functionality that I should be taking advantage of?
Thanks
EDIT2: So guys I fired up my debian machine and the code works flawlessly, it's only in windows that it's doing this. Is this some sort of bug? I was under the impression that the socket library abstracted all the stuff beneath the hood to make the interface identical between OS.
You should call the shutdown method of the socket, before closing it on the client side, like this:
s.shutdown(socket.SHUT_RDWR)
s.close()
That way it won't allow further sending / receiving, check out the Python docs about it, it's nicely explained there.
UPDATE: fixed client code
import socket
host = 'localhost'
port = 50000
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host,port))
msg = ''
while msg !='quit':
msg = raw_input()
s.send(msg)
#data = s.recv(size)
s.shutdown(socket.SHUT_RDWR)
s.close()
I would like to implement a TCP server using a Python script.
The server should basically do the following task:
It will be cyclically polled by a remote client, then reads certain data on its local workstation and sends it to the client polling.
One can assume the following:
There is always only one client connecting to the server (via Ethernet)
Client and server are running on a Windows platform
Python 2.5.1 will be used for implementation
Goal:
The server should be as efficient as possible with respect to reading/writing data from/to the client
The server shall stress the local workstation as less as possible
The server shall not cause system instability
Since I have only a little experience with the topic, I would like to discuss here how my current (very simple) code could be optimized to meet the requirements previously mentioned.
So far I have the following:
import socket
import sys
port_number = 12345
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = (socket.gethostname(), port_number)
sock.bind(server_address)
sock.listen(1)
while True:
connection, client_address = sock.accept()
try:
while True:
data = connection.recv(4096)
if data:
connection.sendall(getdesireddata(data))
else:
break
finally:
connection.close()
Thank you for all your responses.
First of all, there's minor bug in the code. The line...
data = connection.recv(4096)
...will attempt to read up to 4k from a single TCP packet, but if the client sends more than 4k, or decides to break up the data into several packets, you may not get all the data in a single call.
You'd typically continue to read data from the socket, appending to a buffer, until either you have a complete message as defined by your protocol specification, or until the remote host closes the outbound half of its TCP connection.
You'll also have issues with the server getting stuck if the client crashes, and doesn't shut down the socket properly. To avoid this, you'll need to look at either non-blocking IO, which would be something like...
connection.setblocking(0)
buffer = ''
while 1:
try:
data = connection.recv(4096)
if not data:
# Client has closed outbound connection
break
else:
# Append to buffer
buffer += data
except socket.error, e:
code = e.args[0]
if code == 11:
# Got EWOULDBLOCK/EAGAIN
time.sleep(0.1)
else:
# Got another error
raise
do_something_with(buffer)
...or look at using the Python select module to do the same thing.
With regards to the lesser concern of performance, I don't see any major improvements you can make with such a small code sample.
I currently have a client/server pair coded against PyBlueZ. Right now the server can connect to sequential clients - it will work until its completed with a client, then it will begin listening for another client.
However, what I really want is to run client communication in separate threads so I have multiple clients at the same time. When I try a 2nd client connection, however, PyBlueZ advertises the same port that the first client is currently using. I am setting up connections like this:
self.port = bluetooth.PORT_ANY
print "Accepting clients..."
self.server_sock=bluetooth.BluetoothSocket( bluetooth.RFCOMM )
self.server_sock.bind(("",self.port))
self.server_sock.listen(5)
print "listening on port %d" % self.port
bluetooth.advertise_service( self.server_sock, MY_SERVICE, MY_UUID )
client_sock,address = self.server_sock.accept()
print "Accepted connection from ",address
commThread = ServerThread(client_sock, self.bn_id, self.bn_name, self.bn_thumbnail)
Again, this code works fine for sequential connections, but when I try it in parallel my client gets a "busy" response from the server's bluetooth system. On the client side I output the port its trying to connect to and it always shows port "1".
Is there a limitation in PyBlueZ which only allows for a single connection? Or am I doing something wrong here for parallel connections?
I think your problem has nothing to do with the Bluetooth client part of the code.
You were right to show the Bluetooth server code. What you should try to change:
Only advertise a service once, and once only (No need to advertise it for every server thread)
Allocate a different server channel for each thread. (On RFCOMM connection, there usually is a RFCOMM manager which allocates a new server channel per each socket. I think in your case you might have to do that manually.) Change this code
self.port = bluetooth.PORT_ANY
Try channels 1, 2 and so on and see if it works! Then all you have to do is keep track of the allocated channels.
Please let me know if it worked!