So I'm messing around with a very simple echo server/client kind of deal. I'm eventually going to be making a game server, but for now I'm just trying to setup the basic parts.
Here's the server: http://pastebin.com/qtfrMGur
Here's the client: http://pastebin.com/3VK3VxPX
So, my problem is that when the client disconnects, I get socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host. I was under the impression that select.select() returned false(or an empty list, whatever) when the readable socket list had no pending information. This seems to be the cause of the error, but then why is select.select() returning true(on line 26) if the client is no longer there? The actual error occurs when I try to retrieve some info in line 27. I'm really confused by this, assuming the client-side of the socket is closed, shouldn't it just return [], and then go to the else on line 31 and destroy that client?
I don't really know what else to post, this seems like a very elementary problem, usually I use a more abstract library like twisted that makes it easy, but for this project I must use traditional BSD sockets. If you guys have any other info you need just ask.
Thanks.
EDIT: So I put a print statement in there to print the value of select.select(...)[0], like so:
for cl in clients:
ready = select.select([cl], [], [], .1)[0]
print ready
if ready:
data = cl.recv(size)
if data:
print data
#cl.send(data)
else:
print "Client Quit\n"
clients.remove(cl)
cl.shutdown(socket.SHUT_RDWR)
cl.close()
print len(clients)
As you can see I do a proper shutdown now as well. The problem is, that right as the client quits, select.select(...) returns the client socket as a readable source, so cl.recv gets called on it, which then throws an error because cl is closed on the other end. At least this is my suspicion.
So, could somebody please explain what I may be doing wrong here? Why does the client send a last dying message, is it some kind of functionality that I should be taking advantage of?
Thanks
EDIT2: So guys I fired up my debian machine and the code works flawlessly, it's only in windows that it's doing this. Is this some sort of bug? I was under the impression that the socket library abstracted all the stuff beneath the hood to make the interface identical between OS.
You should call the shutdown method of the socket, before closing it on the client side, like this:
s.shutdown(socket.SHUT_RDWR)
s.close()
That way it won't allow further sending / receiving, check out the Python docs about it, it's nicely explained there.
UPDATE: fixed client code
import socket
host = 'localhost'
port = 50000
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host,port))
msg = ''
while msg !='quit':
msg = raw_input()
s.send(msg)
#data = s.recv(size)
s.shutdown(socket.SHUT_RDWR)
s.close()
Related
I am rather a noob here, but trying to setup a script where I can poll a socket, and when no socket data has been sent, a loop continues to run and do other things. I have been playing with several examples I found using select(), but no matter how I organize the code, it seems to stop on or near the server.recv() line and wait for a response. I want to skip out of this if no data has been sent by a client, or if no client connection exists.
Note that this application does not require the server script to send any reply data, if it makes any difference.
The actual application is to run a loop and animate some LEDs (which needs root access to the I/O on a Raspberry Pi). I am going to send this script data from another separate script via sockets that will pass in control parameters for the animations. This way the external script does not require root access.
So far the sending and receiving of data works great, I just can't get loop to keep spinning in the absence of incoming data. It is my understanding that this is what select() was intended to allow, but the examples I've found don't seem to be working that way.
I have attempted adding server.setblocking(0) a few different places to no avail. (If I understand correctly a non-blocking instance should allow the code to skip over the recv() if no data has been sent, but I may be off on this).
I have based my code on an example here:
http://ilab.cs.byu.edu/python/select/echoserver.html
Here is the server side script followed by the client side script.
Server Code: sockselectserver.py
#!/usr/bin/env python
import select
import socket
import sys
server = socket.socket()
host = socket.gethostname()
port = 20568
size = 1024
server.bind((host,port))
server.listen(5)
input = [server,sys.stdin]
running = 1
while running:
inputready,outputready,exceptready = select.select(input,[],[])
for s in inputready:
if s == server:
# handle the server socket
client, address = server.accept()
input.append(client)
elif s == sys.stdin:
# handle standard input
junk = sys.stdin.readline()
running = 0
else:
# handle all other sockets
data = s.recv(size)
if data:
s.send(data)
else:
s.close()
input.remove(s)
print "looping"
server.close()
Client Code: skclient.py
#!/usr/bin/python # This is client.py file
import socket # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 20568 # Reserve a port for your service.
s.connect((host, port))
data = "123:120:230:51:210:120:55:12:35:24"
s.send(data)
print s.recv(1024)
s.close # Close the socket when done
What I would like to achieve by this example is to see "looping" repeated forever, then when the client script sends data, see that data print, then see the "looping" resume printing over and over. That would tell me it's doing what is intended I can take it from there.
Interesting enough, when I test this as is, whenever I run the client, I see "looping" printed 3 times on the screen, then no more. I don't fully understand what is happening inside the select, but I'd assume it would only print 1 time.
I tried moving the inputready.. select.select() around to different places but found it appears to need to be called each time, otherwise the server stops responding (for example if it is called once prior to the endless while: loop).
I'm hoping this can be made simple enough that it can be taught to other hacker types in a maker class, so I'm hopeful I don't need to get too crazy with multi-threading and more elaborate solutions. As a last resort I'm considering logging all my parameters to mySQL from the external script then using this script to query them back out of tables. I've got experience there and would probably work, but it seems this socket angle would be a more direct solution.
Any help very much appreciated.
Great news. This was an easy fix, wanted to post in case anyone else needed it. The suggestion from acw1668 above got me going.
Simply added a timeout of "0" to the select.select() like this:
inputready,outputready,exceptready = select.select(input,[],[],0)
This is in the python docs but somehow I missed it. Link here: https://docs.python.org/2/library/select.html
Per the docs:
The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks.
I tested the same code as above, adding a delay of 5 seconds using time.sleep(5) right after the print "looping" line. With the delay, if no data or client is present the code just loops every 5 seconds and prints "looping" to the screen. If I kick off the client script during the 5 second delay, it pauses and the message is processed the next time the 5 second delay ends. Occasionally it doesn't respond the very next loop, but rather the loop following. I assume this is because the first time through the server.accept is running and the next time through the s.recv() is running which actually exchanges the data.
I'm currently working with python's socket library for the first time and i'm not very experienced with computer networking.
I'm able to connect to the server and the tcp handshake has happened as viewed by wireshark. After establishing a connection to the server(I have no control over the server), the connection stays open for a while, but after a small amount of time, the server sends a "FIN, ACK" and the connection is terminated. I'm trying to understand how I can keep this connection alive while the client is capable of reaching the server.
Looking at a tcp connection, it seems a packet can be sent every so often. Maybe a sort of keep alive message. I had thought using socket.send('hello') every 5 seconds in another thread would keep the connection with the server open, but I still get the "FIN, ACK" after some time.
In the documentation I found a setsockopt() but using this made no noticeable difference. I've tried client.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) both before and after the connection is made. I don't completely understand how this method is supposed to work, so maybe I used it incorrectly. There isn't much mention of this. I read somewhere about it being broken on windows. I don't know the truth in that.
What am I missing? The documentation for sockets doesn't seem to have anything about this unless I may have missed something.
import socket
import time
import threading
SERVER_IP = 'THE SERVER'
SERVER_PORT = SERVER_PORT
SOURCE_IP = socket.gethostname()
SOURCE_PORT = 57004
KEEP_ALIVE_INTERVAL = 5
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def keep_alive(interval):
data = 'hello'
while True:
client.send(data)
time.sleep(interval)
client.connect((SERVER_IP, SERVER_PORT))
t = threading.Thread(target=keep_alive, args = (KEEP_ALIVE_INTERVAL,))
t.start()
while True:
data = client.recv(1024)
if not data:
break
print data
client.close()
For enabling keep alive there is a duplicate question at How to change tcp keepalive timer using python script?
Keep in mind some servers and intermediate proxies forcibly close long lived connections regardless of keep alives being used or not, in which case you will see a FIN,ACK after X amount of time no matter what.
I have a few test clients that are encountering the same issue each time. The clients can connect, and they can send their first message, but after that the server stops responding to that client. I suspect that the problem is related to s.accept(), but I'm not sure exactly what is wrong or how to work around it.
def startServer():
host = ''
port = 13572
backlog = 5
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(backlog)
print "Close the command prompt to stop Gamelink"
while 1:
try:
client, address = s.accept()
data = client.recv(size)
if data:
processData(data)
client.send("OK")
else:
print "Disconnecting from client at client's request"
client.close()
except socket.error, (value, message):
if s:
print "Disconnecting from client, socket issue"
s.close()
print "Error opening socket: " + message
break
except:
print "Gamelink encountered a problem"
break
print "End of loop"
client.close()
s.close()
The server is intended to be accessed across a local network, and it needs to be light weight and very quick to respond, so if another implementation (such as thread based) would be better for meeting those requirements please let me know. The intended application is to be used as a remote gaming keyboard, thus the need for low resource use and high speed.
Writing a server using socket directly will be hard. As Keith says, you need to multiplex the connections somehow, like with select or poll or threads or fork. You might think you need only one connection, but what will you do when something hiccups and the connection is lost? Will your server be able to respond to reconnection attempts from the client if it hasn't yet realized the connection is lost?
If your networking needs are basic, you might be able to let something else handle all the listening and accepting and forking stuff for you. You don't specify a platform, but examples of such programs are launchd on Mac OS and xinetd on Linux. The details differ between these tools, but basically you configure them, in some configuration file, to listen for a connection on some port. When they get it, they take care of setting up the connection, then they exec() your program with stdin and stdout aimed at the socket, so you can simply use all the basic IO you probably already know like print and sys.stdin.read().
The trouble with solutions like xinitd and launchd is that for each new connection, they must fork() and exec() a new instance of your program. These are relatively heavy operations so a large number of connections or a high rate of new connections might hit the limits of your server. But worse, since each connection is in a separate process, sharing data between them is hard. Also, most solutions you might find to communicate between processes involve a blocking API, and now you are back to the problem of multiplexing with select or threads or similar.
If that doesn't meet your needs, I think you are better off learning to use a higher-level networking framework which will handle all the problems you will inevitably encounter if you go down the path of socket. One such framework I'd suggest is Twisted. Beyond handling the mundane details of handling connections, and the more complex task of multiplexing IO between them, you will also have a huge library of tools that will make implementing your protocol much easier.
I'm making a program that needs to recieve a connection hash from a server. When I use:
connhash = s.recv(1024)
I get this error:
[Errno 10054] An existing connection was forcibly closed by the remote host
Is this my fault or the servers fault?
Here is some of the code leading up to s.recv()
stringfmt = u'%(user)s;%(host)s:%(port)d'
string = stringfmt % data
structfmt = '!bh'
encoded = string.encode('utf-16BE')
packetbytes = struct.pack(structfmt, 2, len(encoded))+encoded
s.send(packetbytes)
connhash = s.recv(1024)
I am using Python v 2.7
EDIT: This is for Minecraft just so you know.
It sounds like the remote server doesn't like your connection and cuts you off. This could mean you've made a protocol mistake (i.e., the commands you are sending are incorrect), or you may not have logged in successfully, or your IP may have been banned, or many other similar things.
To debug it, you could try using something like telnet to replicate the connection and see where the error occurs (if it doesn't, then there is something wrong with your implementation; if it does, there is something wrong with your understanding of the protocol, or you are blocked from using the server).
Alternatively, use a packet capture tool like Wireshark to look at what packets are being sent and received, and see if that shows the problem.
I'm working on a slightly larger project of my own and I need to make a localhost proxy in python.
The way I wrote mine is that there's a TCP server (using socket and SOCK_STREAM) on port 8080 on the localhost. It accepts a request from the local host, using slicing, string.find(), and gethostbyname() finds that target IP, so it opens up another TCP socket, sends the request and recv's a reply. After that, it relays the reply back to the localhost proxy which in turn throws it back at the browser.
This is the code with ample debugging messages and a debug file to collect the requests of the browser and the replies received back (also note this is just a prototype, hence the limited for loop instead of a while 1 loop):
import socket
local = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
f = open('test.txt', 'a')
local.bind(('localhost', 8080))
local.listen(5)
for i in xrange(20):
print '=====%d=====\n' % i
out = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
data, addr = local.accept()
print 'Connection accepted'
buffer = data.recv(4096)
print 'data recieved'
f.write('=============================================================\n')
f.write(buffer)
end = buffer.find('\n')
print buffer
#print buffer[:end]
host = buffer[:end].split()[1]
end = host[7:].find('/')
print host[7:(end+7)]
host_ip = socket.gethostbyname(host[7:(end+7)])
#print 'remote host: ' + host + ' IP: ' + host_ip
print 'sending buffer to remote host'
out.connect((host_ip, 80))
out.sendall(buffer)
print 'recieving data from remote host'
reply = out.recv(4096)
out.close()
print 'data recieved from remote host'
f.write('+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n')
f.write(reply)
f.write('\n\n\n')
print 'sending data back to local host'
data.sendall(reply)
print 'data sent'
local.close()
out.close()
f.close()
Now my problem is that it seems to work fine for the first few requests, it gets the html and a few images but at some point it always stops at the "data received" point and quits, because it gets no data ie. the buffer is empty. The browser still shows it's loading elements of the page, but when it stops and I look at the text log file, I see that the buffer was empty, meaning that the browser didn't submit anything to the proxy?
I am guessing that the issue lies somewhere in how a browser submits requests and my script not reacting properly to this behavior.
I know I could use the Twist framework, however I want to learn to write this kinda stuff myself. I've been reading about SocketServer and I might use that, but I have no clue if it'll solve the issue because frankly, I don't really understand what's causing the issue here. Is my script too slow for the browser? Do servers send more than one answer and my receiving socket should listen for more packets? Is my buffer size (4096) too small?
I'd really appreciate a nudge in the right direction.
Thanks!
Well, I managed to answer my question. What I suspected previously was partly true - the browser was waiting for something and that something was replies.
I fired up wire shark, did some experiments and I noticed that my proxy makes a lot of ugly TCP RST appear in wireshark. I also noticed that in a normal connection, a lot of the server replies are split up into a few different packets.
Basically, my program wasn't getting all the answers back from the server because the out.recv was getting just one part of the reply. The obvious answer was to make a loop and listen for all the replies. I found the perfect solution at http://www.binarytides.com/receive-full-data-with-the-recv-socket-function-in-python/ .
I quickly recoded my program a bit and it works like a charm. Now I can move on forward with my whole project.
I hope this might help anyone else in the future with a similar issue.