In Python, I would like to use socket.connect() on a socket that I have set to non-blocking. When I try to do this, the method always throws a BlockingIOError. When I ignore the error (as below) the program executes as expected. When I set the socket to non-blocking after it is connected, there are no errors. When I use select.select() to ensure the socket is readable or writable, I still get the error.
testserver.py
import socket
import select
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setblocking(0)
host = socket.gethostname()
port = 1234
sock.bind((host, port))
sock.listen(5)
while True:
select.select([sock], [], [])
con, addr = sock.accept()
message = con.recv(1024).decode('UTF-8')
print(message)
testclient.py
import socket
import select
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setblocking(0)
host = socket.gethostname()
port = 1234
try:
sock.connect((host, port))
except BlockingIOError as e:
print("BlockingIOError")
msg = "--> From the client\n"
select.select([], [sock], [])
if sock.send(bytes(msg, 'UTF-8')) == len(msg):
print("sent ", repr(msg), " successfully.")
sock.close()
terminal 1
$ python testserver.py
--> From the client
terminal 2
$ python testclient.py
BlockingIOError
sent '--> From the client\n' successfully.
This code works correctly except for the BlockingIOError on the first connect(). The documentation for the error reads like this: Raised when an operation would block on an object (e.g. socket) set for non-blocking operation.
How do I properly connect() with a socket set to non-blocking? Can I make connect() non-blocking? Or is it appropriate to just ignore the error?
When using socket.connect with a non-blocking socket, it is somewhat expected to get a BlockingIOError at first. See TCP Connect error 115 Operation in Progress What is the Cause? for an explanation of the cause. Basically, the socket isn't ready yet and raises BlockingIOError: [Errno 115] Operation now in progress, also known as EINPROGRESS.
The solution is to either catch and ignore the exception or to use socket.connect_ex instead of socket.connect because that method doesn't raise an exception. Especially note the last sentence from its description in the Python docs:
socket.connect_ex(address)
Like connect(address), but return an error indicator instead of raising an exception for errors returned by the C-level connect() call (other problems, such as “host not found,” can still raise exceptions). The error indicator is 0 if the operation succeeded, otherwise the value of the errno variable. This is useful to support, for example, asynchronous connects.
Source: https://docs.python.org/3/library/socket.html#socket.socket.connect_ex
If you want to keep using socket.connect, you can catch and ignore the responsible EINPROGRESS error:
>>> import socket
>>>
>>> # bad
>>> s = socket.socket()
>>> s.setblocking(False)
>>> s.connect(("127.0.0.1", 8080))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
BlockingIOError: [Errno 115] Operation now in progress
>>>
>>> # good
>>> s = socket.socket()
>>> s.setblocking(False)
>>> try:
... s.connect(("127.0.0.1", 8080))
... except OSError as exc:
... if exc.errno != 115: # EINPROGRESS
... raise
...
>>>
The trick here is that when the select completes the first time, then you need to call sock.connect again. The socket is not connected until you have received a successful return status from connect.
Just add these two lines after the first call to select completes:
print("first select completed")
sock.connect((host, port))
EDIT:
Followup. I was wrong to have stated that an additional call to sock.connect is required. It is however a good way to discover whether the original non-blocking call to connect succeeded if you wish to handle the connection failure in its own code path.
The traditional way of achieving this in C code is explained here: Async connect and disconnect with epoll (Linux)
This involves calling getsockopt. You can do this in python too but the result you get back from sock.getsockopt is a bytes object. And if it represents a failure, you then need to convert it into an integer errno value and map that to a string (or exception or whatever you require to communicate the issue to the outside world). Calling sock.connect again maps the errno value to an appropriate exception already.
Solution 2:
You can also simply defer calling sock.setblocking(0) until after the connect has completed.
Related
Good evening,
I am trying to make a basic implementation of the clients-to-server model using sockets.
Basically, I want the client to retrieve a small piece of information (IE: MESSAGE key) to the server. To accomplish this, I've used threading.
When the client sends the request to receive the info, there is no error on the client side.
However, the server sends an error
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Users\Stock\AppData\Local\Continuum\anaconda3\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:\Users\Stock\AppData\Local\Continuum\anaconda3\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "server.py", line 40, in listeningForRetriever
dataBytes = conn.recv(1024)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
After doing some research, this thread seemed pretty relevant, https://stackoverflow.com/a/49289545/6902431, but I had already implemented the suggested fixes.
server.py
import socket
import threading
from Utility import *
HOST = '127.0.0.1' # Standard loopback interface address (localhost)
PORT = 65432 # Port to listen on (non-privileged ports are > 1023)
PORT_A = 22221
PORT_B = 22222
PORT_C = 22223
bank = {'me':'you'}
def listeningForRetriever():
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind((HOST, PORT_B))
s.listen()
conn, addr = s.accept()
with conn:
while True:
dataBytes = conn.recv(1024)
dataString = bytesToString(dataBytes)
try:
retrievedValueBytes = bytesToString(bank[cleanGET(dataString)])
print(f"Value: {retrievedValueBytes} has been retrieved with Key {cleanGET(dataString)}")
conn.sendall(retrievedValueBytes)
except:
print("SERVER: Errored out")
conn.sendall(stringToBytes("ERROR! Value retrieval for command: " + dataString + " doesn't exist!"))
retrieverThread = threading.Thread(target=listeningForRetriever)
retrieverThread.start()
retriever.py
#Retrieves values from server using keys
#retriever.py --key=username
import sys
import socket
from Utility import *
# total arguments
n = len(sys.argv)
print("Total arguments passed:", n)
# Arguments passed
print("\nName of Python script:", sys.argv[0])
print("\nArguments passed:", end = " ")
key = ''
value = ''
for i in range(1, n):
print(sys.argv[i], end = " ")
if '--key=' in sys.argv[i]:
data = sys.argv[i].replace('--key=','')
key = data
print()
if key == '':
print("ERROR! Invalid syntax.")
print("Try something like: Bob.py --key=username")
else:
print(f"Key: {key} {type(key)}")
#TODO: Send key to storage of server
HOST = '127.0.0.1' # The server's hostname or IP address
PORT = 22222 # The port used by the server
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(stringToBytes("GET key"+"="+key))
print("DEBUG 1")
value = s.recv(1024)
print("DEBUG 2")
print(f"Retrieved Value: {value}")
print(f"Value Retrieved: {value}")
s.close()
What am I overlooking/misunderstanding?
I could (more or less...) reproduce and fix. But I may be wrong, because your code uses an Utility module that you failed to show so I could only guess from the names what bytesToString, stringToBytes and cleanGET were supposed to do.
First there is a misunderstanding about how TCP actually behaves. It is a streaming protocol, that guarantees that all sent bytes will be received an in same order. But there is no guarantee on the packets themselves: they may be splitted and/or reassembled by the network. So you must use a higher level way to delimit a message. It does not exhibit any problem here because short packets are often left unmodified, but the protocol does not guarantee that so you could experience weird errors later. Here as you send a single message, you could shut down the socket to signal the end of the message: the receiver will see a 0 size packet at that point.
Next (and the actual cause of your problem), you loop after accepting a connection. So you get a message, successfully process it, read again on a socket that will be closed by the peer and get a 0 size byte string. It probably causes a key error when looking in the bank dict, but you silently swallow that error message (which is BAD), so you try to write on a socket closed by the peer, which causes the connection to be aborted. And finally on next recv you get the error that you see.
What you should have learned from that:
never ever use a silent try: ... except: ... that swallows any exception without letting you know what has happened. Always limit the filtered exception to the smaller expected set, or (at least and at dev time) display the caught exception to make sure you have not inadvertently caught an unwanted one.
use a higher level protocol to delimit messages on TCP
Now for the fixes:
retriever.py
...
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(stringToBytes("GET key"+"="+key))
s.shutdown(socket.SHUT_WR) # signals the end of the message
print("DEBUG 1")
...
server.py
...
s.listen()
conn, addr = s.accept()
with conn:
# read until the end of a message signaled by a 0 size packet
dataBytes = b''
while True:
b = conn.recv(1024)
if len(b) == 0:
break
dataBytes += b
dataString = bytesToString(dataBytes)
try:
retrievedValueBytes = stringToBytes(bank[cleanGET(dataString)])
print(f"Value: {retrievedValueBytes} has been retrieved with Key {cleanGET(dataString)}")
conn.sendall(retrievedValueBytes)
except Exception as e:
print(e)
print("SERVER: Errored out")
conn.sendall(stringToBytes("ERROR! Value retrieval for command: " + dataString + " doesn't exist!"))
...
Here the thread stops after one single message. If you want the server to be able to accept many clients, you can loop over accept:
s.listen()
while True:
conn, addr = s.accept()
...
Your server is coded to listen in a loop (while True / conn.recv), but your
client only listens once, then exits, which closes the connection.
So the message on the server side is actually not vague at all, it's perfectly to
the point: it's telling you that the connection has been closed. It has indeed,
by the client.
(To be very specific, your retriever code uses a socket as a context manager,
meaning that when you exit the 'with' bock, the socket gets automatically
closed - you can remove the s.close() call, BTW)
Your code works fine, you should be getting the data back (I did, but your post
doesn't say anything about that). The only thing missing is the exception
handling on the server side to gracefully handle the client disconnection.
I know there is a similar question already, but none of the solutions solve my problem. Over ssh I am starting a script on a remote client with
nohup python script.py &
This script contains the following:
TCP_PORT = 5005
host = ""
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.settimeout(40)
s.bind((host, TCP_PORT))
s.listen(0)
c, addr = s.accept()
...some code...
try:
while True:
c.send(str(1).ljust(16).encode())
except Exception as e:
print("exiting main")
print(e)
c.close()
s.close()
When I run the code two times in e row, the second time I always get the above mentioned error. The log of the python output:
exiting main
[Errno 32] Broken pipe
Traceback (most recent call last):
File "LogImages.py", line 204, in <module>
main(interv)
File "LogImages.py", line 114, in main
s.bind((host, TCP_PORT))
OSError: [Errno 98] Address already in use
So obviously the process calls c.close() and s.close(). So how can the address still be in use?
Closing a socket just releases the handle to any underlying connection. It can still take the implementation some amount of time to complete the orderly shutdown of the connection and, during that time, the address is still in use.
For example, if you have an active connection and the other side isn't reading from it, the implementation will give it time to read the data that was sent. During that time, the address is still in use.
I need some help. I have a simple server:
host="localhost"
port=4447
from socket import *
import thread
def func():
while 1:
data = conn.recv(1024)
if not data:
continue
else:
print("%s said: %s")%(player, data)
conn.close()
s=socket(AF_INET, SOCK_STREAM)
s.bind((host,port))
s.listen(2)
print("Waiting for clients on localhost, port %s")%port
while 1:
conn, addr = s.accept()
player = addr[1]
print(conn)
thread.start_new_thread(func,())
And a simple client:
import socket
TCP_IP = '127.0.0.1'
TCP_PORT = 4447
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while 1:
data = raw_input("Input: ")
s.send(data)
So when I connect to the server I can type anything and it is printed in the server's terminal. When I open another terminal and start second client I can also type anything and it is sent to the server, but when I go back to the first client's terminal and type several messages, it returns:
Traceback (most recent call last):
File "Client.py", line 18, in <module>
s.send(data)
socket.error: [Errno 32] Broken pipe
So I fixed that with adding conn as a parameter in func(), but I don't understand why this error happened? Could anyone please explain it to me?
Thanks!
Your func, apart from needing a better name, uses global state in your program to communicate with a client. No matter how many threads you start to handle client connections, there's still only one global conn variable. Each time a new client connects, your main thread loop rebinds conn to the new connection. The old socket is thrown away and automatically closed by the Python runtime.
You can fix this by removing the use of global variables to track per-connection state. A better route to explore, though, is Twisted.
I am writing a connector using UDP in Python 3.3
When I am sending data to the UDP port, everything works fine. The problem is that when I am not sending any data, I get an error generated by the receiving port once per minute that says "timed out". While debugging, I used the socket.gettimeout() function and it returned 'None'.
Why am I getting this timeout error? Any help would be greatly appreciated!
import socket
from EventArgs import EventArgs
import logging
class UDPServer(object):
"""description of class"""
def __init__(self, onMessageReceivedEvent = '\x00'):
self.__onMessageReceivedEvent = onMessageReceivedEvent
self.__s = '\x00'
self.__r = '\x00'
def openReceivePort(self,port):
try:
self.__r = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.__r.bind(("",port))
print ("opening port: ", port)
except socket.error as e:
logging.getLogger("ConnectorLogger").critical(e)
def openBroadcastPort(self):
try:
self.__s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.__s.bind(("",2101))
print ("opening port: ", 2101)
except socket.error as e:
logging.getLogger("ConnectorLogger").critical(e)
def closePorts():
if self.__r != '\x00':
self.__r.close()
if self.__s != '\x00':
self.__s.close()
def getUDPData(self):
try:
data, addr = self.__r.recvfrom(1024)
if self.__onMessageReceivedEvent != '\x00':
eventArgs = EventArgs()
eventArgs.Addr = addr
eventArgs.Data = data
self.__onMessageReceivedEvent.fire(self, eventArgs)
except socket.error as e:
logging.getLogger("ConnectorLogger").critical(e)
def send(self,ipAddress,port,message):
try:
self.__s.sendto(message.encode(),(ipAddress,23456))
except socket.error as e:
logging.getLogger("ConnectorLogger").critical(e)
I figured out the answer to my own problem. I was using the default configuration for socket.setblocking which is 0 (non-blocking). The documentation says that using this configuration is the equivalent of using a settimeout value of 0. If I use a blocking socket, it is the equivalent of using a settimeout value of 'None'. Once I changed to a blocking socket I no longer saw this error.
socket.setblocking(flag)-Set blocking or non-blocking mode of the socket: if flag is 0, the socket is set to non- blocking, else to blocking mode. Initially all sockets are in blocking mode. In non-blocking mode, if a recv() call doesn’t find any data, or if a send() call can’t immediately dispose of the data, a error exception is raised; in blocking mode, the calls block until they can proceed. s.setblocking(0) is equivalent to s.settimeout(0.0); s.setblocking(1) is equivalent to s.settimeout(None)*
I have the following code:
class Server:
def __init__(self, port, isWithThread):
self.isWithThread = isWithThread
self.port = port
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setblocking(0)
log.info("Socket created...")
def __enter__(self):
self.sock.bind(('127.0.0.1', self.port))
self.sock.listen(5)
log.info("Listening on %s:%s", '127.0.0.1', self.port)
return self
def __exit__(self, type, value, traceback):
self.sock.setblocking(1)
self.sock.close()
log.info("Socket closed.")
log.info("Bye")
def run(self):
#client, addr = self.sock.accept()
log.info('Selecting...')
readers, writers, errors = select.select([self.sock], [], [], 10)
log.debug(readers)
if readers:
client, addr = readers[0].accept()
log.info('Client: %s', client.recv(2048).decode())
client.sendall("Hippee!".encode())
client.close()
log.info("Disconnected from %s:%s", *addr)
What's interesting about this is that when I have the select.select and setblocking(0), it ends out keeping the address in use. If I remove the setblocking code and change the run function to:
def run(self):
client, addr = self.sock.accept()
log.info('Client: %s', client.recv(2048).decode())
client.sendall("Hippee!".encode())
client.close()
log.info("Disconnected from %s:%s", *addr)
Then I can immediately re-run the server. With the select() call, I get the following error:
python3.3 test.py server
Socket created...
Traceback (most recent call last):
File "test.py", line 89, in <module>
with Server(12345, False) as s:
File "test.py", line 57, in __enter__
self.sock.bind(('127.0.0.1', self.port))
OSError: [Errno 98] Address already in use
So why does it appear that the select is keeping my socket open, and how do I ensure it closes?
Magic. Magic is the only reason to see any difference with or without select.select(). According to this page, the reason that a socket will stay in use even after calling .close() is that the TIME_WAIT has not yet expired.
The solution is to use .setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1).
I tried this but it didn't work, and so I asked this question. Later, I realized that hey, I knew that things like Flask or SimpleHTTPServer will allow you to restart the server immediately. So I used the source, and examined the library code contained in socketserver.py. It was here that I discovered the use of .setsocketopt() but the call came before the call to .bind().
To explain setsocketopt(), let's see what does the docs say?
socket.setsockopt(level, optname, value)
Set the value of the given socket option (see the Unix manual page setsockopt(2)). The needed symbolic constants are defined in the socket module (SO_* etc.). The value can be an integer or a string representing a buffer. In the latter case it is up to the caller to ensure that the string contains the proper bits (see the optional built-in module struct for a way to encode C structures as strings)
level refers to the level of the TCP/IP stack you want to talk about. In this case we don't want the IP layer but the socket itself. The socket option is SO_REUSEADDR, and we're setting the flag (value=1). So somewhere down in the kernel or drivers, we're effectively saying, "SHHhhhhhh... It's OK. I don't care that you're in TIME_WAIT right now. I want to .bind() to you anyway."
So I changed up my code to have:
sock.setsocketopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('127.0.0.1', self.port))
And it works perfectly.
\o/