I have a simple TCP server. There is an issue that occurs when I connect to the server using the Google Chrome browser: after a request, it seems to open another socket to send the next request there quickly next time. When such a socket is opened, I cannot restart my server gracefully. I can use the allow_reuse_address hack but it is not the thing I want to do because the port is still busy after the server shutdown, and if I wish another server to listen to this port, it is unable to do this.
I am sure there may be other cases when I don't want my port to be held by a nasty client and I would like to have the ability to force disconnect it. If I force to disconnect the server and then try to connect again, I get
OSError: [Errno 98] Address already in use
Here is an example. Let us imagine some "dirty client" connecting to my server and doing nothing (simply to consume the resources of my server). I want to shut down my server (let us say, for maintenance). I cannot do this until the client disconnects. I wish to disconnect it manually from the server-side.
from http.server import BaseHTTPRequestHandler
from socket import socket, AF_INET, SOCK_STREAM
from socketserver import TCPServer
from threading import Thread
from time import sleep, time
def dirty_client():
client_socket = socket(AF_INET, SOCK_STREAM)
client_socket.connect(("127.0.0.1", 8080))
sleep(10)
client_socket.close()
if __name__ == "__main__":
# Spawn TCP server
server = TCPServer(("127.0.0.1", 8080), BaseHTTPRequestHandler)
thread = Thread(target=server.serve_forever)
thread.start()
# Block it with a connection (should be on client side)
hang_thread = Thread(target=dirty_client)
hang_thread.start()
sleep(1)
# Let it hang
start = time()
print("Shutting down...")
server.shutdown()
server.server_close()
print(f"Success in {time() - start} seconds")
thread.join()
hang_thread.join()
The server hangs on the moment the request handler reads the socket. There is no timeout for reading and I cannot close the file during the read. The server itself can be shut down during the read, but the port will not be freed in this case.
It is possible to use BaseRequestHandler in this example — after it, the port is also in use. Though, BaseHTTPRequestHandler waits until the port is freed.
Related
I am writing a UDP server application that serves as a back end to Teltonika FMB630 car mounted devices.
I already took care of the protocol specifics and decoding, the problem I am facing relates to the UDP socket used.
My UDP server has to send an acknowledgement to the client device upon receiving a message (that is the protocol), however, if I send those ACKs, the server socket stops receiving data after a while.
The server's UDP socket object is passed to an concurrent.futures.ThreadPoolExecutor that fires a function (send_ack) that sends the ACK, however this is not the issue because I tried calling send_ack in the main thread, after receiving data and the same issue occurs.
I suspect the problem is the remote device somehow breaks the connection or the ISP or MNO doesn't route the reply packet (this is a GPRS device) and then the socket.send() method that is used to send the acknowledge, somehow freezes other socket operations, specifically recvfrom_into called in the main thread loop.
I wrote two scripts to illustrate the situation:
udp_test_echo.py :
#!/usr/env/bin python
import socket
import concurrent.futures
def send_ack(sock, addr, ack):
print("Sending ACK to {}".format(addr))
sock.connect(addr)
print("connected to {}".format(addr))
sock.send(ack)
print("ACK sent to {}".format(addr))
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(("127.0.0.1", 1337))
data = bytearray([0] * 10)
executor = concurrent.futures.ThreadPoolExecutor(max_workers=4)
while True:
print("listening")
nbytes, address = s.recvfrom_into(data)
print("Socket Data received {} bytes Address {}".format(nbytes, address))
print("Data received: ", data, " Echoing back to client")
executor.submit(send_ack, s, address, data[:nbytes])
udp_test_client.py:
#!/usr/env/bin python
import socket
import time
import random
def get_random_bytes():
return bytearray([random.randint(0,255) for b in range(10)])
ip = "127.0.0.1"
port = 1337
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect((ip, port))
while True:
stuff_to_send = get_random_bytes()
print("Sending stuff", stuff_to_send)
s.sendall(stuff_to_send)
print("reply: ", s.recvfrom(10))
time.sleep(0.1)
Running udp_test_echo.py in one terminal and udp_test_client.py in another, we see normal operation but if you Ctrl+C the test client and re run it, you will see that the server doesn't respond until it is restarted.
Is there a way to timeout a specific sending operation from a specific call to socket.send() method without affecting other calls ? (I want my socket.recvfrom_into call to block on the main thread)
If I settimeout on the entire socket object, I am going to have to deal with many exceptions while waiting for data in the main thread and I don't like to have to rely on exceptions for proper program operation.
The culprit was the socket.connect() call in send_ack, when being called on the server's socket object it causes the socket to no longer be bound and listen on the port specified in the start of the program.
Instead the send_ack function was changed to be:
def send_ack(sock, addr, ack):
print("Sending ACK to {}".format(addr))
sock.sendto(ack, addr)
print("ACK sent to {}".format(addr))
socket.sendto(data, address) uses the existing connection instead of starting a new one.
I've a python Socket server running and also socket clients.
Now, for example say there are 3 clients connected to same server. Please find below the code of the server.
#!/usr/bin/python # This is server.py file
import socket # Import socket module
import threading
serversocket = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 1234 # Reserve a port for your service.
serversocket.bind((host, port)) # Bind to the port
serversocket.listen(5)
print("Bound the port ",port,"on Machine : ",host,", and ready to accept connections.\n")
def clientThread(connection):
while True:
data=connection.recv(1024)
if not data:
break
connection.send("Thanks")
connection.close()
def sendMessage(connection, message):
connection.send(message)
while 1:
connection, address = serversocket.accept()
start_new_thread(clientthread, (connection,))
serversocket.close();
Now, I need to call sendMessage for a particular client, say out of clients A,B and C, send it to B. In this case, how do I identify the thread and call that function?
You can use Queues and multiple threads per connection to solve this problem.
Basic outline:
Each client connection spawns two threads - one to monitor client input and another which monitors a Queue. Items placed on the queue will be sent to the client. Each client connection will have its own output queue.
You'll also need a global dictionary to map a client name to their output queue.
To send a message to a particular client, find the client's output queue and add the message to it.
You'll also need a way to shutdown the output thread for a client. A common approach is to use a sentinel value (like None) on the queue to inform the output thread to exit its processing loop. When the client's input thread detects EOF it can place the sentinel value on the client's output queue and eventually the output thread will shut itself down.
On Windows 7:
Given this server code:
# in server.py
if __name__ == '__main__':
serversock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# uncommenting this won't help
#serversock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serversock.bind(('',8888))
serversock.listen(5)
# accept and receive dummy data from client
clientsock,address = serversock.accept()
data = clientsock.recv(1024)
# as long as calc.exe is running, I can't do this again
subprocess.Popen(r"c:\windows\system32\calc.exe")
# letting client close first still won't help
time.sleep(3)
# closing won't help either
clientsock.close()
serversock.close()
And the client code
# in client.py
if __name__ == '__main__':
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 8888))
s.sendall('Hello, world')
# close early to help prevent TIME_WAIT on server, but doesn't help
s.close()
Running server first then client will launch the calculator app.
While the calculator app is still running, I can't run server again. It will complain about
python server.py (ok)
python client.py (ok)
python server.py (boom!)
socket.error: [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted
If I close the Calculator app, running server is ok again...
This does not happen on Mac.
Enabling SO_REUSEADDR will only make the error go away, but the server is unreachable from the client.
In the example above, I specifically let the client close first so that the server socket don't go into TIME_WAIT.
So the questions:
Am I running into the TIME_WAIT problem on the server?
Are any sockets/filedescriptors left unclosed in the server?
Why SO_REUSEADDR won't help in this case? could the client be coming from the same port?
Could the child process be hanging on to some descriptors?
What can I do about this?
The SOLUTION:
The problem IS with the parent process of Calculator holding on to some file descriptor.
So adding close_fds=True to the Popen will ensure everything is released properly.
subprocess.Popen(r"c:\windows\system32\calc.exe", close_fds=True)
Closing socket after subprocess.Popen leaves socket in TIME_WAIT as long as child process is still running
No it doesn't. It leaves it in TIME_WAIT for a fixed amount of time, 2 or 4 minutes. After the close it has nothing to do with the child process at all.
I'm playing around with sockets in python, just for the purpose of learning about them. However I am really annoyed with the following problem:
import socket
soc = socket.socket(socket.AF_INET)
soc.bind(('localhost',8000))
soc.listen(0)
client = soc.accept()
While the socket is waiting for a connection, pressing ctrl-c does not quit the application.
How can I quit the application?
A similar issue was addressed in these two questions, but there the accept method was called from a separate thread and the problem was how to make ctrl-c kill that thread. Here the accept method is called from the main thread.
Edit: I am running python 3.3.0 on Win7 64 bit.
You should use CTRL + Break. That should kill it.
I couldn't find a way to kill the application using ctrl-c or any other way except for killing it through the task manager, so I wrote a workaround:
import socket,os
from threading import Thread
class socketListener(Thread):
def run(self):
soc = socket.socket(socket.AF_INET)
soc.bind(('localhost',8000))
soc.listen(0)
client = soc.accept()
pid = os.getpid()
sl = socketListener()
sl.start()
input('Socket is listening, press any key to abort...')
os.kill(pid,9)
This runs the script in a separate thread, while waiting for a keystroke in the main thread. Once the user presses a key, the entire application is killed.
"serversocket" module provides the standard solution. I tested Control-C on Windows, it worked.
This is the link, serversocket example
The Control-C handling is even mentioned in the comment of the code
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
Here is the complete code from the above link:
import socketserver
class MyTCPHandler(socketserver.BaseRequestHandler):
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server:
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()
If we wanted to re-invent the wheel, we would do a select() or poll() on the listener socket, with a timeout 0.5 seconds.
To save time of the other people searching for this topic, if you laptop keyboard does not have a break button, please try
Ctrl + Fn + F6
or
Ctrl + F6
After being myself confronted to the same problem I found a little workaround, it might not be the cleanest way but at least it works for me :
import socket
from select import select
#create an INET, STREAMing socket
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#bind the socket to localhost
serversocket.bind(('localhost', 8000))
While 1:
serversocket.listen(5)
ready, _, _ = select([serversocket], [], [], 1) #Timeout set to 1 seconds
if ready:
(clientsocket, address) = serversocket.accept()
#Do something with client
else:
#Do nothing, just loop again
By using select you will wait a change on the socket fd until the end of the timeout. As I said this might not be the cleanest way but Ctrl-c will be catch at the end of the timeout.
Portability alert: On Unix, select works both with the sockets and files. On Windows, select works with sockets only.
I'm teaching myself Python networking, and I recalled that back when I was teaching myself threading, I came across this page, so I copied the scripts, updated them for Python 3.1.1 and ran them. They worked perfectly.
Then I made a few modifications. My goal is to do something simple:
The client pickles an integer and sends it to the server.
The server receives the pickled integer, unpickles it, doubles it, then pickles it and sends it back to the client.
The client receives the pickled (and doubled) integer, unpickles it, and outputs it.
Here's the server:
import pickle
import socket
import threading
class ClientThread(threading.Thread):
def __init__(self, channel, details):
self.channel = channel
self.details = details
threading.Thread.__init__ ( self )
def run(self):
print('Received connection:', self.details[0])
request = self.channel.recv(1024)
response = pickle.dumps(pickle.loads(request) * 2)
self.channel.send(response)
self.channel.close()
print('Closed connection:', self.details [ 0 ])
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('', 2727))
server.listen(5)
while True:
channel, details = server.accept()
ClientThread(channel, details).start()
And here is the client:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
for x in range(10):
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
The server runs fine, and when I run the client it successfully connects and starts sending integers and receiving them back doubled as expected. However, very quickly it exceptions out:
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python30\lib\threading.py", line 507, in _bootstrap_inner
self.run()
File "C:\Users\Imagist\Desktop\server\client.py", line 13, in run
print('Received:',repr(pickle.loads(client.recv(1024))))
socket.error: [Errno 10053] An established connection was aborted by the softwar
e in your host machine
The server continues to run and receives connections just fine; only the client crashes. What's causing this?
EDIT: I got the client working with the following code:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
for x in range(10):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
However, I still don't understand what's going on. Isn't this just opening and closing the socket a bunch of times? Shouldn't there be time limitations to that (you shouldn't be able to open a socket so soon after closing it)?
Your client is now correct - you want to open the socket send the data, receive the reply and then close the socket.
The error original error was caused by the server closing the socket after it sent the first response which caused the client to receive a connection closed message when it tried to send the second message on the same connection.
However, I still don't understand
what's going on. Isn't this just
opening and closing the socket a bunch
of times?
Yes. This is acceptable, if not the highest performance way of doing things.
Shouldn't there be time
limitations to that (you shouldn't be
able to open a socket so soon after
closing it)?
You can open a client socket as quickly as you like as every time you open a socket you will get a new local port number, meaning that the connections won't interfere. In the server code above, it will start a new thread for each incoming connection.
There are 4 parts to every IP connection (source_address, source_port, destination_address, destination_port) and this quad (as it is known) must change for ever connection. Everything except source_port is fixed for a client socket so that is what the OS changes for you.
Opening server sockets is more troublesome - if you want to open a new server socket quickly, your
server.bind(('', 2727))
Above then you need to read up on SO_REUSEADDR.