Develop Reference

How to bypass the GIL and handle the KeyboardInterrupt?

I'm wanting to exit an UDP-server on KeyboardInterrupt, but this is not working immediately. when doing some research I stumbled on this question, where someone suggests that it is a issue with the GIL. Python processes calling into C APIs can block the Python interpreter from processing the interrupt. is there any way the unblock this in python?
the code I have is the following:
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt: #works after new data is send to server
print('Interrupted', file=sys.stderr)
try:
sys.exit(0)
except SystemExit:
os._exit(0)
main:
def main():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_address = ('localhost', 2000)
s.bind(server_address)
running = True
print('Ready for connection')
while running:
try:
msg = ""
data, address = s.recvfrom(2000) #look for timeout and interrupt
msg = data.decode('utf-8')
if not data:
break
print(f'received {msg}')
if msg[:5] == "PING-":
print('sending PONG back to the client')
message = 'PONG-' + msg[5:len(msg)]
s.sendto(message.encode('utf-8'), address)
else:
print('bad format')
except socket.error as socketerror:
print(f'Error: {socketerror}', file=sys.stderr)
s.close()
When i'm not running the client and press ctrl-c nothing happens, the program just does nothing.
When I start up the client and send a UDP message, the interrupt is perfectly handled when the server gets a UDP message from the client. The UDP message then is dropped and the client handles it as a timeout. After the timeout the client has a lost connection.
The goal is to get the server exit even without the client running and without using ctrl-break.

How to handle KeyboardException raised in threading module?

I'm creating a server using the socket and threading module. The server allows for multiple connections to be made. I've been doing some error handling (as shown in the code below). However, there is one error that I cannot figure out how to handle.
Code (server.py):
import socket
import threading
# import pyaudio
# import json
from os import system
system('clear')
host = socket.gethostbyname(socket.gethostname())
port = 5000
buffer_size = 2048
connected_clients = {}
# Thread for listening and accepting new connections
def client_listener():
while True:
try:
# Accepts new client
connection, address = server_socket.accept()
# Creates a new thread for each connected client
handle_client_thread = threading.Thread(target=handle_client, args=(connection, address))
handle_client_thread.start()
except ConnectionAbortedError:
pass
except OSError:
pass
except KeyboardInterrupt:
pass
# Handles each connected client in a separate thread
def handle_client(client, client_address):
try:
# Receives room number
room_number = client.recv(buffer_size).decode('utf-8')
print(f'\nConnection from {client_address[0]}:{client_address[1]} ({room_number})')
# Adds new client to 'connected_clients'
connected_clients[room_number] = client
print(connected_clients)
while True:
data = client.recv(buffer_size).decode('utf-8')
# If no data is received, close the socket
if not data:
print(f'{client_address[0]}:{client_address[1]} ({room_number}) Disconnected')
connected_clients.pop(room_number)
break
# Print received data
else:
print(f'{client_address[0]} ({room_number}): {data}')
except KeyboardInterrupt:
client.close()
connected_clients.pop(room_number)
with socket.socket() as server_socket:
try:
# Prevents [Errno 48: Address already in use]
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# Binds socket to host and port
server_socket.bind((host, port))
server_socket.listen()
print(f'Server hosted on {host}:{port}')
# Starts new thread for listening for new clients
client_listener_thread = threading.Thread(target=client_listener, name=client_listener)
client_listener_thread.start()
# Prevents main thread from closing
while True:
pass
except socket.error as error_message:
print(str(error_message))
except KeyboardInterrupt:
print(': Closing server...\n')
pass
The error message is raised when I close the server with a keyboard interrupt (^C). When I first press ^C, the server is closed. However, the second ^C seems to raise an exception within the threading module.
Error message:
Server hosted on 10.108.249.108:5000
^C: Closing server...
^CException ignored in: <module 'threading' from '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py'>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 1273, in _shutdown
t.join()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 1032, in join
self._wait_for_tstate_lock()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
How would I go about handling this KeyboardInterrupt in a seamless way such that the program is closed without errors on a single ^C? The desired result would be having the server close the connections with each connected client, close the threads, then finally shut down the server itself.

When you press Ctrl-C on your keyboard while your script is running on the console, you send a SIGINT to the script process. You can use the signal python module to handle such signals:
import signal
import sys
def my_sig_handler(sig, frame):
print('Received interrupt signal')
sys.exit(0)
signal.signal(signal.SIGINT, my_sig_handler) # Register your signal handler for SIGINT
Once you do this, any time your running script receives an interrupt signal, my_sig_handler will be called. There are several kinds of signals defined in Linux, and it is generally a good idea to add signal handlers in your code for some of the common ones, so that it can clean up and exit gracefully.
More information about signals here.

Try/Except not catching UnboundLocalError on KeyboardInterrupt

I have a simple file transfer server that uses socket, It has an infinite listening to clients loop in the Main() func, so i surrounded it with Try/Except with KeyboardInterrupt so i would be able to properly close all the sockets and connections when CTRL+C-ing out
def Main():
try:
#various variable initations
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.bind((host,port))
print 'Socket bound to host - {0} and port {1}'.format(host,port)
sock.listen(5)
print 'Waiting for connections...'
while True:
conn, addr = sock.accept()
print 'Client IP:',str(addr)
#getting data from client and making the server do the appropriate functions
conn.close()
sock.close()
except(KeyboardInterrupt): # redundancy to make sure that a keyboard interrupt to close the program also closes the sockets and connections
conn.close()
sock.close()
print 'Manual Close'
sys.exit()
Now when a client connects and does whatever and i close it via keyboard interrupt it works fine, printing me the 'Manual Close'
But when i close via keyboardinterrupt before a client connects it gives me this error: UnboundLocalError: local variable 'conn' referenced before assignment
I understand the conn doesn't get assigned if a client doesn't connect but i thought that any errors under except get ignored

You can just put the functions within the except block inside another try/except and tell it to ignore the exception with pass
except(KeyboardInterrupt): # redundancy to make sure that a keyboard interrupt to close the program also closes the sockets and connections
try:
conn.close()
sock.close()
except:
pass
print 'Manual Close'
sys.exit()

Python long-lived socket connection weirdness

I've implemented some code that allows a client to connect to a socket server, introduces itself and the server then goes into an infinite loop which sends "commands" (strings) to the client from a Redis list. The server uses the Redis 'blpop' method to block until a string arrives which is then sent off to the client and the response awaited.
However, in testing (with a python client socket script on another local workstation) I find that if I break the client connection (Ctrl+c) to simulate an interruption in the connectivity, the server happily writes the next received string to the client, reports an empty response but ONLY throws the broken pipe exception when a second string is written :/ Thus, two writes are "lost" before anything is caught. Here's my code:
# Create global Redis resource
rds_cnx = redis.StrictRedis(host='localhost', port=6379, db=6)
def initialise_server():
""" Setup server socket """
try:
srv_skt = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv_skt.bind((IP, PORT))
srv_skt.listen(1)
print("Listening on:[{}]".format(IP, PORT))
return srv_skt
except socket.error as skt_err: # e.g. port in use
print("Could not initialise tcp server:[{}]".format(skt_err))
sys.exit(1)
except Exception as exp:
print("Unable to setup server socket:[{}]".format(exp))
sys.exit(1)
def main():
server_socket = initialise_server()
while True:
client_socket, remote_address = server_socket.accept()
try:
# Block and wait for connection and data
initial_data = client_socket.recv(1024).decode()
print("Connection from [{}] - Data:[{}]".format(remote_address, initial_data))
while True:
wait_for_queue_command(client_socket)
except (BrokenPipeError, socket.error, Exception) as sck_exp:
print("Exception in client loop:[{}]".format(sck_exp))
continue
except KeyboardInterrupt:
# Close client socket
client_socket.shutdown(2)
client_socket.close()
print('Caught Ctrl+c ... Shutting down.')
break
# Tear down context
server_socket.shutdown(2) # Param ref: 0 = done receiving, 1 = done sending, 2 = both
server_socket.close()
def wait_for_queue_command(client_skt):
""" Blocking while waiting for command for Redis list
:param client_skt: socket
:return: None
"""
print('Waiting for command...')
queue_cmd = rds_cnx.blpop('queuetest', 0)
print("Received something from the queue:")
pprint(queue_cmd)
try:
#client_skt.settimeout(15)
client_skt.send(queue_cmd[1])
# Block for response
response_data = client_skt.recv(1024).decode()
print("Response:[{}]".format(response_data))
except BrokenPipeError as brkn_p:
print('Outbound write detected "Broken Pipe":[{}]'.format(brkn_p))
''' Here one would decide to either re-schedule the command or
ignore the error and move on to the next command. A "pause"
(sleep) could also useful?
'''
raise
except socket.timeout as sck_tmo:
print('Socket timed out:[{}]'.format(sck_tmo))
except socket.error as sck_err:
print('Socket timed out:[{}]'.format(sck_err))
raise
print('Command handling complete.')
Is there any better way to handle such a situation? I've had a cursory look at Twisted but it seems very difficult to achieve the specific blocking behavior and other code that might be implemented to handle specific responses from the client.

Can't recover from a stack overflow

This is a little script I made while learning Python, but for some reason it tells me the it can't recover from the stack over flow. This happens when the another server disconnect.
The script:
#/user/bin/python
import os
import socket
import subprocess
import errno
import threading
s = socket.socket()
host = '192.168.1.6'
port = 9999
def connect():
try:
s.connect((host,port))
except Exception as msg:
print("ERROR HAPPEND 2 ")
connect()
else:
Work()
def Work():
while True:
data = s.recv(1024)
print("Data : " + data.decode('utf-8'))
if data[:2].decode("utf-8") == 'cd':
os.chdir(data[3:].decode('utf-8'))
if len(data) >0:
cmd = subprocess.Popen(data[:].decode('utf-8'), shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
output_bytes = cmd.stdout.read() + cmd.stderr.read()
output_str = str(output_bytes , "utf-8")
s.send(str.encode(output_str + str(os.getcwd()) + '> '))
else:
s.shutdown(socket.SHUT_RDWR)
s.close()
thread1 = threading.Thread(target = connect)
thread1.start()
break
connect()

This code is wrong:
def connect():
try:
s.connect((host,port))
except Exception as msg:
print("ERROR HAPPEND 2 ")
connect()
else:
Work()
If connection fails for some reason (refused, or even syntax error in the try/except block since you're not filtering the exception type), then you're printing the error message and try again by calling recursively your function.
Since the socket error is very likely to happen again since you're retrying immediately the same operation without changing anything (starting the other program for instance!), you get a stack overflow very quickly.
Fix, first step: let your connection crash with a proper error message
def connect():
s.connect((host,port))
Work()
Fix, second step: if you think that the connection can be established later, you can catch the exception, wait a while and retry, for example like this:
def connect():
while True:
try:
s.connect((host,port))
break # connection OK, proceeed to Work
except ConnectionRefusedError as e:
print("{}, retry in 10s ...".format(str(e)))
time.sleep(10)
Work()
In your case, just after the socket is closed, you create another thread that calls connect, and fails to do so, recursively, which explains the problem you're experiencing when disconnecting the other side.