I am trying to connect to two different websockets and print out their messages concurrently but am having a hard time figuring out the best approach. I have tried stuff with asyncio and threading but cant get the messages to print out together
Here is what I have so far:
async def socket1():
async with websockets.connect("wss:socket1...") as websocket:
await websocket.send(json.dumps(
{"msg": "Hello1"}
))
while True:
msg = await websocket.recv()
print("From socket 1")
async def socket2():
async with websockets.connect("wss:socket2...") as websocket:
await websocket.send(json.dumps(
{"msg": "Hello2"}
))
while True:
msg = await websocket.recv()
print("From socket 2")
I want the sockets to print something out like this:
From socket 1
From socket 1
From socket 2
From socket 1
From socket 2
From socket 1
etc.
Related
Im trying to send and receive at the same time from websockets. But the while-loop is blocking my received websockets. Im using python 3.6 so I cannot use asyncio.run but have to stick to asyncio.get_event_loop()
websocket.recv()
receives the message only once. After that it is blocked by the while loop
Here is my code.
import asyncio
import websockets
import json
async def hello(websocket, path):
name = await websocket.recv()
print(f"< {name}")
count = 2
while True:
count = count + 1
counting = json.dumps(count)
await websocket.send(counting)
await asyncio.sleep(0.1)
start_server = websockets.serve(hello, "192.168.1.104", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
How can I separate the processes and run them in parallell with asyncio?
I'm trying to understand how to use asyncio streams for multiple connections that will keep sending messages until a predefined condition or a socket timeout. Looking at Python docs, they provide the following example for a TCP server based on asyncio streams:
import asyncio
async def handle_echo(reader, writer):
data = await reader.read(100)
message = data.decode()
addr = writer.get_extra_info('peername')
print(f"Received {message!r} from {addr!r}")
print(f"Send: {message!r}")
writer.write(data)
await writer.drain()
print("Close the connection")
writer.close()
async def main():
server = await asyncio.start_server(
handle_echo, '127.0.0.1', 8888)
addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
print(f'Serving on {addrs}')
async with server:
await server.serve_forever()
asyncio.run(main())
What I'm trying to do is more complex and it looks more like so (a lot of it is pseudocode, written in capital letters or with implementation omitted):
import asyncio
async def io_control(queue):
while true:
...
# do I/O control in this function ...
async def data_processing(queue):
while true:
...
# perform data handling
async def handle_data(reader, writer):
data = await reader.read()
message = data.decode()
addr = writer.get_extra_info('peername')
print(f"Received {message!r} from {addr!r}")
#do stuff with a queue - pass messages to other two async functions as needed
#keep open until something happens
if(ERROR or SOCKET_TIMEOUT):
writer.close()
async def server(queue):
server = await asyncio.start_server(
handle_data, '127.0.0.1', 8888)
addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
print(f'Serving on {addrs}')
async with server:
await server.serve_forever()
async def main():
queue_io = asyncio.Queue()
queue_data = asyncio.Queue()
asyncio.run(server(queue_data))
asyncio.run(data_handling(queue_data))
asyncio.run(io_control(queue_io))
asyncio.run(main())
Does this look feasible? I'm not used to working with co-routines (I'm coming from more of a multi-threading paradigm), so I'm not sure if what I'm doing is right or if I have to explicitly include yields or do any extra stuff.
If I understand correctly, you just need the TCP server to be able to handle multiple concurrent connections. The start_server function should already give you everything you need.
The first parameter client_connected_cb is a coroutine function called whenever a client establishes a connection. If you introduce a loop into that function (in your example code handle_data), you can keep the connection open until some criterion is met. What conditions exactly should lead to closing the connection is up to you, and the implementation details will obviously depend on that. The simplest approach I can imagine is something like this:
import asyncio
import logging
log = logging.getLogger(__name__)
async def handle_data(reader, writer):
while True:
data = (await reader.readline()).decode().strip()
if not data:
log.debug("client disconnected")
break
response = await your_data_processing_function(data)
writer.write(response.encode())
await writer.drain()
...
async def main():
server = await asyncio.start_server(handle_data, '127.0.0.1', 8888)
async with server:
await server.serve_forever()
if __name__ == '__main__':
asyncio.run(main())
There is theoretically no limit for the number of concurrent connections.
If your client_connected_cb is a coroutine function, each new connection will schedule a new task for the event loop. That is where the concurrency comes from. The magic then happens at the point of awaiting new data from the client; that is where the event loop can switch execution to another coroutine. All this happens behind the scenes, so to speak.
If you want to introduce a timeout, you could wrap the awaitable readline coroutine in a wait_for for example and then catch the TimeoutError exiting the loop.
Hope this helps.
I have a python socket server using asyncio and websockets. When the websocket is active 100+ devices will connect and hold their connection waiting for commands/messages.
There are two threads the first thread accepts connections and adds their details to a global variable then waits for messages from the device:
async def thread1(websocket, path):
client_address = await websocket.recv()
CONNECTIONS[client_address] = websocket
async for message in websocket:
... do something with message
start_server = websockets.serve(thread1, host, port)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.ensure_future(thread2())
asyncio.get_event_loop().run_forever()
The second thread processes some user data and once it needs to send a command it accesses a global variable to get the websocket info:
thread2()
...some data processing
soc = CONNECTIONS[ipaddress]
await soc.send("some message")
My question: What's the best way to allow another thread to send messages?
I can keep the global variable safe using thread locking and a function made only to process that data, however global variables aren't ideal. I cannot send information between threads since thread1 is stuck waiting to receive messages.
The first thing I would like to say is the incorrect use of the term thread. You use asyncio and here the concept is used - coroutine (coroutine is wrapped into a asyncio task). How it differs from threads can be found, for example, here.
The websockets server spawns a new task for each incoming connection (there are the same number of connections and spawned tasks). I don't see anything wrong with the global object, at least in a small script. However, below I gave an example where I placed this in a separate class.
Also, in this case, special synchronization between coroutines is not required, since they are implemented through cooperative multitasking (in fact, all are executed in one thread, transferring control at certain points.)
Here is a simple example in which the server stores a dictionary of incoming connections and starts a task that every 2 seconds, notifies all clients and sends them the current time. The server also prints confirmation from clients to the console.
# ws_server.py
import asyncio
import websockets
import datetime
class Server:
def __init__(self, host, port):
self.host = host
self.port = port
self.connections = {}
self.is_active = False
self.server = None
async def start(self):
self.is_active = True
self.server = await websockets.serve(self.handler, self.host, self.port)
asyncio.create_task(self.periodic_notifier())
async def stop(self):
self.is_active = False
self.server.close()
await self.wait_closed()
async def wait_closed(self):
await self.server.wait_closed()
async def handler(self, websocket, path):
self.connections[websocket.remote_address] = websocket
try:
async for message in websocket:
print(message)
except ConnectionClosedError as e:
pass
del self.connections[websocket.remote_address]
print(f"Connection {websocket.remote_address} is closed")
async def periodic_notifier(self):
while self.is_active:
await asyncio.gather(
*[ws.send(f"Hello time {datetime.datetime.now()}") for ws in self.connections.values()],
return_exceptions=True)
await asyncio.sleep(2)
async def main():
server = Server("localhost", 8080)
await server.start()
await server.wait_closed()
asyncio.run(main())
# ws_client.py
import asyncio
import websockets
async def client():
uri = "ws://localhost:8080"
async with websockets.connect(uri) as websocket:
async for message in websocket:
print(message)
await websocket.send(f"ACK {message}")
asyncio.run(client())
I am learning how to use the websockets package for python 3.6 with asyncio.
Using the Websockets Getting Started example, here are my server and client code (both running in two separate console using python <script>)
wsserver.py
import asyncio
import websockets
msg_queue = asyncio.Queue()
async def consumer_handler(websocket):
global msg_queue
while True:
message = await websocket.recv()
print("Received message {}".format(message))
await msg_queue.put("Hello {}".format(message))
print("Message queued")
async def producer_handler(websocket):
global msg_queue
while True:
print("Waiting for message in queue")
message = await msg_queue.get()
print("Poped message {}".format(message))
websocket.send(message)
print("Message '{}' sent".format(message))
async def handler(websocket, path):
print("Got a new connection...")
consumer_task = asyncio.ensure_future(consumer_handler(websocket))
producer_task = asyncio.ensure_future(producer_handler(websocket))
done, pending = await asyncio.wait([consumer_task, producer_task]
, return_when=asyncio.FIRST_COMPLETED)
print("Connection closed, canceling pending tasks")
for task in pending:
task.cancel()
start_server = websockets.serve(handler, 'localhost', 5555)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
wsclient.py
import asyncio
import websockets
async def repl():
async with websockets.connect('ws://localhost:5555') as websocket:
while True:
name = input("\nWhat's your name? ")
await websocket.send(name)
print("Message sent! Waiting for server answer")
greeting = await websocket.recv()
# never goes here
print("> {}".format(greeting))
asyncio.get_event_loop().run_until_complete(repl())
During the execution, the server is doing what is expected of him :
Wait for a client message
Queue 'Hello $message'
Dequeue it
Send the dequeued message back to the sender
The client does work up to the waiting of the server response :
Wait for a user input
Send it to the server
Wait answer from the server <-- Holds on indefinitely
Print it & loop
Here are the console outputs of the execution :
Server
Got a new connection...
Waiting for message in queue
Received message TestName
Message queued
Poped message Hello TestName
Message 'Hello TestName' sent
Waiting for message in queue
Client
What's your name? TestName
Message sent! Waiting for server answer
_
What am I missing?
Server-side, you're missing an await on the websocket.send(message) line.
To find those kind of bugs, start your program with the PYTHONASYNCIODEBUG environment variable, like: PYTHONASYNCIODEBUG=1 python3 wsserver.py which prints:
<CoroWrapper WebSocketCommonProtocol.send() running at […]/site-packages/websockets/protocol.py:301, created at wsserver.py:23> was never yielded from
I need to write some async code which runs a subprocess as part of its tasks. Even though I am using asyncio.subprocess my code is still blocking. My server looks like this:
import asyncio
import asyncio.subprocess
import websockets
async def handler(websocket, path):
while True:
data = await websocket.recv()
print('I received a message')
player = await asyncio.create_subprocess_exec(
'sleep', '5',
stdin=asyncio.subprocess.DEVNULL,
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.DEVNULL)
await player.wait()
print('Finished waiting')
server = websockets.serve(handler, '0.0.0.0', '8000')
asyncio.get_event_loop().run_until_complete(server)
asyncio.get_event_loop().run_forever()
And a very basic client:
import asyncio
import websockets
async def client():
async with websockets.connect('ws://localhost:8000') as websocket:
for i in range(5):
await websocket.send('message')
await asyncio.sleep(0.5)
asyncio.get_event_loop().run_until_complete(client())
I would expect the output to look like this:
I received a message
I received a message
I received a message
I received a message
I received a message
Finished waiting
Finished waiting
Finished waiting
Finished waiting
Finished waiting
But instead I get this:
I received a message
Finished waiting
I received a message
Finished waiting
I received a message
Finished waiting
I received a message
Finished waiting
I received a message
Finished waiting
With a 5 second wait after each "I received a message" line.
The line await player.wait() does not block other async operations, but waits for 5 seconds!
If you don't want to wait for the response, try using ensure_future() instead:
# add:
async def wait_for_player(player, path):
print("Waiting...", path)
await player.wait()
print("Done", path)
# and replace await player.wait() with:
asyncio.ensure_future(wait_for_player(player, path))
You can actually also move create_subprocess_exec() to wait_for_player().
To see your code is not blocking see try these:
Client:
import asyncio
import websockets
async def client(n):
async with websockets.connect('ws://localhost:8000/{}/'.format(n)) as websocket:
print(n, "start")
for i in range(5):
print(n, i)
await websocket.send('message')
await asyncio.sleep(0.5)
print(n, "done")
tasks = [client(i) for i in range(5)]
asyncio.get_event_loop().run_until_complete(asyncio.wait(tasks))
Server:
import asyncio
import asyncio.subprocess
import random
import websockets
async def handler(websocket, path):
try:
while True:
data = await websocket.recv()
pause = random.randint(1, 5)
print('I received a message', path, "Pausing:", pause)
player = await asyncio.create_subprocess_exec(
'sleep', str(pause),
stdin=asyncio.subprocess.DEVNULL,
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.DEVNULL)
await player.wait()
print('Finished waiting', path)
except websockets.ConnectionClosed:
print("Connection closed!", path)
server = websockets.serve(handler, '0.0.0.0', '8000')
asyncio.get_event_loop().run_until_complete(server)
asyncio.get_event_loop().run_forever()
Your ws server seems ok. Actually it is your client that is blocking. If you want to test the async behavior of your server, You need to make asynchronous requests. The for loop in your client blocks the thread. So remove it and instead, use asyncio.gather to run your client() method 5 times asynchronously
import asyncio
import websockets
async def client():
async with websockets.connect('ws://localhost:8000') as websocket:
await websocket.send('message')
await asyncio.sleep(0.5)
tasks = asyncio.gather(*[client() for i in range(5)])
asyncio.get_event_loop().run_until_complete(tasks)