I wrote a code in python 3.9.0 which opens a text file, read line by line and host it at web socket for clients to consume. It works just fine. The problem is, when I consume the file from the web socket server it is really slow. I can see that server sends it faster in thousands but client is receiving it slow. I am not sure what am I doing wrong with my client code. Here is my client code
async def consumeStream():
while True:
async with websockets.connect('ws://localhost:6800') as ws:
ingestCounter = 0
while True:
try:
consumedStream = await ws.recv()
except websockets.ConnectionClosed:
print(color.red+'Websocket connection closed, retrying !'+color.reset)
break
if ingestCounter % 100 == 0:
print('\r'+f'Consuming: {ingestCounter}', end='')
ingestCounter += 1
await asyncio.sleep(0)
Is streaming good idea with web sockets? Should I use Flask or FastAPI with yield ? What is the best practice for web sockets streaming and consuming ?
Related
After seeing a video about a youtuber who uses a websocket server to interacte with an entity in a video game (minecraft) video : https://www.youtube.com/watch?v=pwKRbsDbxqc&t=25s&ab_channel=Ottomated
I wanted to recreate the same idea, but with some little tweaks:
I wanted the websocket server to be host in python
I wanted to be able to interact without any html or javascript
So I built the websocket server:
#!/usr/bin/env python
import asyncio
import websockets
async def server(websocket, path):
data = await websocket.recv()
while True:
msg = input('type the command: ')
await websocket.send(msg)
start_server = websockets.serve(server, "localhost", 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
(really simple)
and the client side (in lua):
print('connecting')
local ws, err = http.websocket("ws://localhost:8765")
if not ws then
return printError(err)
ws.close()
end
ws.send("message")
repeat
command = ws.receive()
load(command)()
until command == "stop"
ws.close()
The only thing is that after the first message gets executed by the lua programm, it stops processing the other messages.
I don't know what I did wrong!
Am I doing it the wrong way? or maybe there is a better way?
well i am still learning sockets and improving my server-side skill but it is easy you just need to make it a loop 'while true:' or i prefer a controllable variable 'kim = True
while kim :' then you can make false or true to close or open the loop under certian conditions and i think you need somthing like threads to not block the other clients input
well i didnt study web-server but the problem is as i think its that you only receive one message because its not in a loop so it will excute the code and then close
I have a Python3 program that runs a "while True"-loop until stopped, which occasionally saves data to a MySQL database. I am creating an administrative website, separate from the Python program, where I will be able to observe this data.
I now want to be able to be notified, on the website, when changes have been made to the database. My thought was to set up a websocket connection, so that the Python program can send a message through the socket to all connected clients, i.e. all open browsers, if there has been any changes to the database table.
I have done something similar before, but in that case I had to wait for a websocket connection before the "while True"-loop would start. In the new scenario I want to be able to have multiple website clients at once, and let them connect at any time, as well as disconnect without interrupting the Python programs loop.
This is a simplified version of my previous code, which I now want to update to be able to run both with & without websocket clients.
import asyncio
import websockets
socket_server = websockets.serve(run, "127.0.0.1", 5055)
asyncio.get_event_loop().run_until_complete(socket_server)
console_log("Waiting for socket connection...")
asyncio.get_event_loop().run_forever()
async def run(ws):
while True:
db_has_updated = do_stuff()
if db_has_updated:
await ws.send(data)
I just can't seem to be able to come up with the right search terms to find a solution, so I'm asking here instead.
I figured it out, finally! Here is my solution with a websocket server running in a separate thread from the other logic. I'm probably changing some things to make it neater, but this does everything I need. Feel free to ask any questions.
Be aware that this blocks when messaging all the connected clients. That is the way I needed it to work, but you could always thread/subprocess the logic/data-gen part of the program if you want it to run completely asynchronously.
#!/usr/bin/env python3
import asyncio
import websockets
import threading
import time
import random
def gen_data():
print("Generating data...")
time.sleep(3)
data = random.randint(1, 10)
return data
async def send(client, data):
await client.send(data)
async def handler(client, path):
# Register.
print("Websocket Client Connected.", client)
clients.append(client)
while True:
try:
print("ping", client)
pong_waiter = await client.ping()
await pong_waiter
print("pong", client)
time.sleep(3)
except Exception as e:
clients.remove(client)
print("Websocket Client Disconnected", client)
break
clients = []
start_server = websockets.serve(handler, "localhost", 5555)
asyncio.get_event_loop().run_until_complete(start_server)
threading.Thread(target = asyncio.get_event_loop().run_forever).start()
print("Socket Server Running. Starting main loop.")
while True:
data = str(gen_data())
message_clients = clients.copy()
for client in message_clients:
print("Sending", data, "to", client)
try:
asyncio.run(send(client, data))
except:
# Clients might have disconnected during the messaging process,
# just ignore that, they will have been removed already.
pass
I have a server built in Python that uses Sanic and websockets to routinely broadcast data to clients:
#app.websocket("/")
async def websocket(request, ws):
app.ws_clients.add(ws)
await ws.send(json.dumps("hello from climate server!"))
while True:
try:
data = dict()
time_of_reading = time.ctime(time.time())
data['climateData'] = sensor.read_data()
data['systemData'] = get_system_data()
data['timestamp'] = time_of_reading
await broadcast(json.dumps(data))
time.sleep(10) # changing this to asyncio.sleep() causes the msgs to send sporatically
except KeyboardInterrupt:
sensor.clear()
pass
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8080, workers=1, debug=False)
and my broadcast function which attempts to send a message, or removes a client from app.ws_clients if there is a ConnectionClosed error:
async def broadcast(message):
for ws in app.ws_clients:
try:
await ws.send(message)
print('attempting data send') # this line runs, but the clients don't receive the messages
except websockets.ConnectionClosed:
clients_to_remove.add(ws)
except KeyboardInterrupt:
sensor.clear()
pass
if (len(clients_to_remove) > 0):
await remove_dead_clients(clients_to_remove)
async def remove_dead_clients(clients_to_remove):
for client in clients_to_remove:
app.ws_clients.remove(client)
clients_to_remove.clear()
The client is able to connect just fine, and the server prints that it is trying to broadcast, but no message is ever received by the client.
I am using this broadcast function from another server that I wrote, and it works perfectly there. The difference with that one is that it only sends data when a client requests it. I feel the issue here is that the async cannot handle both continually broadcasting and removing clients simultaneously. I tried changing time.sleep() to asyncio.sleep(), but that only succeeded in successfully sending the messages dozens at a time, and then nothing at all for awhile.
Is there a pattern I could implement that would meet my needs, where I can send messages in an endless loop and also asynchronously manage connected clients?
For anyone that happens to stumble across this in the future - I had forgotten to use the await keyword prior to asyncio.sleep().
I'm trying to build a multiplayer game (to be precise a Card game) in Python via websockets, but I'm currently failing in the very early steps.
What I'm trying to do:
Clients connect to the websocket, until a certain number is reached
The client sends an input to the server
The server responds to each client separately and simultaneously
What works:
Letting the client connect and storing a websocket per client works as expected, but then I'm a bit stuck.
What I've tried:
client.py
import asyncio
import websockets
import random
def random_name():
return "".join([random.choice("abcdefghijkl") for _ in range(5)])
async def main():
uri = "ws://localhost:1235"
print("Starting...")
async with websockets.connect(uri) as ws:
client_name = random_name()
await ws.send(client_name)
print(await ws.recv()) # server sends :client_name registered
print(await ws.recv()) # server sends: 3 people registered
print(await ws.recv()) # server sends: Waiting for input
inp = input()
await ws.send(inp)
asyncio.get_event_loop().run_until_complete(main())
server.py (very naive approach)
import websockets
import asyncio
clients = []
async def main(ws, *args, **kwargs):
while True:
print(f"Waiting for new connection! {ws}")
client_name = await ws.recv()
clients.append((client_name, ws, ws.remote_address))
await ws.send(f"{client_name} registered on {ws.remote_address}")
if len(clients) >= 3:
registered_clients = ', '.join([c[0] for c in clients])
print(f"3 clients ({registered_clients}) registered")
break
for name, ws_, _ in clients:
await ws_.send(f"3 clients registered ({registered_clients})")
await ws_.send(f"Waiting for input")
for _ in clients:
inp = await ws_.recv()
print(ws_.remote_address, inp)
start_server = websockets.serve(main, "localhost", "1235")
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
This fails with RuntimeError: cannot call recv while another coroutine is already waiting for the next message. It seems that I cannot wait for the answer of each connected clients individually. Even though not perfect, I thought I can distinguish each client by looking at their remote addresses. For doing so, I modified the main function to
server.py, main function (naive approach)
async def main(ws, *args, **kwargs):
while True:
print(f"Waiting for new connection! {ws}")
client_name = await ws.recv()
clients.append((client_name, ws, ws.remote_address))
await ws.send(f"{client_name} registered on {ws.remote_address}")
if len(clients) >= 3:
registered_clients = ', '.join([c[0] for c in clients])
print(f"3 clients ({registered_clients}) registered")
break
for name, ws_ in clients:
await ws_.send(f"3 clients registered ({registered_clients})")
await ws_.send(f"Waiting for input")
for _ in range(len(clients)):
inp = await ws.recv()
print(ws.remote_address, inp)
This doesn't work neither, cause the loop doesn't break for all connected clients.
My questions:
How can my examples be repaired?
Is the websockets package the right framework for me? Two things let me think of that:
In case of sockets, I can bind one client to the socket instance and directly read from one instance (which it seems I cannot easily do here).
Even though I'm pretty sure, that one can achieve what I want, it seems that there's a lot code involved for doing that (!?)
But somehow I'd like to stick to websockets in general (because of also having the possibility to connect to communicate wit a browser). And sticking to the websockets package in particular, would have the advantage for me to get to know the asyncio module better. But if someone says, that I should rather switch to aiohttp, flask-sockets, tornado etc. I'd like to do so as well.
I'm trying to create a unidirectional websocket connection between the client and server (python). The libraries I've currently been prototyping with are websockets and simplesocketserver. I've got a hello world example getting from the server to the client, but I need to be able to send data from the backend to the client unprompted from the client. All of the websockets examples seem to show the server listening to the client and then responding.
So far I've tried:
Using websockets 8.0, sending data from server to client unprompted, this works but only with hard-coded strings, I don't understand how to send real data on demand unprompted from the client
Using simplesocketserver in the same exact manner
Started investigating server sent events - is this more appropriate?
Example from the websockets documentation:
import asyncio
import websockets
async def hello(websocket, path):
name = await websocket.recv()
print(f"< {name}")
greeting = f"Hello {name}!"
await websocket.send(greeting)
print(f"> {greeting}")
start_server = websockets.serve(hello, "localhost", 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Note: the need for unidirectional communication is due to existing architecture.
Any guidance or resources on this would be great, I hope I'm overlooking something easy. Thank you!
I came across a similar issue. After some trial and error I found a solution that works for me, maybe so it does for you. I've tested this on Linux and with Firefox and mobile Safari (parallel).
The InputThread waits for input (words) on the command line and writes them into a list, that is sent to all connected clients each time a new string is appended or a client is connected.
Code snippet
import asyncio
import websockets
import json
from threading import Thread
words = []
clients = []
async def register_client(websocket, path):
# register new client in list and keep connection open
clients.append(websocket)
await send_to_all_clients()
while True:
await asyncio.sleep(10)
async def send_to_all_clients():
global words
for i, ws in list(enumerate(clients))[::-1]:
try:
await ws.send(json.dumps({"words": words}))
except websockets.exceptions.ConnectionClosedError:
# remove if connection closed
del clients[i]
class InputThread(Thread):
def run(self):
global words
async def sending_loop():
while True:
i = input()
if not i.strip():
continue
words.append({"options": i.strip().slit()})
await send_to_all_clients()
asyncio.run(sending_loop())
InputThread().start()
asyncio.get_event_loop().run_until_complete(
websockets.serve(register_client, "localhost", 8765))
asyncio.get_event_loop().run_forever()