I'm using a python websockets library in order to create a websocket server. My goals is to export a synchronous API, as it's going to be used outside of python.
As such, I need, at least at the beginning, start() and stop() methods. So it seems right to create a websocket server class for that.
Main issue it that the way to create (and start) a server through the library is by awaiting, and so these methods should encouraged to be async, which I try to avoid.
The following code work perfectly when I run the main() function.
When runnig server = WebsocketServer(); server.start(1234) through ipython shell I can't seem to connect through a client code. What am I missing?
class WebsocketServer():
def __init__(self):
self._server = None
def start(self, port):
start_server = websockets.serve(self._handle_client,
host='localhost',
port=port)
self._server = asyncio.get_event_loop().run_until_complete(start_server)
def stop(self):
if self._server:
self._server.close()
self._server = None
async def _handle_client(self, client, path):
async for buffer in client:
await self._handle_buffer(client, buffer)
async def _handle_buffer(self, client, buffer):
print(buffer)
await client.send(buffer)
def main():
server = WebsocketServer()
server.start(1234)
asyncio.get_event_loop().run_forever()
The synchronous interface into the IO loop is via tasks. Scheduling methods return a future that can be synchronously waited on if needed. The run an event loop section of the docs features a combo for synchronous shutdown in the bottom.
When running inside an iPython shell, one option is to spawn a daemon background thread for the IO loop and register an atexit callback to synchronously shutdown the IO loop when the shell exits.
Another option is to "borrow" shell's thread once in a while for the IO tasks (only works for short tasks, of cause) using the UI Event Loop integration point described here describing how to borrow the shell thread for the IO.
You are missing the last line from you main function.
asyncio.get_event_loop().run_forever()
Nothing happens when the loop is not running. So, the server won't be running unless you run the loop.
Related
I want to implement paho-mqtt in which it should process the incoming messages asynchronously.
I have implement gmqtt with asyncio which runs perfectly fine, but as far as I understand paho-mqtt is better to use rather than gmqtt (Link : https://www.emqx.io/blog/comparision-of-python-mqtt-client).
gmqtt w/ asyncio :
def assign_callbacks_to_client(self, client):
""" Helper function which sets up client's callbacks. """
client.on_connect = self.on_connect
client.on_message = self.on_message
client.on_disconnect = self.on_disconnect
client.on_subscribe = self.on_subscribe
async def subscriber(self, mqtt_name):
""" Connect to mqtt-broker. """
sub_client = MQTTClient(mqtt_name)
self.assign_callbacks_to_client(sub_client)
logging.info("connecting")
await sub_client.connect(host=config.MQTT_HOST, port=int(config.MQTT_PORT))
return sub_client
could you please let me know, how to implement paho with asyncio library?
Will using loop_start asynchronously, as I understand everytime it executes it starts a new thread in background.
loop_start() only create a single background thread that all the callbacks will be run on, you should not be doing long running tasks directly in these callbacks as they will block all other actions of the client.
If you want to process incoming messages without blocking then you will need to implement your own thread pool and just use the on_message() callback to push messages to this thread pool.
I have a class that contains a function that I would like to be able to invoke by invoking a flask-resful endpoint. Is there a way to define an asynchronous function that would await/subscribe to this endpoint to be called? I can make changes to the flask app (but can't switch to SocketIO) as well if required or write some sort of async requests function. I can only work with the base Anaconda 3.7 library and I don't have any additional message brokers installed or available.
class DaemonProcess:
def __init__(self):
pass
async def await_signal():
signal = await http://ip123/signal
self.(process_signal) # do stuff with signal
For context, this isn't the main objective of the process. I simply want to be able to use this to tell my process remotely or via UI to shut down worker processes either gracefully or forcefully. The only other idea I came up with is pinging a database table repeatedly to see if a signal has been inserted, but time is of the essence and would require pinging at too short of intervals in my opinion and an asynchronous approach would be favored. The database would be SQLite3 and it doesn't appear to support update_hook callbacks.
Here's sample pattern to send a singal and process it:
import asyncio
import aiotools
class DaemonProcess
async def process(reader, writer):
data = await reader.read(100)
writer.write(data)
print(f"We got a message {data} - time to do something about it.")
await writer.drain()
writer.close()
#aiotools.server
async def worker(loop, pidx, args):
server = await asyncio.start_server(echo, '127.0.0.1', 8888,
reuse_port=True, loop=loop)
print(f'[{pidx}] started')
yield # wait until terminated
server.close()
await server.wait_closed()
print(f'[{pidx}] terminated')
def start(self):
aiotools.start_server(myworker, num_workers=4)
if __name__ == '__main__':
# Run the above server using 4 worker processes.
d = DaemonProcess()
d.start()
if you save it in a file, for example, process.py, you should be able to start it:
python3 process.py
Now once you have this daemon in background, you should be able to ping it (see a sample client below):
import asyncio
async def tcp_echo_client(message):
reader, writer = await asyncio.open_connection('127.0.0.1', 8888)
print(f'Send: {message!r}')
writer.write(message.encode())
await writer.drain()
data = await reader.read(100)
print(f'Received: {data.decode()!r}')
print('Close the connection')
writer.close()
await writer.wait_closed()
And now, somewhere in your Flask view, you should be able to invoke:
asyncio.run(tcp_echo_client('I want my daemon to do something for me'))
Notice this all used localhost 127.0.0.1 and port 8888, so those to be made available unless you have your own ports and IPs, then you'll need to configure them accordingly.
Also notice the use of aiotools which is a module providing a set of common asyncio patterns (daemons, etc...).
I used asyncio for my non-stop server in python and implemented
connection_made , connection_lost , data_received
funtions in my ServerClientProtocol
I used this class first beacause of using multiple times repeatedly sending data to socket class socket
got closed and program exited
and second becuase I thought its async and have parallel answering multiple coming sockets in same time,
but it's not.
how should I use that in one async thread and parallel answering socket?
this is my code:
class ServerClientProtocol(asyncio.Protocol):
def connection_made(self,transport):
self.transport = transport
def connection_lost(self,exc):
pass
def data_received(self, data):
server.server(self,data)
def main(*args):
loop = get_event_loop()
coro = loop.create_server(ServerClientProtocol, '127.0.0.1', 50008)
srv = loop.run_until_complete(coro)
loop.run_forever()
if __name__ == '__main__':
main()
server.server() might be blocking the other connections. If this is a long-running call, try using asyncio.start_server (example here) instead, and call server.server() using await loop.run_in_executor(None, server.server, data)
I am writing a Python 3.5 program which handles some signals and serves this data to a small amount of websocket clients.
I want the websocket server and the signal handling to happen in the same program, therefore I am using threading.
The problem is I don't know how to send data from the worker thread to the client.
The Websocket server is implemented with a simple library called "websockets". The server is set up and clients can connect and talk to the server within the "new websocket client has connected" handler.
The server is set up with the help of an event loop:
start_server = websockets.serve(newWsHandler, host, port)
loop = asyncio.get_event_loop()
loop.run_until_complete(start_server)
loop.run_forever()
Because I want my program to do signal handling too, and loop.run_forever() is a blocking call, I create an endless worker thread before I start my server. This works as expected.
When the worker thread detects a signal change, it has to alert the connected websocket clients. But a simple client.send() does not work. Putting await in front of it does not work either (since that only works within coroutines, I think). I tried making a separate "async def" function and adding it to the event loop, but it gets a bit complicated because it's not on the same thread.
So the main question is: what is the best way send something to a websocket client from a worker thread? I don't receive anything in response.
EDIT:
It will probably help if I add some mock code.
def signalHandler():
#check signals
...
if alert:
connections[0].send("Alert") #NEED HELP HERE
async def newWsHandler(websocket, path):
connections.append(websocket)
while True:
#keep the connection open until the client disconnects
msg = await websocket.recv()
#top level
connections = []
...
start_server = websockets.serve(newWsHandler, host, port)
signalThread = Thread(target = signalHandler)
signalThread.setDaemon(True)
signalThread.start()
loop = asyncio.get_event_loop()
loop.run_until_complete(start_server)
loop.run_forever()
I have a tkinter based GUI program running in Python 3.4.1. I have several threads running in the program to get JSON data from various urls. I am wanting to add some WebSocket functionality to be able to allow program to act as a server and allow several clients to connect to it over a WebSocket and exchange other JSON data.
I am attempting to use the Autobahn|Python WebSocket server for asyncio.
I first tried to run the asyncio event loop in a separate thread under the GUI program. However, every attempt gives 'AssertionError: There is no current event loop in thread 'Thread-1'.
I then tried spawning a process with the standard library multiprocessing package that ran the asyncio event loop in another Process. When I try this I don't get any exception but the WebSocket server doesn't start either.
Is it even possible to run an asyncio event loop in a subprocess from another Python program?
Is there even a way to integrate an asyncio event loop into a currently multithreaded/tkinter program?
UPDATE
Below is the actual code I am trying to run for an initial test.
from autobahn.asyncio.websocket import WebSocketServerProtocol
from autobahn.asyncio.websocket import WebSocketServerFactory
import asyncio
from multiprocessing import Process
class MyServerProtocol(WebSocketServerProtocol):
def onConnect(self, request):
print("Client connecting: {0}".format(request.peer))
def onOpen(self):
print("WebSocket connection open.")
def onMessage(self, payload, isBinary):
if isBinary:
print("Binary message received: {0} bytes".format(len(payload)))
else:
print("Text message received: {0}".format(payload.decode('utf8')))
## echo back message verbatim
self.sendMessage(payload, isBinary)
def onClose(self, wasClean, code, reason):
print("WebSocket connection closed: {0}".format(reason))
def start_server():
factory = WebSocketServerFactory("ws://10.241.142.27:6900", debug = False)
factory.protocol = MyServerProtocol
loop = asyncio.get_event_loop()
coro = loop.create_server(factory, '10.241.142.27', 6900)
server = loop.run_until_complete(coro)
loop.run_forever()
server.close()
loop.close()
websocket_server_process = Process(target = start_server)
websocket_server_process.start()
Most of it is straight from the Autobahn|Python example code for asyncio. If I try to run it as a Process it doesn't do anything, no client can connect to it, if I run netstat -a there is no port 6900 being used. If just use start_server() in the main program it creates the WebSocket Server.
First, you're getting AssertionError: There is no current event loop in thread 'Thread-1'. because asyncio requires each thread in your program to have its own event loop, but it will only automatically create an event loop for you in the main thread. So if you call asyncio.get_event_loop once in the main thread it will automatically create a loop object and set it as the default for you, but if you call it again in a child thread, you'll get that error. Instead, you need to explicitly create/set the event loop when the thread starts:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
Once you've done that, you should be able to use get_event_loop() in that specific thread.
It is possible to start an asyncio event loop in a subprocess started via multiprocessing:
import asyncio
from multiprocessing import Process
#asyncio.coroutine
def coro():
print("hi")
def worker():
loop = asyncio.get_event_loop()
loop.run_until_complete(coro())
if __name__ == "__main__":
p = Process(target=worker)
p.start()
p.join()
Output:
hi
The only caveat is that if you start an event loop in the parent process as well as the child, you need to explicitly create/set a new event loop in the child if you're on a Unix platform (due to a bug in Python). It should work fine on Windows, or if you use the 'spawn' multiprocessing context.
I think it should be possible to start an asyncio event loop in a background thread (or process) of your Tkinter application and have both the tkinter and asyncio event loop run side-by-side. You'll only run into issues if you try to update the GUI from the background thread/process.
The answer by #dano might be correct, but creates an new process which is unnessesary in most situations.
I found this question on Google because i had the same issue myself. I have written an application where i wanted an websocket api to not run on the main thread and this caused your issue.
I found my alternate sollution by simply reading about event loops on the python documentation and found the asyncio.new_event_loop and asyncio.set_event_loop functions which solved this issue.
I didn't use AutoBahn but the pypi websockets library, and here's my solution
import websockets
import asyncio
import threading
class WebSocket(threading.Thread):
#asyncio.coroutine
def handler(self, websocket, path):
name = yield from websocket.recv()
print("< {}".format(name))
greeting = "Hello {}!".format(name)
yield from websocket.send(greeting)
print("> {}".format(greeting))
def run(self):
start_server = websockets.serve(self.handler, '127.0.0.1', 9091)
eventloop = asyncio.new_event_loop()
asyncio.set_event_loop(eventloop)
eventloop.run_until_complete(start_server)
eventloop.run_forever()
if __name__ == "__main__":
ws = WebSocket()
ws.start()
"Is there even a way to integrate an asyncio event loop into a currently multithreaded/tkinter program?"
Yes, run your tkinter program with an asyncio event loop. Proof of concept.
'''Proof of concept integrating asyncio and tk loops.
Terry Jan Reedy
Run with 'python -i' or from IDLE editor to keep tk window alive.
'''
import asyncio
import datetime as dt
import tkinter as tk
loop = asyncio.get_event_loop()
root = tk.Tk()
# Combine 2 event loop examples from BaseEventLoop doc.
# Add button to prove that gui remain responsive between time updates.
# Prints statements are only for testing.
def flipbg(widget, color):
bg = widget['bg']
print('click', bg, loop.time())
widget['bg'] = color if bg == 'white' else 'white'
hello = tk.Label(root)
flipper = tk.Button(root, text='Change hello background', bg='yellow',
command=lambda: flipbg(hello, 'red'))
time = tk.Label(root)
hello.pack()
flipper.pack()
time.pack()
def hello_world(loop):
hello['text'] = 'Hello World'
loop.call_soon(hello_world, loop)
def display_date(end_time, loop):
print(dt.datetime.now())
time['text'] = dt.datetime.now()
if (loop.time() + 1.0) < end_time:
loop.call_later(1, display_date, end_time, loop)
else:
loop.stop()
end_time = loop.time() + 10.1
loop.call_soon(display_date, end_time, loop)
# Replace root.mainloop with these 4 lines.
def tk_update():
root.update()
loop.call_soon(tk_update) # or loop.call_later(delay, tk_update)
# Initialize loop before each run_forever or run_until_complete call
tk_update()
loop.run_forever()
I have experimentally run IDLE with those 4 extra lines, with a slowdown only noticeable when syntax highlighting 1000s of lines.