I'm setting up a listener for Hyperledger Sawtooth events with a pyzmq dealer socket and the provided asyncio functionality. Currently futures are returned but only sometimes finished, even though messages are sent to the Socket.
Weirdly this works for the connection message (only when sleeping before it as shown below) but not for event messages. I implemented this already with JavaScript and it works without problems. It seems that the issue does not lie with Sawtooth but rather in pyzmq's implementation of asyncio functionality or in my code.
class EventListener:
def __init__(self):
...
ctx = Context.instance()
self._socket = ctx.socket(zmq.DEALER)
self._socket.connect("tcp://127.0.0.1:4004")
async def subscribe(self):
...
await self._socket.send_multipart([connection_msg])
async def receive(self):
# Necessary wait otherwise the future is never finished
await asyncio.sleep(0.1)
resp = await self._socket.recv_multipart()
handle_response(resp)
async def listen(self):
while True:
# here sleep is not helping
# await asyncio.sleep(0.1)
# follwing await is never finished
resp = await self._socket.recv_multipart()
handle_response(resp)
...
listener = listener.EventListener()
await asyncio.gather(
listener.receive(), listener.subscribe())
await asyncio.create_task(listener.listen())
...
Debugging shows that the returned Future object is never changed from a pending to a finished state. So, is my code incorrect, do I need to await messages differently or is it possible that something is wrong with pyzmq's asyncio functionality? Also, why do I need to sleep in receive(), isn't that why we have asyncio?
There are too many queries, this answer may not address all of them. Hope at least this will help others looking for a way to setup event listeners.
The Hyperledger Sawtooth python SDK provides option for clients to subscribe to the events. The SDK part of code that does what you're trying to do can be found at https://github.com/hyperledger/sawtooth-sdk-python/blob/master/sawtooth_sdk/messaging/stream.py
The example code to use the Hyperledger Sawtooth python SDK for event subscription can be found here https://github.com/danintel/sawtooth-cookiejar/blob/master/events/events_client.py
Related
I'm currently writing a discord bot which needs to be able to run a task that could take anywhere from a few seconds to a minute while still being responsive to other commands. Forgive me if this is a fairly simple question, but I haven't been able to find a solution that worked yet.
Here is an abridged version of the code
class StableCog(commands.Cog, name='Stable Diffusion', description='Create images from natural language.'):
def __init__(self, bot):
self.text2image_model = Text2Image()
self.bot = bot
#commands.slash_command(description='Create an image.')
async def dream(self, -- a ton of arguments -- ):
print(f'Request -- {ctx.author.name}#{ctx.author.discriminator} -- Prompt: {query}')
asyncio.get_event_loop().create_task(src.bot.queue_system.dream_async( -- a ton of arguments -- ))
inside queue_system.py
async def dream_async(-- a ton of arguments --):
await ctx.interaction.response.send_message('Added to queue! You are # in queue')
embed = discord.Embed()
try:
#lots of code, I've removed it since it doesn't have anything to do with the async
await ctx.channel.send(embed=embed, file=discord.File(fp=buffer, filename=f'{seed}.png'))
except Exception as e:
embed = discord.Embed(title='txt2img failed', description=f'{e}\n{traceback.print_exc()}', color=embed_color)
await ctx.channel.send(embed=embed)
However, the discord bot becomes unresponsive until the code in queue_system.py finishes running. Every solution I've tried so far hasn't worked correctly since I'm trying to create a thread to run an asynchronous method. What would be the best way to do so? Ignore the name queue_system.py, it isn't quite a queue system yet, I'm just working out how to run the dream method asynchronously before I work that out.
What is blocking the event loop in the dream_async coroutine ? In that coroutine if you're calling some (non async) functions that have the potential to block the loop, that's an issue, but the real culprit must be in the "lots of code" part :)
A good option would be to use run_in_executor() to run the non async code in a threadpool, and therefore prevent that code from blocking dream_async.
def blocking_stuff(arg):
# this will run in a thread
...
return 'something'
async def dream_async(-- a ton of arguments --):
loop = asyncio.get_event_loop()
await ctx.interaction.response.send_message('Added to queue! You are # in queue')
embed = discord.Embed()
try:
# Run the blocking part in a threadpool
result = await loop.run_in_executor(None, blocking_stuff, 'test')
await ctx.channel.send(embed=embed, file=discord.File(fp=buffer, filename=f'{seed}.png'))
except Exception as e:
embed = discord.Embed(title='txt2img failed', description=f'{e}\n{traceback.print_exc()}', color=embed_color)
await ctx.channel.send(embed=embed)
Hope i didn't misunderstand you.
I am trying simultaneously to listen for a telegram or a discord message, whatever the first comes in. For Telegram I'm using Telethon:
async def listentg():
tgclient = TelegramClient('anon', conntg.tg_api_id, conntg.tg_api_hash)
#tgclient.on(events.NewMessage(chats=conntg.canaltg, pattern=patternmatch))
async def tg_event_handler(event):
print("Telegram message listened")
await tgclient.disconnect()
await tgclient.start()
await tgclient.run_until_disconnected()
And for Discord I'm using Discum
async def listendc():
dcclient = discum.Client(token=conndc.tokendc, log=False)
#dcclient.gateway.command
def dc_event_handler(resp):
if resp.event.message:
print("Discord message listened")
dcclient.gateway.close()
dcclient.gateway.run()
I understand that for running simultaneously CPU blocking code i have to (https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor) use python processes, but I don't know how to do this, just wait for the first one to return the value.
You don't really need to have them in parallel for simultaneously, it is easier to have them concurrent (but still simultaneosly waiting). You can use Python Thread-based parallelism instead.
I want to submit data to MongoDB in a non-blocking way. Other questions on this website recommend either:
Phrasing this problem as a single-producer, single-consumer problem with queue communication.
Using Motor with asyncio and create_task.
All examples in the Motor documentation use asyncio.get_event_loop() and run_until_complete(). However, this is a blocking way of running co-routines. I would like to instead use the non-blocking method of create_task.
Here's what I've tried:
client = motor.motor_asyncio.AsyncIOMotorClient(my_login_url)
async def do_find_one():
doc = client.my_db.my_collection.find_one()
pprint.pprint(doc)
async def main():
task = asyncio.create_task(do_find_one())
print("other things")
await task
print("done")
asyncio.run(main())
However, instead of a MongoDB entry, task just prints a pending Future which I don't know how to get the value of. How am I supposed to be using motor with create_task?
I am currently checking out the Python discord wrapper found here but it doesn't seem to work due to the above mentioned error. I tried calling the nest_asyncio.apply() before running the client.run() function but it doesn't seem to work either as mentioned in this other question. This question is possibly a duplicate but couldn't add a comment in the previous one.
I tried this:
nest_asyncio.apply()
client = discord.Client()
client.run(bot_token)
as well as this to no avail:
client = discord.Client()
nest_asyncio.apply(client.run(bot_token))
I've faced a similar, if not the same issue a few months ago, and what I found on a comment to a github issue ultimately led to the following:
class DiscordAccessor:
'''class to handle discord authentication and async loop handling
Attributes
----------
dc_coroutine
the coroutine to start the client
dc_thread
the thread to keep the coroutine alive
Methods
-------
start_loop
starts the async loop'''
dc_loop = asyncio.new_event_loop()
client = discord.Client(loop=dc_loop)
def __init__(self):
self.dc_coroutine = DiscordAccessor.client.start('YOUR_TOKEN')
self.dc_thread = threading.Thread(target=self.start_loop,
args=(self.dc_loop, self.dc_coroutine))
self.dc_thread.start()
def start_loop(self, loop, coro):
'''starts the async loop
Parameters
----------
loop
the asyncio loop
coro
the coroutine'''
loop.run_until_complete(coro)
This class wraps the discord client into it's own thread and event loop. You'd call your client something like:
dc = DiscordAccessor()
dc.client.whatever_you_want_to_call()
I have a server script running:
async def give_time(websocket,path):
while True:
await websocket.send(str(datetime.datetime.now()))
await asyncio.sleep(3)
start_server = websockets.serve(give_time, '192.168.1.32', 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
which is working fine, just sending the current time every 3 seconds.
I can receive that string from the client which runs this code:
async def hello(): #takes whatever text comes through the websocket and displays it in the socket_text label.
async with websockets.connect('ws://wilsons.lan:8765') as ws:
while True:
text = await ws.recv()
logger.info('message received through websocket:{}'.format(text))
socket_text.configure(text=text) #socket_text is a tkinter object
loop = asyncio.new_event_loop()
def socketstuff():
asyncio.set_event_loop(loop)
asyncio.get_event_loop().run_until_complete(hello())
t = threading.Thread(target=socketstuff,daemon=True)
It runs in a thread so that I can run tkinter.mainloop in the main thread. This is the first time I ever used threading so I may be getting it wrong, but it seems to work at present.
What I need to do, is to be able to send a message down the websocket based on a tkinter event - currently just clicking a button next to a text box, but eventually more complex things. The clicking part works fine.
I'm having a lot of trouble with the sending of the message. I've tried a lot of different things, with and without async and await, although that may have just been panic.
The main problem seems to be that I can't access ws from outside that hello() function. This makes sense as I'm using the with context manager. However, if I just use ws = websockets.connect('ws://host') then I get a websockets.py35.client.Connect object which wen I try to use the send (or indeed recv) methods, I get a object has no attribute 'send' error.
I hope this is enough info - will happily post anything else required!
It turns out that the best way of solving this is not to use threads.
This post helped me solve it. It shows that you can run, in a coroutine, one iteration of the tkinter mainloop:
async def do_gui(root,interval=0.05):
while True:
root.update()
await asyncio.sleep(interval)
However, the best way of getting a tkinter event to spawn a websocket message is to use asyncio.queue. Making the tkinter callback add an item to the queue using put_nowait() and having a coroutine which runs at the same time as do_gui which uses await queue.get() to get the message from the queue works for me.