Here's what I want to do.
I have multiple asynchronous functions and a separate asynchronous function let's say main. I want to call this main function with every other function I call.
I'm using this structure in a telegram bot, and functions are called upon a certain command. But I want to run main on any incoming message including the messages with commands as mentioned above where another function is also called. So in that case, I wanna run both (first command specific function then main function)
I believe this can be done using threading.RLock() as suggested by someone, but I can't figure out how.
What's the best approach for this?
You could use aiotelegram in combination with asyncio's create_task().
While Threads can also do the job, they don't seem to be as good as asynchronous execution.
You can choose any telegram framework that provides an async context like Bot.run() does in aiotelegram, or you can even implement your own API client, just make sure you run on an asynchronous (ASGI) context.
The main idea then is to call asyncio.create_task() to fire up the main() function in parallel with the rest of the function that runs the Telegram Bot command.
Here's an example (note I've use my_main() instead main()):
import asyncio
from aiotg import Bot, Chat
bot = Bot(api_token="...")
async def other_command():
#Replace this with the actual logic
await asyncio.sleeep(2)
async def my_main():
# Replace this with the actual parallel task
await asyncio.sleep(5)
#bot.command(r"/echo_command (.+)")
async def echo(chat: Chat, match):
task = asyncio.create_task(my_main())
return chat.reply(match.group(1))
#bot.command(r"/other_command (.+)")
async def other(chat: Chat, match):
task = asyncio.create_task(my_main())
result = other_command()
return chat.reply(result)
bot.run()
It is important to know that with this approach, the tasks are never awaited nor checked for their completion, so Exceptions or failed executions can be difficult to track, as well as any result from main() that needs to be kept.
A simple solution for this is to declare a global dict() where you store the Task(s), so you can get back to them later on (i.e. with a specific Telegram Command, or running always within certain existing Telegram Commands).
Whatever logic you decide to keep track of the tasks, you can check if they're completed, and their results, if any, with Task.done() and Task.result(). See their official doc for further details about how to manage Tasks.
Related
I am creating a discord bot and am wondering how multithreading can be implemented. Basically I have a section of the bot that runs with the on_ready() function which contains an infinite loop that checks the content of a link every 60 seconds, and so far everything works fine, the problem is that while doing this it should also be able to respond to commands like !help but since the program is executing this infinite loop it does not execute all the rest of the code. Since I know that multithreading requires manual execution of the desired function when starting a thread I don't understand how I can implement it with functions such as async def help(ctx) which must be executed in response to a message and which have arguments like ctx or message.
QL;DR
In Python, is it a good practice to use asyncio.run inside a non-main function?
Description of the problem
I have a python code that runs multiple commands using subprocesses.
At the moment I run those subprocesses sequentially and wait until each one of the is finished until I run the next one. I want to start avoiding this using the async keyword and the asyncio library in general.
Now, in python you cannot use the await keyword unless you're in an async function. This forces you to propagate the async prefix to every function that calls on an async function, up until the main function at the top layer. the only way to avoid it that I know is to use asyncio.run. However, in all of the tutorials I saw the only place this function is used is when calling the main function, which doesn't help me avoid this propagation.
I was wondering if there is a real reason not to use asyncio.run in non-main function and avoid making all of my functions async for no reason.
Would love to know the answer!
I'm not very experienced in Python asyncio, although synchronous Python is going well.
I have a function, which creates a task list, and another function which is to be called with tasks in this list:
import asyncio
async def createTasks(some_dict):
coroutines = []
# some_dict can have an arbitrary number of items
for item in some_dict:
coroutines.append(executeOneTask(item))
tasks = await asyncio.gather(*coroutines, return_exceptions=True)
return tasks
async def executeOneTask(item):
# execute a task asynchronously
return
Here's the part where you are free to correct me if I'm wrong.
Now, my understanding of asyncio is that I need an event loop to execute an asynchronous function, which means that to asyncio.gather I need to await it that means this needs to happen inside an async function. OK, so I need an event loop to create my list of asynchronous tasks that I actually want to execute asynchronously.
If my understanding of event loops is correct, I cannot easily add tasks inside an event loop to that same event loop. Let's assume that I have an asynchronous main() function which is supposed to first retrieve a list of asynchronous tasks using createTasks() and then create an amount (equal to list length) of asynchronous tasks to be run by utilizing executeOneTask().
How should I approach the construction of such a main() function? Do I need multiple event loops? Can I create a task list some other way, which enables easier code?
Side note: The approach I have set up here might be a very difficult or upside-down way to solve the problem. What I aim to do is to create a list of asynchronous tasks and then run those tasks asynchronously. Feel free to not follow the code structure above if a smart solution requires that.
Thanks!
You should only use one event loop in the entire application. Start the main function by asyncio.run(main()) and asyncio creates a loop for you. With Python 3.8 you rarely need to access or use the loop directly but with older versions you may obtain it by asyncio.get_event_loop() if using loop methods or some functions that require loop argument.
Do note that IPython (used in Spyder and Jupyter) also runs its own loop, so in those you can directly call and await without calling asyncio.run.
If you only wish to do async programming but don't specifically need to work with asyncio, I would recommend checking out https://trio.readthedocs.io/ which basically does the same things but is much, much easier to use (correctly).
I want to write a library that mixes synchronous and asynchronous work, like:
def do_things():
# 1) do sync things
# 2) launch a bunch of slow async tasks and block until they are all complete or an exception is thrown
# 3) more sync work
# ...
I started implementing this using asyncio as an excuse to learn the learn the library, but as I learn more it seems like this may be the wrong approach. My problem is that there doesn't seem to be a clean way to do 2, because it depends on the context of the caller. For example:
I can't use asyncio.run(), because the caller could already have a running event loop and you can only have one loop per thread.
Marking do_things as async is too heavy because it shouldn't require the caller to be async. Plus, if do_things was async, calling synchronous code (1 & 3) from an async function seems to be bad practice.
asyncio.get_event_loop() also seems wrong, because it may create a new loop, which if left running would prevent the caller from creating their own loop after calling do_things (though arguably they shouldn't do that). And based on the documentation of loop.close, it looks like starting/stopping multiple loops in a single thread won't work.
Basically it seems like if I want to use asyncio at all, I am forced to use it for the entire lifetime of the program, and therefore all libraries like this one have to be written as either 100% synchronous or 100% asynchronous. But the behavior I want is: Use the current event loop if one is running, otherwise create a temporary one just for the scope of 2, and don't break client code in doing so. Does something like this exist, or is asyncio the wrong choice?
I can't use asyncio.run(), because the caller could already have a running event loop and you can only have one loop per thread.
If the caller has a running event loop, you shouldn't run blocking code in the first place because it will block the caller's loop!
With that in mind, your best option is to indeed make do_things async and call sync code using run_in_executor which is designed precisely for that use case:
async def do_things():
loop = asyncio.get_event_loop()
await loop.run_in_executor(None, sync_stuff)
await async_func()
await loop.run_in_executor(None, more_sync_stuff)
This version of do_things is usable from async code as await do_things() and from sync code as asyncio.run(do_things()).
Having said that... if you know that the sync code will run very briefly, or you are for some reason willing to block the caller's event loop, you can work around the limitation by starting an event loop in a separate thread:
def run_async(aw):
result = None
async def run_and_store_result():
nonlocal result
result = await aw
t = threading.Thread(target=asyncio.run, args=(run_and_store_result(),))
t.start()
t.join()
return result
do_things can then look like this:
async def do_things():
sync_stuff()
run_async(async_func())
more_sync_stuff()
It will be callable from both sync and async code, but the cost will be that:
it will create a brand new event loop each and every time. (Though you can cache the event loop and never exit it.)
when called from async code, it will block the caller's event loop, thus effectively breaking its asyncio usage, even if most time is actually spent inside its own async code.
There's one very particular thing about asyncio that I can't seem to understand. And that is the difference between asynchronous and synchronous callbacks. Let me introduce some examples.
Asyncio TCP example:
class EchoServer(asyncio.Protocol):
def connection_made(self, transport):
print('connection made')
def data_received(self, data):
print('data received: ', data.decode())
def eof_received(self):
pass
def connection_lost(self, exc):
print('connection lost:', exc)
Aiohttp example:
async def simple(request):
return Response(text="Simple answer")
async def init(loop):
app = Application(loop=loop)
app.router.add_get('/simple', simple)
return app
loop = asyncio.get_event_loop()
app = loop.run_until_complete(init(loop))
run_app(app, loop=loop)
Those two examples are very similar in functionality but they seem to be both doing it in different way. In the first example, if you want to be notified on some action, you specify a synchronous function (EchoServer.connection_made). However, in the second example, if you want to be notified on some action, you have to define an asynchronous callback function (simple).
I would like to ask what is the difference between these two types of callback. I understand the difference between regular functions and asynchronous functions, but I cannot wrap my head around the callback difference. For example, if I would want to write an asynchrnonous API like aiohttp is, and I would have a function that would do something and call a callback after that, how would I decide if I should demand an asynchronous function to be passed as an argument or just a regular synchronous one?
In aiohttp example you could do asynchronous calls from simple web-handler: access to database, make http requests etc.
In Protocol.data_received() you should call only regular synchronous methods.
UPD
Protocol callbacks are supposed to be synchronous by design.
They are very low-level bridge between sync and async.
You may call async code from them but it requires really tricky coding.
User level asyncio API for sockets etc. are streams: https://docs.python.org/3/library/asyncio-stream.html
When you introduce your own callback system must likely you need asynchronous callback unless you are %100 sure why the callback will never want to call async code.
Regular functions (def) and coroutines (async def) have a different signatures. It's hard to change a required signature, especially if your code has published as a library and you cannot control all users of your callback.
P.S.
The same is true for any public API methods.
The hardest lesson I've learned during development of my libraries is: .close() method should be a coroutine even initially it calls sync functions only, e.g. socket.close().
But later you perhaps will want to add a graceful shutdown which requires a waiting for current activity finishing and so on.
If your users had call your api as obj.close() now they should use await obj.close() but it's backward incompatible change!