I am pretty new with Async.io and I am using it with Discord.py to create a bot. Once a day, I need to update a spreadsheet, but the problem is that the spreadsheet has gotten a little long so it now triggers the loop's default timeout. Is there anyway to overcome this? I have seen run_until_complete but as you see below there is a await asyncio.sleep(86400) which from my understanding will not work with wait until complete because it will wait for a day? I would also be fine with just changing the timeout for that function and then changing it back after it is complete, but I have not been able to find any resources.
Here is the function that needs to repeat everyday:
async def updateSheet():
while True:
print("Updating Sheet at " + datetime.now().strftime("%H:%M"))
user.updateAllUsers(os.getenv('CID'), os.getenv('CS'), subs) #This is the function that takes too long
print("Done Updating")
await asyncio.sleep(86400)
and here is how I am adding it to the loop (because I am using Discord.py):
#client.event
async def on_ready():
print('We have logged in as {0.user}'.format(client))
client.loop.create_task(updateSheet())
Any and all help will be appreciated since as long as this is down my project loses precious time. :)
If something is blocking, the direct method would be trying to convert it to a task, which might not be possible in your case. So we would have to use something like APS to schedule jobs.
sched = Scheduler()
sched.start()
#sched.cron_schedule(day='mon-fri')
def task():
user.updateAllUsers(os.getenv('CID'), os.getenv('CS'), subs)
Make sure you do this in a separate file, and use async scheduler for tasks.
You can simply measure how much time does the function take to execute and simply subtract it from 86400
import time
async def updateSheet():
while True:
start = time.monotonic()
print("Updating Sheet at " + datetime.now().strftime("%H:%M"))
user.updateAllUsers(os.getenv('CID'), os.getenv('CS'), subs) #This is the function that takes too long
end = time.monotonic()
total = end - start
sleep_time = 86400 - total
await asyncio.sleep(sleep_time)
I really suggest you that you run the blocking functions in a non-blocking way, refer to one of my previous answers for more info, (What does "blocking" mean)
Related
I need to have two function in my Python 3.11 code.
One function must be sync, it retrive some data from a local machine so i need to wait to finish.
Another function must be async, it get the data from the first function and send to the server. Since i don't know how many time can be (5 seconds to 30 seconds) this function must doesn't interrupt the first one
Pratically, the second function start always when the first finish but the first always start and don't care about the second one. This code run H24
My attempt:
import time
import asyncio
async def task1():
print("Recover data... waiting")
time.sleep(3)
print("End data recover")
return "slow"
async def task2(p):
print("I'm so" + p)
time.sleep(10)
print("END--->")
async def main():
while True:
print("create task1 and wait to finish")
x = await task1()
print("create task2 and not wait to finishing")
asyncio.create_task(task2(x))
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.create_task(main())
loop.run_forever()
I dont' need to use asyncio like a requirement, i just want to meet the goal without run out all the memory of the machine. Thanks
basically, "yes". Like in: that is the way you do it. If you need some task to complete before goind on with code in a given place, the thing to do is to wait for that data - if it is in an async func, then you use the await keyword, just as depicted in the code above.
If there are other tasks in parallel to calling main, then it would be nice if the code in task1 would return execution to the asyncio.loop while it waits for its result - That code could run in another thread, with the use of await asyncio.run_in_executor(sync_call, args) or simply await asyncio.sleep(...)`
I have a function that make a post request with a lot of treatment. All of that takes 30 seconds.
I need to execute this function every 6 mins. So I used asyncio for that ... But it's not asynchrone my api is blocked since the end of function ... Later I will have treatment that takes 5 minutes to execute.
def update_all():
# do request and treatment (30 secs)
async run_update_all():
while True:
await asyncio.sleep(6 * 60)
update_all()
loop = asyncio.get_event_loop()
loop.create_task(run_update_all())
So, I don't understand why during the execute time of update_all() all requests comming are in pending, waiting for the end of update_all() instead of being asynchronous
I found an answer with the indication of larsks
I did that :
def update_all():
# Do synchrone post request and treatment that take long time
async def launch_async():
loop = asyncio.get_event_loop()
while True:
await asyncio.sleep(120)
loop.run_in_executore(None, update_all)
asyncio.create_task(launch_async())
With that code I'm able to launch a synchrone function every X seconds without blocking the main thread of FastApi :D
I hope that will help other people in the same case than me.
My discord bot has command that makes the bot it say "I'm running" every 30 seconds. However that makes the main thread sleep making the bot commands unusable.
the code is as follows:
while true:
await channel.send("I'm running")
time.sleep(30)
I tried using https://github.com/Rapptz/discord.py/issues/82 and https://docs.python.org/3/library/asyncio-task.html#asyncio.run_coroutine_threadsafe as a reference, but I couldn't understand how it worked.
You should use some non-blocking sleep. Instead of using time.sleep you could for example try await asyncio.sleep(30) Maybe it works straight?
If not, then you must implement manual timer to call channel.send after certain time has passed with if statements for example. Asynchronous methods might be confusing at first.
Examples about threading you checked, are outdated, since discord.py uses async methods nowadays.
If its purpose is only to send a certain message every 30 seconds. You could also create a task which runs every 30 seconds. You can start/stop the task anytime you want.
Example:
class example_class():
def __init__():
self.channel = None
async def some_command_starting_our_task(ctx):
self.channel = ctx.channel
botStatus.start() # Used to start the task
async def some_command_stopping_our_task():
botStatus.stop() # Used to stop the task
#task.loop(seconds=30)
async def botStatus():
self.channel.send("Im running")
For more information about tasks you can read the documentation.
In one of my projects, I need to run three different database updater functions at different intervals.
For instance, function one needs to run every 30 seconds, function two needs to run every 60 seconds and function 3 every 5 minutes (notably due to API call restrictions).
I've been trying to achieve this in python, looking up every possible solution but I cannot seem to find anything that works for my use case. I am rather fresh in python.
Here is (somewhat) what I have, using asyncio.
import asyncio
def updater1(url1, url2, time):
print(f"Doing my thing here every {time} seconds")
def updater2(url1, url2, time):
print(f"Doing my thing here every {time} seconds")
def updater3(url, time):
print(f"Doing my thing here every {time} seconds")
async def func1():
updater1(rankUrl, statsUrl, 30)
await asyncio.sleep(30)
async def func2():
updater2(rankUrl, statsUrl, 60)
await asyncio.sleep(60)
async def func3():
updater3(url, 300)
await asyncio.sleep(300)
# Initiate async loops
while True:
asyncio.run(func1())
asyncio.run(func2())
asyncio.run(func3())
The issue is that these tasks run one after each other, while what I am trying to achieve is that they run independently from each other, with a start time when the script is initiated, and respective to their own individual loop times
Any idea on how this could be done is much appreciated - I am open to new concepts and ideas if you have any for me to explore :)
Don't use asyncio.run() on individual coroutines, as async.run() is itself not asynchronous. The call to asyncio.run() won't return until the funcN() coroutine is done.
Create a single top-level coroutine that then runs others as tasks:
async def main():
task1 = asyncio.create_task(func1())
task2 = asyncio.create_task(func2())
task3 = asyncio.create_task(func3())
await asyncio.wait([task1, task2, task3])
The above kicks off three independent tasks, then waits for all 3 to complete.
If I have some function, which does a lot of calculations, and it can take a while, is it good to use asyncio.sleep() between the parts of calculations to release event loop (to prevent blocking event loop)?
import asyncio
async def long_function(a, b, c):
# some calculations
await asyncio.sleep(0) # release event loop
# some another calculations
await asyncio.sleep(0) # release event loop
Is there another, more better way to solve such problems? Some best practices, maybe?
TL;DR just use loop.run_in_executor to do blocking work
To understand why it doesn't help, let's first make a class that does something with the event loop. Like:
class CounterTask(object):
def __init__(self):
self.total = 0
async def count(self):
while True:
try:
self.total += 1
await asyncio.sleep(0.001) # Count ~1000 times a second
except asyncio.CancelledError:
return
This will simply count around 1000 times a second, if the event loop is completely open to it.
Naive
Just to demonstrate the worst way, let's start the counter task and naively run an expensive function without any thought to the consequences:
async def long_function1():
time.sleep(0.2) # some calculations
async def no_awaiting():
counter = CounterTask()
task = asyncio.create_task(counter.count())
await long_function1()
task.cancel()
print("Counted to:", counter.total)
asyncio.run(no_awaiting())
Output:
Counted to: 0
Well that didn't do any counting! Notice, we never awaited at all. This function is just doing synchronous blocking work. If the counter was able to run in the event loop by itself we should have counted to about 200 in that time. Hmm, so maybe if we split it up and leverage asyncio to give control back to the event loop it can count? Let's try that...
Splitting it up
async def long_function2():
time.sleep(0.1) # some calculations
await asyncio.sleep(0) # release event loop
time.sleep(0.1) # some another calculations
await asyncio.sleep(0) # release event loop
async def with_awaiting():
counter = CounterTask()
task = asyncio.create_task(counter.count())
await long_function2()
task.cancel()
print("Counted to:", counter.total)
asyncio.run(with_awaiting())
Output:
Counted to: 1
Well I guess that's technically better. But ultimately this shows the point: The asyncio event loop shouldn't do any blocking processing. It is not intended to solve those issues. The event loop is helplessly waiting for your next await. But the run_in_executor does provide a solution for this, while keeping our code in the asyncio style.
Executor
def long_function3():
time.sleep(0.2) # some calculations
async def in_executor():
counter = CounterTask()
task = asyncio.create_task(counter.count())
await asyncio.get_running_loop().run_in_executor(None, long_function3)
task.cancel()
print("Counted to:", counter.total)
asyncio.run(in_executor())
Output:
Counted to: 164
Much better! Our loop was able to continue going while our blocking function was doing things as well, by the good old-fashion way of threads.