According to https://realpython.com/async-io-python/,
The keyword await passes function control back to the event loop. (It suspends the execution of the surrounding coroutine.) If Python encounters an await f() expression in the scope of g(), this is how await tells the event loop, “Suspend execution of g() until whatever I’m waiting on—the result of f()—is returned. In the meantime, go let something else run.”
import asyncio
import time
async def spend_time(name):
print(f"spending time for {name}")
for i in range(5):
time.sleep(1)
print(f"{name} sleep {i}")
async def first():
print("first task...")
await spend_time("first")
print("first task finished")
async def second():
print("second task...")
await spend_time("second")
print("second task finished")
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(first(), second()))
For code above, I expect that after printing "first task", it will give control to another coroutines and print "second task", because it await, but it's not.
first task...
spending time for first
first sleep 0
first sleep 1
first sleep 2
first sleep 3
first sleep 4
first task finished
second task...
spending time for second
second sleep 0
second sleep 1
second sleep 2
second sleep 3
second sleep 4
second task finished
Why doesn't it suspend the execution of first task?
time.sleep is a blocking call (it doesn't return the control back to the event loop), you should use asyncio.sleep instead:
import asyncio
async def spend_time(name):
print(f"spending time for {name}")
for i in range(5):
await asyncio.sleep(1) # awaiting, returning the control
print(f"{name} sleep {i}")
async def first():
print("first task...")
await spend_time("first")
print("first task finished")
async def second():
print("second task...")
await spend_time("second")
print("second task finished")
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(first(), second()))
loop.close()
Output:
first task...
spending time for first
second task...
spending time for second
first sleep 0
second sleep 0
first sleep 1
second sleep 1
first sleep 2
second sleep 2
first sleep 3
second sleep 3
first sleep 4
first task finished
second sleep 4
second task finished
Related
I recently started studying threads in python, and I ran into this problem: I need the "two" function to finish executing after executing the function one in the thread, but the join method does not work, apparently because of the while true loop in the third function. I tried using queue, but it didn't work either.
the code itself:
from threading import Thread,Event
def one():
event.set()
thr.join()
for i in range(3):
print('some print')
time.sleep(1)
def two():
t = Thread(target=one)
t.start()
#with t.join() here the program does not work at all, same thing with event.set()
print('func two finished')
def three(callback, event):
c = 0
while True:
c += 1
time.sleep(1)
print('func 3 is working')
if c == 5:
two()
if event.is_set():
callback(c)
print('func 3 is stopped')
break
def callback(t):
print('callback ',t)
def thread(callback):
global event, thr
event = Event()
thr = Thread(target=three, args=(callback, event,))
thr.start()
thr.join()
thread(callback)
current output:
func 3 is working
func 3 is working
func 3 is working
func 3 is working
func 3 is working
func two finished
callback 5
func 3 is stopped
some print
some print
some print
expected:
func 3 is working
func 3 is working
func 3 is working
func 3 is working
func 3 is working
callback 5
func 3 is stopped
some print
some print
some print
func two finished
After running the code I understand by "not working" you mean the program finished before all prints are printed.
The reason is that you join the thr thread twice, one of them by the main thread.
The sequence of return of join is not guaranteed.
When the main thread finished, all threads created by the program also finish, so they terminated no matter what.
Same thing when setting the event, it makes the main thread exit and kill the remaining threads.
To do what you intend, you should wait for the one thread in the main thread.
from threading import Thread,Event
def one():
event.set()
thr.join()
for i in range(3):
print('some print')
time.sleep(1)
def two():
t = Thread(target=one)
t.start()
#with t.join() here the program does not work at all, same thing with event.set()
print('func two finished')
def three(callback, event):
c = 0
while True:
c += 1
time.sleep(1)
print('func 3 is working')
if c == 5:
two()
if event.is_set():
callback(c)
print('func 3 is stopped')
break
def callback(t):
print('callback ',t)
def thread(callback):
global event, thr
event = Event()
thr = Thread(target=three, args=(callback, event,))
thr.start()
thr.join()
thread(callback)
Note that as other said, this might be nice for learning purpesses but has to be modified if you want to actually use it in real life code.
Your program creates a deadlock if you un-comment that t.join() call in function two;
The thr thread cannot finish until after the t thread has finished because the thr thread calls t.join() in function two.
The t thread cannot finish until after the thr thread has finished because the t thread calls thr.join() in function one.
Neither thread can finish until after the other thread finishes. Therefore, neither thread can ever finish.
Why does one join the thr thread?
def one():
event.set()
thr.join() # What were you trying to do here?
for i in range(3):
print('some print')
time.sleep(1)
Your program will give the output you wanted if you comment out that join call, and uncomment the t.join() call in function two.
The sequence you need is obtained by a small permutation of commands. But it is not clear why you need threads if everything is done sequentially.
from threading import Thread, Event
import time
def one(event):
event.set()
for i in range(3):
print('some print')
time.sleep(1)
def two(event):
t = Thread(target=one, args=(event,))
t.start()
t.join()
print('func two finished')
def three(event):
c = 0
while True:
c += 1
time.sleep(1)
print('func 3 is working')
if c == 5:
callback(c)
print('func 3 is stopped')
two(event)
break
def callback(t):
print('callback ', t)
def thread():
event = Event()
thr = Thread(target=three, args=(event,))
thr.start()
thread()
--------------------------------
func 3 is working
func 3 is working
func 3 is working
func 3 is working
func 3 is working
callback 5
func 3 is stopped
some print
some print
some print
func two finished
This is a comment, not an answer.
This makes no sense:
t = Thread(target=one, args=(event,))
t.start()
t.join()
There's no point in starting a new thread if you aren't going to do something concurrently with the thread. Either do this,
t = Thread(target=one, args=(event,))
t.start()
do_something_else_concurrently_with_thread_t(...)
t.join()
Or just just call the function instead of creating a new thread to call it:
one(event)
If you don't want concurrency, then you don't need threads.
I can't figure how to stop loop after one task is finished. In sample when WsServe count to 5 I expect loop to close. But instead stop I got RuntimeError: Cannot close a running event loop
#!/usr/bin/env python
import asyncio
async def rxer():
i=0
while True:
i+=1
print ('Rxer ',i)
await asyncio.sleep(1)
async def WsServe():
for i in range(5):
print ('WsServe',i)
await asyncio.sleep(1)
print ('Finish')
loop.stop()
loop.close()
loop=asyncio.get_event_loop()
loop.create_task(rxer())
loop.run_until_complete(WsServe())
loop.run_forever()
The error comes from calling loop.close() from inside the loop. You don't need to bother with loop.close(), loop.stop() is quite sufficient to stop the loop. loop.close() is only relevant when you want to ensure that all the resources internally acquired by the loop are released. It is not needed when your process is about to exit anyway, and removing the call to loop.close() indeed eliminates the error.
But also, loop.stop() is incompatible with run_until_complete(). It happens to work in this code because the coroutine returns immediately after calling loop.stop(); if you added e.g. an await asyncio.sleep(1) after loop.stop(), you'd again get a (different) RuntimeError.
To avoid such issues, I suggest that you migrate to the newer asyncio.run API and avoid both run_until_complete and stop. Instead, you can just use an event to terminate the main function, and the loop with it:
# rxer() defined as before
async def WsServe(stop_event):
for i in range(5):
print ('WsServe',i)
await asyncio.sleep(1)
print ('Finish')
stop_event.set()
await asyncio.sleep(1)
async def main():
asyncio.get_event_loop().create_task(rxer())
stop_event = asyncio.Event()
asyncio.get_event_loop().create_task(WsServe(stop_event))
await stop_event.wait()
asyncio.run(main())
# python 3.6 and older:
#asyncio.get_event_loop().run_until_complete(main())
Check commented lines of your implementation as below:
import asyncio
async def rxer():
i=0
while True:
i+=1
print ('Rxer ',i)
await asyncio.sleep(1)
async def WsServe():
for i in range(5):
print ('WsServe',i)
await asyncio.sleep(1)
print ('Finish')
#loop.stop()
#loop.close()
loop=asyncio.get_event_loop()
loop.create_task(rxer())
loop.run_until_complete(WsServe())
#loop.run_forever()
And here is the output:
Rxer 1
WsServe 0
Rxer 2
WsServe 1
Rxer 3
WsServe 2
Rxer 4
WsServe 3
Rxer 5
WsServe 4
Rxer 6
Finish
I'm using python 3.6 and trying to use asyncio to run tasks concurrently. I thought asyncio.gather and ensure future would be the tools to use, but it does not seem to be working as I thought it would. Could someone give me pointers?
Here is my code:
import time
import asyncio
async def f1():
print('Running func 1')
time.sleep(4)
print('Returning from func 1')
return 1
async def f2():
print('Running func 2')
time.sleep(6)
print('Returning from func 2')
return 2
async def f3():
print('Running func 3')
time.sleep(1)
print('Returning from func 3')
return 3
async def foo():
calls = [
asyncio.ensure_future(f())
for f in [f1, f2, f3]
]
res = await asyncio.gather(*calls)
print(res)
loop = asyncio.get_event_loop()
start = time.time()
loop.run_until_complete(foo())
end = time.time()
print(f'Took {end - start} seconds')
print('done')
I would expect the 3 functions to run independently of each other, but each one seems to be blocked behind the other. This is the ouput I get
Running func 1
Returning from func 1
Running func 2
Returning from func 2
Running func 3
Returning from func 3
[1, 2, 3]
Took 11.009816884994507 seconds
done
I would have expected it to take 6 seconds, with the bottleneck being f2.
First off, welcome to StackOverflow.
When you run code inside event loop, this code MUST use async libraries or be run in executor if you want not to block the entire process.
In this way, event loop can send the task background to be executed in a worker or in the event loop itself in case you use async libraries. Meanwhile, event loop can attend new function o portion of code and repeat the same process.
Once any background task has finished, event loop catch them and return its value.
In your case, if you use async library of sleep you should obtain expected results. For example:
async def f1():
print('Running func 1')
await asyncio.sleep(4)
print('Returning from func 1')
return 1
I didn't read this tutorial, but I hope it should be interesting for finding your solution.
I'm trying to time, and record in the program, how long it takes to complete a single iteration of an async generator + for loop, without resorting to global state. For example, given
import asyncio
async def run():
async for page in pull():
await push(page)
async def pull():
for i in range(0, 3):
print(f'Start of iteration {i}')
await asyncio.sleep(1)
yield i
async def push(i):
await asyncio.sleep(1)
print(f'End of iteration {i}')
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
that outputs
Start of iteration 0
End of iteration 0
Start of iteration 1
End of iteration 1
Start of iteration 2
End of iteration 2
I would like to record the time between each Start of iteration i and the next End of iteration i.
How can this be done? Ideally without global state, and ideally where the timing code is somewhat decoupled from the rest (e.g. with decorators)
Instead of passing around the i as an integer, you can pass a structure with some state. Here is an example of augmenting the i with some timing information:
Timing info:
class TimerInt(int):
def __new__(cls, *args, **kwargs):
my_int = super(TimerInt, cls).__new__(cls, *args, **kwargs)
my_int._start_time = time.time()
return my_int
#property
def time_since_create(self):
return time.time() - self._start_time
This gives the int a time_since_create property that can be used for some crude timing.
Test Code:
import asyncio
import time
async def run():
async for page in pull():
await push(page)
async def pull():
for i in range(0, 3):
i = TimerInt(i)
print(f'Start of iteration {i}')
await asyncio.sleep(1)
yield i
async def push(i):
await asyncio.sleep(1)
print(f'End of iteration {i}: {i.time_since_create:.2f}')
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
Test Results:
Start of iteration 0
End of iteration 0: 2.01
Start of iteration 1
End of iteration 1: 2.00
Start of iteration 2
End of iteration 2: 2.00
If you just care about the costed time, maybe you can calculate them in run. There is an example, I think it should work as if we just look at this async for loop, the process is lineal.
At first, end - start should equal to the time from the first iteration of pull() to the end of push(page). And then use the end time as the second start time, so this time end - start should equal to the time from the second iteration of pull() to the end of push(page) again.
async def run():
start = time.time()
async for page in pull():
await push(page)
end = time.time()
print("Cost {end - start} seconds")
start = end
If I am wrong, correct me please.
The best solution I've found so far is to just use the sleep() function. I'd like to run my own callback function when the event of a timer expiration happens. Is there any event-driven way to go about it?
from time import sleep
# Sleep for a minute
time.sleep(60)
There's a built-in simple solution, using the threading module:
import threading
timer = threading.Timer(60.0, callback)
timer.start() # after 60 seconds, 'callback' will be called
## (in the meanwhile you can do other stuff...)
You can also pass args and kwargs to your callback. See here.
I think it could be really simple. Take a look at this example. It works even in a python console!
from threading import Thread
from time import sleep
# Function to be called when the timer expires
def myFunction():
print 'Did anyone call me?'
# Function with the timer
def myTimer(seconds):
sleep(seconds)
myFunction()
# Thread that will sleep in background and call your function
# when the timer expires.
myThread = Thread(target=myTimer, args=(4,))
myThread.start()
Put whatever amount of seconds you want, and keep working with the console or running the main thread/programm. You will notice that the function will be called when the timer comes to an end.
Edit
Another good example, considering the comment from #tarabyte is the one where the function is called only depending on the value of some variable or flag. I hope this would then be the answer #tarabyte is looking for.
from threading import Thread
from time import sleep
myFlag = False
# Function to be called when the flag turns on
def myFunction():
print 'Did anyone call me?'
def myTimer():
global myFlag
while True:
if myFlag:
myFunction()
myFlag = False
else:
sleep(1)
# Thread that will sleep in background and call your function
# when the myFlag turns to be True
myThread = Thread(target=myTimer)
myThread.start()
# Then, you can do whatever you want and later change the value of myFlag.
# Take a look at the output inside ipython when the value of myFlag is changed.
In [35]: myFlag
Out[35]: False
In [36]: myFlag = True
In [37]: Did anyone call me?
Sometimes a simple solution is best, even if it polls the time. I have used this to great success before - it doesn't block if your thread doesn't stop on it.
I think I would manage this most simply by checking times, since this is so much more simple and resource economical than working out a separate threaded solution:
def event_minute_later(event):
print(time.time()) # use for testing, comment out or delete for production
return event + 60 < time.time()
And usage:
>>> event = time.time()
>>> print(event)
1393962502.62
>>> event_minute_later(event)
1393962526.73
False
>>> event_minute_later(event)
1393962562.9
True
Since Python 3.7 (and older versions have reached end of life by now) the asyncio built-in module lets you add a Python sleep() call asynchronously:
import asyncio
async def test():
print("Hello ... but wait, there is more!")
await asyncio.sleep(3)
print("... in the async world!")
Here's some proof that it is non-blocking (courtesy of RealPython):
import asyncio
# Jupyter Notebook users need to allow
# nesting of the asyncio event loop
import nest_asyncio
nest_asyncio.apply()
import time
async def workload(text, duration):
while duration > 0:
# run sleep and yield control
# back to the event loop (for one cycle)
await asyncio.sleep(1)
print(f'{text} counter: sleeping {duration} seconds')
duration -= 1
async def main():
# send the workload() coroutine to the background,
# to let it run concurrently with other tasks,
# switching between them at await points
task_1 = asyncio.create_task(workload('First', 2))
task_2 = asyncio.create_task(workload('Second', 4))
task_3 = asyncio.create_task(workload('Third', 8))
print(f"Started: {time.strftime('%X')}")
# create await points for each
# of the concurrent tasks
await task_1
await task_2
await task_3
print(f"Ended: {time.strftime('%X')}")
if __name__ == '__main__':
asyncio.run(main())
Output:
Started: 09:07:21
First counter: sleeping 2 seconds
Second counter: sleeping 4 seconds
Third counter: sleeping 8 seconds
First counter: sleeping 1 seconds
Second counter: sleeping 3 seconds
Third counter: sleeping 7 seconds
Second counter: sleeping 2 seconds
Third counter: sleeping 6 seconds
Second counter: sleeping 1 seconds
Third counter: sleeping 5 seconds
Third counter: sleeping 4 seconds
Third counter: sleeping 3 seconds
Third counter: sleeping 2 seconds
Third counter: sleeping 1 seconds
Ended: 09:07:29