My code execution does not reach the print statement: print("I want to display after MyClass has started")
Why is this? I thought the purpose of await asyncio.sleep() is to unblock execution of code so that subsequent lines of code can run. Is that not the case?
import asyncio
class MyClass:
def __init__(self):
self.input = False
asyncio.run(self.start())
print("I want to display after MyClass has started") #This line is never reached.
async def start(self):
while True:
print("Changing state...")
if self.input:
print("I am on.")
break
await asyncio.sleep(1)
m = MyClass()
m.input = True #This line is never reached! Why?
print("I want to display after MyClass is started")
When I execute, it keeps printing "Changing state...". Even when I ctrl+c to quit, the execution continues as shown below. How can I properly terminate the execution? Sorry, I am new to python.
EDIT:
I appreciate the common use of asyncio is for running two or more separate functions asynchronously. However, my class is one which will be responding to changes in its state. For example, I intend to write code in the setters to do stuff when the class objects attributes change -WHILE still having a while True event loop running in the background. Is there not any way to permit this? I have tried running the event loop in it's own thread. However, that thread then dominates and the class objects response times run into several seconds. This may be due to the GIL (Global Interpreter Lock) which we can do nothing about. I have also tried using multiprocessing, but then I lose access to the properties and methods of the object as parallel process run in their own memory spaces.
In the init method of MyClass you invoke asyncio.run() - this method will execute the required routine until that routine terminates. In your case, since the main method includes a while True loop, it will never terminate.
Here is a slight modification of your code that perhaps shows the concurrency effect you're after -
import asyncio
class MyClass:
def __init__(self):
self.input = False
asyncio.run(self.main())
print("I want to display after MyClass has been initialized.") # This line is never reached.
async def main(self):
work1 = self.work1()
work2 = self.work2()
await asyncio.gather(work1, work2)
async def work1(self):
for idx in range(5):
print('doing some work 1...')
await asyncio.sleep(1)
async def work2(self):
for idx in range(5):
print('doing some work 2...')
await asyncio.sleep(1)
m = MyClass()
print("I want to display after MyClass is terminated")
using: python 3.8 on windows 10
I am working on a script that runs in a while loop.
At the end of the loop I want it to wait for 5 seconds for the user to give input and then restart, or exit if they do not give input.
I've never actually used it before but I assumed asyncio would be useful because it allows for the definition of awaitables. However bringing things together is proving more difficult than I'd anticipated.
import keyboard, asyncio, time as t
async def get():
keyboard.record('enter', True)
return True
async def countdown(time):
while time > 0:
print(f'This program will exit in {int(time)} seconds. Press enter to start over.', end='\r')
await asyncio.sleep(1)
# t.sleep(1)
time -= 1
print(f'This program will exit in 0 seconds. Press enter to start over.', end='\r')
return False
async def main():
running = True
while running:
# other code
clock = asyncio.create_task(countdown(5))
check = asyncio.create_task(get())
done, pending = await asyncio.wait({clock, check}, return_when=asyncio.FIRST_COMPLETED)
running = next(iter(done)).result()
print(running, end='\r')
print('bye')
asyncio.run(main())
As it stands, the process doesn't end if I wait for five seconds. It also doesn't visibly count down (my best, and most ridiculous, guess is that it may be looping too fast in main? Try holding down the "enter" key - with and without printing running).
Also, when I switch to time.sleep, the display works fine, but it doesn't seem as though the countdown function ever returns.
Previously I'd tried using an input statement instead of keyboard.record; that blocks. I had also tried using asyncio.wait_for; but the timeout never came, though again, the "enter" key was registered (that said, it wouldn't have printed the countdown even if it had worked). I also tried asyncio.as_completed but I was unable to parse anything useful from the iterable. Happy to be wrong there though!
Also, bonus points if you can generalize the countdown to non-integer time spans <3
The wait_for approach:
async def main():
running = True
while running:
try:
running = await asyncio.wait_for(get(), 5)
except asyncio.TimeoutError:
running = False
print(running)
print('bye')
asyncio.run(main())
as_completed
async def main():
running = True
while running:
## other code
clock = asyncio.create_task(countdown(5))
check = asyncio.create_task(get())
for f in asyncio.as_completed({check, clock}):
# print(f.cr_frame)
# print(f.cr_await, f.cr_running, f.cr_origin, f.cr_frame, f.cr_code)
print(f.cr_await, f.cr_running, f.cr_origin)
print('bye')
Also, bonus points if you can generalize the countdown to non-integer time :P
cheers!
If you look in the docs for keyboard.record it says:
Note: this is a blocking function.
This is why your process didn't end after 5 seconds. It was blocking at keyboard.record('enter', True). If you are going to stick with the keyboard module, what you need to do is create a hook on the 'enter' key. I put a quick demo together with your code:
import asyncio
import keyboard
class Program:
def __init__(self):
self.enter_pressed = asyncio.Event()
self._loop = asyncio.get_event_loop()
async def get(self):
await self.enter_pressed.wait()
return True
#staticmethod
async def countdown(time):
while time > 0:
print(f'This program will exit in {int(time)} seconds. Press enter to start over.')
await asyncio.sleep(1)
# t.sleep(1)
time -= 1
print(f'This program will exit in 0 seconds. Press enter to start over.')
return False
def notify_enter(self, *args, **kwargs):
self._loop.call_soon_threadsafe(self.enter_pressed.set)
async def main(self):
running = True
while running:
# other code
keyboard.on_press_key('enter', self.notify_enter)
self.enter_pressed.clear()
clock = asyncio.create_task(self.countdown(5))
check = asyncio.create_task(self.get())
done, pending = await asyncio.wait({clock, check}, return_when=asyncio.FIRST_COMPLETED)
keyboard.unhook('enter')
for task in pending:
task.cancel()
running = next(iter(done)).result()
print(running)
print('bye')
async def main():
program = Program()
await program.main()
asyncio.run(main())
There is callback created notify_enter, that sets an asyncio.Event whenever it's fired. The get() task waits for this event to trigger before it exits. Since I didn't know what your other code is doing, we don't bind a hook to the enter key's key_down event until right before you await both tasks and we unbind it right after one of the tasks completes. I wrapped everything up in a class so the event is accessible in the callback, since there isn't a way to pass parameters in.
I have the following code:
import time
def wait10seconds():
for i in range(10):
time.sleep(1)
return 'Counted to 10!'
print(wait10seconds())
print('test')
Now my question is how do you make print('test') run before the function wait10seconds() is executed without exchanging the 2 lines.
I want the output to be the following:
test
Counted to 10!
Anyone know how to fix this?
You can use Threads for this
like:
from threading import Thread
my_thread = Thread(target=wait10seconds) # Create a new thread that exec the function
my_thread.start() # start it
print('test') # print the test
my_thread.join() # wait for the function to end
You can use a Timer. Taken from the Python docs page:
def hello():
print("hello, world")
t = Timer(30.0, hello)
t.start() # after 30 seconds, "hello, world" will be printed
if you are using python 3.5+ you can use asyncio:
import asyncio
async def wait10seconds():
for i in range(10):
await asyncio.sleep(1)
return 'Counted to 10!'
print(asyncio.run(wait10seconds()))
asyncio.run is new to python 3.7, for python 3.5 and 3.6 you won't be able to use asyncio.run but you can achieve the same thing by working with the event_loop directly
I need to create a Runner class that can asynchronously runs multiple jobs and return the results one all jobs are completed.
Here is the dummy code snippet which I already have
import asyncio
class Runner:
def __init__(self):
self.jobs_loop = asyncio.new_event_loop()
async def slow_process(self, *args):
... # run slow proces with args
await asyncio.sleep(5)
return "result"
def batch_run(self, count, *args):
results = self.jobs_loop.run_until_complete(asyncio.gather(
*[self.slow_process(*args) for _ in range(count)],
loop=self.jobs_loop,
return_exceptions=True
))
return results
runner = Runner()
print(runner.batch_run(10))
When the batch_run is called, it schedules count jobs and blocks until finished.
This, however, doesn't work when called from a running event loop raising
RuntimeError: Cannot run the event loop while another loop is running
How can I make batch_run work when called from existing event loop (e.g. Jupyter Console) as well as being callable from outside of the event loop.
EDIT
Ideally the function would look like this
def batch_run(self, count, *args):
loop = asyncio.get_event_loop()
jobs = asyncio.gather(...)
if not loop.is_running():
return loop.run_until_complete(jobs)
else:
future = asyncio.ensure_future(jobs, loop=loop)
# loop.wait_until_completed(future)
return future.result()
if only loop.wait_until_completed existed.
I was wondering if there's any library for asynchronous method calls in Python. It would be great if you could do something like
#async
def longComputation():
<code>
token = longComputation()
token.registerCallback(callback_function)
# alternative, polling
while not token.finished():
doSomethingElse()
if token.finished():
result = token.result()
Or to call a non-async routine asynchronously
def longComputation()
<code>
token = asynccall(longComputation())
It would be great to have a more refined strategy as native in the language core. Was this considered?
Something like:
import threading
thr = threading.Thread(target=foo, args=(), kwargs={})
thr.start() # Will run "foo"
....
thr.is_alive() # Will return whether foo is running currently
....
thr.join() # Will wait till "foo" is done
See the documentation at https://docs.python.org/library/threading.html for more details.
You can use the multiprocessing module added in Python 2.6. You can use pools of processes and then get results asynchronously with:
apply_async(func[, args[, kwds[, callback]]])
E.g.:
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
pool = Pool(processes=1) # Start a worker processes.
result = pool.apply_async(f, [10], callback) # Evaluate "f(10)" asynchronously calling callback when finished.
This is only one alternative. This module provides lots of facilities to achieve what you want. Also it will be really easy to make a decorator from this.
As of Python 3.5, you can use enhanced generators for async functions.
import asyncio
import datetime
Enhanced generator syntax:
#asyncio.coroutine
def display_date(loop):
end_time = loop.time() + 5.0
while True:
print(datetime.datetime.now())
if (loop.time() + 1.0) >= end_time:
break
yield from asyncio.sleep(1)
loop = asyncio.get_event_loop()
# Blocking call which returns when the display_date() coroutine is done
loop.run_until_complete(display_date(loop))
loop.close()
New async/await syntax:
async def display_date(loop):
end_time = loop.time() + 5.0
while True:
print(datetime.datetime.now())
if (loop.time() + 1.0) >= end_time:
break
await asyncio.sleep(1)
loop = asyncio.get_event_loop()
# Blocking call which returns when the display_date() coroutine is done
loop.run_until_complete(display_date(loop))
loop.close()
It's not in the language core, but a very mature library that does what you want is Twisted. It introduces the Deferred object, which you can attach callbacks or error handlers ("errbacks") to. A Deferred is basically a "promise" that a function will have a result eventually.
You can implement a decorator to make your functions asynchronous, though that's a bit tricky. The multiprocessing module is full of little quirks and seemingly arbitrary restrictions – all the more reason to encapsulate it behind a friendly interface, though.
from inspect import getmodule
from multiprocessing import Pool
def async(decorated):
r'''Wraps a top-level function around an asynchronous dispatcher.
when the decorated function is called, a task is submitted to a
process pool, and a future object is returned, providing access to an
eventual return value.
The future object has a blocking get() method to access the task
result: it will return immediately if the job is already done, or block
until it completes.
This decorator won't work on methods, due to limitations in Python's
pickling machinery (in principle methods could be made pickleable, but
good luck on that).
'''
# Keeps the original function visible from the module global namespace,
# under a name consistent to its __name__ attribute. This is necessary for
# the multiprocessing pickling machinery to work properly.
module = getmodule(decorated)
decorated.__name__ += '_original'
setattr(module, decorated.__name__, decorated)
def send(*args, **opts):
return async.pool.apply_async(decorated, args, opts)
return send
The code below illustrates usage of the decorator:
#async
def printsum(uid, values):
summed = 0
for value in values:
summed += value
print("Worker %i: sum value is %i" % (uid, summed))
return (uid, summed)
if __name__ == '__main__':
from random import sample
# The process pool must be created inside __main__.
async.pool = Pool(4)
p = range(0, 1000)
results = []
for i in range(4):
result = printsum(i, sample(p, 100))
results.append(result)
for result in results:
print("Worker %i: sum value is %i" % result.get())
In a real-world case I would ellaborate a bit more on the decorator, providing some way to turn it off for debugging (while keeping the future interface in place), or maybe a facility for dealing with exceptions; but I think this demonstrates the principle well enough.
Just
import threading, time
def f():
print "f started"
time.sleep(3)
print "f finished"
threading.Thread(target=f).start()
My solution is:
import threading
class TimeoutError(RuntimeError):
pass
class AsyncCall(object):
def __init__(self, fnc, callback = None):
self.Callable = fnc
self.Callback = callback
def __call__(self, *args, **kwargs):
self.Thread = threading.Thread(target = self.run, name = self.Callable.__name__, args = args, kwargs = kwargs)
self.Thread.start()
return self
def wait(self, timeout = None):
self.Thread.join(timeout)
if self.Thread.isAlive():
raise TimeoutError()
else:
return self.Result
def run(self, *args, **kwargs):
self.Result = self.Callable(*args, **kwargs)
if self.Callback:
self.Callback(self.Result)
class AsyncMethod(object):
def __init__(self, fnc, callback=None):
self.Callable = fnc
self.Callback = callback
def __call__(self, *args, **kwargs):
return AsyncCall(self.Callable, self.Callback)(*args, **kwargs)
def Async(fnc = None, callback = None):
if fnc == None:
def AddAsyncCallback(fnc):
return AsyncMethod(fnc, callback)
return AddAsyncCallback
else:
return AsyncMethod(fnc, callback)
And works exactly as requested:
#Async
def fnc():
pass
You could use eventlet. It lets you write what appears to be synchronous code, but have it operate asynchronously over the network.
Here's an example of a super minimal crawler:
urls = ["http://www.google.com/intl/en_ALL/images/logo.gif",
"https://wiki.secondlife.com/w/images/secondlife.jpg",
"http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif"]
import eventlet
from eventlet.green import urllib2
def fetch(url):
return urllib2.urlopen(url).read()
pool = eventlet.GreenPool()
for body in pool.imap(fetch, urls):
print "got body", len(body)
Something like this works for me, you can then call the function, and it will dispatch itself onto a new thread.
from thread import start_new_thread
def dowork(asynchronous=True):
if asynchronous:
args = (False)
start_new_thread(dowork,args) #Call itself on a new thread.
else:
while True:
#do something...
time.sleep(60) #sleep for a minute
return
You can use concurrent.futures (added in Python 3.2).
import time
from concurrent.futures import ThreadPoolExecutor
def long_computation(duration):
for x in range(0, duration):
print(x)
time.sleep(1)
return duration * 2
print('Use polling')
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(long_computation, 5)
while not future.done():
print('waiting...')
time.sleep(0.5)
print(future.result())
print('Use callback')
executor = ThreadPoolExecutor(max_workers=1)
future = executor.submit(long_computation, 5)
future.add_done_callback(lambda f: print(f.result()))
print('waiting for callback')
executor.shutdown(False) # non-blocking
print('shutdown invoked')
The newer asyncio running method in Python 3.7 and later is using asyncio.run() instead of creating loop and calling loop.run_until_complete() as well as closing it:
import asyncio
import datetime
async def display_date(delay):
loop = asyncio.get_running_loop()
end_time = loop.time() + delay
while True:
print("Blocking...", datetime.datetime.now())
await asyncio.sleep(1)
if loop.time() > end_time:
print("Done.")
break
asyncio.run(display_date(5))
Is there any reason not to use threads? You can use the threading class.
Instead of finished() function use the isAlive(). The result() function could join() the thread and retrieve the result. And, if you can, override the run() and __init__ functions to call the function specified in the constructor and save the value somewhere to the instance of the class.
The native Python way for asynchronous calls in 2021 with Python 3.9 suitable also for Jupyter / Ipython Kernel
Camabeh's answer is the way to go since Python 3.3.
async def display_date(loop):
end_time = loop.time() + 5.0
while True:
print(datetime.datetime.now())
if (loop.time() + 1.0) >= end_time:
break
await asyncio.sleep(1)
loop = asyncio.get_event_loop()
# Blocking call which returns when the display_date() coroutine is done
loop.run_until_complete(display_date(loop))
loop.close()
This will work in Jupyter Notebook / Jupyter Lab but throw an error:
RuntimeError: This event loop is already running
Due to Ipython's usage of event loops we need something called nested asynchronous loops which is not yet implemented in Python. Luckily there is nest_asyncio to deal with the issue. All you need to do is:
!pip install nest_asyncio # use ! within Jupyter Notebook, else pip install in shell
import nest_asyncio
nest_asyncio.apply()
(Based on this thread)
Only when you call loop.close() it throws another error as it probably refers to Ipython's main loop.
RuntimeError: Cannot close a running event loop
I'll update this answer as soon as someone answered to this github issue.
You can use process. If you want to run it forever use while (like networking) in you function:
from multiprocessing import Process
def foo():
while 1:
# Do something
p = Process(target = foo)
p.start()
if you just want to run it one time, do like that:
from multiprocessing import Process
def foo():
# Do something
p = Process(target = foo)
p.start()
p.join()