My example class uses only two methods,
async def run(): creates a asyncio.Future() and awaits it
def stop(): sets the result of the Future() created by the run() method
The idea is to use the python signal handler to call the stop() function once a signal is received. The stop() then sets the result of the Future() the run() is waiting for. So far so good, but this does not work. Actually, the run() method never notices that the Future() is done.
Example code:
import asyncio
import signal
class Foo:
def __init__(self):
self._stop = None
async def run(self):
print(f"1. starting foo")
self._stop = asyncio.Future()
await self._stop
print(f"4. 'stop' called, canceling running tasks...")
def stop(self):
print(f"3. stopping foo")
self._stop.set_result(None)
f = Foo()
loop = asyncio.new_event_loop()
def signal_handler(_, __):
print(f"2. signal handler: call Foo.stop()")
f.stop()
signal.signal(signal.SIGINT, signal_handler)
loop.run_until_complete(f.run())
print(f"5. bye")
and the output is:
1. starting foo
2. signal handler: call Foo.stop()
3. stopping foo
That's it. The fourth print entry is never called. Although the self._stop Future is done after setting the result.
Does anyone have any idea what I am doing wrong and what I am missing?
Any help would be greatly appreciated!
I cannot reproduce your problem, but it is quite possible that it'll happen on different operating systems.
Per loop.add_signal_handler:
Unlike signal handlers registered using signal.signal(), a callback registered with this function is allowed to interact with the event loop.
The usual culprit that causes those issues is the event loop not waking up from outside input.
There are 2 solutions for your issue:
If you're using unix, change signal.signal() to loop.add_signal_handler().
If not, try changing f.stop() to loop.call_soon_threadsafe(f.stop). It will make sure that the event loop wakes up correctly.
Irrespective to that, you have a couple of different issues arising from using asyncio.new_event_loop() and not assigning the loop to the thread or cleaning up correctly. I suggest you to use asyncio.run().
Related
I have a asyncio running loop, and from the coroutine I'm calling a sync function, is there any way we can call and get result from an async function in a sync function
tried below code, it is not working
want to print output of hel() in i() without changing i() to async function
is it possible, if yes how?
import asyncio
async def hel():
return 4
def i():
loop = asyncio.get_running_loop()
x = asyncio.run_coroutine_threadsafe(hel(), loop) ## need to change
y = x.result() ## this lines
print(y)
async def h():
i()
asyncio.run(h())
This is one of the most commonly asked type of question here. The tools to do this are in the standard library and require only a few lines of setup code. However, the result is not 100% robust and needs to be used with care. This is probably why it's not already a high-level function.
The basic problem with running an async function from a sync function is that async functions contain await expressions. Await expressions pause the execution of the current task and allow the event loop to run other tasks. Therefore async functions (coroutines) have special properties that allow them to yield control and resume again where they left off. Sync functions cannot do this. So when your sync function calls an async function and that function encounters an await expression, what is supposed to happen? The sync function has no ability to yield and resume.
A simple solution is to run the async function in another thread, with its own event loop. The calling thread blocks until the result is available. The async function behaves like a normal function, returning a value. The downside is that the async function now runs in another thread, which can cause all the well-known problems that come with threaded programming. For many cases this may not be an issue.
This can be set up as follows. This is a complete script that can be imported anywhere in an application. The test code that runs in the if __name__ == "__main__" block is almost the same as the code in the original question.
The thread is lazily initialized so it doesn't get created until it's used. It's a daemon thread so it will not keep your program from exiting.
The solution doesn't care if there is a running event loop in the main thread.
import asyncio
import threading
_loop = asyncio.new_event_loop()
_thr = threading.Thread(target=_loop.run_forever, name="Async Runner",
daemon=True)
# This will block the calling thread until the coroutine is finished.
# Any exception that occurs in the coroutine is raised in the caller
def run_async(coro): # coro is a couroutine, see example
if not _thr.is_alive():
_thr.start()
future = asyncio.run_coroutine_threadsafe(coro, _loop)
return future.result()
if __name__ == "__main__":
async def hel():
await asyncio.sleep(0.1)
print("Running in thread", threading.current_thread())
return 4
def i():
y = run_async(hel())
print("Answer", y, threading.current_thread())
async def h():
i()
asyncio.run(h())
Output:
Running in thread <Thread(Async Runner, started daemon 28816)>
Answer 4 <_MainThread(MainThread, started 22100)>
In order to call an async function from a sync method, you need to use asyncio.run, however this should be the single entry point of an async program so asyncio makes sure that you don't do this more than once per program, so you can't do that.
That being said, this project https://github.com/erdewit/nest_asyncio patches the asyncio event loop to do that, so after using it you should be able to just call asyncio.run in your sync function.
My code runs N number of threads. I want to stop specific threads on some condition but the remaining threads should continue running. I am doing some operation once each thread finishes its job. Is there a way to stop running thread in Python 3.
My current code is implemented in Python2 which does this by "_Thread__stop()". Is there any identical thing in Python3?
The practice is to "signal" the thread that it is time to finish and then the thread needs to exit. This is not killing like you kill a process but a regular state machine behavior of your thread function.
For example, suppose your thread is lopping. You should insert an if statement inside the loop that instructing the thread function to break or return if stop is True. The stop variable should be a shared variable with the main thread (or the thread who need to stop out thread) that will change it to True. usually after this, the stopper thread will want to wait for the thread completion by join()
It's a bad habit to kill a thread, better is to create a "flag" which will tell you when your thread made its work done.
Consider the following example:
import threading
import random
class CheckSomething(threading.Thread):
def __init__(self, variable):
super(CheckSomething, self).__init__()
self.start_flag = threading.Event()
self.variable = variable
def check_position(self, variable):
x = random.randint(100)
if variable == x:
self.stop_checking()
def run(self):
while True:
self.check_position(self.variable)
def stop_checking():
self.start_flag.set()
def stopped():
return self.start_flag.is_set()
The set() method of Event() set its status to True. More you can read in docs: https://docs.python.org/3.5/library/threading.html
So you need to call stop_checking() when you meet a condition where you want exit.
I have two Python threads that share some state, A and B. At one point, A submits a callback to be run by B on its loop with something like:
# This line is executed by A
loop.call_soon_threadsafe(callback)
After this I want to continue doing something else, but I want to make sure that callback has been run by B before doing so. Is there any way (besides standard threading synchronization primitives) to make A wait for the completion of the callback? I know call_soon_threadsafe returns a asyncio.Handle object that can cancel the task, but I am not sure whether this can be used for waiting (I still don't know much about asyncio).
In this case, this callback calls loop.close() and cancels the remaining tasks, and after that, in B, after loop.run_forever() there is a loop.close(). So for this use case in particular a thread-safe mechanism that allows me to know from A when the loop has been effectively closed would also work for me - again, not involving a mutex/condition variable/etc.
I know that asyncio is not meant to be thread-safe, with very few exceptions, but I wanted to know if a convenient way to achieve this is provided.
Here is a very small snippet of what I mean in case it helps.
import asyncio
import threading
import time
def thread_A():
print('Thread A')
loop = asyncio.new_event_loop()
threading.Thread(target=thread_B, args=(loop,)).start()
time.sleep(1)
handle = loop.call_soon_threadsafe(callback, loop)
# How do I wait for the callback to complete before continuing?
print('Thread A out')
def thread_B(loop):
print('Thread B')
asyncio.set_event_loop(loop)
loop.run_forever()
loop.close()
print('Thread B out')
def callback(loop):
print('Stopping loop')
loop.stop()
thread_A()
I have tried this variation with asyncio.run_coroutine_threadsafe but it does not work, instead thread A hangs forever. Not sure if I am doing something wrong or it is because I am stopping the loop.
import asyncio
import threading
import time
def thread_A():
global future
print('Thread A')
loop = asyncio.new_event_loop()
threading.Thread(target=thread_B, args=(loop,)).start()
time.sleep(1)
future = asyncio.run_coroutine_threadsafe(callback(loop), loop)
future.result() # Hangs here
print('Thread A out')
def thread_B(loop):
print('Thread B')
asyncio.set_event_loop(loop)
loop.run_forever()
loop.close()
print('Thread B out')
async def callback(loop):
print('Stopping loop')
loop.stop()
thread_A()
Callbacks are set and (mostly) forget. They are not intended to be used for something you need to get a result back from. This is why the handle produced only lets you cancel a callback (this callback is no longer needed), nothing more.
If you need to wait for a result from an asyncio-managed coroutine in another thread, use a coroutine and schedule it as a task with asyncio.run_coroutine_threadsafe(); this gives you a Future() instance, which you can then wait for to be done.
However, stopping the loop with run_coroutine_threadsafe() does require the loop to handle one more round of callbacks than it'll actually be able to run; the Future returned by run_coroutine_threadsafe() would otherwise not be notified of the state change of the task it scheduled. You can remedy this by running asyncio.sleep(0) through loop.run_until_complete() in thread B before closing the loop:
def thread_A():
# ...
# when done, schedule the asyncio loop to exit
future = asyncio.run_coroutine_threadsafe(shutdown_loop(loop), loop)
future.result() # wait for the shutdown to complete
print("Thread A out")
def thread_B(loop):
print("Thread B")
asyncio.set_event_loop(loop)
loop.run_forever()
# run one last noop task in the loop to clear remaining callbacks
loop.run_until_complete(asyncio.sleep(0))
loop.close()
print("Thread B out")
async def shutdown_loop(loop):
print("Stopping loop")
loop.stop()
This is, of course, slightly hacky and depends on the internals of callback management and cross-threading task scheduling to not change. As the default asyncio implementation stands, running a single noop task is plenty for several rounds of callbacks creating more callbacks being handled, but alternative loop implementations may handle this differently.
So for shutting down the loop, you may be better off using thread-based coordination:
def thread_A():
# ...
callback_event = threading.Event()
loop.call_soon_threadsafe(callback, loop, callback_event)
callback_event.wait() # wait for the shutdown to complete
print("Thread A out")
def thread_B(loop):
print("Thread B")
asyncio.set_event_loop(loop)
loop.run_forever()
loop.close()
print("Thread B out")
def callback(loop, callback_event):
print("Stopping loop")
loop.stop()
callback_event.set()
Is there any way (besides standard threading synchronization primitives) to make A wait for the completion of the callback?
Normally you'd use run_coroutine_threadsafe, as Martijn initially suggested. But your use of loop.stop() makes the callback somewhat specific. Given that, you are probably best off using the standard thread synchronization primitives, which are in this case very straightforward and can be completely decoupled from the callback implementation and the rest of your code. For example:
def submit_and_wait(loop, fn, *args):
"Submit fn(*args) to loop, and wait until the callback executes."
done = threading.Event()
def wrap_fn():
try:
fn(*args)
finally:
done.set()
loop.call_soon_threadsafe(wrap_fn)
done.wait()
Instead of using loop.call_soon_threadsafe(callback), use submit_and_wait(loop, callback). The threading synchronization is there, but completely hidden inside submit_and_wait.
I have an asyncio based class which I want to unit test. Using tornado.testing.AsyncTestCase this works quite well and easily. However, one specific method of my class uses asyncio.ensure_future to schedule execution of another method. This never finishes in the AsyncTestCase, because the default test runner uses the tornado KQueueIOLoop event loop, not an asyncio event loop.
class TestSubject:
def foo(self):
asyncio.ensure_future(self.bar())
async def bar(self):
pass
class TestSubjectTest(AsyncTestCase):
def test_foo(self):
t = TestSubject()
# here be somewhat involved setup with MagicMock and self.stop
t.foo()
self.wait()
$ python -m tornado.testing baz.testsubject_test
...
[E 160627 17:48:22 testing:731] FAIL
[E 160627 17:48:22 base_events:1090] Task was destroyed but it is pending!
task: <Task pending coro=<TestSubject.bar() running at ...>>
.../asyncio/base_events.py:362: RuntimeWarning: coroutine 'TestSubject.bar' was never awaited
How can I use a different event loop to run the tests on to ensure my task will actually be executed? Alternatively, how can I make my implementation event loop-independent and cross-compatible?
Turns out to be simple enough...
class TestSubjectTest(AsyncTestCase):
def get_new_ioloop(self): # override this method
return tornado.platform.asyncio.AsyncIOMainLoop()
I was trying this before, but directly returned asyncio.get_event_loop(), which didn't work. Returning Tornado's asyncio loop wrapper does the trick.
I've got a script that runs an infinite loop and adds things to a database and does things that I can't just stop halfway through, so I can't just press Ctrl+C and stop it.
I want to be able to somehow stop a while loop, but let it finish it's last iteration before it stops.
Let me clarify:
My code looks something like this:
while True:
do something
do more things
do more things
I want to be able to interrupt the while loop at the end, or the beginning, but not between doing things because that would be bad.
And I don't want it to ask me after every iteration if I want to continue.
Thanks for the great answers, I'm super grateful but my implementation doesn't seem to be working:
def signal_handler(signal, frame):
global interrupted
interrupted = True
class Crawler():
def __init__(self):
# not relevant
def crawl(self):
interrupted = False
signal.signal(signal.SIGINT, signal_handler)
while True:
doing things
more things
if interrupted:
print("Exiting..")
break
When I press Ctrl+C the program just keeps going ignoring me.
What you need to do is catch the interrupt, set a flag saying you were interrupted but then continue working until it's time to check the flag (at the end of each loop). Because python's try-except construct will abandon the current run of the loop, you need to set up a proper signal handler; it'll handle the interrupt but then let python continue where it left off. Here's how:
import signal
import time # For the demo only
def signal_handler(signal, frame):
global interrupted
interrupted = True
signal.signal(signal.SIGINT, signal_handler)
interrupted = False
while True:
print("Working hard...")
time.sleep(3)
print("All done!")
if interrupted:
print("Gotta go")
break
Notes:
Use this from the command line. In the IDLE console, it'll trample on IDLE's own interrupt handling.
A better solution would be to "block" KeyboardInterrupt for the duration of the loop, and unblock it when it's time to poll for interrupts. This is a feature of some Unix flavors but not all, hence python does not support it (see the third "General rule")
The OP wants to do this inside a class. But the interrupt function is invoked by the signal handling system, with two arguments: The signal number and a pointer to the stack frame-- no place for a self argument giving access to the class object. Hence the simplest way to set a flag is to use a global variable. You can rig a pointer to the local context by using closures (i.e., define the signal handler dynamically in __init__(), but frankly I wouldn't bother unless a global is out of the question due to multi-threading or whatever.
Caveat: If your process is in the middle of a system call, handling an signal may interrupt the system call. So this may not be safe for all applications. Safer alternatives would be (a) Instead of relying on signals, use a non-blocking read at the end of each loop iteration (and type input instead of hitting ^C); (b) use threads or interprocess communication to isolate the worker from the signal handling; or (c) do the work of implementing real signal blocking, if you are on an OS that has it. All of them are OS-dependent to some extent, so I'll leave it at that.
the below logic will help you do this,
import signal
import sys
import time
run = True
def signal_handler(signal, frame):
global run
print("exiting")
run = False
signal.signal(signal.SIGINT, signal_handler)
while run:
print("hi")
time.sleep(1)
# do anything
print("bye")
while running this, try pressing CTRL + C
To clarify #praba230890's solution: The interrupted variable was not defined in the correct scope. It was defined in the crawl function and the handler could not reach it as a global variable, according to the definition of the handler at the root of the program.
Here is edited example of the principle above. It is the infinitive python loop in a separate thread with the safe signal ending. Also has thread-blocking sleep step - up to you to keep it, replace for asyncio implementation or remove.
This function could be imported to any place in an application, runs without blocking other code (e.g. good for REDIS pusub subscription). After the SIGINT catch the thread job ends peacefully.
from typing import Callable
import time
import threading
import signal
end_job = False
def run_in_loop(job: Callable, interval_sec: int = 0.5):
def interrupt_signal_handler(signal, frame):
global end_job
end_job = True
signal.signal(signal.SIGINT, interrupt_signal_handler)
def do_job():
while True:
job()
time.sleep(interval_sec)
if end_job:
print("Parallel job ending...")
break
th = threading.Thread(target=do_job)
th.start()
You forgot to add global statement in crawl function.
So result will be
import signal
def signal_handler(signal, frame):
global interrupted
interrupted = True
class Crawler():
def __init__(self):
... # or pass if you don't want this to do anything. ... Is for unfinished code
def crawl(self):
global interrupted
interrupted = False
signal.signal(signal.SIGINT, signal_handler)
while True:
# doing things
# more things
if interrupted:
print("Exiting..")
break
I hope below code would help you:
#!/bin/python
import sys
import time
import signal
def cb_sigint_handler(signum, stack):
global is_interrupted
print("SIGINT received")
is_interrupted = True
if __name__ == "__main__":
is_interrupted = False
signal.signal(signal.SIGINT, cb_sigint_handler)
while True:
# do stuff here
print("processing...")
time.sleep(3)
if is_interrupted:
print("Exiting..")
# do clean up
sys.exit(0)