I'm trying to test some asyncio functionality on Windows. I'm using Python 3.7.
This complains bitterly about some NotImplementedError
import asyncio
import os
import time
import sys
#this works
async def say_after(delay, what):
await asyncio.sleep(delay)
print(what)
#this doesn't
async def test_async(num):
print('Task #{0} start'.format(num))
proc = await asyncio.create_subprocess_shell(
'C:/Python37/python test_repl.py',
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
stdin=asyncio.subprocess.PIPE )
stdout, stderr = await proc.communicate("sleep 10")
cmd = 'python'
print(f'[{cmd!r} exited with {proc.returncode}]')
if stdout:
print(f'[stdout]\n{stdout.decode()}')
if stderr:
print(f'[stderr]\n{stderr.decode()}')
async def test_loop():
task1 = asyncio.create_task(
test_async(1))
task2 = asyncio.create_task(
test_async(2))
print(f"started at {time.strftime('%X')}")
await task1
await task2
print(f"finished at {time.strftime('%X')}")
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
asyncio.run(test_loop())
I've tried several loop types (Protractor and Default). I've tried different functions through the shell and directly invoking the program (_shell and _exec in the subprocess lingo). Nothing seems to work. For a much hype I've read about asyncio, it can't possibly be a Linux only thing. I must be doing something wrong. Can you please point me in the right direction.
The specific error I get is the following:
--------------Async--------------------
started at 22:39:55
Task #1 start
Task #2 start
Traceback (most recent call last):
File "multirun.py", line 45, in <module>
asyncio.run(test_loop())
File "C:\Python37\lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python37\lib\asyncio\base_events.py", line 568, in run_until_complete
return future.result()
File "multirun.py", line 38, in test_loop
await task1
File "multirun.py", line 20, in test_async
stdin=asyncio.subprocess.PIPE)
File "C:\Python37\lib\asyncio\subprocess.py", line 202, in create_subprocess_shell
stderr=stderr, **kwds)
File "C:\Python37\lib\asyncio\base_events.py", line 1486, in subprocess_shell
protocol, cmd, True, stdin, stdout, stderr, bufsize, **kwargs)
File "C:\Python37\lib\asyncio\base_events.py", line 444, in _make_subprocess_transport
raise NotImplementedError
NotImplementedError
Task exception was never retrieved
future: <Task finished coro=<test_async() done, defined at multirun.py:12> exception=NotImplementedError()>
Traceback (most recent call last):
File "multirun.py", line 20, in test_async
stdin=asyncio.subprocess.PIPE)
File "C:\Python37\lib\asyncio\subprocess.py", line 202, in create_subprocess_shell
stderr=stderr, **kwds)
File "C:\Python37\lib\asyncio\base_events.py", line 1486, in subprocess_shell
protocol, cmd, True, stdin, stdout, stderr, bufsize, **kwargs)
File "C:\Python37\lib\asyncio\base_events.py", line 444, in _make_subprocess_transport
raise NotImplementedError
NotImplementedError
Also, I got the same error when trying to run the notepad.
As of Python 3.8, this issue should no longer exist, as the proactor event loop is now the default event loop on Windows. Original answer follows below.
The problem is that, despite appearances, you're not actually using the ProactorEventLoop. asyncio.run() creates a new event loop based on the current loop creation policy, which you've never changed. Creating a new loop for each run is normally a feature because it guarantees cleanup of the resources associated with the loop - but in this case it's incompatible with the example from the documentation. (Edit: the example has since been removed because ProactorEventLoop was made the default.)
A quick fix is to change asyncio.run(test_loop()) to loop.run_until_complete(test_loop()). A better fix is to set the event loop policy to one that creates the proactor loop.
Related
How do I run asynchronous function in blocking style? I am not allowed to modify signature of my mock function f1() and can't easy switch to async def and so can't use await expression.
async def cc2():
await asyncio.sleep(1)
await asyncio.to_thread(print, "qwerty")
def f1():
t = asyncio.create_task(cc2())
# wait until the task finished before returning
async def main():
f1() # typical call from many places of a program
asyncio.run(main())
I tried asyncio.get_running_loop().run_until_complete(t), but the hack does not work and I get the next error.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "<stdin>", line 2, in main
File "<stdin>", line 3, in f1
File "/usr/lib/python3.9/asyncio/base_events.py", line 618, in run_until_complete
self._check_running()
File "/usr/lib/python3.9/asyncio/base_events.py", line 578, in _check_running
raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
How do I run asynchronous function in blocking style?
If you are in sync code, you can call it with asyncio.run(async_function()). If you are in async code, you can await async_function() or asyncio.create_task(async_function()), with the latter scheduling it to run in the background. You are not allowed to use asyncio.run() (or run_until_complete, even with a newly created event loop object) inside async code because it blocks and could halt the outer event loop.
But if you need it for testing purposes, you can always do something like:
# XXX dangerous - could block the current event loop
with concurrent.futures.ThreadPoolExecutor() as pool:
pool.submit(asyncio.run, async_function()).result()
I'm trying to use django-channels 2 to create websockets. I need to run a async method which should return the output of a command, so I can pass the data back to the user on my website. My problem is that it will not let me run it and comes out with the error:
asyncio.run() cannot be called from a running event loop
What am I doing wrong and what can I do about it?
consumers.py
class ChatConsumer(AsyncConsumer):
async def websocket_connect(self, event):
await self.send({
"type": "websocket.accept"
})
user = self.scope['user']
get_task_id = self.scope['url_route']['kwargs']['task_id']
await asyncio.run(self.run("golemcli tasks show {}".format(get_task_id)))
await self.send({
"type": "websocket.send",
"text": "hey"
})
async def websocket_receive(self, event):
print("receive", event)
async def websocket_disconnect(self, event):
print("disconnected", event)
async def run(self, cmd):
proc = await asyncio.create_subprocess_shell(
cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE)
stdout, stderr = await proc.communicate()
print(f'[{cmd!r} exited with {proc.returncode}]')
if stdout:
print(f'[stdout]\n{stdout.decode()}')
if stderr:
print(f'[stderr]\n{stderr.decode()}')
Traceback:
Exception inside application: asyncio.run() cannot be called from a running event loop
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/sessions.py", line 183, in __call__
return await self.inner(receive, self.send)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/middleware.py", line 41, in coroutine_call
await inner_instance(receive, send)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/consumer.py", line 62, in __call__
await await_many_dispatch([receive], self.dispatch)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/utils.py", line 52, in await_many_dispatch
await dispatch(result)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/consumer.py", line 73, in dispatch
await handler(message)
File "/Users/golemgrid/Documents/GitHub/GolemGrid/overview/consumers.py", line 19, in websocket_connect
await asyncio.run(self.run("golemcli tasks show {}".format(get_task_id)))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 34, in run
"asyncio.run() cannot be called from a running event loop")
asyncio.run() cannot be called from a running event loop
UPDATE 2
When using the following snippet:
await self.run("golemcli tasks show {}".format(get_task_id)
It returns following traceback:
Exception inside application: Cannot add child handler, the child watcher does not have a loop attached
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/sessions.py", line 183, in __call__
return await self.inner(receive, self.send)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/middleware.py", line 41, in coroutine_call
await inner_instance(receive, send)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/consumer.py", line 62, in __call__
await await_many_dispatch([receive], self.dispatch)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/utils.py", line 52, in await_many_dispatch
await dispatch(result)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/channels/consumer.py", line 73, in dispatch
await handler(message)
File "/Users/golemgrid/Documents/GitHub/GolemGrid/overview/consumers.py", line 19, in websocket_connect
await self.run("golemcli tasks show {}".format(get_task_id))
File "/Users/golemgrid/Documents/GitHub/GolemGrid/overview/consumers.py", line 37, in run
stderr=asyncio.subprocess.PIPE)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/subprocess.py", line 202, in create_subprocess_shell
stderr=stderr, **kwds)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 1503, in subprocess_shell
protocol, cmd, True, stdin, stdout, stderr, bufsize, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/unix_events.py", line 193, in _make_subprocess_transport
self._child_watcher_callback, transp)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/unix_events.py", line 924, in add_child_handler
"Cannot add child handler, "
Cannot add child handler, the child watcher does not have a loop attached
You are already within an async runloop so depending on what you want to do you can either.
Handle the run call in the consumers run loop
(Requires python 3.8 due to a bug)
await self.run("golemcli tasks show {}".format(get_task_id)) no need for a nested runloop.
This will mean if any new messages come to the consumer while the subprocess is doing its work they will be queued up and not be executed until the subprocess finishes. (this is by fair the simplest solution).
this will mean your hey message will be sent after that run method finishes.
Create a nested runloop
(Requires python 3.8 due to a bug)
If you want your consumer to be able to handle new messages (being sent to it) while your subprocess. (this is a lot more complex and you should not do this unless you need this feature for your websocket connection).
async def websocket_connect(self, event):
...
# start the run
self.run_task = asyncio.create_task(self.run(...))
...
async def websocket_receive(self, event):
print("receive", event)
async def websocket_disconnect(self, event):
print("disconnected", event)
if not self.run_task.done():
# Clean up the task for the queue we created
self.run_task.cancel()
try:
# let us get any exceptions from the nested loop
await self.run_task
except CancelledError:
# Ignore this error as we just triggered it
pass
else:
# throw any error from this nested loop
self.run_task.result()
Run as a blocking task but in a threadpool
Change run to be a sync task.
# 2. Run in a custom thread pool:
loop = asyncio.get_running_loop()
with concurrent.futures.ThreadPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, self.run, "your cmd")
print('custom thread pool', result)
As a separate security note your code could easily let someone run bash commands on your server since you are taking the raw string from the task id in the url. I suggest adding some validation here, possibly if you have a task entry in your db using the url value to lookup the tasks in the db then using the id value from the record when you build your command, or if that is not possible at minimum ensure that the task_id has a very strict regex validation to ensure it cant contain any chars that you do not expect.
When I run the following python code:
import asyncio
import logging
logging.basicConfig(level=logging.DEBUG)
async def read_future(fut):
print(await fut)
async def write_future(fut):
fut.set_result('My Value')
async def main():
loop = asyncio.get_running_loop()
fut = loop.create_future()
asyncio.gather(read_future(fut), write_future(fut))
asyncio.run(main(), debug=True)
Instead of read_future waiting for the result of fut to be set, the program crashes with the following error:
DEBUG:asyncio:Using selector: KqueueSelector
ERROR:asyncio:_GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=CancelledError() created at /Users/<user>/.pyenv/versions/3.7.4/lib/python3.7/asyncio/tasks.py:642>
source_traceback: Object created at (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/<user>/.pyenv/versions/3.7.4/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/Users/<user>/.pyenv/versions/3.7.4/lib/python3.7/asyncio/base_events.py", line 566, in run_until_complete
self.run_forever()
File "/Users/<user>/.pyenv/versions/3.7.4/lib/python3.7/asyncio/base_events.py", line 534, in run_forever
self._run_once()
File "/Users/<user>/.pyenv/versions/3.7.4/lib/python3.7/asyncio/base_events.py", line 1763, in _run_once
handle._run()
File "/Users/<user>/.pyenv/versions/3.7.4/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "<stdin>", line 4, in main
File "/Users/<user>/.pyenv/versions/3.7.4/lib/python3.7/asyncio/tasks.py", line 766, in gather
outer = _GatheringFuture(children, loop=loop)
File "/Users/<user>/.pyenv/versions/3.7.4/lib/python3.7/asyncio/tasks.py", line 642, in __init__
super().__init__(loop=loop)
concurrent.futures._base.CancelledError
DEBUG:asyncio:Close <_UnixSelectorEventLoop running=False closed=False debug=True>
What am I doing wrong in this code? I want to be able to await the Future fut and continue after the Future has a value/exception set.
Your problem is that asyncio.gather is itself async (returns an awaitable); by not awaiting it, you never handed control back to the event loop, nor did you store the awaitable, so it was immediately cleaned up, implicitly cancelling it, and by extension, all of the awaitables it controlled.
To fix, just make sure you await the results of the gather:
await asyncio.gather(read_future(fut), write_future(fut))
Try it online!
From https://docs.python.org/3/library/asyncio-future.html#asyncio.Future.result:
result()
Return the result of the Future.
If the Future is done and has a result set by the set_result() method, the result value is returned.
If the Future is done and has an exception set by the set_exception() method, this method raises the exception.
If the Future has been cancelled, this method raises a CancelledError exception.
If the Future’s result isn’t yet available, this method raises a InvalidStateError exception.
(Bold added.)
I'm not sure why the Future is getting cancelled, but that seems to be the cause of the issue.
I'm trying to create an async subprocess and pass a variable to stdin but trio.run_process and trio.open_process both give an attribute error. It doesn't specify what is causing the error.
Traceback (most recent call last):
File "./update.py", line 122, in pkgUpdates
trio.run(self.asetup, password)
File "/home/user/.local/lib/python3.7/site-packages/trio/_core/_run.py", line 1444, in run
raise runner.main_task_outcome.error
File "./update.py", line 65, in asetup
nursery.start_soon(self.upProc, password, 'update')
File "/home/user/.local/lib/python3.7/site-packages/trio/_core/_run.py", line 506, in __aexit__
raise combined_error_from_nursery
File "./update.py", line 75, in upProc
await trio.open_process(['sudo', '-S', 'apt-get', cmd, '-y'],
File "/home/user/.local/lib/python3.7/site-packages/trio/_deprecate.py", line 125, in __getattr__
raise AttributeError(name)
AttributeError: open_process
I've also tried using trio.Process with stdin=io.BytesIO(password) but this gives an io.UnsupportedOperation error. Passing only bytes also gives an error.
The function is:
async def upProc(self, password, cmd):
await trio.open_process(['sudo', '-S', 'apt-get', cmd, '-y'],
stdin=password.encode(), capture_stdout=True)
if (cmd == 'update'):
await self.upProc(password, 'upgrade')
return
The reason you're getting an AttributeError on open_process and run_process is that they were only added in Trio v0.12.0, and you're probably using v0.11.0. This isn't your fault – I only released v0.12.0 about 20 minutes ago :-). But I guess you were looking at the docs for the development version, and that threw you off.
Assuming you can upgrade to v0.12.0, I think you want:
await trio.run_process(['sudo', '-S', 'apt-get', cmd, '-y'], stdin=password_bytes)
This starts the process, waits for it to complete, and then returns some information about the process, all in one shot.
I have a textbased interface (asciimatics module) for my program that uses asyncio and discord.py module and occasionally when my wifi adapter goes down I get an exception like so:
Task exception was never retrieved
future: <Task finished coro=<WebSocketCommonProtocol.run() done, defined at /home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py:428> exception=ConnectionResetError(104, 'Connection reset by peer')>
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 434, in run
msg = yield from self.read_message()
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 456, in read_message
frame = yield from self.read_data_frame(max_size=self.max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 511, in read_data_frame
frame = yield from self.read_frame(max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 546, in read_frame
self.reader.readexactly, is_masked, max_size=max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/framing.py", line 86, in read_frame
data = yield from reader(2)
File "/usr/lib/python3.5/asyncio/streams.py", line 670, in readexactly
block = yield from self.read(n)
File "/usr/lib/python3.5/asyncio/streams.py", line 627, in read
yield from self._wait_for_data('read')
File "/usr/lib/python3.5/asyncio/streams.py", line 457, in _wait_for_data
yield from self._waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/selector_events.py", line 662, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
This exception is non-fatal and the program is able to re-connect despite it - what I want to do is prevent this exception from dumping to stdout and mucking up my text interface.
I tried using ensure_future to handle it but it doesn't seem to work. Am I missing something:
#asyncio.coroutine
def handle_exception():
try:
yield from WebSocketCommonProtocol.run()
except Exception:
print("SocketException-Retrying")
asyncio.ensure_future(handle_exception())
#start discord client
client.run(token)
Task exception was never retrieved - is not actually exception propagated to stdout, but a log message that warns you that you never retrieved exception in one of your tasks. You can find details here.
I guess, most easy way to avoid this message in your case is to retrieve exception from task manually:
coro = WebSocketCommonProtocol.run() # you don't need any wrapper
task = asyncio.ensure_future(coro)
try:
#start discord client
client.run(token)
finally:
# retrieve exception if any:
if task.done() and not task.cancelled():
task.exception() # this doesn't raise anything, just mark exception retrieved
The answer provided by Mikhail is perfectly acceptable, but I realized it wouldn't work for me since the task that is raising the exception is buried deep in some module so trying to retrieve it's exception is kind've difficult. I found that instead if I simply set a custom exception handler for my asyncio loop (loop is created by the discord client):
def exception_handler(loop,context):
print("Caught the following exception")
print(context['message'])
client.loop.set_exception_handler(exception_handler)
client.run(token)