Develop Reference

Python asyncio loop.run_forever()

I try to do a trading bot and the most things work fine. But everytime the internet connection is gone for a short time, the code fails. I use asyncio run_forever() function and I think that the code should run forever until it gets stopped, but it does not work.
Here is my code:
import json
async def listen():
url = "wss://phemex.com/ws"
async with websockets.connect(url) as ws:
sub_msg = json.dumps({
"id": sequence,
"method": "kline.subscribe",
"params": [symbol, kline_interval],
})
await ws.send(sub_msg)
while True:
msg = await ws.recv()
msg = json.loads(msg)["kline"]
And I call the loop like this:
loop = asyncio.get_event_loop()
try:
loop.create_task(listen())
asyncio.ensure_future(listen())
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
print("Closing Loop")
loop.close()
As soon as the connection is lost, there is the following error:
Task exception was never retrieved
future: <Task finished coro=<listen() done, defined at c:\Users\danis\Desktop\AWS\Dateien\Crypto_Bot_Telegram_PSAR_PHEMEX_inkl_websocket_reconnect.py:351> exception=ConnectionClosedError('code = 1006 (connection closed abnormally [internal]), no reason',)>
Traceback (most recent call last):
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\protocol.py", line 750, in transfer_data
message = await self.read_message()
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\protocol.py", line 819, in read_message
frame = await self.read_data_frame(max_size=self.max_size)
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\protocol.py", line 895, in read_data_frame
frame = await self.read_frame(max_size)
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\protocol.py", line 975, in read_frame
extensions=self.extensions,
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\framing.py", line 55, in read
data = await reader(2)
File "C:\Users\danis\.conda\envs\python36\lib\asyncio\streams.py", line 674, in readexactly
yield from self._wait_for_data('readexactly')
File "C:\Users\danis\.conda\envs\python36\lib\asyncio\streams.py", line 464, in _wait_for_data
yield from self._waiter
File "C:\Users\danis\.conda\envs\python36\lib\asyncio\selector_events.py", line 714, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [WinError 10054] Eine vorhandene Verbindung wurde vom Remotehost geschlossen
How can I run this code forever?

In asyncio the event loop is responsible of run asynchronous tasks, but it doesn't handle errors they can throw. There are different ways of run the loop, you can run it for execute specific tasks or run it forever, so it keeps running awaiting for new tasks.
In your case the task listen is throwing an uncaught exception (ConnectionResetError), the event loop noitifies you about that with the traceback but it keeps running, maybe with other tasks (that means run forever). You are just notified, and like an error happened in your coroutine, the task stops running.
Solution:
Handle the error of the traceback in your courutine, you must make sure it will run forever, the event loop doesn't do that.
async def listen():
url = "wss://phemex.com/ws"
while True:
try:
async with websockets.connect(url) as ws:
sub_msg = "{\"id\":" + str(
sequence) + ", \"method\": \"kline.subscribe\", \"params\":[\"" + symbol + "\","
+ str(kline_interval) + "]}"
await ws.send(sub_msg)
while True:
msg = await ws.recv()
msg = json.loads(msg)["kline"]
except ConnectionResetError:
print("ConnectionResetError, reconnecting...")

Your loop run forever, but it can't stop the program to crash on unhandled exception. In your case you try to recv data from client that probably lost connection. You can catch this exception by yourself!
while True:
async with websockets.connect(url) as ws:
sub_msg = "{\"id\":" + str(
sequence) + ", \"method\": \"kline.subscribe\", \"params\":[\"" + symbol + "\","
+ str(kline_interval) + "]}"
await ws.send(sub_msg)
while True:
try:
msg = await ws.recv()
except ConnectionResetError:
break
msg = json.loads(msg)["kline"]

'task() takes 0 positional arguments but 1 was given' for python asyncio function

Within my python code, I am trying to design a piece of client code that connects to a WebSockets Server every second and then prints the timestamp and the obtained value from the server in a .csv file. This is given below:
import asyncio
import websockets
import logging
import datetime
import time
starttime = time.time() # start value for timed data acquisition
logger = logging.getLogger("websockets")
logger.setLevel(logging.INFO) # Switch to DEBUG for full error information
logger.addHandler(logging.StreamHandler())
class Timer: # class for asynchronous (non-blocking) counter
def __init__(self, interval, first_immediately, callback):
self._interval = interval
self._first_immediately = first_immediately
self._callback = callback
self._is_first_call = True
self._ok = True
self._task = asyncio.ensure_future(self._job())
print("init timer done")
async def _job(self):
try:
while self._ok:
if not self._is_first_call or not self._first_immediately:
await asyncio.sleep(self._interval)
await self._callback(self)
self._is_first_call = False
except Exception as ex:
print(ex)
def cancel(self):
self._ok = False
self._task.cancel()
async def test():
async with websockets.connect(
"ws://198.162.1.177:80/", ping_interval=None
) as websocket:
await websocket.send(
str(1.001)
) # send a message to the websocket server
response = (
await websocket.recv()
) # wait to get a response from the server
print(response)
dataline_pv1 = (
datetime.datetime.today().isoformat()
+ ","
+ str(response)
+ ","
+ str(0)
+ "\n"
) # format and assemble data line
file_name_pv1 = (
"{:%Y%m%d}".format(datetime.datetime.today()) + "_flow.csv"
) # generate file name
with open(
file_name_pv1, "a"
) as etherm_file1: # append dataline to file
etherm_file1.write(dataline_pv1)
# asyncio.get_event_loop().run_forever(test()) # run until test() is finished while True:
timer = Timer(interval=1, first_immediately=True, callback=test)
loop = asyncio.get_event_loop()
try:
asyncio.ensure_future(test())
loop.run_forever()
except KeyboardInterrupt:
timer.cancel()
pass
finally:
print("Closing Loop")
loop.close()
When this runs, I obtain the following message of my terminal (however the code does not crash):
test() takes 0 positional arguments but 1 was given
I have seen from this question (TypeError: takes 0 positional arguments but 1 was given) that this error occurs when a Class object is not defined properly, but my error seems to be occurring outside of a class framework. In addition, the desired .csv file is produced, however only one line is printed to the file, and does not repeat every second as desired.
What am I missing here? (also I am a complete novice with asyncio programming)
UPDATE: After changing the definition of test() to async def test(timer=None), my code now runs as expected and outputs the values to a .csv file every second (roughly), but still throws up an error. Specifically:
Task exception was never retrieved
future: <Task finished coro=<test() done, defined at flowmeterclient_v2.py:36> exception=ConnectionRefusedError(111, "Connect call failed ('198.162.1.177', 80)")>
Traceback (most recent call last):
File "flowmeterclient_v2.py", line 37, in test
async with websockets.connect("ws://198.162.1.177:80/", ping_interval=None) as websocket:
File "/usr/lib64/python3.6/site-packages/websockets/legacy/client.py", line 604, in __aenter__
return await self
File "/usr/lib64/python3.6/site-packages/websockets/legacy/client.py", line 622, in __await_impl__
transport, protocol = await self._create_connection()
File "/usr/lib64/python3.6/asyncio/base_events.py", line 798, in create_connection
raise exceptions[0]
File "/usr/lib64/python3.6/asyncio/base_events.py", line 785, in create_connection
yield from self.sock_connect(sock, address)
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect
return (yield from fut)
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb
raise OSError(err, 'Connect call failed %s' % (address,))
ConnectionRefusedError: [Errno 111] Connect call failed ('198.162.1.177', 80)

Your Timer code passes the timer itself as the first argument in
await self._callback(self)
You may wish to change the signature of test to
async def test(timer=None):
so you can call it with or without the timer.

I think you don't really need the Timer here at all. Simply have an asyncio task that loops forever and has an asyncio.sleep() internally.
This also doesn't reconnect to the websocket server for each request, like your previous code did.
import asyncio
import websockets
import logging
import datetime
logger = logging.getLogger("websockets")
logger.setLevel(logging.INFO) # Switch to DEBUG for full error information
logger.addHandler(logging.StreamHandler())
async def test():
async with websockets.connect(
"ws://198.162.1.177:80/",
ping_interval=None,
) as websocket:
while True:
await websocket.send(str(1.001))
response = await websocket.recv()
print(response)
now = datetime.datetime.now()
dataline_pv1 = f"{now.isoformat()},{response},0\n"
file_name_pv1 = f"{now:%Y%m%d}_flow.csv"
with open(file_name_pv1, "a") as etherm_file1:
etherm_file1.write(dataline_pv1)
await asyncio.sleep(1)
asyncio.run(test())
Following up on comments, if you actually do need to reconnect for each request, you can refactor this like so:
import asyncio
import websockets
import logging
import datetime
logger = logging.getLogger("websockets")
logger.setLevel(
logging.INFO
) # Switch to DEBUG for full error information
logger.addHandler(logging.StreamHandler())
async def get_data():
async with websockets.connect(
"ws://198.162.1.177:80/",
ping_interval=None,
) as websocket:
await websocket.send(str(1.001))
response = await websocket.recv()
return response
def save_data(response):
now = datetime.datetime.now()
dataline_pv1 = f"{now.isoformat()},{response},0\n"
file_name_pv1 = f"{now:%Y%m%d}_flow.csv"
with open(file_name_pv1, "a") as etherm_file1:
etherm_file1.write(dataline_pv1)
async def test():
while True:
response = await get_data()
save_data(response)
await asyncio.sleep(1)
asyncio.run(test())

Python asyncio: RuntimeError Non-thread-safe operation invoked on an event loop other than the current one

I have setup a Python program that relies on asyncio and socketio to transfer messages between chat participants using two different chat systems (one interface being a CRM backend and the other being a javascript chat widget on a website).
The relevant part of the code is below. I need to open a separate thread for each independent chat. Therefore, in order to open a channel to the backend, I create a separate thread and run a "long_polling" function in it, which constantly checks for new messages from the backend via pull_messages() and sends them to the widget asynchronously via socketio AsyncServer (forwarding messages from widget to backend works fine and I leave it out here):
thread = threading.Thread(target=long_poll, args=())
thread.daemon = True
thread.start()
with the long_poll() function being defined as
self.server = AsyncServer(async_mode="sanic")
async def send_aio(self, msg):
await self.server.emit(msg)
def long_poll(self):
while chat:
response = self.pull_messages(...)
if response == "/end":
chat = False
asyncio.set_event_loop(asyncio.new_event_loop())
loop = asyncio.get_event_loop()
loop.run_until_complete(self.send_aio(response))
loop.close()
When the long_poll function is executed, just after the human response is fetched, I get the following error:
ERROR asyncio - Task exception was never retrieved
future: <Task finished coro=<AsyncServer._emit_internal() done, defined at /usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py:344> exception=RuntimeError('Non-thread-safe operation invoked on an event loop other than the current one',)>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 354, in _emit_internal
binary=None))
File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 365, in _send_packet
await self.eio.send(sid, encoded_packet, binary=False)
File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_server.py", line 89, in send
binary=binary))
File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_socket.py", line 74, in send
await self.queue.put(pkt)
File "/usr/local/lib/python3.6/asyncio/queues.py", line 141, in put
return self.put_nowait(item)
File "/usr/local/lib/python3.6/asyncio/queues.py", line 153, in put_nowait
self._wakeup_next(self._getters)
File "/usr/local/lib/python3.6/asyncio/queues.py", line 74, in _wakeup_next
waiter.set_result(None)
File "uvloop/loop.pyx", line 1251, in uvloop.loop.Loop.call_soon
File "uvloop/loop.pyx", line 644, in uvloop.loop.Loop._check_thread
RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one
Strange thing is, when I test locally via docker, this error does not happen, only once I migrate to Kubernetes (images and all other settings are the same).
What I have tried so far to analyze the situation:
Put logger.info("Using thread: {}".format(threading.current_thread().name)) into the long_poll() function, just before the error occurs. It tells me code is run in Thread-n (n is always 1 locally and usually some higher integer in the Kubernetes Pod), which is different from MainThread where other parts of my code with asyncio run, therefore I thought I am safe (I am aware the above asyncio code is not thread-safe). In case you have any suggestions, let me know.
UPDATE:
As user4815162342 suggested, instead of creating and destroying the event loops within the while loop, I created a single event loop in a dedicated thread and then pass the coroutine I need to run to the loop in the thread via asyncio.run_coroutine_threadsafe(). Not sure whether I did everything correct here, but now this part of the code is blocking the rest of the program (e.g. no more messages from user to backend are possible...)
self.server = AsyncServer(async_mode="sanic")
async def send_aio(self, msg):
await self.server.emit(msg)
def long_poll(self, loop):
while chat:
response = self.pull_messages(...)
if response == "/end":
chat = False
future = asyncio.run_coroutine_threadsafe(self.send_io(response), loop)
_ = future.result()
In the main program:
...
loop = asyncio.new_event_loop()
lpt = LongPollingObject()
threading.Thread(target=lpt.long_poll, args=(loop,), daemon=True).start()
...

Combining 2 asyncio based code segments

I'm using the Autobahn asyncio system (to talk the Websocket WAMP protocol), which works fine and I can handle incoming RPC calls and pubsub.
My problem is I now have to connect TCP sockets and send information over these sockets as soon as an RPC call comes in through the Autobahn part.
The autobahn part works fine like this :
from autobahn.asyncio.component import Component, run
from asyncio import sleep
from autobahn.asyncio.wamp import ApplicationSession, ApplicationRunner
#comp.on_join
async def joined(session, details):
print("Connected to websocket")
def on_message(msg):
msg = json.loads(msg)
print(msg)
def some_rpc(with_data):
print("Doing something with the data")
return json.dumps({'status': 'OK'})
try:
session.subscribe(on_message, u'some_pubsub_topic')
session.register(some_rpc, u'some_rpc_call')
print("RPC and Pubsub initialized")
except Exception as e:
print("could not subscribe to topic: {0}".format(e))
if __name__ == "__main__":
run([comp])
However now I need to be able to connect to multiple regular TCP sockets :
class SocketClient(asyncio.Protocol):
def __init__(self, loop):
self.data = b''
self.loop = loop
def connection_made(self, transport):
self.transport = transport
print('connected')
def data_received(self, data):
print('Data received: {!r}'.format(data.decode()))
def send(self, data):
self.transport.write(data)
def connection_lost(self, exc):
print('The server closed the connection')
print('Stop the event loop')
self.loop.stop()
loop = asyncio.get_event_loop()
c=loop.create_connection(lambda: SocketClient(loop),
'192.168.0.219', 6773)
loop.run_until_complete(c)
loop.run_forever()
loop.close()
The problem is that, when I combine both and do this :
def some_rpc(with_data):
c.send('test')
return json.dumps({'status': 'OK'})
It barks at me and tells me :
StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"/usr/lib/python3.5/site-packages/autobahn/wamp/websocket.py", line
95, in onMessage
self._session.onMessage(msg) File "/usr/lib/python3.5/site-packages/autobahn/wamp/protocol.py", line
894, in onMessage
on_reply = txaio.as_future(endpoint.fn, *invoke_args, **invoke_kwargs) File "/usr/lib/python3.5/site-packages/txaio/aio.py", line 400, in
as_future
return create_future_error(create_failure()) File "/usr/lib/python3.5/site-packages/txaio/aio.py", line 393, in
create_future_error
reject(f, error) File "/usr/lib/python3.5/site-packages/txaio/aio.py", line 462, in reject
future.set_exception(error.value) File "/usr/lib64/python3.5/asyncio/futures.py", line 365, in set_exception
raise TypeError("StopIteration interacts badly with generators " TypeError: StopIteration interacts badly with generators and cannot be
raised into a Future
Does anyone have any idea on how to call the send function from within the RPC call function ?

In this code:
c=loop.create_connection(lambda: SocketClient(loop),
'192.168.0.219', 6773)
# [...]
def some_rpc(with_data):
c.send('test')
return json.dumps({'status': 'OK'})
create_connection is a coroutine function, so c contains a coroutine object. Such object does have a send method, but one that is entirely unrelated to sending things over the network. After calling create_connection, you probably want to get the resulting transport with something like:
transport, ignore = loop.run_until_complete(c)
and then use transport.write(), not c.send().

Python Asyncio blocked coroutine

I'm trying to code a simple program based on Asyncio and a Publish/Subscribe design pattern implemented with ZeroMQ. The publisher has 2 coroutines; one that listens for incoming subscriptions, and another one that publishes the value (obtained via an HTTP request) to the subscriber. The subscriber subscribes to a specific parameter (the name of a city in this case), and waits for the value (the temperature in this city).
Here is my code:
publisher.py
#!/usr/bin/env python
import json
import aiohttp
import aiozmq
import asyncio
import zmq
class Publisher:
BIND_ADDRESS = 'tcp://*:10000'
def __init__(self):
self.stream = None
self.parameter = ""
#asyncio.coroutine
def main(self):
self.stream = yield from aiozmq.create_zmq_stream(zmq.XPUB, bind=Publisher.BIND_ADDRESS)
tasks = [
asyncio.async(self.subscriptions()),
asyncio.async(self.publish())]
print("before wait")
yield from asyncio.wait(tasks)
print("after wait")
#asyncio.coroutine
def subscriptions(self):
print("Entered subscriptions coroutine")
while True:
print("New iteration of subscriptions loop")
received = yield from self.stream.read()
first_byte = received[0][0]
self.parameter = received[0][-len(received[0])+1:].decode("utf-8")
# Subscribe request
if first_byte == 1:
print("subscription request received for parameter "+self.parameter)
# Unsubscribe request
elif first_byte == 0:
print("Unsubscription request received for parameter "+self.parameter)
#asyncio.coroutine
def publish(self):
print("Entered publish coroutine")
while True:
if self.parameter:
print("New iteration of publish loop")
# Make HTTP request
url = "http://api.openweathermap.org/data/2.5/weather?q="+self.parameter
response = yield from aiohttp.request('GET', url)
assert response.status == 200
content = yield from response.read()
# Decode JSON string
decoded_json = json.loads(content.decode())
# Get parameter value
value = decoded_json["main"]["temp"]
# Publish fetched values to subscribers
message = bytearray(self.parameter+":"+str(value),"utf-8")
print(message)
pack = [message]
print("before write")
yield from self.stream.write(pack)
print("after write")
yield from asyncio.sleep(10)
test = Publisher()
loop = asyncio.get_event_loop()
loop.run_until_complete(test.main())
subscriber.py
#!/usr/bin/env python
import zmq
class Subscriber:
XSUB_CONNECT = 'tcp://localhost:10000'
def __init__(self):
self.context = zmq.Context()
self.socket = self.context.socket(zmq.XSUB)
self.socket.connect(Subscriber.XSUB_CONNECT)
def loop(self):
print(self.socket.recv())
self.socket.close()
def subscribe(self, parameter):
self.socket.send_string('\x01'+parameter)
print("Subscribed to parameter "+parameter)
def unsubscribe(self, parameter):
self.socket.send_string('\x00'+parameter)
print("Unsubscribed to parameter "+parameter)
test = Subscriber()
test.subscribe("London")
while True:
print(test.socket.recv())
And here is the output :
Subscriber side :
$ python3 subscriber.py
Subscribed to parameter London
b'London:288.15'
Publisher side :
$ python3 publisher.py
before wait
Entered subscriptions coroutine
New iteration of subscriptions loop
Entered publish coroutine
subscription request received for parameter London
New iteration of subscriptions loop
New iteration of publish loop
bytearray(b'London:288.15')
before write
And the program is stuck there.
As you can see, "before write" appears in the output and the message is sent, but "after write" doesn't appear. So, I figured that an exception was probably raised and caught somewhere in the self.stream.write(pack) call stack.
If I send a KeyboardInterrupt to the Publisher, here is what I get:
Traceback (most recent call last):
File "publisher.py", line 73, in <module>
loop.run_until_complete(test.main())
File "/usr/lib/python3.4/asyncio/base_events.py", line 304, in run_until_complete
self.run_forever()
File "/usr/lib/python3.4/asyncio/base_events.py", line 276, in run_forever
self._run_once()
File "/usr/lib/python3.4/asyncio/base_events.py", line 1136, in _run_once
event_list = self._selector.select(timeout)
File "/usr/lib/python3.4/selectors.py", line 432, in select
fd_event_list = self._epoll.poll(timeout, max_ev)
KeyboardInterrupt
Task exception was never retrieved
future: <Task finished coro=<publish() done, defined at publisher.py:43> exception=TypeError("'NoneType' object is not iterable",)>
Traceback (most recent call last):
File "/usr/lib/python3.4/asyncio/tasks.py", line 236, in _step
result = coro.send(value)
File "publisher.py", line 66, in publish
yield from self.stream.write(pack)
TypeError: 'NoneType' object is not iterable
Task was destroyed but it is pending!
task: <Task pending coro=<subscriptions() running at publisher.py:32> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.4/asyncio/tasks.py:399]>
So I guess my problem actually is this error: TypeError: 'NoneType' object is not iterable, but I have no clue what's causing it.
What is going wrong here?

The issue is that you're trying to yield from the call to self.stream.write(), but stream.write isn't actually a coroutine. When you call yield from on an item, Python internally calls iter(item). In this case, the call to write() is returning None, so Python is trying to do iter(None) - hence the exception you see.
To fix it, you should just call write() like a normal function. If you want to actually wait until the write is flushed and sent to the reader, use yield from stream.drain() after you make the call to write():
print("before write")
self.stream.write(pack)
yield from self.stream.drain()
print("after write")
Also, to make sure that exception in publish get raised without needing to Ctrl+C, use asyncio.gather instead of asyncio.wait:
yield from asyncio.gather(*tasks)
With asyncio.gather, any exception thrown by a task inside tasks will be re-raised.