I am trying out some asyncio examples found on the web:
Proxybroker example
When I run this first example:
"""Find and show 10 working HTTP(S) proxies."""
import asyncio
from proxybroker import Broker
async def show(proxies):
while True:
proxy = await proxies.get()
if proxy is None: break
print('Found proxy: %s' % proxy)
proxies = asyncio.Queue()
broker = Broker(proxies)
tasks = asyncio.gather(
broker.find(types=['HTTP', 'HTTPS'], limit=10),
show(proxies))
loop = asyncio.get_event_loop()
loop.run_until_complete(tasks)
I get the error:
RuntimeError: This event loop is already running
But the loop completes as expected.
I'm new to concurrent code so any explanation / pseudo code of what is occurring would be appreciated.
I install this package,and run it passed, no error occured,are use a ide? try to run it on cli,or move it another directory
Related
I want to implement a class with the possibility to start various websockets in different threads to retrieve market data and update the class attributes. I am using the kucoin-python-sdk library to that purpose.
The below works fine in spyder, however when I set my script to run via a conda batch it fails with the following errors over and over.
Thank you.
<Task finished name='Task-4' coro=<ConnectWebsocket._run() done,>
defined at > path\lib\site-packages\kucoin\websocket\websocket.py:33>>
exception=RuntimeError("can't register atexit after shutdown")> got
an> exception can't register atexit after shutdown pending name='Task-3' coro=<ConnectWebsocket._recover_topic_req_msg()
running> at>
path\lib\site-packages\kucoin\websocket\websocket.py:127>>
wait_for=> cancel ok.> _reconnect
over.
<Task finished name='Task-7' coro=<ConnectWebsocket._run() done,
defined at>> path\lib\site-packages\kucoin\websocket\websocket.
py:33>> exception=RuntimeError("can't register atexit after shutdown")> got an> exception can't register atexit after shutdown pending name='Task-6' coro=<ConnectWebsocket._recover_topic_req_msg() running> at
path\lib\site-packages\kucoin\websocket\websocket.py:127>> wait_for=> cancel ok.> _reconnect over.
Hence wondering:
Does the issue come from the Kucoin package or is my implementation of threads/asyncio incorrect ?
How to explain the different behavior between Spyder execution and conda on the same environment ?
Python 3.9.13 | Spyder 5.3.3 | Spyder kernel 2.3.3 | websocket 0.2.1 | nest-asyncio 1.5.6 | kucoin-python 1.0.11
Class_X.py
import asyncio
import nest_asyncio
nest_asyncio.apply()
from kucoin.client import WsToken
from kucoin.ws_client import KucoinWsClient
from threading import Thread
class class_X():
def __init__(self):
self.msg= ""
async def main(self):
async def book_msg(msg):
self.msg = msg
client = WsToken()
ws_client = await KucoinWsClient.create(None, client, book_msg, private=False)
await ws_client.subscribe(f'/market/level2:BTC-USDT')
while True:
await asyncio.sleep(20)
def launch(self):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(self.main())
instance = class_X()
t = Thread(target=instance.launch)
t.start()
Batch
call path\anaconda3\Scripts\activate myENV
python "path1\class_X.py"
conda deactivate
I want to say it's your implementation but I haven't tried using that client the way you're doing it. Here's a pared down skeleton of what I'm doing to implement that kucoin-python in async.
import asyncio
from kucoin.client import WsToken
from kucoin.ws_client import KucoinWsClient
from kucoin.client import Market
from kucoin.client import User
from kucoin.client import Trade
async def main():
async def handle_event(msg):
if '/market/snapshot:' in msg['topic']:
snapshot = msg['data']['data']
## trade logic here using snapshot data
elif msg['topic'] == '/spotMarket/tradeOrders':
print(msg['data'])
else:
print("Unhandled message type")
print(msg)
async def unsubscribeFromPublicSnapsot(symbol):
ksm.unsubscribe('/market/snapshot:' + symbol)
async def subscribeToPublicSnapshot(symbol):
try:
print("subscribing to " + symbol)
await ksm.subscribe('/market/snapshot:' + symbol)
except Exception as e:
print("Error subscribing to snapshot for " + doc['currency'])
print(e)
pubClient = WsToken()
print("creating websocket client")
ksm = await KucoinWsClient.create(None, pubClient, handle_event, private=False)
# for private topics pass private=True
privateClient = WsToken(config["tradeKey"], config["tradeSecret"], config["tradePass"])
ksm_private = await KucoinWsClient.create(None, privateClient, handle_event, private=True)
# Always subscribe to BTC-USDT
await subscribeToPublicSnapshot('BTC-USDT')
# Subscribe to the currency-BTC spot market for each available currency
for doc in tradeable_holdings:
if doc['currency'] != 'BTC': # Don't need to resubscribe :D
await subscribeToPublicSnapshot(doc['currency'] + "-BTC")
# Subscribe to spot market trade orders
await ksm_private.subscribe('/spotMarket/tradeOrders')
if __name__ == "__main__":
print("Step 1: Kubot initialzied")
print("Step 2: ???")
print("Step 2: Profit")
loopMain = asyncio.get_event_loop()
loopMain.create_task(main())
loopMain.run_forever()
loopMain.close()
As you can probably guess, "tradeable_holdings" is a list of symbols I'm interested in that I already own. You'll also notice I'm using the snapshot instead of the market/ticker subscription. I think at 100ms updates on the ticker, it could quickly run into latency and race conditions - at least until I figure out how to deal with those. So I opted for the snapshot which only updates every 2 seconds and for the less active coins, not even that often.
Anyway, I'm not to where it's looking to trade but I'm quickly getting to that logic.
Hope this helps you figure your implementation out even though it's different.
I create redis pool the following way:
async def create_redis_connection_pool(app) -> aioredis.Redis:
redis = aioredis.from_url(
"redis://localhost", encoding="utf-8", decode_responses=True, max_connections=10,
)
app["redis"] = redis
try:
yield
finally:
loop = asyncio.get_event_loop()
await loop.create_task(app["redis"].close())
Then I use the function when I create the Aiohttp app:
def init() -> web.Application:
app = web.Application()
...
app.cleanup_ctx.append(create_redis_connection_pool)
...
return app
When I start server, do at least one request which use redis pool and then do Cnrl+C I get the following warning message:
sys:1: RuntimeWarning: coroutine 'Connection.disconnect' was never awaited
How to solve the issue and gracefully close Redis connection pool? I do tests in Mac OS.
If you're using redis==4.2.0 (from redis import asyncio as aioredis) or later,
pass close_connection_pool=True when you call .close():
await app["redis"].close(close_connection_pool=True)
Otherwise, for aioredis==2.0.1 (latest version as of this answer) or earlier,
call .connection_pool.disconnect() after .close():
await app["redis"].close()
await app["redis"].connection_pool.disconnect()
Reference: https://github.com/aio-libs/aioredis-py/pull/1256
How can i launch an application which does below 2 things:
Expose rest endpoint via FastAPI.
Run a seperate thread infintely (rabbitmq consumer - pika) waiting for request.
Below is the code through which i am launching the fastAPI server. But when i try to run a thread, before execution of below line, it says coroutine was never awaited.
How can both be run parallely?
Maybe this is not the answer you are looking for.
There is a library called celery that makes multithreading easy to manage in python.
Check it out:
https://docs.celeryproject.org/en/stable/getting-started/introduction.html
import asyncio
from concurrent.futures import ThreadPoolExecutor
from fastapi import FastAPI
import uvicorn
app = FastAPI()
def run(corofn, *args):
loop = asyncio.new_event_loop()
try:
coro = corofn(*args)
asyncio.set_event_loop(loop)
return loop.run_until_complete(coro)
finally:
loop.close()
async def sleep_forever():
await asyncio.sleep(1000)
#
async def main():
loop = asyncio.get_event_loop()
executor = ThreadPoolExecutor(max_workers=2)
futures = [loop.run_in_executor(executor, run, sleep_forever),
loop.run_in_executor(executor, run, uvicorn.run, app)]
print(await asyncio.gather(*futures))
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Note: This may hinder your FastAPI performance. Better approach would be to use a Celery Task
I'm not really a Python expert, so excuse me if this is really obvious. I'm trying to run a script using asyncio. Relevant bits of code:
import websockets
import asyncio
stream = websockets.connect(<resource_uri>)
async def main():
async with stream as receiver:
while True:
data = receiver.recv()
# do stuff
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
When I run this, I get:
DeprecationWarning: There is no current event loop
loop = asyncio.get_event_loop()
Similarly, using
'''
loop = asyncio.get_running_loop()
'''
instead, I get
RuntimeError: no running event loop.
Any ideas? I guess it's something to do with main() not running in the correct thread...
I'm using Python 3.10.
Newer code should use just asyncio.run(main()) - that will automatically create a new instance loop and "run until complete" on the awaitable.
Try this, works for me on 3.8 (and probably originally got from someone smarter than me that posted it on here!!)
try:
loop = asyncio.get_running_loop()
except RuntimeError: # 'RuntimeError: There is no current event loop...'
loop = None
if loop and loop.is_running():
print('Async event loop already running. Adding coroutine to the event loop.')
tsk = loop.create_task(main())
# ^-- https://docs.python.org/3/library/asyncio-task.html#task-object
# Optionally, a callback function can be executed when the coroutine completes
tsk.add_done_callback(
lambda t: print(f'Task done with result = {t.result()}'))
else:
print('Starting new event loop')
asyncio.run(main())
It seems there's an issue with websockets and Python 3.10. Looks like I'll have to find a workaround. Thanks for your time!
I try to create a client which uses a asyncio.Queue to feed the messages I want to send to the server. Receiving data from websocket server works great. Sending data which is just generated by the producer works, too. For explaning what works and what fails, first here's my code:
import sys
import asyncio
import websockets
class WebSocketClient:
def __init__(self):
self.send_queue = asyncio.Queue()
#self.send_queue.put_nowait('test-message-1')
async def startup(self):
await self.connect_websocket()
consumer_task = asyncio.create_task(
self.consumer_handler()
)
producer_task = asyncio.create_task(
self.producer_handler()
)
done, pending = await asyncio.wait(
[consumer_task, producer_task],
return_when=asyncio.ALL_COMPLETED
)
for task in pending:
task.cancel()
async def connect_websocket(self):
try:
self.connection = await websockets.client.connect('ws://my-server')
except ConnectionRefusedError:
sys.exit('error: cannot connect to backend')
async def consumer_handler(self):
async for message in self.connection:
await self.consumer(message)
async def consumer(self, message):
self.send_queue.put_nowait(message)
# await self.send_queue.put(message)
print('mirrored message %s now in queue, queue size is %s' % (message, self.send_queue.qsize()))
async def producer_handler(self):
while True:
message = await self.producer()
await self.connection.send(message)
async def producer(self):
result = await self.send_queue.get()
self.send_queue.task_done()
#await asyncio.sleep(10)
#result = 'test-message-2'
return result
if __name__ == '__main__':
wsc = WebSocketClient()
asyncio.run(wsc.startup())
Connecting works great. If I send something from my server to the client, this works great too and prints the message in consumer(). But producer never gets any message I put in send_queue inside consumer().
The reason why I chose send_queue.put_nowait in consumer() was that I wanted to prevent deadlocks. If I use the line await self.send_queue.put(message) line instead of self.send_queue.put_nowait(message) it makes no difference.
I thought, maybe the queue dos not work at all, so I filled something to the queue just at creation in __init__(): self.send_queue.put_nowait("test-message-1"). This works and is sent to my server. So the basic concept of the queue and await queue.get() works.
I als thought, maybe there is some issue with the producer, so let's just randomly generate messages during runtime: result = "test-message-2" instead of result = await self.send_queue.get(). This works too: every 10 seconds 'test-message-2' is sent to my server.
EDIT: This also happens if I try to add stuff from another source to the queue on the fly. I build a small asyncio socket server which pushes any message to the queue, which works great, and you can see the messages I added from the other source with qsize() in consumer(), but still no successfull queue.get(). So the queue itself seems to work, just not get(). This is btw the reason for the queue, too: I would like to send data from quite different sources.
So, this is the point where I'm stuck. My wild guess is that the queue I use in producer() is not the same as in consumer(), something which happens at threading quite easily if you use non-thread-safe queues like asyncio.Queue, but as I understood it I don't use threading at all, just coroutines. So, what else went wrong here?
Just for the context: it's a Ubuntu 20.04 python 3.8.2 inside a docker container.
Thanks,
Ernesto
Just for the records - the solution for my problem was quite simple: I defined send_queue outside the event loop created by my websocket client. So it called events.get_event_loop() and got its own loop - which was not part of the main loop and therefore never called, therefore await queue.get() really never got anything back.
In normal mode, you don't see any message which is a hint to this issue. But, python documentation to the rescue: for course they mentioned it at https://docs.python.org/3/library/asyncio-dev.html : logging.DEBUG gave the hints I needed to find the problem.
It should look like this:
class WebSocketClient:
async def startup(self):
self.send_queue = asyncio.Queue()
await self.connect_websocket()
Then the queue is defined inside the main loop.