I'm quite new to all things async in python. I have a certain code I would like to run while an async slack RTM client is listening on messages with a dedicated callback, like this:
RTM_CLIENT.start()
while True:
...
except Exception as e:
...
finally:
RTM_CLIENT.stop()
the callback function:
#slack.RTMClient.run_on(event='message')
def listen(**payload):
...
The RTM_CLIENT.start() function returns a future object.
I'm not getting any message events though. Am I doing something wrong?
This solves it(thread sync):
import re
import slack
import time
import asyncio
import concurrent
from datetime import datetime
#slack.RTMClient.run_on(event='message')
async def say_hello(**payload):
data = payload['data']
print(data.get('text'))
def sync_loop():
while True:
print("Hi there: ", datetime.now())
time.sleep(5)
async def slack_main():
loop = asyncio.get_event_loop()
rtm_client = slack.RTMClient(token='x', run_async=True, loop=loop)
executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
await asyncio.gather(
loop.run_in_executor(executor, sync_loop),
rtm_client.start()
)
if __name__ == "__main__":
asyncio.run(slack_main())
Related
Binance API & python-binance offers async functionality for non-blocking execution as per discussed in Async basics for Binance.
I am using BinanceSocketManager listening (async non-blocking) to live data via websocket.
In scenarios like network intermittent connection lost, I wish to add an auto-reconnect feature to my project. But I can't seems to find any info with BinanceSocketManager. I was only able to find a guide which uses ThreadedWebsocketManager, but it was not an async implementation.
Does anyone know how to implement a Binance websocket disconnect detection and auto-reconnect mechanism?
Here is some code of what I have so far:
import asyncio
from binance import AsyncClient, BinanceSocketManager
async def main():
client = await AsyncClient.create()
await kline_listener(client)
async def kline_listener(client):
bm = BinanceSocketManager(client)
async with bm.kline_socket(symbol='BTCUSDT') as stream:
while True:
res = await stream.recv()
print(res)
# a way detect websocket error/disconnect, callback 'disconnect_callback'
async def disconnect_callback():
await client.close_connection()
await main() # restart client and kline socket
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
In case someone else is looking at this, for this, you should be looking at the BinanceAPIException. Code could look something like this then:
from binance import AsyncClient, BinanceSocketManager
from binance.exceptions import BinanceAPIException
async def main():
client = await AsyncClient.create()
bm = BinanceSocketManager(client, user_timeout=60)
# start any sockets here, i.e a trade socket
kline_candles = bm.kline_socket('BNBUSDT', interval=client.KLINE_INTERVAL_1MINUTE)
# start receiving messages
try:
status = await client.get_system_status()
print(status['msg'])
async with kline_candles as stream:
for _ in range(5):
res = await stream.recv() # create/await response
await process_message(msg=res, client=client) # process message
except BinanceAPIException as e:
print(e)
await disconnect_callback(client=client)
async def disconnect_callback(client):
await client.close_connection() # close connection
time.sleep(60) # wait a minute before restarting
await main() # restart client and kline socket
async def process_message(msg, client):
if msg['e'] == 'error':
await disconnect_callback(client=client)
print('ERROR OCCURED')
else:
candle = msg['k'] # get only the candle info within the general dict
start_time = datetime.utcfromtimestamp(candle['t']/1000).strftime('%Y-%m-%d %H:%M:%S')
close_time = datetime.utcfromtimestamp(candle['T']/1000).strftime('%Y-%m-%d %H:%M:%S')
print(f'__ start: {start_time}, close: {close_time}')
print(msg)
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
The disconnect has not been tested yet, but I assume this will work. If anyone has any additional notes, just let me know.
I have tested above code and it proves rather stable.
Here are some improvements I have made.
I'm not sure what happens if your internet connection is completely gone when this line is executed:
client = await AsyncClient.create()
This could probably be solved like this (I'm open for better ideas):
while True:
try:
client = await AsyncClient.create()
except Exception as error_msg:
print(f"error: {error_msg}")
# should we add a sleep here?
# time.sleep(3)
else:
print("finally got through the loop")
break
Surrounding this with a try/except is a good idea:
bm = BinanceSocketManager(client, user_timeout=60)
The call to stream.recv() should be extended with asyncio.wait_for() to cover the situation, when there is no data coming in for a longer period of time. It usually means there's something wrong.
async with kline_candles as stream:
for _ in range(5):
try:
res = await asyncio.wait_for(stream.recv(), timeout=60) # create/await response
await process_message(msg=res, client=client) # process message
except (asyncio.TimeoutError, websockets.exceptions.ConnectionClosed, asyncio.exceptions.CancelledError, asyncio.exceptions.TimeoutError) as error_msg_1:
print(f"Error! in main loop 1:\n{error_msg_1}")
await disconnect_callback(client=client)
I am writing a bot and I need to implement the following functionality: the bot once every 10 minutes(for example) parse a certain URL and if there were changes from the previous call, writes to the chat.
Since the bot is also engaged in other things, I decided to loop the parsing in the function with sleep at the end. If there are changes, I try to send a message to the chat, but then a problem happens.
Since a successful combination of circumstances does not arise from an event in the chat, I can't pull the "entity" from the "event" for the "send_message" function. therefore, we have to get through the "get_entity" function and links to the chat as a parameter, but for some reason this does not work from another stream. below is a simplified code:
import threading, queue
from time import sleep
import asyncio
from telethon.sync import TelegramClient, events
import config as cfg
bot = TelegramClient('Bot', cfg.api_id, cfg.api_hash)
#bot.on(events.NewMessage(pattern=r'^(?i)(idchat){1}$'))
async def echoidchat(event):
channelaa = await bot.get_entity('https://t.me/elvistest')
await bot.send_message(channelaa, 'ответ')
def parseurls():
for x in range(10):
q.put(x)
pass
async def pre_sendmsg():
while True:
try:
msg = q.get_nowait()
except Exception as e:
await asyncio.sleep(1.0)
else:
await sendmsg(msg)
q.task_done()
async def sendmsg(msg):
channel = await bot.get_entity('https://t.me/elvistest')
await bot.send_message(channel, f'ответ из другого потока {msg}')
if __name__ == '__main__':
q = queue.Queue()
parseurls()
bot.start(bot_token=cfg.bot_token)
threading.Thread(target=asyncio.run, daemon=True, args=(pre_sendmsg(),)).start()
bot.run_until_disconnected()
The thing is that on the line " boot.get_entity" nothing happens. The script execution is lost somewhere and does not go further, that is, the next line with "bot. send_message" is simply not executed. however, "def echoidchat" is working at this time.
Well done!
This is work like I want.
import random
import threading, queue
from time import sleep
import asyncio
from telethon import TelegramClient, events
import config as cfg
bot = TelegramClient('Bot', cfg.api_id, cfg.api_hash)
#bot.on(events.NewMessage(pattern=r'^(?i)(idchat){1}$'))
async def echoidchat(event):
await bot.send_message(event.chat, 'ответ')
async def parseurls():
while True:
ts = abs(int(random.random()*10))
print(f'parseurls({ts})')
await sendmsg(ts)
await asyncio.sleep(ts)
async def sendmsg(msg):
print(f'sendmsg({msg}) - start')
channel = await bot.get_entity('https://t.me/elvistest')
await bot.send_message(channel, f'ответ из другого потока {msg}')
print(f'sendmsg({msg}) - done')
def main():
bot.start(bot_token=cfg.bot_token)
loop = asyncio.get_event_loop()
tasks = [
loop.create_task(parseurls()),
loop.create_task(bot.run_until_disconnected()),
]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()
if __name__ == '__main__':
main()
I'm trying to run 2 async functions forever. Can someone help me? My code and error message is provided below.
Code:
import websockets
import asyncio
import json
import time
async def time(loop):
while True:
millis = await int(round(time.time() * 1000))
print(millis)
await asyncio.sleep(0.001)
async def stream(loop):
async with websockets.connect('wss://www.bitmex.com/realtime?subscribe=trade:XBTUSD')
as websocket:
while True:
try:
data = await websocket.recv()
data = json.loads(data)
print(data['data'][-1]['price'])
except KeyError:
pass
except TypeError:
pass
loop = asyncio.get_event_loop()
async def main():
await asyncio.gather(loop.run_until_complete(stream(loop)),
loop.run_until_complete(time(loop)))
if __name__ == "__main__":
asyncio.run(main())
Error:
Exception has occurred: RuntimeError
Cannot run the event loop while another loop is running
Well, there are few errors with your snippet code:
You can't name your first function as time because it'll generate a conflict with time built-in function
Why are you passing loop as parameter if you're not gonna use it ? It's useless.
You can't await a function if it's not awaitable i.e. int is a synchronous built-in method.
Making the proper corrections it'll be something like this:
import websockets
import asyncio
import json
import time
async def another_name():
while True:
millis = int(round(time.time() * 1000))
print(millis)
await asyncio.sleep(0.001)
async def stream():
async with websockets.connect('wss://www.bitmex.com/realtime?subscribe=trade:XBTUSD')
as websocket:
while True:
try:
data = await websocket.recv()
data = json.loads(data)
print(data['data'][-1]['price'])
except KeyError:
pass
except TypeError:
pass
await asyncio.sleep(0.001) #Added
if __name__ == "__main__":
loop = asyncio.get_event_loop()
coros = []
coros.append(another_name())
coros.append(stream())
loop.run_until_complete(asyncio.gather(*coros))
The line await asyncio.sleep(0.001) in stream() function is compulsory otherwise it won't never let the another_name() function runs.
I'm trying to read messages from Azure ServiceBus Topics using async/await and then forward the content to another application via HTTP. My code is simple:
import asyncio
from aiohttp import ClientSession
from azure.servicebus.aio.async_client import ServiceBusService
bus_service = ServiceBusService(service_namespace=..., shared_access_key_name=..., shared_access_key_value=...)
async def watch(topic_name, subscription_name):
print('{} started'.format(topic_name))
message = bus_service.receive_subscription_message(topic_name, subscription_name, peek_lock=False, timeout=1)
if message.body is not None:
async with ClientSession() as session:
await session.post('ip:port/endpoint',
headers={'Content-type': 'application/x-www-form-urlencoded'},
data={'data': message.body.decode()})
async def do():
while True:
for topic in ['topic1', 'topic2', 'topic3']:
await watch(topic, 'watcher')
if __name__ == "__main__":
asyncio.run(do())
I want to look for messages (forever) from various topics and when a message arrives send the POST. I import the aio package from azure which should work in an async way. After many attempts, the only solution I got is this with while True and setting the timeout=1. This is not what I wanted, I'm doing it sequentially.
azure-servicebus version 0.50.3.
This is my first time with async/await probably I'm missing something...
Any solution/suggestions?
Here's how you'll do it with the latest major version v7 of servicebus
Please take a look a the async samples to send and receive subscription messages
https://github.com/Azure/azure-sdk-for-python/blob/04290863fa8963ec525a0b2f4079595287e15d93/sdk/servicebus/azure-servicebus/samples/async_samples/sample_code_servicebus_async.py
import os
import asyncio
from aiohttp import ClientSession
from azure.servicebus.aio import ServiceBusClient
connstr = os.environ['SERVICE_BUS_CONNECTION_STR']
topic_name = os.environ['SERVICE_BUS_TOPIC_NAME']
subscription_name = os.environ['SERVICE_BUS_SUBSCRIPTION_NAME']
async def watch(topic_name, subscription_name):
async with ServiceBusClient.from_connection_string(conn_str=servicebus_connection_str) as servicebus_client:
subscription_receiver = servicebus_client.get_subscription_receiver(
topic_name=topic_name,
subscription_name=subscription_name,
)
async with subscription_receiver:
message = await subscription_receiver.receive_messages(max_wait_time=1)
if message.body is not None:
async with ClientSession() as session:
await session.post('ip:port/endpoint',
headers={'Content-type': 'application/x-www-form-urlencoded'},
data={'data': message.body.decode()})
async def do():
while True:
for topic in ['topic1', 'topic2', 'topic3']:
await watch(topic, 'watcher')
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(do())
You will have to use the package : azure.servicebus.aio
They have the below modules for async :
We will have to use the Receive handler class - it can instantiated with get_receiver() method. With this object you will be able to iterate through the message Asynchronously. Spun up a sample script which does that you could further optimise it :
from azure.servicebus.aio import SubscriptionClient
import asyncio
import nest_asyncio
nest_asyncio.apply()
Receiving = True
#Topic 1 receiver :
conn_str= "<>"
name="Allmessages1"
SubsClient = SubscriptionClient.from_connection_string(conn_str, name)
receiver = SubsClient.get_receiver()
#Topic 2 receiver :
conn_str2= "<>"
name2="Allmessages2"
SubsClient2 = SubscriptionClient.from_connection_string(conn_str2, name2)
receiver2 = SubsClient2.get_receiver()
#obj= SubscriptionClient("svijayservicebus","mytopic1", shared_access_key_name="RootManageSharedAccessKey", shared_access_key_value="ySr+maBCmIRDK4I1aGgkoBl5sNNxJt4HTwINo0FQ/tc=")
async def receive_message_from1():
await receiver.open()
print("Opening the Receiver for Topic1")
async with receiver:
while(Receiving):
msgs = await receiver.fetch_next()
for m in msgs:
print("Received the message from topic 1.....")
print(str(m))
await m.complete()
async def receive_message_from2():
await receiver2.open()
print("Opening the Receiver for Topic2")
async with receiver2:
while(Receiving):
msgs = await receiver2.fetch_next()
for m in msgs:
print("Received the message from topic 2.....")
print(str(m))
await m.complete()
loop = asyncio.get_event_loop()
topic1receiver = loop.create_task(receive_message_from1())
topic2receiver = loop.create_task(receive_message_from2())
I have created two tasks to facilitate the concurrency. You could refer this post to get more clarity on them.
Output :
I have a manager to cache some user settings. I want to cleanup it every hour for inactive users (10 seconds in my example). I try to use aiojobs for this. I spawn the same job inside the job coroutine.
from aiohttp import web
from aiojobs.aiohttp import setup, get_scheduler
import asyncio
async def cleanup(scheduler):
await asyncio.sleep(10)
print('do cleanup')
await scheduler.spawn(cleanup(scheduler))
async def handler(request):
if not request.app['init']:
scheduler = get_scheduler(request)
await scheduler.spawn(cleanup(scheduler))
request.app['init'] = True
return web.Response(text = 'ok')
def main():
app = web.Application()
app.router.add_get('/', handler)
setup(app)
app['init'] = False
web.run_app(app, host='127.0.0.1', port = 8000)
main()
Is it a good solution? Should I create my own scheduler because my job does not relate to a request?
I want some background tasks to run in the same loop as the aiohttp web server, even before any http requests arrive. It looks like aiojobs doesn't help me so I'm using something that looks like this. I'm using a janus queue because my real application makes blocking calls from another thread. I don't know aiosync well, so this may be the blind leading the blind.
import asyncio
from aiohttp import web
from aiojobs.aiohttp import setup
import janus
async def ticket_maker(q: janus.Queue):
counter = 1
while True:
print(f'Made ticket {counter}')
await q.async_q.put(counter)
await asyncio.sleep(1)
counter += 1
async def handler(request):
q: janus.Queue = request.app.get('Q')
ticket = await q.async_q.get()
return web.Response(text=f'You got ticket {ticket}')
def main():
q = janus.Queue()
app = web.Application()
asyncio.get_event_loop().create_task(ticket_maker(q))
app.router.add_get('/', handler)
app['Q'] = q
setup(app)
web.run_app(app, port=8080)
if __name__ == '__main__':
main()