I'm trying to read messages from Azure ServiceBus Topics using async/await and then forward the content to another application via HTTP. My code is simple:
import asyncio
from aiohttp import ClientSession
from azure.servicebus.aio.async_client import ServiceBusService
bus_service = ServiceBusService(service_namespace=..., shared_access_key_name=..., shared_access_key_value=...)
async def watch(topic_name, subscription_name):
print('{} started'.format(topic_name))
message = bus_service.receive_subscription_message(topic_name, subscription_name, peek_lock=False, timeout=1)
if message.body is not None:
async with ClientSession() as session:
await session.post('ip:port/endpoint',
headers={'Content-type': 'application/x-www-form-urlencoded'},
data={'data': message.body.decode()})
async def do():
while True:
for topic in ['topic1', 'topic2', 'topic3']:
await watch(topic, 'watcher')
if __name__ == "__main__":
asyncio.run(do())
I want to look for messages (forever) from various topics and when a message arrives send the POST. I import the aio package from azure which should work in an async way. After many attempts, the only solution I got is this with while True and setting the timeout=1. This is not what I wanted, I'm doing it sequentially.
azure-servicebus version 0.50.3.
This is my first time with async/await probably I'm missing something...
Any solution/suggestions?
Here's how you'll do it with the latest major version v7 of servicebus
Please take a look a the async samples to send and receive subscription messages
https://github.com/Azure/azure-sdk-for-python/blob/04290863fa8963ec525a0b2f4079595287e15d93/sdk/servicebus/azure-servicebus/samples/async_samples/sample_code_servicebus_async.py
import os
import asyncio
from aiohttp import ClientSession
from azure.servicebus.aio import ServiceBusClient
connstr = os.environ['SERVICE_BUS_CONNECTION_STR']
topic_name = os.environ['SERVICE_BUS_TOPIC_NAME']
subscription_name = os.environ['SERVICE_BUS_SUBSCRIPTION_NAME']
async def watch(topic_name, subscription_name):
async with ServiceBusClient.from_connection_string(conn_str=servicebus_connection_str) as servicebus_client:
subscription_receiver = servicebus_client.get_subscription_receiver(
topic_name=topic_name,
subscription_name=subscription_name,
)
async with subscription_receiver:
message = await subscription_receiver.receive_messages(max_wait_time=1)
if message.body is not None:
async with ClientSession() as session:
await session.post('ip:port/endpoint',
headers={'Content-type': 'application/x-www-form-urlencoded'},
data={'data': message.body.decode()})
async def do():
while True:
for topic in ['topic1', 'topic2', 'topic3']:
await watch(topic, 'watcher')
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(do())
You will have to use the package : azure.servicebus.aio
They have the below modules for async :
We will have to use the Receive handler class - it can instantiated with get_receiver() method. With this object you will be able to iterate through the message Asynchronously. Spun up a sample script which does that you could further optimise it :
from azure.servicebus.aio import SubscriptionClient
import asyncio
import nest_asyncio
nest_asyncio.apply()
Receiving = True
#Topic 1 receiver :
conn_str= "<>"
name="Allmessages1"
SubsClient = SubscriptionClient.from_connection_string(conn_str, name)
receiver = SubsClient.get_receiver()
#Topic 2 receiver :
conn_str2= "<>"
name2="Allmessages2"
SubsClient2 = SubscriptionClient.from_connection_string(conn_str2, name2)
receiver2 = SubsClient2.get_receiver()
#obj= SubscriptionClient("svijayservicebus","mytopic1", shared_access_key_name="RootManageSharedAccessKey", shared_access_key_value="ySr+maBCmIRDK4I1aGgkoBl5sNNxJt4HTwINo0FQ/tc=")
async def receive_message_from1():
await receiver.open()
print("Opening the Receiver for Topic1")
async with receiver:
while(Receiving):
msgs = await receiver.fetch_next()
for m in msgs:
print("Received the message from topic 1.....")
print(str(m))
await m.complete()
async def receive_message_from2():
await receiver2.open()
print("Opening the Receiver for Topic2")
async with receiver2:
while(Receiving):
msgs = await receiver2.fetch_next()
for m in msgs:
print("Received the message from topic 2.....")
print(str(m))
await m.complete()
loop = asyncio.get_event_loop()
topic1receiver = loop.create_task(receive_message_from1())
topic2receiver = loop.create_task(receive_message_from2())
I have created two tasks to facilitate the concurrency. You could refer this post to get more clarity on them.
Output :
Related
I am writing a bot and I need to implement the following functionality: the bot once every 10 minutes(for example) parse a certain URL and if there were changes from the previous call, writes to the chat.
Since the bot is also engaged in other things, I decided to loop the parsing in the function with sleep at the end. If there are changes, I try to send a message to the chat, but then a problem happens.
Since a successful combination of circumstances does not arise from an event in the chat, I can't pull the "entity" from the "event" for the "send_message" function. therefore, we have to get through the "get_entity" function and links to the chat as a parameter, but for some reason this does not work from another stream. below is a simplified code:
import threading, queue
from time import sleep
import asyncio
from telethon.sync import TelegramClient, events
import config as cfg
bot = TelegramClient('Bot', cfg.api_id, cfg.api_hash)
#bot.on(events.NewMessage(pattern=r'^(?i)(idchat){1}$'))
async def echoidchat(event):
channelaa = await bot.get_entity('https://t.me/elvistest')
await bot.send_message(channelaa, 'ответ')
def parseurls():
for x in range(10):
q.put(x)
pass
async def pre_sendmsg():
while True:
try:
msg = q.get_nowait()
except Exception as e:
await asyncio.sleep(1.0)
else:
await sendmsg(msg)
q.task_done()
async def sendmsg(msg):
channel = await bot.get_entity('https://t.me/elvistest')
await bot.send_message(channel, f'ответ из другого потока {msg}')
if __name__ == '__main__':
q = queue.Queue()
parseurls()
bot.start(bot_token=cfg.bot_token)
threading.Thread(target=asyncio.run, daemon=True, args=(pre_sendmsg(),)).start()
bot.run_until_disconnected()
The thing is that on the line " boot.get_entity" nothing happens. The script execution is lost somewhere and does not go further, that is, the next line with "bot. send_message" is simply not executed. however, "def echoidchat" is working at this time.
Well done!
This is work like I want.
import random
import threading, queue
from time import sleep
import asyncio
from telethon import TelegramClient, events
import config as cfg
bot = TelegramClient('Bot', cfg.api_id, cfg.api_hash)
#bot.on(events.NewMessage(pattern=r'^(?i)(idchat){1}$'))
async def echoidchat(event):
await bot.send_message(event.chat, 'ответ')
async def parseurls():
while True:
ts = abs(int(random.random()*10))
print(f'parseurls({ts})')
await sendmsg(ts)
await asyncio.sleep(ts)
async def sendmsg(msg):
print(f'sendmsg({msg}) - start')
channel = await bot.get_entity('https://t.me/elvistest')
await bot.send_message(channel, f'ответ из другого потока {msg}')
print(f'sendmsg({msg}) - done')
def main():
bot.start(bot_token=cfg.bot_token)
loop = asyncio.get_event_loop()
tasks = [
loop.create_task(parseurls()),
loop.create_task(bot.run_until_disconnected()),
]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()
if __name__ == '__main__':
main()
Quart is a Python web framework which re-implements the Flask API on top of the asyncio coroutine system of Python. In my particular case, I have a Quart websocket endpoint which is supposed to have not just one source of incoming events, but two possible sources of events which are supposed to continue the asynchronous loop.
An example with one event source:
from quart import Quart, websocket
app = Quart(__name__)
#app.websocket("/echo")
def echo():
while True:
incoming_message = await websocket.receive()
await websocket.send(incoming_message)
Taken from https://pgjones.gitlab.io/quart/
This example has one source: the incoming message stream. But what is the correct pattern if I had two possible sources, one being await websocket.receive() and another one being something along the lines of await system.get_next_external_notification() .
If either of them arrives, I'd like to send a websocket message.
I think I'll have to use asyncio.wait(..., return_when=FIRST_COMPLETED), but how do I make sure that I miss no data (i.e. for the race condition that websocket.receive() and system.get_next_external_notification() both finish almost exactly at the same time) ? What's the correct pattern in this case?
An idea you could use is a Queue to join the events together from different sources, then have an async function listening in the background to that queue for requests. Something like this might get you started:
import asyncio
from quart import Quart, websocket
app = Quart(__name__)
#app.before_serving
async def startup():
print(f'starting')
app.q = asyncio.Queue(1)
asyncio.ensure_future(listener(app.q))
async def listener(q):
while True:
returnq, msg = await q.get()
print(msg)
await returnq.put(f'hi: {msg}')
#app.route("/echo/<message>")
async def echo(message):
while True:
returnq = asyncio.Queue(1)
await app.q.put((returnq, message))
response = await returnq.get()
return response
#app.route("/echo2/<message>")
async def echo2(message):
while True:
returnq = asyncio.Queue(1)
await app.q.put((returnq, message))
response = await returnq.get()
return response
Using Quart I am trying to receive data from one client via a websocket, then have the Quart websocket server send it to a different client via websocket.
The two clients will be alone sharing the same url, other pairs of clients will have their own urls. This echo test works for both clients individually:
#copilot_ext.websocket('/ws/<unique_id>')
async def ws(unique_id):
while True:
data = await websocket.receive()
await websocket.send(f"echo {data}")
I have tried broadcasting using the example here https://pgjones.gitlab.io/quart/tutorials/websocket_tutorial.html#broadcasting although I can catch and print the different websockets, have not had much luck sending data from one client to the other :(
connected_websockets = set()
def collect_websocket(func):
#wraps(func)
async def wrapper(*args, **kwargs):
global connected_websockets
send_channel, receive_channel = trio.open_memory_channel(2)
connected_websockets.add(send_channel)
try:
return await func(send_channel, *args, **kwargs)
finally:
connected_websockets.remove(send_channel)
return wrapper
#copilot_ext.websocket('/ws/<unique_id>')
#collect_websocket
async def ws(que, unique_id):
while True:
data = await websocket.receive()
for send_channel in connected_websockets:
await send_channel.send(f"message {data}")
print(send_channel)
Just storing the websocket object and iterating through them doesn't work either
connected_websockets = set()
#copilot_ext.websocket('/ws/<unique_id>')
async def ws(unique_id):
global connected_websockets
while True:
data = await websocket.receive()
connected_websockets.add(websocket)
for websockett in connected_websockets:
await websockett.send(f"message {data}")
print(type(websockett))
I think this snippet can form the basis of what you want to achieve. The idea is that rooms are a collection of queues keyed by the room id. Then each connected client has a queue in the room which any other clients put messages to. The send_task then runs in the background to send any messages to the client that are on its queue. I hope this makes sense,
import asyncio
from collections import defaultdict
from quart import Quart, websocket
app = Quart(__name__)
websocket_rooms = defaultdict(set)
async def send_task(ws, queue):
while True:
message = await queue.get()
await ws.send(message)
#app.websocket("/ws/<id>/")
async def ws(id):
global websocket_rooms
queue = asyncio.Queue()
websocket_rooms[id].add(queue)
try:
task = asyncio.ensure_future(send_task(websocket._get_current_object(), queue))
while True:
message = await websocket.receive()
for other in websocket_rooms[id]:
if other is not queue:
await other.put(message)
finally:
task.cancel()
await task
websocket_rooms[id].remove(queue)
I am using websocket library in python for one of my projects. It works for me but I am curious to know how it works. I could not find this in the documentation. Specifically, for the example given in the docs,
#!/usr/bin/env python
# WS server example that synchronizes state across clients
import asyncio
import json
import logging
import websockets
logging.basicConfig()
STATE = {"value": 0}
USERS = set()
def state_event():
return json.dumps({"type": "state", **STATE})
def users_event():
return json.dumps({"type": "users", "count": len(USERS)})
async def notify_state():
if USERS: # asyncio.wait doesn't accept an empty list
message = state_event()
await asyncio.wait([user.send(message) for user in USERS])
async def notify_users():
if USERS: # asyncio.wait doesn't accept an empty list
message = users_event()
await asyncio.wait([user.send(message) for user in USERS])
async def register(websocket):
USERS.add(websocket)
await notify_users()
async def unregister(websocket):
USERS.remove(websocket)
await notify_users()
async def counter(websocket, path):
# register(websocket) sends user_event() to websocket
await register(websocket)
try:
await websocket.send(state_event())
async for message in websocket:
data = json.loads(message)
if data["action"] == "minus":
STATE["value"] -= 1
await notify_state()
elif data["action"] == "plus":
STATE["value"] += 1
await notify_state()
else:
logging.error("unsupported event: {}", data)
finally:
await unregister(websocket)
start_server = websockets.serve(counter, "localhost", 6789)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Whenever, I disconnect, the event unregister() gets fired. How does websocket come to know that I have disconnected?
My guess is the line async for message in websocket: has to do something with it but I do not know the details. Any insight on it will be appreciated.
With the websockets module, disconnections are handled as exceptions. For example, if you wanted to handle different kinds of disconnections, you could do something like this:
try:
await websocket.send(state_event())
async for message in websocket:
# Your code...
except websockets.exceptions.ConnectionClosedOK:
print("Client disconnected OK")
except websockets.exceptions.ConnectionClosedError:
print("Client disconnected unexpectedly")
finally:
await unregister(websocket)
More information can be found in the documentation.
I tried out the example below (from this page):
nc = NATS()
await nc.connect(servers=["nats://demo.nats.io:4222"])
future = asyncio.Future()
async def cb(msg):
nonlocal future
future.set_result(msg)
await nc.subscribe("updates", cb=cb)
await nc.publish("updates", b'All is Well')
await nc.flush()
# Wait for message to come in
msg = await asyncio.wait_for(future, 1)
But this only seems useful for receiving one message. How would I subscribe and keep receiving messages?
I've also seen the package example, but it seems to just play both sides of the conversation, then quit.
You can find a long running service example as well: https://github.com/nats-io/nats.py/blob/master/examples/service.py
import asyncio
from nats.aio.client import Client as NATS
async def run(loop):
nc = NATS()
async def disconnected_cb():
print("Got disconnected...")
async def reconnected_cb():
print("Got reconnected...")
await nc.connect("127.0.0.1",
reconnected_cb=reconnected_cb,
disconnected_cb=disconnected_cb,
max_reconnect_attempts=-1,
loop=loop)
async def help_request(msg):
subject = msg.subject
reply = msg.reply
data = msg.data.decode()
print("Received a message on '{subject} {reply}': {data}".format(
subject=subject, reply=reply, data=data))
await nc.publish(reply, b'I can help')
# Use queue named 'workers' for distributing requests
# among subscribers.
await nc.subscribe("help", "workers", help_request)
print("Listening for requests on 'help' subject...")
for i in range(1, 1000000):
await asyncio.sleep(1)
try:
response = await nc.request("help", b'hi')
print(response)
except Exception as e:
print("Error:", e)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(run(loop))
loop.run_forever()
loop.close()
Don't know much about python, but looks like you are just waiting for one message and program ends. You should look at the subscriber example here. As you can see, there is a loop to wait forever or for the SIGTERM signal.