I have set up an endpoint in Express.js that can start a long "for" loop and return a response before the loop finishes (thanks to Promises which push - I believe - the function to message queue):
const express = require('express')
const app = express()
const port = 3000
const long_task = () => {
for (let i=0; i<100000; i++) {
return new Promise(r => r(console.log(i)))
}
}
app.get('/', (req, res) => {
long_task()
res.send('endpoint returned')
})
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
})
(for now it does not matter that the server will not be able to process any other requests before the loop finishes).
Now, I want to mimic this in a Python framework, preferably FastAPI, due to its async capabilities.
So far I have this:
async def async_print(value):
print(value)
async def long_task():
for i in range(10000):
await async_print(i)
#app.get('/')
async def testing_long_task():
await long_task()
return 'endpoint returned'
But, as expected, the code will pause at await long_task(), and only return a response once the loop is finished.
And if I replace await long_task() with just long_task() I will get an error that the coroutine long_task was never awaited.
Is there any way to achieve such async for loop in FastAPI without using its background tasks or task queue like Celery?
Related
I want to make a websocket server using Fastapi then get a message from a manager then broadcast it to all clients.
The messages are just text.
Here is the process:
The manager connects to server.
The clients connect to server.
The manager send a message to server and server receive the message(could send OK status or response for error handling)
The server broadcast the message to all clients(no matter to manager or not)
The manager and clients are preferred to be MQL4/5 Experts.
A simple topology of the system:
The server is using fastapi sockets
FastAPI websocket
The simple server code like The manager of system:
from typing import List
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
from fastapi.responses import HTMLResponse
app = FastAPI()
class ConnectionManager:
def __init__(self):
self.active_connections: List[WebSocket] = []
async def connect(self, websocket: WebSocket):
await websocket.accept()
self.active_connections.append(websocket)
def disconnect(self, websocket: WebSocket):
self.active_connections.remove(websocket)
async def send_personal_message(self, message: str, websocket: WebSocket):
await websocket.send_text(message)
async def broadcast(self, message: str):
for connection in self.active_connections:
await connection.send_text(message)
manager = ConnectionManager()
#app.get("/")
async def get():
return (f"Hi!")
#app.websocket("/ws/{client_id}")
async def websocket_endpoint(websocket: WebSocket, client_id: int):
await manager.connect(websocket)
try:
while True:
data = await websocket.receive_text()
await manager.send_personal_message(f"You wrote: {data}", websocket)
await manager.broadcast(f"Client #{client_id} says: {data}")
except WebSocketDisconnect:
manager.disconnect(websocket)
await manager.broadcast(f"Hello everyone")
The connection manager in the above code can broadcast a message to all client(in that case, it is used for disconnection handling, but it is not my case).
Also, a sample piece of code for the client in MQL5:
int socket=SocketCreate();
if(socket!=INVALID_HANDLE)
{
if(SocketConnect(socket,"127.0.0.1",9091,1000)) {
Print("Connected to "," 127.0.0.1",":",9091);
string tosend = "heartbeat";
string received = socksend(socket, tosend) ? socketreceive(socket, 10) : "";
Print(received);
string sep="_"; // A separator as a character
ushort u_sep; // The code of the separator character
string result[]; // An array to get strings
u_sep=StringGetCharacter(sep,0);
//--- Split the string to substrings
int k=StringSplit(received,u_sep,result);
Print(result[0]);
Print(result[1]);
Print(result[2]);
}
}
//---
bool socksend(int sock,string request) {
char req[];
int len=StringToCharArray(request,req)-1;
if(len<0) return(false);
return(SocketSend(sock,req,len)==len);
}
//---
string socketreceive(int sock,int timeout) {
char rsp[];
string result="";
uint len;
uint timeout_check=GetTickCount()+timeout;
do {
len=SocketIsReadable(sock);
if(len) {
int rsp_len;
rsp_len=SocketRead(sock,rsp,len,timeout);
if(rsp_len>0) {
result+=CharArrayToString(rsp,0,rsp_len);
}
}
} while((GetTickCount()<timeout_check) && !IsStopped());
return result;
}
I want to know how can I connect the manager to system the get it's message and broadcast it.
I want to know how can I make the system.
I will try to explain what I am doing with an example, say I am building a weather client. The browser sends a message over websocket, eg:
{
"city": "Chicago",
"country": "US"
}
The server queries the weather every 5 minutes and updates the browser back with the latest data.
Now the browser could send another message, eg:
{
"city": "Bangalore",
"country": "IN"
}
Now I the server should STOP updating the weather details of Chicago and start updating the details about Bangalore, i.e. simultaneously send / receive messages over websocket. How should I go about implementing this?
Currently I have this but this only updates the browser on receiving an event:
#app.websocket("/ws")
async def read_webscoket(websocket: WebSocket):
await websocket.accept()
weather_client = WeatherClient(client)
while True:
data = await websocket.receive_json()
weather = await weather_client.weather(data)
await websocket.send_json(weather.dict())
If I move websocket.receive_json() outside the loop, I won't be able to continuously listen to the message from browser. I guess I need to spin up two asyncio tasks but I am not quite able to nail down the implementation since I am new to asynchronous way of programming.
The simplest way to do this is like you mentioned moving the reading outside of the loop in a separate task. In this paradigm you'll need to update a local variable with the latest data, making your code look something like this:
#app.websocket("/ws")
async def read_webscoket(websocket: WebSocket):
await websocket.accept()
json_data = await websocket.receive_json()
async def read_from_socket(websocket: WebSocket):
nonlocal json_data
async for data in websocket.iter_json():
json_data = data
asyncio.create_task(read_from_socket(websocket))
while True:
print(f"getting weather data for {json_data}")
await asyncio.sleep(1) # simulate a slow call to the weather service
Note I've used the iter_json asynchronous generator, which amounts to an infinite loop over receive_json.
This will work but may have a bug depending on your requirements. Imagine that the weather service takes 10 seconds to complete and in that time the user sends three requests for different cities over the socket. In the code above you'll only get the latest city the user sent. That might be fine for your application, but if you need to keep track of all that the user sent you'll need to use a queue. In this paradigm you'll have one task reading data and putting it on the queue and one task getting data from the queue and querying the weather service. You'll then run these concurrently with gather.
#app.websocket("/wsqueue")
async def read_webscoket(websocket: WebSocket):
await websocket.accept()
queue = asyncio.queues.Queue()
async def read_from_socket(websocket: WebSocket):
async for data in websocket.iter_json():
print(f"putting {data} in the queue")
queue.put_nowait(data)
async def get_data_and_send():
data = await queue.get()
while True:
if queue.empty():
print(f"getting weather data for {data}")
await asyncio.sleep(1)
else:
data = queue.get_nowait()
print(f"Setting data to {data}")
await asyncio.gather(read_from_socket(websocket), get_data_and_send())
In this way, you won't lose data the user sends. In the example above, I only get weather data for the latest the user requests, but you still have access to all data sent.
EDIT: To answer your question in the comments, a queue approach is probably best to cancel tasks when new requests come in. Basically move the long-running task you want to be able to cancel into its own coroutine function (in this example read_and_send_to_client) and run it as a task. When new data comes in, if that task is not finished, cancel it and then create a new one.
async def read_and_send_to_client(data):
print(f'reading {data} from client')
await asyncio.sleep(10) # simulate a slow call
print(f'finished reading {data}, sending to websocket client')
#app.websocket("/wsqueue")
async def read_webscoket(websocket: WebSocket):
await websocket.accept()
queue = asyncio.queues.Queue()
async def read_from_socket(websocket: WebSocket):
async for data in websocket.iter_json():
print(f"putting {data} in the queue")
queue.put_nowait(data)
async def get_data_and_send():
data = await queue.get()
fetch_task = asyncio.create_task(read_and_send_to_client(data))
while True:
data = await queue.get()
if not fetch_task.done():
print(f'Got new data while task not complete, canceling.')
fetch_task.cancel()
fetch_task = asyncio.create_task(read_and_send_to_client(data))
await asyncio.gather(read_from_socket(websocket), get_data_and_send())
I'm setting up a Discord server for me and my friends, and I'm in need of a bot able to add 20 reactions to any message in which a certain trigger phrase was typed.
I used to have a bot that could do that, but it has been updated and limited to 3 reactions, which isn't working for me.
So here I am, with my very basic understanding of programming, trying to setup a very basic bot for that sole purpose.
I already tried a simple bot proposition found on YT, but it didn't work at all, and I'm at a loss to know why and how to fix it.
So this was the proposition I tried and didn't manage to make functional:
import discord
from discord.ext import commands
from discord.ext.commands import Bot
import asyncio
bot = commands.Bot(command_prefix='!')
#bot.event
async def on_ready():
print ("Ready to react, boss !")
#bot.event
async def on_message(message):
if(message.channel.id == "550373218758688790"):
await bot.add_reaction(message, ":war_tank:552569109108490252")
bot.run("NTY5OTQ0NTMyMzIyNjE1MzI2.XL4IBg.WH-Ms1DWKJN8qGBBLAxdGye0q2I")
So this one was supposed to react to every message with a reaction, and I was planning on working from there, but it didn't even work.
Instead, I got a message in the cmd log for each message and that looked like this:
Ignoring exception in on_message
Traceback (most recent call last):
File "G:\Jeux\Discord Bot\lib\site-packages\discord\client.py", line 255, in _run_event
await coro(*args, **kwargs)
File "G:\Bureau\TotemReact Bot\bot.py", line 16, in on_message
await bot.add_reaction(message, ":war_tank:552569109108490252")
AttributeError: 'Bot' object has no attribute 'add_reaction'
So if anyone with actual skills (considering mine, I will be easily impressed) can point me in the right direction, I will be more than happy.
Thank you for your time and attention.
You're looking at old tutorials. Client.add_reaction was moved to Message.add_reaction in discord.py 1.0
The functionality you describe could look something like:
default_emojis = [
"\N{GRINNING FACE}",
"\N{KEYCAP DIGIT ONE}"
]
custom_emojis = [
"war_tank"
]
async def react(message):
for emoji in default_emojis:
await message.add_reaction(emoji)
for emoji in message.guild.emojis:
if emoji.name in custom_emojis:
await message.add_reaction(emoji)
#bot.event
async def on_message(message):
if message.author == bot.user:
return
if "react to me" in message.content.lower():
await react(message)
First of all, you're going to want to change your token now, if you haven't already.
You're using syntax from a version of discord.py, v0.16, that isn't supported anymore.
See the guide for migrating to v1, specifically the Models are Stateful section.
You should use Message.add_reaction rather than Bot.add_reaction.
first register on the replit site
secondly create new repyl and create node.js
and create new file and rename server.js
and copy these codes
for server.js
const express = require('express');
const server = express();
server.all('/', (req, res)=>{
res.setHeader('Content-Type', 'text/html');
res.write('<link href="https://fonts.googleapis.com/css?family=Roboto Condensed" rel="stylesheet"> <style> body {font-family: "Roboto Condensed";font-size: 22px;} <p>Hosting Active</p>');
res.end();
})
function keepAlive(){
server.listen(3000, ()=>{console.log("Server is online!")});
}
module.exports = keepAlive;
for index.js
const Discord = require('discord.js');
const client = new Discord.Client();
const mySecret = process.env['mysecrettoken']
const keepAlive = require("./server");
client.on('message', msg => {
if (msg.attachments.size > 0) { //Make sure there are attachments at all
var react = false; //Do we react to the message? Default to false
msg.attachments.forEach(attachment => { //Check each attachment to see if it's a jpg, png, or jpeg
if (attachment.url.includes(".jpg") || attachment.url.includes(".png") || attachment.url.includes(".jpeg")) {
react = true; //It's an image! We want to react to the message
};
});
if (react === true) { //React to the message
msg.react('837608362719969291');
msg.react('837608381283827734');
msg.react('837608396659097640');
msg.react('837608412337668096');
msg.react('837608426821124096');
};
};
}
)
keepAlive();
client.login(process.env.mysecrettoken);
create your first discord bot and give it admin rights. You can watch youtube videos about it how create dc bot and give it to admin rights.
click this site https://discord.com/developers/applications/
create your dc bot give admin rights regenerate and copy your toke paste to replit secret values
copy url add monitoring sites and your discord bot is ready to use 7/24 using
copy your token and create your mysecrettoken
monitoring sites example : https://uptimerobot.com
😁😁😁😁
your dc bot is ready
I'm using websockets in an python project i'm working on. The websocket is being run in an thread and given 2 queue's from the parent thread. I'm using javascript to connect to the websocket server.
I'm able to get messages from the parent thread throughself.ssi.get(True) and passed them through to the javascript websocket client.
But i'm unable to receive messages from the client. When i use zaproxy i can see the messages going through. On the websocket server i'm also able to see the packets arrive on the interface. Python throws no error and logger.setLevel(logging.DEBUG) does not show messages arriving the same way as i'm able to see messages being send.
I've been trying to solve this but i've run out of ideas to find the problem, any help is welcome.
Python websocket server:
import websockets
import logging
import asyncio
import ssl
class websocket:
def __init__(self,ssi,sso):
self.ssi = ssi
self.sso = sso
logger = logging.getLogger('websockets')
logger.setLevel(logging.DEBUG)
# logger.addHandler(logging.FileHandler('debug.log'))
logger.addHandler(logging.StreamHandler())
sslc = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
sslc.load_cert_chain(
'keys/wss.server.crt',
'keys/wss.server.key')
loop = asyncio.new_event_loop()
wsrv = websockets.serve(
self.handler,
host='0.0.0.0',
port=9000,
ssl=sslc,
loop=loop)
loop.run_until_complete(wsrv)
loop.run_forever()
async def handler(self, wss, path):
consumer_task = asyncio.ensure_future(self.consumerHandler(wss, path))
producer_task = asyncio.ensure_future(self.producerHandler(wss, path))
done, pending = await asyncio.wait(
[consumer_task, producer_task],
return_when=asyncio.FIRST_COMPLETED,)
for task in pending:
task.cancel()
async def producerHandler(self, wss, path):
while True:
msg = await self.producer()
await wss.send(str(msg))
async def consumerHandler(self, wss, path):
async for msg in wss:
await self.consumer(msg)
async def producer(self):
return self.ssi.get(True)
async def consumer(self, msg):
self.sso.put(msg.data)
Javascript client:
var ws;
function ws_init() {
ws = new WebSocket("wss://pri.local:9000/");
ws.onopen = function(e) {
output("connected");
};
ws.onmessage = function(e) {
output("i: " + e.data);
};
ws.onclose = function() {
output("disconnect");
};
ws.onerror = function(e) {
output("onerror");
console.log(e)
};
}
function onSubmit() {
var input = document.getElementById("input");
ws.send(input.value);
output("o: " + input.value);
input.value = "";
input.focus();
}
function onCloseClick() {
ws.close();
}
function output(str) {
var log = document.getElementById("log");
var escaped = str.replace(/&/, "&").replace(/</, "<").
replace(/>/, ">").replace(/"/, """); // "
log.innerHTML = escaped + "<br>" + log.innerHTML;
}
I think the issue is that you're mixing the usage of the queue library and asyncio.queue.
queue is thread safe and so is a good mechanism for communicating between threads, but it doesn't have an async API, so you're blocking the websocket thread when you call self.ssi.get(True) which prevents any of the other websocket code from running.
asyncio.queue has the API you want (you can await queue.get()), but unfortunately isn't thread safe (it's designed for use within single threaded async applications).
You may be able to use loop.run_in_executor to await the blocking queue.get(True) call. See here for an example https://carlosmaniero.github.io/asyncio-handle-blocking-functions.html
I am trying to implement a simple web sockets server in Python by using this module. For learning purposes, the server should reply with a reversed version of what it received. For example, if the client sends "Hello Server", the server should respond with "revreS olleH". My code is based off the documentation here
Since an example of a consumer() and producer() function/coroutine wasn't provided in the documentation, I took a stab at creating them but think I am misunderstanding something not obvious to me. The code is currently returning the string 'nothing' instead of the reversed version of what the client sent.
FYI, since the machine I am using has Python 3.4.3, the code had to be adjusted to accommodate for that version. That's why you'll see newer code commented out, for now. Lots of documentation is included too as I learn this stuff.
Now, the codez...
index.py:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#########################
# Dependencies
#########################
# asyncio
# websockets
#########################
# Modules
#########################
import asyncio
import websockets
#########################
# Functions
#########################
# async indicates an asynchronous function.
# Calling them doesn't actually run them,
# but instead a coroutine object is returned,
# which can then be passed to the event loop to be executed later on.
# Python ≥ 3.5: async def producer(reply):
#asyncio.coroutine
def producer(reply=None):
"""Sends the reply to producer_handler."""
if reply is None:
return 'nothing'
else:
return reply
# Python ≥ 3.5: async def consumer(message):
#asyncio.coroutine
def consumer(message):
"""Reverses message then sends it to the producer."""
reply = message[::-1]
#await producer(reply)
yield from producer(reply)
# async def consumer_handler(websocket):
#asyncio.coroutine
def consumer_handler(websocket):
"""Handles incoming websocket messages."""
while True:
# await calls an asynchronous function.
#message = await websocket.recv()
message = yield from websocket.recv()
# Python ≥ 3.5: await consumer(message)
yield from consumer(message)
#async def producer_handler(websocket):
#asyncio.coroutine
def producer_handler(websocket):
"""Handles outgoing websocket messages."""
while True:
#message = await producer()
message = yield from producer()
#await websocket.send(message)
yield from websocket.send(message)
#async def handler(websocket, path):
#asyncio.coroutine
def handler(websocket, path):
"""Enables reading and writing messages on the same websocket connection."""
# A Future is an object that is supposed to have a result in the future.
# ensure_future:
# schedules the execution of a coroutine object,
# wraps it in a future, then returns a Task object.
# If the argument is a Future, it is returned directly.
# Python ≥ 3.5
#consumer_task = asyncio.ensure_future(consumer_handler(websocket))
#producer_task = asyncio.ensure_future(producer_handler(websocket))
consumer_task = asyncio.async(consumer_handler(websocket))
producer_task = asyncio.async(producer_handler(websocket))
# .wait:
# wait for the Futures and coroutine objects given
# by the sequence futures to complete. Coroutines will be
# wrapped in Tasks. Returns two sets of Future: (done, pending).
#done, pending = await asyncio.wait(
done, pending = yield from asyncio.wait(
# The futures.
[consumer_task, producer_task],
# FIRST_COMPLETED: the function will return when
# any future finishes or is cancelled.
return_when=asyncio.FIRST_COMPLETED,
)
for task in pending:
task.cancel()
#########################
# Start script
#########################
def main():
# Creates a WebSocket server.
start_server = websockets.serve(handler, '127.0.0.1', 8000)
# Get the event loop for the current context.
# Run until the Future is done.
asyncio.get_event_loop().run_until_complete(start_server)
# Run until stop() is called.
asyncio.get_event_loop().run_forever()
#########################
# Script entry point.
#########################
if __name__ == '__main__':
main()
index.html:
<!DOCTYPE html>
<html>
<head>
<title>WebSocket demo</title>
</head>
<body>
<script>
// Create the websocket.
var ws = new WebSocket("ws://127.0.0.1:8000/"),
messages = document.createElement('ul');
// Called when the websocket is opened.
ws.onopen = function(event) {
ws.send('Hello Server!');
};
// Called when a message is received from server.
ws.onmessage = function(event) {
var messages = document.getElementsByTagName('ul')[0],
message = document.createElement('li'),
content = document.createTextNode(event.data);
message.appendChild(content);
messages.appendChild(message);
};
document.body.appendChild(messages);
</script>
</body>
</html>
Not completely sure on this, but I think you misinterpreted the docs. The consumer shouldn't be calling the producer.
The "Hello Server!" the HTML file sends goes through consumer_handler to consumer to producer, but the yield from statements means that the reversed string ends up back in the consumer_handler, as the result of yield from consumer(message).
On the other hand, producer_handler calls producer many times without an argument (from message = yield from producer()), which is what creates the nothing that gets sent to the HTML file. It doesn't receive the consumer's string.
Instead, there should be a queue or something where the consumer pushes to and the producer takes from, like in this example.
Thanks.