Develop Reference

Combining Restapi and Websockets

I have a rest api server which makes call to some other Apis,I am accessing the data I get from the server on a react js frontend,But for certain usecases I need to fetch real time data from backed,is there any way do this together,below is my code
from flask import Flask,request
from flask_cors import CORS
from tuya_connector import TuyaOpenAPI, TUYA_LOGGER
app = Flask(__name__)
CORS(app)
#app.get("/api/device/<string:deviceid>")
def getdata(deviceid):
ACCESS_ID = ""
ACCESS_KEY = ""
API_ENDPOINT = ""
# Enable debug log
# Init OpenAPI and connect
openapi = TuyaOpenAPI(API_ENDPOINT, ACCESS_ID, ACCESS_KEY)
openapi.connect()
# Set up device_id
DEVICE_ID = deviceid
# Call APIs from Tuya
# Get the device information
response = openapi.get("/v1.0/devices/{}".format(DEVICE_ID))
return response
I want to have traditional request response service along with real time data fetching

Websockets endpoints are exactly what you're looking for. If that is not too late, I'd recommend switching to FastAPI which supports WebSockets "natively" (out-of-the-box) - https://fastapi.tiangolo.com/advanced/websockets
If you need to keep using Flask, there are a few packages that allow you to add WebSockets endpoints: https://flask-sock.readthedocs.io/en/latest/
With FastAPI, this is that simple:
#app.get("/")
async def get():
return {"msg": "This is a regular HTTP endpoint"}
#app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
while True:
data = await websocket.receive_text()
await websocket.send_text(f"Message text was: {data}")

How to use Blocks correctly to load AWS S3 credentials in Prefect?

I am using Prefect. And I tried to download a file from S3.
When I hard coded the AWS credentials, the file can be downloaded successfully:
import asyncio
from prefect_aws.s3 import s3_download
from prefect_aws.credentials import AwsCredentials
from prefect import flow, get_run_logger
#flow
async def fetch_taxi_data():
logger = get_run_logger()
credentials = AwsCredentials(
aws_access_key_id="xxx",
aws_secret_access_key="xxx",
)
data = await s3_download(
bucket="hongbomiao-bucket",
key="hm-airflow/taxi.csv",
aws_credentials=credentials,
)
logger.info(data)
if __name__ == "__main__":
asyncio.run(fetch_taxi_data())
Now I tried to load the credentials from Prefect Blocks.
I created a AWS Credentials Block:
However,
aws_credentials_block = AwsCredentials.load("aws-credentials-block")
data = await s3_download(
bucket="hongbomiao-bucket",
key="hm-airflow/taxi.csv",
aws_credentials=aws_credentials_block,
)
throws the error:
AttributeError: 'coroutine' object has no attribute 'get_boto3_session'
And
aws_credentials_block = AwsCredentials.load("aws-credentials-block")
credentials = AwsCredentials(
aws_access_key_id=aws_credentials_block.aws_access_key_id,
aws_secret_access_key=aws_credentials_block.aws_secret_access_key,
)
data = await s3_download(
bucket="hongbomiao-bucket",
key="hm-airflow/taxi.csv",
aws_credentials=credentials,
)
throws the error:
AttributeError: 'coroutine' object has no attribute 'aws_access_key_id'
I didn't find any useful document about how to use it.
Am I supposed to use Blocks to load credentials? If it is, what is the correct way to use Blocks correctly in Prefect? Thanks!

I just found the snippet in the screenshot in the question misses an await.
After adding await, it works now!
aws_credentials_block = await AwsCredentials.load("aws-credentials-block")
data = await s3_download(
bucket="hongbomiao-bucket",
key="hm-airflow/taxi.csv",
aws_credentials=aws_credentials_block,
)
UPDATE:
Got an answer from Michael Adkins on GitHub, and thanks!
await is only needed if you're writing an async flow or task. For users writing synchronous code, an await is not needed (and not possible). Most of our users are writing synchronous code and the example in the UI is in a synchronous context so it does not include the await.
I saw the source code at
https://github.com/PrefectHQ/prefect/blob/1dcd45637914896c60b7d49254a34e95a9ce56ea/src/prefect/blocks/core.py#L601-L604
#classmethod
#sync_compatible
#inject_client
async def load(cls, name: str, client: "OrionClient" = None):
# ...
So I think as long as the function has the decorator #sync_compatible, it means it can be used as both async and sync functions.

Fastapi actual library for limits

I need to add limits for my fastapi endpoints. I'm currently using Fastapi-limiter. But there is a problem. The developer no longer supports the project. As well as slowapi (request pulls hang for almost a year). I'm currently using the patched version, of course. I would like to know if there are analogues known to you or something else to solve my problem. As an example, I attach just a piece of code (usual endpoint)
I want to thank you in advance for the time devoted to my problem, thank you.
from fastapi import APIRouter, Query, HTTPException, Depends, Request
from fastapi_limiter import FastAPILimiter, default_identifier
from fastapi_limiter.depends import RateLimiter
router = APIRouter()
limits_enabled = settings.limits_enabled
async def custom_identifier(request: Request) -> None:
return None
if limits_enabled == 1:
#router.on_event("startup")
async def startup():
redis = await aioredis.from_url(
settings.redis_url,
encoding="utf-8",
decode_responses=True,
password=settings.redis_pass
)
await FastAPILimiter.init(redis)
#router.on_event("shutdown")
async def shutdown():
await FastAPILimiter.close()
limits = [Depends(RateLimiter(times=settings.limiter_times,
seconds=settings.limiter_seconds, identifier=custom_identifier))]
#router.get("/test", dependencies=limits, responses=example_vehicle_types_responses,
response_model=schemas.test,
description="test")
async def get_test():
return "test"

Code hangs when trying to run code, telegram bot deployment

I am trying to compile some python code and run it as a telegram bot.
When I run it on VS Code or in the CMD console (python main.py) it freezes.
However, when I broke up the code individually and run, it works well.
1) Can any PyGod point out where im doing wrong?
2) I realised this happens when I try to redeploy a telegram bot too. Anyone have ideas on this?
from telegram.ext import Updater, InlineQueryHandler, CommandHandler
from telegram.ext.dispatcher import run_async
import requests
import re
def get_url():
contents = requests.get('https://dog.ceo/api/breed/retriever/golden/images/random').json()
url = contents['message']
return url
def get_image_url():
allowed_extension = ['jpg','jpeg','png']
file_extension = ''
while file_extension not in allowed_extension:
url = get_url()
file_extension = re.search("([^.]*)$",url).group(1).lower()
return url
#run_async
def goodboy(update, context):
url = get_image_url()
chat_id = update.message.chat_id
context.bot.send_photo(chat_id=chat_id, photo=url)
def main():
updater = Updater('YOUR_TOKEN', use_context=True)
dp = updater.dispatcher
dp.add_handler(CommandHandler('goodboy',goodboy))
updater.start_polling()
updater.idle()
if __name__ == '__main__':
main()

How can I fix coroutine was never awaited?

I have a RESTFUL Flask API I am serving with gunicorn and I'm trying to continue running parse_request() after sending a response to whoever made a POST request so they're not left waiting for it to finish
I'm not too sure if this will even achieve what I want but this is the code I have so far.
from threading import Thread
import subprocess
from flask import Flask
import asyncio
application = Flask(__name__)
async def parse_request(data):
try:
command = './webscraper.py -us "{user}" -p "{password}" -url "{url}"'.format(**data)
output = subprocess.check_output(['bash','-c', command])
except Exception as e:
print(e)
#application.route('/scraper/run', methods=['POST'])
def init_scrape():
try:
thread = Thread(target=parse_request, kwargs={'data': request.json})
thread.start()
return jsonify({'Scraping this site: ': request.json["url"]}), 201
except Exception as e:
print(e)
if __name__ == '__main__':
try:
application.run(host="0.0.0.0", port="8080")
except Exception as e:
print(e)
I am sending a POST request similar to this.
localhost:8080/scraper/run
data = {
"user": "username",
"password": "password",
"url": "www.mysite.com"
}
The error I get when sending a POST request is this.
/usr/lib/python3.6/threading.py:864: RuntimeWarning: coroutine 'parse_request' was never awaited
self._target(*self._args, **self._kwargs)

So first things first, why are you calling webscraper.py with subprocess? This is completely pointless. Because webscraper.py is a python script you should be importing the needed functions/classes from webscraper.py and using them directly. Calling it this way is totally defeating what you are wanting to do.
Next, your actual question you have got mixed up between async and threading. I suggest you learn more about it but essentially you want something like the following using Quart which is an async version of Flask, it would suit your situation well.
from quart import Quart, response, jsonify
import asyncio
from webscraper import <Class>, <webscraper_func> # Import what you need or
import webscraper # whatever suits your needs
app = Quart(__name__)
async def parse_request(user, password, url):
webscraper_func(user, password, url)
return 'Success'
#app.route('/scraper/run', methods=['POST'])
async def init_scrape():
user = request.args.get('user')
password = request.args.get('password')
url = request.args.get('url')
asyncio.get_running_loop().run_in_executor(
None,
parse_request(user, password, url)
)
return 'Success'
if __name__ == '__main__':
app.run(host='0.0.0.0', port='8080')