Develop Reference

How to use Blocks correctly to load AWS S3 credentials in Prefect?

I am using Prefect. And I tried to download a file from S3.
When I hard coded the AWS credentials, the file can be downloaded successfully:
import asyncio
from prefect_aws.s3 import s3_download
from prefect_aws.credentials import AwsCredentials
from prefect import flow, get_run_logger
#flow
async def fetch_taxi_data():
logger = get_run_logger()
credentials = AwsCredentials(
aws_access_key_id="xxx",
aws_secret_access_key="xxx",
)
data = await s3_download(
bucket="hongbomiao-bucket",
key="hm-airflow/taxi.csv",
aws_credentials=credentials,
)
logger.info(data)
if __name__ == "__main__":
asyncio.run(fetch_taxi_data())
Now I tried to load the credentials from Prefect Blocks.
I created a AWS Credentials Block:
However,
aws_credentials_block = AwsCredentials.load("aws-credentials-block")
data = await s3_download(
bucket="hongbomiao-bucket",
key="hm-airflow/taxi.csv",
aws_credentials=aws_credentials_block,
)
throws the error:
AttributeError: 'coroutine' object has no attribute 'get_boto3_session'
And
aws_credentials_block = AwsCredentials.load("aws-credentials-block")
credentials = AwsCredentials(
aws_access_key_id=aws_credentials_block.aws_access_key_id,
aws_secret_access_key=aws_credentials_block.aws_secret_access_key,
)
data = await s3_download(
bucket="hongbomiao-bucket",
key="hm-airflow/taxi.csv",
aws_credentials=credentials,
)
throws the error:
AttributeError: 'coroutine' object has no attribute 'aws_access_key_id'
I didn't find any useful document about how to use it.
Am I supposed to use Blocks to load credentials? If it is, what is the correct way to use Blocks correctly in Prefect? Thanks!

I just found the snippet in the screenshot in the question misses an await.
After adding await, it works now!
aws_credentials_block = await AwsCredentials.load("aws-credentials-block")
data = await s3_download(
bucket="hongbomiao-bucket",
key="hm-airflow/taxi.csv",
aws_credentials=aws_credentials_block,
)
UPDATE:
Got an answer from Michael Adkins on GitHub, and thanks!
await is only needed if you're writing an async flow or task. For users writing synchronous code, an await is not needed (and not possible). Most of our users are writing synchronous code and the example in the UI is in a synchronous context so it does not include the await.
I saw the source code at
https://github.com/PrefectHQ/prefect/blob/1dcd45637914896c60b7d49254a34e95a9ce56ea/src/prefect/blocks/core.py#L601-L604
#classmethod
#sync_compatible
#inject_client
async def load(cls, name: str, client: "OrionClient" = None):
# ...
So I think as long as the function has the decorator #sync_compatible, it means it can be used as both async and sync functions.

Testing asynchronous FastAPI endpoints with dependencies

I've encountered this problem, and I can't see any solution, though it must be a common one. So, maybe I'm missing something here.
I'm working on FastAPI app with asynchronous endpoints and asynchronous connection with database. Database connection is passed as a dependency. I want to write some asynchronous tests for said app.
engine = create_async_engine(connection_string, echo=True)
def get_session():
return sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
#router.post("/register")
async def register(
user_data: UserRequest,
authorize: AuthJWT = Depends(),
async_session: sessionmaker = Depends(get_session),
):
"""Register new user."""
if authorize.get_jwt_subject():
raise LogicException("already authorized")
session: AsyncSession
async with async_session() as session:
query = await session.execute(
select(UserModel).where(UserModel.name == user_data.name)
)
...
I'm using AsyncSession to work with database. So in my test, db connection also has to be asynchronous.
engine = create_async_engine(
SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False}
)
app.dependency_overrides[get_session] = lambda: sessionmaker(
engine, class_=AsyncSession, expire_on_commit=False
)
#pytest.mark.asyncio
async def test_create_user():
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
async with AsyncClient(app=app, base_url="http://test") as ac:
response = await ac.post(
"/register",
json={"name": "TestGuy", "password": "TestPass"},
)
assert response.status_code == 200, response.text
When running the test, I get the following error:
...
coin_venv\lib\site-packages\fastapi\routing.py:217: in app
solved_result = await solve_dependencies(
coin_venv\lib\site-packages\fastapi\dependencies\utils.py:529: in solve_dependencies
solved = await run_in_threadpool(call, **sub_values)
AttributeError: module 'anyio' has no attribute 'to_thread'
I concluded that error appears only when there is a dependency in an endpoint. Weird part is that I don't even have anyio in my environment.
So, is there a way to test asynchronous FastAPI endpoints with dependencies and asynchronous db connection? Surely, there must be something, it's not like this situation is something unique...
UPD: I tried using decorator #pytest.mark.anyio and also have installed trio and anyio. Now pytest seem to discover two distinct tests in this one:
login_test.py::test_create_user[asyncio]
login_test.py::test_create_user[trio]
Both fails, first one with what seems to be a valid error in my code, and second one with:
RuntimeError: There is no current event loop in thread 'MainThread'.
I guess it is true, though I don't really know if pytest creates eventloop to test async code. Anyway, I don't need the second test, why it is here, and how can I get rid of it?

It turned out, I can specify backend to run tests like this:
#pytest.fixture
def anyio_backend():
return 'asyncio'
So, now I have only the right tests running)

pytest runs on different eventloop (not get_running_loop), and so, when you try to run it in the same context, it raises exception. I suggest you to consider using nest_asyncio (https://pypi.org/project/nest-asyncio/), so that pytest can run in the same eventloop.
import nest_asyncio
nest_asyncio.apply()

Google Cloud Pricing API - Python

I am unable to run the following code. Basically I want to list services and their prices...
import requests
import asyncio
class_initiate= billing_v1.services.cloud_catalog.CloudCatalogAsyncClient()
result = asyncio.run(class_initiate.list_services())```
RuntimeError: Task <Task pending coro=<CloudCatalogAsyncClient.list_services() running at local/lib/python3.7/site-packages/google/cloud/billing_v1/services/cloud_catalog/async_client.py:185> cb=[_run_until_complete_cb() at /usr/lib/python3.7/asyncio/base_events.py:158]> got Future <Task pending coro=<UnaryUnaryCall._invoke() running at /usr/local/lib/python3.7/dist-packages/grpc/aio/_call.py:489>> attached to a different loop

#brownkhusra You may want to create an async main function, and then pass it to the asyncio.run method. In this way, you ensured that the entire logic in your application will be run on the same event loop.
async def main():
class_initiate= billing_v1.services.cloud_catalog.CloudCatalogAsyncClient()
result = await class_initiate.list_services()
asyncio.run(main())

Use something like this to get prices for specific services.
In my case I looked the price for CPU/RAM in us-cenytral-1.
#!/usr/bin/env python3
from google.cloud import billing_v1
import asyncio
SERVICE_COMPUTE_DISPLAY_NAME = "Compute Engine"
REGION = "us-central1"
INSTANCE_DESCR = "N2 Custom Instance"
async def main():
catalog = billing_v1.services.cloud_catalog.CloudCatalogAsyncClient()
result = await catalog.list_services()
compute_name = None
for service in result.services:
if service.display_name == SERVICE_COMPUTE_DISPLAY_NAME:
compute_name = service.name
result = await catalog.list_skus(parent=compute_name)
for sku in result.skus:
if REGION in sku.service_regions:
if sku.description.startswith(INSTANCE_DESCR):
for rate in sku.pricing_info[0].pricing_expression.tiered_rates:
if rate.unit_price.currency_code == "USD":
print(sku.description)
print(rate.unit_price.nanos / 1_000_000_000)
asyncio.run(main())

How to fetch a record from db asynchronously while unit testing?

In this unit test I would like to check if the device has been created and that the expiry date is 7 days in future.
database.py
import databases
database = databases.Database(settings.sqlalchemy_database_uri)
Unit Test:
from database.database import database
def test_successful_register_expiry_set_to_seven_days():
response = client.post(
"/register/",
headers={},
json={"device_id": "u1"},
)
assert response.status_code == 201
query = device.select(whereclause=device.c.id == "u1")
d = database.fetch_one(query)
assert d.expires_at == datetime.utcnow().replace(microsecond=0) + timedelta(days=7)
Because d is a coroutine object it fails with the message:
AttributeError: 'coroutine' object has no attribute 'expires_at'
And I can't use await inside a unit test.
d = await database.fetch_one(query)
What am I missing, please?

Well it is never awaited, your code returns the coroutine before that semaphore going in to the scheduler.
If you are using an asynchronous driver, you need to await it. Are there any workarounds for this? Yes.
You can use asyncio.run(awaitable) to run the coroutine inside an event loop.
import asyncio
d = asyncio.run(database.fetch_one(query))
If you have an currently running event loop you may want to use that event loop instead. You can achieve that by asyncio.get_event_loop(), which will run the function inside the running loop.
import asyncio
asyncio.get_event_loop().run_until_complete(database.fetch_one(query))
You can also use #pytest.mark.asyncio decorator (see documentation).
#pytest.mark.asyncio
async def dummy():
await some_awaitable()

Concurrent HTTP and SQL requests using async Python 3

first time trying asyncio and aiohttp.
I have the following code that gets urls from the MySQL database for GET requests. Gets the responses and pushes them to MySQL database.
if __name__ == "__main__":
database_name = 'db_name'
company_name = 'company_name'
my_db = Db(database=database_name) # wrapper class for mysql.connector
urls_dict = my_db.get_rest_api_urls_for_specific_company(company_name=company_name)
update_id = my_db.get_updateid()
my_db.get_connection(dictionary=True)
for url in urls_dict:
url_id = url['id']
url = url['url']
table_name = my_db.make_sql_table_name_by_url(url)
insert_query = my_db.get_sql_for_insert(table_name)
r = requests.get(url=url).json() # make the request
args = [json.dumps(r), update_id, url_id]
my_db.db_execute_one(insert_query, args, close_conn=False)
my_db.close_conn()
This works fine but to speed it up How can I run it asynchronously?
I have looked here, here and here but can't seem to get my head around it.
Here is what I have tried based on #Raphael Medaer's answer.
async def fetch(url):
async with ClientSession() as session:
async with session.request(method='GET', url=url) as response:
json = await response.json()
return json
async def process(url, update_id):
table_name = await db.make_sql_table_name_by_url(url)
result = await fetch(url)
print(url, result)
if __name__ == "__main__":
"""Get urls from DB"""
db = Db(database="fuse_src")
urls = db.get_rest_api_urls() # This returns list of dictionary
update_id = db.get_updateid()
url_list = []
for url in urls:
url_list.append(url['url'])
print(update_id)
asyncio.get_event_loop().run_until_complete(
asyncio.gather(*[process(url, update_id) for url in url_list]))
I get an error in the process method:
TypeError: object str can't be used in 'await' expression
Not sure whats the problem?
Any code example specific to this would be highly appreciated.

Make this code asynchronous will not speed it up at all. Except if you consider to run a part of your code in "parallel". For instance you can run multiple (SQL or HTTP) queries in "same time". By doing asynchronous programming you will not execute code in "same time". Although you will get benefit of long IO tasks to execute other part of your code while you're waiting for IOs.
First of all, you'll have to use asynchronous libraries (instead of synchronous one).
mysql.connector could be replaced by aiomysql from aio-libs.
requests could be replaced by aiohttp
To execute multiple asynchronous tasks in "parallel" (for instance to replace your loop for url in urls_dict:), you have to read carefully about asyncio tasks and function gather.
I will not (re)write your code in an asynchronous way, however here are a few lines of pseudo code which could help you:
async def process(url):
result = await fetch(url)
await db.commit(result)
if __name__ == "__main__":
db = MyDbConnection()
urls = await db.fetch_all_urls()
asyncio.get_event_loop().run_until_complete(
asyncio.gather(*[process(url) for url in urls]))