Develop Reference

FastAPI: How to get raw URL path from request?

I have a GET method with requested parameter in path:
#router.get('/users/{user_id}')
async def get_user_from_string(user_id: str):
return User(user_id)
Is it possible to get base url raw path (i.e., '/users/{user_id}') from the request?
I have tried to use the following way:
path = [route for route in request.scope['router'].routes if
route.endpoint == request.scope['endpoint']][0].path
But it doesn't work and I get:
AttributeError: 'Mount' object has no attribute 'endpoint'

The below solution worked fine for me
using the string replace with count parameter replaces the first occurence only. And request.path_params will return the path parameters in the sequence you take it in the request.
def get_raw_path(request):
path = request.url.path
for key, val in request.path_params.items():
path = path.replace(val, F'{{{key}}}',1)
return path

As per FastAPI documentation:
As FastAPI is actually Starlette underneath, with a layer of several
tools on top, you can use Starlette's Request object directly when you
need to.
Thus, you can use Request object to get the URL path. For instance:
from fastapi import Request
#app.get('/users/{user_id}')
def get_user(user_id: str, request: Request):
return request.url.path
Output (if the received user_id was 1):
/users/1
Update
If, however, what you need is the original route path, i.e., /users/{user_id}, you could use the below. The way it works is by getting the root_path first—which would normally be an empty string, unless you have mounted sub-application(s) to the top-level app (e.g., app.mount("/subapi", subapi)), and hence, you need the result to be prefixed with that specific path /subapi—and then append to it the route's path , which you can get from the APIRoute object. Example:
from fastapi import Request
#app.get('/users/{user_id}')
def get_user(user_id: str, request: Request):
path = request.scope['root_path'] + request.scope['route'].path
return path
Output:
/users/{user_id}

I'm working on implementing this for OpenTelemetry and the way to get the original route with the data that's available is as follows:
def get_route_from_request(req):
root_path = req.scope.get("root_path", "")
route = scope.get("route")
if not route:
return None
path_format = getattr(route, "path_format", None)
if path_format:
return f"{route_path}{path_format}"
return None
Note that the accepted answer is not returning what was asked, as it returns the path as received by the server.
None of the other answers deal with mounted apps.
And finally, answers checking the name of the endpoint are also wrong as the same function could be used in different endpoints.

Having found both the answers to not work I'll share what I use.
It's not great as if there's shared values in path params it will not work
path = request.url.path
for key, val in request.path_params.items():
path = path.replace(val, F'{{{key}}}')

You can use the APIRout object property in the request to get the actual path
example:
raw_path = request.scope['route'].path
#'/user/{id}'

InvalidS3ObjectException when calling the AnalyzeDocument operation:

InvalidS3ObjectException when calling the AnalyzeDocument operation: Unable to get object metadata from S3. Check object key, region and/or access permissions."
I keep getting this error. Over. And. Over. This program worked with my test cases of what I'm bringing in, the json with a {"body":"imagename.jpg"}. But the very moment I try to utilize the actual code my JS brings in, I get this error. The thing that confuses me is that I've checked the regions and they are fine. I went into my account and created users with full access to all AWS and S3 features, and utilized those logins, I've used my root account, everything. All I'm trying to do is access an image from my s3 bucket. Why won't it work? Below is my code. It works if I utilize the test case I provided above, but the moment I try and use the website it's connected to, it doesn't work.
def main(event, context):
key_map, value_map, block_map = get_kv_map(event) #Take map variables in to get the key and value map we need.
It goes to this function...
def get_kv_map(event):
filePath = event
fileExt = filePath.get('body')
s3 = boto3.resource('s3', region_name='us-east-1')
bucket = s3.Bucket('react-images-ex')
obj = bucket.Object(bucket)
client = boto3.client('textract') #We utilize boto3's textract lib
response = client.analyze_document(Document={'S3Object': {'Bucket': 'react-images-ex', 'Name': fileExt}}, FeatureTypes=['FORMS'])
# Get the text blocks
blocks=response['Blocks'] #We make a blocks variable that will be the blocks we find in the document
# get key and value maps
key_map = {}
value_map = {}
block_map = {}
for block in blocks: #Traverse the blocks found in the document
block_id = block['Id'] #Set variable for blockId to the Id's found on that block location
block_map[block_id] = block #Make the block map at that ID be the block variable
if block['BlockType'] == "KEY_VALUE_SET": #if we see that the type of block we're on is a key and value set pair, we check if it's a key or not. If it's not a key, we know it's a value. We send it to the respective map.
if 'KEY' in block['EntityTypes']:
key_map[block_id] = block
else:
value_map[block_id] = block
return key_map, value_map, block_map #Return the maps we need after they're filled.
I have been told before this code is fine, and it should work. So why exactly is it that I get this error?

Based on the comments.
The issue with body was that it was json string, not actual json object.
The solution was to parse the string into json:
fileExt = json.loads(filePath.get('body'))

Try awscli to see if you can access the image in s3:
aws s3 ls s3://react-images-ex/<some-fileExt>
Either you are parsing the fileExt wrongly, or you don't have S3 permission to access the file. The awscli command will help to verify this.

boto3 put_object with new key python

I want to save a csv file ("test.csv") in S3 using boto3.
my bucket is "outputS3Bucket" and the key is "folder/newFolder".
I want to check if "newFolder" exists and if not to create it.
import boto3
client = boto3.client('s3')
s3 = boto3.resource('s3')
bucket = s3.Bucket("outputS3Bucket")
result = client.list_objects(Bucket='outputS3Bucket',Prefix="folder/newFolder")
if len(result)==0:
key = bucket.new_key("folder/newFolder")
newKey = key + "/" + "test.csv"
client.put_object(Bucket="outputS3Bucket", Key=newKey, Body=content)
# put_object path: 's3://outputS3Bucket/folder/newFolder/test.csv'
I have few problems:
if I don't write the full key name (such as "folder/ne") and there is a "neaFo" folder instead it still says it exists.
key = bucket.new_key("folder/newFolder")
AttributeError: 's3.Bucket' object has no attribute 'new_key'

Firstly, according to boto3 documentation, it's preferred to use the new API method - list_objects_v2() instead to list a bucket's objects.
I suggest using a simple boolean function to check whether a folder exist (makes your code cleaner and more readable).
for question 1, you can check if the prefix ends with '/' character and append it if not, - this will make sure your are looking for EXACT match and not Starts With.
Sample Function:
def bucket_folder_exists(client, bucket, path_prefix):
# make path_prefix exact match and not path/to/folder*
if list(path_prefix)[-1] is not '/':
path_prefix += '/'
# check if 'Contents' key exist in response dict - if it exist it indicate the folder exists, otherwise response will be None
response = client.list_objects_v2(Bucket=bucket, Prefix=path_prefix).get('Contents')
if response:
return True
return False
Sample Implementation:
if bucket_folder_exists(client, 'outputS3Bucket', 'folder/newFolder'):
pass # Do something if folder already exist
else:
pass # Do something if folder does not exist
Regarding your second question, I added a comment - it seems your code mentions bucket variable\object used as key = bucket.new_key("folder/newFolder"), however bucket is not set anywhere in your code, -> according to the error you are getting, it looks like a s3.Bucket object, which doesn't have the the new_key attribute defined.

Amazon S3 boto - how to delete folder?

I created a folder in s3 named "test" and I pushed "test_1.jpg", "test_2.jpg" into "test".
How can I use boto to delete folder "test"?

Here is 2018 (almost 2019) version:
s3 = boto3.resource('s3')
bucket = s3.Bucket('mybucket')
bucket.objects.filter(Prefix="myprefix/").delete()

There are no folders in S3. Instead, the keys form a flat namespace. However a key with slashes in its name shows specially in some programs, including the AWS console (see for example Amazon S3 boto - how to create a folder?).
Instead of deleting "a directory", you can (and have to) list files by prefix and delete. In essence:
for key in bucket.list(prefix='your/directory/'):
key.delete()
However the other accomplished answers on this page feature more efficient approaches.
Notice that the prefix is just searched using dummy string search. If the prefix were your/directory, that is, without the trailing slash appended, the program would also happily delete your/directory-that-you-wanted-to-remove-is-definitely-not-t‌​his-one.
For more information, see S3 boto list keys sometimes returns directory key.

I feel that it's been a while and boto3 has a few different ways of accomplishing this goal. This assumes you want to delete the test "folder" and all of its objects Here is one way:
s3 = boto3.resource('s3')
objects_to_delete = s3.meta.client.list_objects(Bucket="MyBucket", Prefix="myfolder/test/")
delete_keys = {'Objects' : []}
delete_keys['Objects'] = [{'Key' : k} for k in [obj['Key'] for obj in objects_to_delete.get('Contents', [])]]
s3.meta.client.delete_objects(Bucket="MyBucket", Delete=delete_keys)
This should make two requests, one to fetch the objects in the folder, the second to delete all objects in said folder.
https://boto3.readthedocs.org/en/latest/reference/services/s3.html#S3.Client.delete_objects

A slight improvement on Patrick's solution. As you might know, both list_objects() and delete_objects() have an object limit of 1000. This is why you have to paginate listing and delete in chunks. This is pretty universal and you can give Prefix to paginator.paginate() to delete subdirectories/paths
client = boto3.client('s3', **credentials)
paginator = client.get_paginator('list_objects_v2')
pages = paginator.paginate(Bucket=self.bucket_name)
delete_us = dict(Objects=[])
for item in pages.search('Contents'):
delete_us['Objects'].append(dict(Key=item['Key']))
# flush once aws limit reached
if len(delete_us['Objects']) >= 1000:
client.delete_objects(Bucket=bucket, Delete=delete_us)
delete_us = dict(Objects=[])
# flush rest
if len(delete_us['Objects']):
client.delete_objects(Bucket=bucket, Delete=delete_us)

You can use bucket.delete_keys() with a list of keys (with a large number of keys I found this to be an order of magnitude faster than using key.delete).
Something like this:
delete_key_list = []
for key in bucket.list(prefix='/your/directory/'):
delete_key_list.append(key)
if len(delete_key_list) > 100:
bucket.delete_keys(delete_key_list)
delete_key_list = []
if len(delete_key_list) > 0:
bucket.delete_keys(delete_key_list)

If versioning is enabled on the S3 bucket:
s3 = boto3.resource('s3')
bucket = s3.Bucket('mybucket')
bucket.object_versions.filter(Prefix="myprefix/").delete()

If one needs to filter by object contents like I did, the following is a blueprint for your logic:
def get_s3_objects_batches(s3: S3Client, **base_kwargs):
kwargs = dict(MaxKeys=1000, **base_kwargs)
while True:
response = s3.list_objects_v2(**kwargs)
# to yield each and every file: yield from response.get('Contents', [])
yield response.get('Contents', [])
if not response.get('IsTruncated'): # At the end of the list?
break
continuation_token = response.get('NextContinuationToken')
kwargs['ContinuationToken'] = continuation_token
def your_filter(b):
raise NotImplementedError()
session = boto3.session.Session(profile_name=profile_name)
s3client = session.client('s3')
for batch in get_s3_objects_batches(s3client, Bucket=bucket_name, Prefix=prefix):
to_delete = [{'Key': obj['Key']} for obj in batch if your_filter(obj)]
if to_delete:
s3client.delete_objects(Bucket=bucket_name, Delete={'Objects': to_delete})

#Deleting a Files Inside Folder S3 using boto3#
def delete_from_minio():
"""
This function is used to delete files or folder inside the another Folder
"""
try:
logger.info("Deleting from minio")
aws_access_key_id='Your_aws_acess_key'
aws_secret_access_key = 'Your_aws_Secret_key'
host = 'your_aws_endpoint'
s3 = boto3.resource('s3', aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key ,
config=boto3.session.Config(signature_version='your_version'),
region_name="your_region",
endpoint_url=host ,
verify=False)
bucket = s3.Bucket('Your_bucket_name')
for obj in bucket.objects.filter(Prefix='Directory/Sub_Directory'):
s3.Object(bucket.name, obj.key).delete()
except Exception as e:
print(f"Error Occurred while deleting from the S3,{str(e)}")
Hope this Helps :)

How to decode a Google App Engine entity Key path str in Python?

In Google App Engine, an entity has a Key. A key can be made from a path, in which case str(key) is an opaque hex string. Example:
from google.appengine.ext import db
foo = db.Key.from_path(u'foo', u'bar', _app=u'baz')
print foo
gives
agNiYXpyDAsSA2ZvbyIDYmFyDA
if you set up the right paths to run the code.
So, how can one take the hex string and get the path back? I thought the answer would be in Key or entity group docs, but I can't see it.

from google.appengine.ext import db
k = db.Key('agNiYXpyDAsSA2ZvbyIDYmFyDA')
_app = k.app()
path = []
while k is not None:
path.append(k.id_or_name())
path.append(k.kind())
k = k.parent()
path.reverse()
print 'app=%r, path=%r' % (_app, path)
when run in a Development Console, this outputs:
app=u'baz', path=[u'foo', u'bar']
as requested. A shorter alternative is to use the (unfortunately, I believe, undocumented) to_path method of Key instances:
k = db.Key('agNiYXpyDAsSA2ZvbyIDYmFyDA')
_app = k.app()
path = k.to_path()
print 'app=%r, path=%r' % (_app, path)
with the same results. But the first, longer version relies only on documented methods.

Once you have the Key object (which can be created by passing that opaque identifier to the constructor), use Key.to_path() to get the path of a Key as a list. For example:
from google.appengine.ext import db
opaque_id = 'agNiYXpyDAsSA2ZvbyIDYmFyDA'
path = db.Key(opaque_id).to_path()