Develop Reference

FastAPI fileupload to S3

I am writing a FastAPI endpoint to upload a file to S3 Object Store. Following is the code snippet. However, I get an exception "expected string or bytes-like object"
I don't want to save the file temporarily and then upload it.
I reead in FastAPI documentation that the UploadFile has file attribute which is an "actual Python file (SpooledTemporaryFile) that you can pass directly to other functions or libraries that expect a "file-like" object". Since the upload_fileobj function of boto3 expects a string or bytes-like argument, I converted the file-like object to BytesIO object using the _file attribute of the SpooledTemporaryFile. But still the error "expected string or bytes-like object"
Any pointers will be appreciated.
#app.post("/uploadFile")
async def upload_file(fileobject: UploadFile = File(...),filename: str = Body(default=None),key: str = Body(default=None),):
extension = fileobject.filename.rsplit(".", 1)[1].lower()
if key is None:
raise HTTPException(status_code=400, MissingError="Key is missing")
if filename is None:
filename = fileobject.filename
if fileobject.filename.endswith(tuple(ALLOWED_EXTENSIONS)):
data = (
fileobject.file._file
) # Using _file attribute convert tempfile.SpooledTemporaryFile to io.BytesIO
try:
# Upload the file to Spaces
upload_result = await client.upload_fileobj(
data,
BUCKET,
f"{key}/{filename}",
ExtraArgs={
"ACL": "public",
"ContentType": FILE_CONTENT_TYPES[extension],
},
)
if upload_result:
object_url = f"{OBJ_URL}/{key}/{file.filename}"
doc = [{"file_url": object_url}]
else:
raise HTTPException(status_code=400, detail="Failed to upload in S3")
except Exception as e:
logging.error(f"Exception while uploading file - {e}")
raise HTTPException(
status_code=400, detail=f"Exception {e} while uploading the file"
)
else:
raise HTTPException(
status_code=400, detail=f"File of type {extension} is not allowed"
)

I had a similar issue, but the issue was with the bucket name, mine was None/empty, the error message is rather annoying, the bucket should be a non empty string.

How do I serialize this JSON response from API in Python?

The response from the API is:
{"states":[{"state_id":1,"state_name":"Andaman and Nicobar Islands"},{"state_id":2,"state_name":"Andhra Pradesh"},{"state_id":3,"state_name":"Arunachal Pradesh"},{"state_id":4,"state_name":"Assam"},{"state_id":5,"state_name":"Bihar"},{"state_id":6,"state_name":"Chandigarh"},{"state_id":7,"state_name":"Chhattisgarh"},{"state_id":8,"state_name":"Dadra and Nagar Haveli"},{"state_id":37,"state_name":"Daman and Diu"},{"state_id":9,"state_name":"Delhi"},{"state_id":10,"state_name":"Goa"},{"state_id":11,"state_name":"Gujarat"},{"state_id":12,"state_name":"Haryana"},{"state_id":13,"state_name":"Himachal Pradesh"},{"state_id":14,"state_name":"Jammu and Kashmir"},{"state_id":15,"state_name":"Jharkhand"},{"state_id":16,"state_name":"Karnataka"},{"state_id":17,"state_name":"Kerala"},{"state_id":18,"state_name":"Ladakh"},{"state_id":19,"state_name":"Lakshadweep"},{"state_id":20,"state_name":"Madhya Pradesh"},{"state_id":21,"state_name":"Maharashtra"},{"state_id":22,"state_name":"Manipur"},{"state_id":23,"state_name":"Meghalaya"},{"state_id":24,"state_name":"Mizoram"},{"state_id":25,"state_name":"Nagaland"},{"state_id":26,"state_name":"Odisha"},{"state_id":27,"state_name":"Puducherry"},{"state_id":28,"state_name":"Punjab"},{"state_id":29,"state_name":"Rajasthan"},{"state_id":30,"state_name":"Sikkim"},{"state_id":31,"state_name":"Tamil Nadu"},{"state_id":32,"state_name":"Telangana"},{"state_id":33,"state_name":"Tripura"},{"state_id":34,"state_name":"Uttar Pradesh"},{"state_id":35,"state_name":"Uttarakhand"},{"state_id":36,"state_name":"West Bengal"}],"ttl":24}
I am trying to send this data to my Telegram bot.
states_url = "https://cdn-api.co-vin.in/api/v2/admin/location/states"
res = requests.get(states_url,headers={'User-Agent':my_headers})
bot.send_message(chat_id = chat_id, text=response)
I am getting this error:
TypeError: Object of type Response is not JSON serializable

You're trying to send the Response object, while you want either the Python object from the response, in which case you can use res.json(), or you want the raw text of the response, in which case you can use res.text.

Python urllib.request.urlopen: AttributeError: 'bytes' object has no attribute 'data'

I am using Python 3 and trying to connect to dstk. I am getting an error with urllib package.
I researched a lot on SO and could not find anything similar to this problem.
api_url = self.api_base+'/street2coordinates'
api_body = json.dumps(addresses)
#api_url=api_url.encode("utf-8")
#api_body=api_body.encode("utf-8")
print(type(api_url))
response_string = six.moves.urllib.request.urlopen(api_url, api_body).read()
response = json.loads(response_string)
If I do not encode the api_url and api_body I get the below:
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1247, in do_request_
raise TypeError(msg)
TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.
However if I try and encode them to utf-8 (uncommenting the lines) then I get the below error:
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 514, in open
req.data = data
AttributeError: 'bytes' object has no attribute 'data'
This seems like a circular error for me and I am not able to resolve it. I did try make to solutions from SO regards to change it to json.load etc but nothing seems to work.

You are encoding both the url and the request body, but only the body should be encoded.
This ought to work:
api_url = self.api_base+'/street2coordinates'
api_body = json.dumps(addresses)
api_body=api_body.encode("utf-8")
response_string = six.moves.urllib.request.urlopen(api_url, api_body).read()
response = json.loads(response_string)
urlopen's arguments are passed to another class to create an opener, and this class does not know whether it has been passed a url or a Request instance. So it checks whether the "url" is a string - if the "url" is a string, it creates a Request, if not it assumes that "url" is a Request instance and tries to set its data attribute, causing the exception that you are seeing.
The code in question is here.

Cannot read urllib error message once it is read()

My problem is with error handling of the python urllib error object. I am unable to read the error message while still keeping it intact in the error object, for it to be consumed later.
response = urllib.request.urlopen(request) # request that will raise an error
response.read()
response.read() # is empty now
# Also tried seek(0), that does not work either.
So this how I intend to use it, but when the Exception bubbles up, the.read() second time is empty.
try:
response = urllib.request.urlopen(request)
except urllib.error.HTTPError as err:
self.log.exception(err.read())
raise err
I tried making a deepcopy of the err object,
import copy
try:
response = urllib.request.urlopen(request)
except urllib.error.HTTPError as err:
err_obj_copy = copy.deepcopy(err)
self.log.exception(
"Method:{}\n"
"URL:{}\n"
"Data:{}\n"
"Details:{}\n"
"Headers:{}".format(method, url, data, err_obj_copy.read(), headers))
raise err
but copy is unable to make a deepcopy and throws an error -
TypeError: __init__() missing 5 required positional arguments: 'url', 'code', 'msg', 'hdrs', and 'fp'.
How do I read the error message, while still keeping it intact in the object?
I do know how to do it using requests, but I am stuck with legacy code and need to make it work with urllib

This is what I did. Worked for me.
When reading the error for the first time, save it to a variable like this: msg = response.read().decode('utf8'). You can then create a new HTTPError instance, with the message, and propagate it.
resp = urllib.request.urlopen(request)
msg = resp.read().decode('utf8')
self.log.exception(msg)
raise HTTPError(resp.url, resp.code, resp.reason, resp.headers, io.BytesIO(bytes(msg, 'utf8')))

The error object may read from the network. Network is not seekable -- you can't go back in the general case.
You could replace err with a new HTTPError instance that reads from a buffer (like io.BytesIO()) instead of the network e.g., (not tested):
content = err.read()
self.log.exception(content)
raise HTTPError(err.url, err.code, err.reason, err.headers, io.BytesIO(content))
Though I'm not sure that you should -- handle the error in a single place instead e.g., reraise a more application specific exception or leave the logging to an upstream handler.

Serialize / Unserialize using json.dumps/loads gives AttributeError: 'unicode' object has no attribute 'read'

I'm attempting to capture payment result info from Amazon FPS, which comes in 2 forms:
User redirected to originating server with GET and query string parameters
Amazon sends POST to originating server with matching parameters
I can't guarantee which request will reach the server first, so I store the first in the DB by serializing either request.GET or request.POST using json.dumps and then attempt to load it later on using json.loads for comparison with the other request:
Initial request:
type = request.META['REQUEST_METHOD']
sub_req = SubscriptionRequest()
params = getattr(request, type)
serialized_params = json.dumps(params)
if type == 'GET': sub_req.client_params = serialized_params
if type == 'POST': sub_req.server_params = serialized_params
sub_req.save()
Followup request:
stored_params = json.load(sub_req.server_params if type == "GET" else sub_req.client_params)
Error:
File "/var/www/test.com/htdocs/apps/subscription/services.py", line 147, in subscription_request_check_or_store
stored_params = json.load(sub_req.server_params if type == "GET" else sub_req.client_params)
File "/usr/lib/python2.7/json/__init__.py", line 274, in load
return loads(fp.read(),
AttributeError: 'unicode' object has no attribute 'read'

Json.load() reads from a file. Use json.loads() to parse a string.