We send a file via API.
When the file is saved locally or on the same ec2 instance, all works fine, and we get the response from the API
When the file is saved on AWS s3, we get the error 'Unicode-objects must be encoded before hashing'
This is the code that works to open and send file from local device but not when getting the file from s3
my_file = self.original_file.open(mode='rb').read()
Related
I am using django-s3direct to file upload
https://github.com/bradleyg/django-s3direct
Using IAM role setting because I upload the file from the server on ECS container.
Now I set the blockPublicAccess of S3 false.
When uploading images from html, there comes error.
https://s3.ap-northeast-1.amazonaws.com/static-resource-v/images/c64d6e593de44aa5b10dcf1766582547/_origin.jpg?uploads (403 (Forbidden)
)
initiate error: static-resource-v/line-assets/images/c64d6e593de44aa5b10dcf1766582547/_origin.jpg AWS Code: AccessDenied, Message:Access Deniedstatus:403
OK, it is understandable.
Browser try to access the for initiation.
However there is any way to upload file from browser when blockPublicAccess is true??
I am trying to upload a file via API gateway to a lambda.
I can get the file data by below
s3 = boto3.client("s3")
# decoding form-data into bytes
post_data = base64.b64decode(event["body"])
but am unclear on how to get the file name from here.
I can get this to work if I use a separate field as the file name, but would prefer to automatically detect it
I have a Django application and form which accepts from a user an Excel(.xlsx) and CSV (.csv) file. I need to save both files to Azure Blob Storage. I found it to be trivial to handle the .csv file but the same code fails when attempting up upload an xlsx file:
from azure.storage.blob import BlobServiceClient
# This code executes successfully when saving a CSV to blob storage
blob_service_client = BlobServiceClient.from_connection_string(os.getenv('STORAGE_CONN_STRING'))
blob_client = blob_service_client.get_blob_client(container="my-container-name", blob=form.cleaned_data.get('name_of_form_field_for_csv_file'))
blob_client.upload_blob(form.cleaned_data.get('name_of_form_field_for_csv_file''))
# This code fails when saving xlsx to blob storage
blob_client = blob_service_client.get_blob_client(container="my-container-name", blob=form.cleaned_data.get('name_of_form_field_for_xlsx_file'))
blob_client.upload_blob(form.cleaned_data.get('name_of_form_field_for_xlsx_file''))
ClientAuthenticationError at /mypage/create/
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
However, I've been unable to figure out how to save the .xlsx file. I--perhaps somewhat naively--assumed I could pass the .xlsx file as-is (like the .csv example above) but I get the error:
ClientAuthenticationError at /mypage/create/
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
I found this SO Answer about the above error, but there's no concensus at all on what the error means and I've been unable to progress much further from that link. However, there was some discussion about sending the data to Azure blob storage as a byte stream. Is this a possible way forward? I should note here that, ideally, I need to process the files in memory as my app is deployed within App Service (my understanding is that I don't have access to a file system in which to create and manipulate files.)
I have also learned that .xlsx files are compressed so do I need to first decompress the file and then send it as a byte stream? If so, has anyone got any experience with this who could point me in the right direction?
Storage account connection string:
STORAGE_CONN_STRING=DefaultEndpointsProtocol=https;AccountName=REDACTED;AccountKey=REDACTED;EndpointSuffix=core.windows.net
Did you try like below:
# Create a local directory to hold blob data
local_path = "./data"
os.mkdir(local_path)
# Create a file in the local data directory to upload and download
local_file_name = str(uuid.uuid4()) + ".xlsx"
upload_file_path = os.path.join(local_path, local_file_name)
# Write text to the file
file = open(upload_file_path, 'w')
file.write("Hello, World!")
file.close()
# Create a blob client using the local file name as the name for the blob
blob_client =
blob_service_client.get_blob_client(container=container_name,
blob=local_file_name)
# Upload the created file
with open(upload_file_path, "rb") as data:
blob_client.upload_blob(data)
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python
For reasons I don't fully understand (comments welcome for an explanation!), I can successfully save a .xlsx file to Azure Blob Storage with:
self.request.FILES['name_of_form_field_for_xlsx_file']
I suspect there's a difference in how csv vs. xlsx files are handled between request.FILES and form.cleaned_data.get() in Django, resulting in an authentication error as per the original question.
The full code to save a .csv and then a .xlsx is (note this is within a FormView):
from azure.storage.blob import BlobServiceClient
# Set connection string
blob_service_client = BlobServiceClient.from_connection_string(os.getenv('STORAGE_CONN_STRING'))
# Upload an xlsx file
blob_client = blob_service_client.get_blob_client(container="my-container", blob=self.request.FILES['xlsx_file'])
blob_client.upload_blob(self.request.FILES['xlsx_file'])
# Upload a CSV file
blob_client = blob_service_client.get_blob_client(container="my-container", blob=form.cleaned_data.get('csv_file'))
blob_client.upload_blob(form.cleaned_data.get('csv_file'))
I'm trying to read file residing inside an s3 bucket and upload it to another s3 bucket through pre-signed url. I have tried to upload single file, that works fine. But if I try to upload list of files in a bucket, data can't be uploaded. It's always giving the response code as 200, but data is uploaded in the target s3 location especially with large files.
My python code is given below.
import boto3
import requests
from io import BytesIO
for file in file_list:#list of file names in the bucket
pre_url=getPreSinedUrl()#for each file method will return a separate pre-signed url to upload file
data_in_bytes = s3.get_object(Bucket="my-bucket", Key=file)['Body'].read()
res = requests.put(pre_url, data=BytesIO(data_in_bytes), headers=my_header)
print(res.status_code)
Any help why the file in s3 is not getting uploaded?
I have a service A, that uploads the file sample.txt to S3 using boto3.
Service A should return HTTP response as 200 only after the file is uploaded (or next line in code needs to be executed only after file is successfully uploaded). How to check for the condition that the file is uploaded.
s3_client = boto3.client('s3')
response = s3_client.upload_file(file_path, BUCKET_NAME, key)
I have tried above code, but the response object is None.
Any help appreciated. Thanks
Scenario is: There is another Service B, that calls Service A and downloads the file sample.txt from S3. Sometimes, the situation is that Service B unable to find the sample.txt in S3 (404- Not found) as Service A file upload didn't finish yet