I am using the boto3 library to create a S3 folder using python.(Want to create a directory 'c' in already existing directory structure like '/a/b'
s3_client=boto3.client('s3')
s3_client =put_object(Bucket=bucket_name, Key='a/b/c/')
I am not getting any error but the directory is also not getting created. I cant really figure out the reason, any suggestions?
Not sure if it is a typo in the code you show but it should give you an error. I think what you are trying to do is:
s3_client = boto3.client('s3')
response = s3_client.put_object(Bucket=bucket_name, Key='a/b/c/')
print("Response: {}".format(response)) # See result of request.
there are no such things as folders or directory in S3
to upload a file to your bucket you could use:
s3_client=boto3.client('s3')
# you have to provide my_binary_data
response = s3_client.put_object(Body=my_binary_data, Bucket=bucket_name, Key='a/b/c/')
where Key represents the name or your file
you can read more aboout Client.put_object here
Related
I come here after a few days of research on the subject,
I have a Django class: Question, the admin can create a question: he has to input a "title" and some "items" + True or False for each item. I want to get the "title" of the question as an mp3 file so I use gTTS (google text to speech) and it works as well while I use it locally!
if(sender == Question and kwargs['update_fields'] == None):
myQuestion = kwargs['instance']
output = gTTS(myQuestion.title, lang="fr")
mySound = './media/questionsSound/Question'+str(myQuestion.id)+'.mp3'
output.save(mySound)
Now I want to save these mp3 files on an s3 space because I have many applications that will call them.
So in the first time, I just tried to replace the path of my sound with the path of my bucket but I got this error message :
[Errno 22] Invalid argument: 'https://[mySpaceName].fra1.digitaloceanspaces.com/[myBucketName]/questionSound/Question95.mp3'
This is the first problem that has no sense for me because the URL exists I tried it with files that I dragged and dropped.
So I tried to find a workaround and I decided to save the file in the original folder ./media/questionsSoud... then to upload it on the s3 space :
import boto3
from boto3 import session
from botocore.client import Config
from boto3.s3.transfer import S3Transfer
session = session.Session()
client = session.client('s3', region_name='fra1', endpoint_url=settings.AWS_S3_ENDPOINT_URL, aws_access_key_id=settings.AWS_ACCESS_KEY_ID, aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
transfer = S3Transfer(client)
transfer.upload_file(mySound, settings.AWS_STORAGE_BUCKET_NAME, "testhello.mp3")
response = client.put_object(ACL='public-read', Bucket=settings.AWS_STORAGE_BUCKET_NAME, Key='testhello.mp3')
And now the problem is different: I get a file on my space but it is "binary/octet-stream" type instead of "audio/mpeg" and it is absolutely empty (I download it and try it) even if mySound is not empty!
Thank you if you read this big question I would be so grateful if you can help me!
** EDIT **
To get a non-empty file I removed the line put_object()
But I still get a "binary/octet" content-type so I found something on GitHub like that for the function upload_file()
mimetype, _ = mimetypes.guess_type(mySound)
print(mimetype)
if mimetype is None:
raise Exception("Failed to guess mimetype")
transfer.upload_file(mySound, settings.AWS_STORAGE_BUCKET_NAME, "testhello.mp3", ExtraArgs={
"ContentType": mimetype
})
But I get this error
upload_file() got an unexpected keyword argument 'ExtraArgs'
Your put_object() command does not specify the data to store, so it is creating a zero-length file. Try removing your line with put_object() and it should work, because the previous line has probably already uploaded the file.
I'm not familiar with using Digital Ocean, but here's the code you would normally use for S3:
s3_client = boto3.client('s3')
s3_client.upload_file('./file.mp3', 'mybucket', 'file.mp3', ExtraArgs={'ACL': 'public-read', 'Content-Type': 'audio/mpeg'})
Thank you for your help
For the people who use DigitalOcean too, its finally not a big deal, you can call the url of the "binary/octet-stream" in your front-end application and it's indeed considered as an audio file
it's 'ContentType': 'audio/mp3' not 'Content-Type'
I have written code on my backend (hosted on Elastic Beanstalk) to retrieve a file from an S3 bucket and save it back to the bucket under a different name. I am using boto3 and have created an s3 client called 's3'.
bucketname is the name of the bucket, keyname is name of the key. I am also using the tempfile module
tmp = tempfile.NamedTemporaryFile()
with open(tmp.name, 'wb') as f:
s3.download_fileobj(bucketname, keyname, f)
s3.upload_file(tmp, bucketname, 'fake.jpg')
I was wondering if my understanding was off (still debugging why there is an error) - I created a tempfile and opened and saved within it the contents of the object with the keyname and bucketname. Then I uploaded that temp file to the bucket under a different name. Is my reasoning correct?
The upload_file() command is expecting a filename (as a string) in the first parameter, not a file object.
Instead, you should use upload_fileobj().
However, I would recommend something different...
If you simply wish to make a copy of an object, you can use copy_object:
response = client.copy_object(
Bucket='destinationbucket',
CopySource='/sourcebucket/HappyFace.jpg',
Key='HappyFaceCopy.jpg',
)
I am uploading images to a folder currently on local . like in site/uploads.
And After searching I got that for uploading images to Amazon S3, I have to do like this
import boto3
s3 = boto3.resource('s3')
# Get list of objects for indexing
images=[('image01.jpeg','Albert Einstein'),
('image02.jpeg','Candy'),
('image03.jpeg','Armstrong'),
('image04.jpeg','Ram'),
('image05.jpeg','Peter'),
('image06.jpeg','Shashank')
]
# Iterate through list to upload objects to S3
for image in images:
file = open(image[0],'rb')
object = s3.Object('rekognition-pictures','index/'+ image[0])
ret = object.put(Body=file,
Metadata={'FullName':image[1]}
)
Clarification
Its my code to send images and name to S3 . But I dont know how to get image in this line of code images=[('image01.jpeg','Albert Einstein'), like how can I get this image in this code from /upload/image01.jpeg . and 2ndly how can I get images from s3 and show in my website image page ?
I know your question is specific to boto3 so you might not like my answer, but it will achieve the same outcome as what you would like to achieve and the aws-cli also makes use of boto3.
See here: http://bigdatums.net/2016/09/17/copy-local-files-to-s3-aws-cli/
This example is from the site and could easily be used in a script:
#!/bin/bash
#copy all files in my-data-dir into the "data" directory located in my-s3-bucket
aws s3 cp my-data-dir/ s3://my-s3-bucket/data/ --recursive
The very first thing, the code snippet you are showing as a reference is not for your use case as I had written that code snippet for batch uploads from boto3 where you have to provide image paths in your script along with metadata for image, so the names in your code snippet are metadata.So upto what i get to understand from your question, you want files in a local folder to be uploaded and want to provide custom names before uploading , so this is how you will do that.
import os
import boto3
s3 = boto3.resource('s3')
directory_in_str="E:\\streethack\\hold"
directory = os.fsencode(directory_in_str)
for file in os.listdir(directory):
filename = os.fsdecode(file)
if filename.endswith(".jpeg") or filename.endswith(".jpg") or filename.endswith(".png"):
strg=directory_in_str+'\\'+filename
print(strg)
print("Enter name for your image : ")
inp_val = input()
strg2=inp_val+'.jpeg'
file = open(strg,'rb')
object = s3.Object('mausamrest','test/'+ strg2) #mausamrest is bucket
object.put(Body=file,ContentType='image/jpeg',ACL='public-read')
else:
continue
programmatically , you have to provide path of folder which is hard-coded in this example in directory_in_str variable. then , this code will iterate over each file searching for image , then it will ask for input for custom name and then it will upload your file.
Moreover, you want to show these images on your website , so public_read for images have been turned on using ACL , so you can directly use s3 links to embedd images in your webpages like this one.
https://s3.amazonaws.com/mausamrest/test/jkl.jpeg
This above file is the one i used to test this code snippet. your images will be availbale like this. Make sure you change bucket name. :)
Using the Resource method:
# Iterate through list to upload objects to S3
bucket = s3.Bucket('rekognition-pictures')
for image in images:
bucket.upload_file(Filename='/upload/' + image[0],
Key='index/' + image[0],
ExtraArgs={'FullName': image[1]}
)
Using the client method:
import boto3
client = boto3.client('s3')
...
# Iterate through list to upload objects to S3
for image in images:
client.upload_file(Filename='/upload/' + image[0],
Bucket='rekognition-pictures',
Key='index/' + image[0],
ExtraArgs={'FullName': image[1]}
)
I have a json file placed in s3. The s3 url is similar to the below one:
https://s3-eu-region-1.amazonaws.com/dir-resources/sample.json
But in pyspark when pass the same, It is not reading the file.
path = "https://s3-eu-region-1.amazonaws.com/dir-resources/sample.json"
df=spark.read.json(path)
But I am able to download it through browser.
Assuming that dir-resources is the name of your bucket, you should be able to access to the file with the following URI:
path = "s3://dir-resources/sample.json"
In some cases, you may have to use the s3n protocol instead:
path = "s3n://dir-resources/sample.json"
I'm attempting to save an image to S3 using boto. It does save a file, but it doesn't appear to save it correctly. If I try to open the file in S3, it just shows a broken image icon. Here's the code I'm using:
# Get and verify the file
file = request.FILES['file']
try:
img = Image.open(file)
except:
return api.error(400)
# Determine a filename
filename = file.name
# Upload to AWS and register
s3 = boto.connect_s3(aws_access_key_id=settings.AWS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
bucket = s3.get_bucket(settings.AWS_BUCKET)
f = bucket.new_key(filename)
f.set_contents_from_file(file)
I've also tried replacing the last line with:
f.set_contents_from_string(file.read())
But that didn't work either. Is there something obvious that I'm missing here? I'm aware django-storages has a boto backend, but because of complexity with this model, I do not want to use forms with django-storages.
Incase you don't want to go for django-storages and just want to upload few files to s3 rather then all the files then below is the code:
import boto3
file = request.FILES['upload']
s3 = boto3.resource('s3', aws_access_key_id=settings.AWS_ACCESS_KEY, aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
bucket = s3.Bucket('bucket-name')
bucket.put_object(Key=filename, Body=file)
You should use django-storages which uses boto internally.
You can either swap the default FileSystemStorage, or create a new storage instance and manually save files. Based on your code example I guess you really want to go with the first option.
Please consider using django's Form instead of directly accessing the request.