My code is to add watermark on video and upload that video directly to s3.
I am able to do add the watermark on video. Upload that video Directly to s3 need to be done.
import moviepy.editor as mp
import boto3
AWS_ACCESS_KEY_ID="aws_key_id"
AWS_SECRET_ACCESS_KEY="aws_secret_access_key"
s3_resource = boto3.resource("s3")
BUCKET = "bucket_name"
video = mp.VideoFileClip('path_of_ video_file_stored')
logo = (mp.ImageClip('logo_file_stored')
.set_duration(video.duration))
final = mp.CompositeVideoClip([video,logo])
final.subclip(0).write_videofile('localpath/filename.mp4')
Instead of writing to local from final.subclip(0).write_videofile(localpath/filename.mp4)
I need to write file directly on s3 instead of first writing on local and then uploading to s3. May I know how to write to s3 directly with code solution of above.
Related
I would like to read .pdf files in S3 bucket, but the problem is that it returns formatted bytes,
Whereas if the file is in .csv or .txt this code works
What's wrong with .pdf files?
the code :
import boto3
s3client = boto3.client('s3')
fileobj = s3client.get_object(
Bucket=BUCKET_NAME,
Key='file.pdf'
)
filedata = fileobj['Body'].read()
contents = filedata
print(contents)
it returns :
b'%PDF-1.4\n%\xd3\xeb\xe9\xe1\n1 0 obj\n<</Title (Architecture technique)\n/Producer (Skia/PDF m99 Google Docs Renderer)>>\nendobj\n3 0 obj\n<</ca 1\n/BM /Normal>>\nendobj\n6 0 obj\n<</Type /XObject\n/Subtype /Image\n/Width 1424\n/Height 500\n/ColorSpace /DeviceRGB\n/SMask 7 0 R\n/BitsPerComponent 8\n/Filter /FlateDecode\n/Length 26885>> stream\nx\x9c\xed\xdd\xeb\x93$Y\x99\xe7\xf7'
another solution that i try but not work too:
import boto3
from PyPDF2 import PdfFileReader
from io import BytesIO
s3 = boto3.resource('s3')
obj = s3.Object(BUCKET_NAME,'file.pdf')
fs = obj.get()['Body'].read()
pdfFile = PdfFileReader(BytesIO(fs))
it's return :
<PyPDF2.pdf.PdfFileReader at 0x7efbc8aead00>
Start by writing some Python code to access a PDF file on your local disk (search for a Python PDF library on the web).
Once you have that working, then you can look at reading the file from Amazon S3.
When reading a file from S3, you have two options:
Use fileobj['Body'].read() (as you already are doing) to obtain the bytes from the file directly, or
Use download_file() to download the file from S3 to the local disk, then process the file from disk
Which method to choose will depend upon the PDF library that you choose to use.
I am uploading images to a folder currently on local . like in site/uploads.
And After searching I got that for uploading images to Amazon S3, I have to do like this
import boto3
s3 = boto3.resource('s3')
# Get list of objects for indexing
images=[('image01.jpeg','Albert Einstein'),
('image02.jpeg','Candy'),
('image03.jpeg','Armstrong'),
('image04.jpeg','Ram'),
('image05.jpeg','Peter'),
('image06.jpeg','Shashank')
]
# Iterate through list to upload objects to S3
for image in images:
file = open(image[0],'rb')
object = s3.Object('rekognition-pictures','index/'+ image[0])
ret = object.put(Body=file,
Metadata={'FullName':image[1]}
)
Clarification
Its my code to send images and name to S3 . But I dont know how to get image in this line of code images=[('image01.jpeg','Albert Einstein'), like how can I get this image in this code from /upload/image01.jpeg . and 2ndly how can I get images from s3 and show in my website image page ?
I know your question is specific to boto3 so you might not like my answer, but it will achieve the same outcome as what you would like to achieve and the aws-cli also makes use of boto3.
See here: http://bigdatums.net/2016/09/17/copy-local-files-to-s3-aws-cli/
This example is from the site and could easily be used in a script:
#!/bin/bash
#copy all files in my-data-dir into the "data" directory located in my-s3-bucket
aws s3 cp my-data-dir/ s3://my-s3-bucket/data/ --recursive
The very first thing, the code snippet you are showing as a reference is not for your use case as I had written that code snippet for batch uploads from boto3 where you have to provide image paths in your script along with metadata for image, so the names in your code snippet are metadata.So upto what i get to understand from your question, you want files in a local folder to be uploaded and want to provide custom names before uploading , so this is how you will do that.
import os
import boto3
s3 = boto3.resource('s3')
directory_in_str="E:\\streethack\\hold"
directory = os.fsencode(directory_in_str)
for file in os.listdir(directory):
filename = os.fsdecode(file)
if filename.endswith(".jpeg") or filename.endswith(".jpg") or filename.endswith(".png"):
strg=directory_in_str+'\\'+filename
print(strg)
print("Enter name for your image : ")
inp_val = input()
strg2=inp_val+'.jpeg'
file = open(strg,'rb')
object = s3.Object('mausamrest','test/'+ strg2) #mausamrest is bucket
object.put(Body=file,ContentType='image/jpeg',ACL='public-read')
else:
continue
programmatically , you have to provide path of folder which is hard-coded in this example in directory_in_str variable. then , this code will iterate over each file searching for image , then it will ask for input for custom name and then it will upload your file.
Moreover, you want to show these images on your website , so public_read for images have been turned on using ACL , so you can directly use s3 links to embedd images in your webpages like this one.
https://s3.amazonaws.com/mausamrest/test/jkl.jpeg
This above file is the one i used to test this code snippet. your images will be availbale like this. Make sure you change bucket name. :)
Using the Resource method:
# Iterate through list to upload objects to S3
bucket = s3.Bucket('rekognition-pictures')
for image in images:
bucket.upload_file(Filename='/upload/' + image[0],
Key='index/' + image[0],
ExtraArgs={'FullName': image[1]}
)
Using the client method:
import boto3
client = boto3.client('s3')
...
# Iterate through list to upload objects to S3
for image in images:
client.upload_file(Filename='/upload/' + image[0],
Bucket='rekognition-pictures',
Key='index/' + image[0],
ExtraArgs={'FullName': image[1]}
)
I'm trying to upload a .wav file to s3.
My currently put_object code looks like this:
client.put_object(
ACL='public-read',
Bucket='test',
Key='test_folder/' + file_name,
ContentType = 'audio/x-wav',
Body='test_folder/music/' + file_name,
StorageClass='STANDARD_IA'
)
I'm uploading it successfully, but when I try to download it and play, I get an error:
Could not determine type of stream
What arguments do I need to pass in order to tell S3 that it's a sound .wav file? When I tried to upload it using boto.s3 using bucket.new_key(key_name) and set_content_from_file, it worked perfectly, but I couldn't set the StorageClass using that function.
Thanks.
In my flask application, I am using a function to upload file to Amazon s3, using Boto.
Its working fine most of the cases, but some times its uploading files as zero byte file with no extension.
Why its failing sometimes,
I am validating user image file in form.
FileField('Your photo',validators=[FileAllowed(['jpg', 'png'], 'Images only!')])
My image upload function.
def upload_image_to_s3(image_from_form):
#upload pic to amazon
source_file_name_photo = secure_filename(image_from_form.filename)
source_extension = os.path.splitext(source_file_name_photo)[1]
destination_file_name_photo = uuid4().hex + source_extension
s3_file_name = destination_file_name_photo
# Connect to S3 and upload file.
conn = boto.connect_s3('ASJHjgjkhSDJJHKJKLSDH','GKLJHASDJGFAKSJDGJHASDKJKJHbbvhjcKJHSD')
b = conn.get_bucket('mybucket')
# Connect to S3 and upload file.
sml = b.new_key("/".join(["myfolder",destination_file_name_photo]))
sml.set_contents_from_string(image_from_form.read())
acl='public-read'
sml.set_acl(acl)
return s3_file_name
How large are your assets? If there is too large of an upload, you may have to multipart/chunk it otherwise it will timeout.
bucketObject.initiate_multipart_upload('/local/object/as/file.ext')
it means you will not be using set_contents_from_string but rather store and upload. You may have to use something to chuck the file, like FileChuckIO.
An example is here if this applies to you : http://www.bogotobogo.com/DevOps/AWS/aws_S3_uploading_large_file.php
Also, you may want to edit your post above and alter your AWS keys.
I'm attempting to save an image to S3 using boto. It does save a file, but it doesn't appear to save it correctly. If I try to open the file in S3, it just shows a broken image icon. Here's the code I'm using:
# Get and verify the file
file = request.FILES['file']
try:
img = Image.open(file)
except:
return api.error(400)
# Determine a filename
filename = file.name
# Upload to AWS and register
s3 = boto.connect_s3(aws_access_key_id=settings.AWS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
bucket = s3.get_bucket(settings.AWS_BUCKET)
f = bucket.new_key(filename)
f.set_contents_from_file(file)
I've also tried replacing the last line with:
f.set_contents_from_string(file.read())
But that didn't work either. Is there something obvious that I'm missing here? I'm aware django-storages has a boto backend, but because of complexity with this model, I do not want to use forms with django-storages.
Incase you don't want to go for django-storages and just want to upload few files to s3 rather then all the files then below is the code:
import boto3
file = request.FILES['upload']
s3 = boto3.resource('s3', aws_access_key_id=settings.AWS_ACCESS_KEY, aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
bucket = s3.Bucket('bucket-name')
bucket.put_object(Key=filename, Body=file)
You should use django-storages which uses boto internally.
You can either swap the default FileSystemStorage, or create a new storage instance and manually save files. Based on your code example I guess you really want to go with the first option.
Please consider using django's Form instead of directly accessing the request.