I am resizing images client side before sending them to my flask app.
The resized image, which is drawn into a canvas to be resized, is sent via a POST request.
In my app the image is decoded via base64:
def resize_image(item):
content = item.split(';')[1]
image_encoded = content.split(',')[1]
body = base64.decodestring(image_encoded.encode('utf-8'))
return body
The imagedata is stored as type String in the body variable. I can save the data to my local machine and it works:
filename = 'some_image.jpg'
with open(filename, 'wb') as f:
print "written"
f.write(body)
What I need is to upload the resized image to AWS3. On one point I need to read() the image contents, but until the image is saved somewhere as a file it is still a String, so it fails:
file_data = request.values['a']
imagedata = resize_image(file_data)
s3 = boto.connect_s3(app.config['MY_AWS_ID'], app.config['MY_AWS_SECRET'], host='s3.eu-central-1.amazonaws.com')
bucket_name = 'my_bucket'
bucket = s3.get_bucket(bucket_name)
k = Key(bucket)
# fails here
file_contents = imagedata.read()
k.key = "my_images/" + "test.png"
k.set_contents_from_string(file_contents)
Unless there is an other solution, I thought I save the image temporarily to my server (Heroku) and upload it and then delete it, how would this work? Deleting afterwards is important here!
set_contents_from_string takes a string as a parameter, you could probably just pass your image string data directly to it for upload to S3
Solution:
Delete this part:
file_contents = imagedata.read()
Use imagedata directly here:
k.set_contents_from_string(imagedata)
If you need to call .read() on your data, but don't need save file on disk use StringIO:
import StringIO
output = StringIO.StringIO()
output.write('decoded image')
output.seek(0)
output.read()
Out[1]: 'decoded image'
Related
I am trying to give the user a "Save as" option when the user clicks the download button in my Django app. When the user clicks the button it kicks-off the following function. The function gets some CSVs from a blob container in Azure and adds them to a zip. That zip should then be offered to download and store in a location of the user's choice.
def create_downloadable_zip():
container_client = az.container_client(container_name=blob_generator.container_name)
blobs = container_client.list_blobs()
zip_file = zipfile.ZipFile(f'{models.AppRun.client_name}.zip', 'w')
for blob in blobs:
if blob.name.endswith(".csv"):
downloaded_blob = container_client.download_blob(blob)
blob_data = downloaded_blob.readall()
zip_file.writestr(blob.name, blob_data)
zip_file.close()
return zip_file
My views.py looks like follow:
def download_file(request):
if request.method == 'POST':
zip = create_downloadable_zip()
response = HttpResponse(zip, content_type='application/zip')
response['Content-Disposition'] = 'attachement;' f'filename={zip}.zip'
return response
#
# else:
# # return a 404 response if this is a POST request
# return HttpResponse(status=404)
return render(request, "download_file.html")
The functionality works, but it returns an empty non-zip file when the "Save as" window pop-ups. However, the actual zip file contains the files is being saved in the root folder of the Django project.
I really don't get why I doesn't return the zip file from memory, but rather directly stores that zip file in root and returns an empty non-zip file with the download functionality.
Someone knows what I am doing wrong?
zipfile is used to open a file, but it is not the actual file, simply a zipfile object as #b-remmelzwaal mentioned. You will need to create a file like object, and return that instead. This can be done using io.BytesIO.
from io import BytesIO
from zipfile import ZipFile
def create_zip():
container_client = az.container_client(container_name=blob_generator.container_name)
blobs = container_client.list_blobs()
buffer = BytesIO()
with ZipFile(buffer, 'w') as zip_file:
for blob in blobs:
if blob.name.endswith(".csv"):
downloaded_blob = container_client.download_blob(blob)
blob_data = downloaded_blob.readall()
zip_file.writestr(blob.name, blob_data)
return buffer.getvalue()
Note we are returning the file like object, not the zip file object. This is because buffer represents the actual file you've created.
You don't have to use a context manager, but I find them very useful.
Also, check your spelling for the line:
# attachment instead attachement
response['Content-Disposition'] = 'attachment;' f'filename={zip}.zip'
BytesIO Documentation
I have a script that takes a a group of images on the local machine, sends it to removeBG using threading. If it is successful, it gets the resultant file and uploads to s3 and grabs the s3 URL. Now we pass this URL to BannerBear to generate a composite which returns another URL that we print to the screen and then get the file so we can write it locally.
But while all the text of each process is being printed to the screen properly, somewhere along the way the writing of the final image locally gets skipped or overwritten by the other file that is being processed. If I go one at a time it works.
Code:
# Check the result of the RemoveBG API request if ok write the file locally and upload to s3
if response.status_code == requests.codes.ok:
with open(
os.path.join(OUTPUT_DIR, os.path.splitext(file)[0] + ".png"), "wb"
) as out:
out.write(response.content)
# Open a file-like object using io.BytesIO
image_data = io.BytesIO(response.content)
# Upload the image data to S3
s3.upload_fileobj(image_data, bucket_name, object_key)
# Get the URL for the uploaded image
image_url = f"https://{bucket_name}.s3.amazonaws.com/{object_key}"
print(f"Image uploaded to S3: {image_url}")
# pass the s3 url to BannerBear
bannerbear(image_url)
# start BannerBear Func
def bannerbear(photo_url):
# Send a POST request to the endpoint URL
responseBB = requests.post(endpoint_url, headers=headers, json=data)
if responseBB.status_code == 200:
# Print the URL of the generated image
print(responseBB.json()["image_url_jpg"])
# Write the image data to a file
filedata = requests.get(responseBB.json()["image_url_jpg"])
with open(
os.path.join(OUTPUT_DIR, "bb", os.path.splitext(file)[0] + ".jpg"), "wb"
) as f:
f.write(filedata.content)
print("BannerBear File Written")
else:
# Something went wrong
print(f"An error occurred: {responseBB.status_code}")
print(response.json()["message"])
# Create a thread pool with a maximum of 4 threads
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
# Iterate through all the files in the image directory
for file in os.listdir(IMAGE_DIR):
# Check if the file is an image
if file.endswith(".jpg") or file.endswith(".png"):
# Submit the task to the thread pool
executor.submit(process_image, file)
This is what I see in the console:
So it's going through all the steps.
I am trying to get a webp image, convert it to jpg and upload it to aws S3 without saving the file to disk (using io.BytesIO and boto3 upload_fileobj) , but with no success. The funny thing is that it works fine if I save the file to local disk and than use boto3 upload melhod.
This works:
r = requests.get(url)
if r.status_code == 200:
file_name = "name.jpeg"
s3 = boto3.client("s3")
webp_file = io.BytesIO(r.content)
im = Image.open(webp_file).convert("RGB")
im.save(
f"{config.app_settings.image_tmp_dir}/{file_name}", "JPEG"
)
s3.upload_file(
f"{config.app_settings.image_tmp_dir}/{file_name}",
config.app_settings.image_S3_bucket,
file_name,
ExtraArgs={"ContentType": "image/jpeg"},
)
This does not work:
r = requests.get(url)
if r.status_code == 200:
file_name = "name.jpeg"
s3 = boto3.client("s3")
webp_file = io.BytesIO(r.content)
im = Image.open(webp_file).convert("RGB")
jpg_file = io.BytesIO()
im.save(
jpg_file, "JPEG"
)
s3.upload_fileobj(
jpg_file,
config.app_settings.image_S3_bucket,
file_name,
ExtraArgs={"ContentType": "image/jpeg"},
)
I can see that the jpg_file has the correct size after im.save, but when the file is uploaded to aws S3 I get empty file.
After calling im.save(jpg_file, "JPEG"), the stream's position is still pointing after the newly-written image data. Anything that tries to read from jpg_file, will start reading from that position and will not see the image data.
You can use the stream's seek() method to move the position back to the start of the stream, before s3.upload_fileobj() tries to read it:
im.save(
jpg_file, "JPEG"
)
# Reset the stream position back to the start of the stream.
jpg_file.seek(0)
s3.upload_fileobj(
jpg_file,
config.app_settings.image_S3_bucket,
file_name,
ExtraArgs={"ContentType": "image/jpeg"},
)
Context
I have made a simple web app for uploading content to a blog. The front sends AJAX requests (using FormData) to the backend which is Bottle running on Python 3.7. Text content is saved to a MySQL database and images are saved to a folder on the server. Everything works fine.
Image processing and PIL/Pillow
Now, I want to enable processing of uploaded images to standardise them (I need them all resized and/or cropped to 700x400px).
I was hoping to use Pillow for this. My problem is creating a PIL Image object from the file object in Bottle. I cannot initialise a valid Image object.
Code
# AJAX sends request to this route
#post('/update')
def update():
# Form data
title = request.forms.get("title")
body = request.forms.get("body")
image = request.forms.get("image")
author = request.forms.get("author")
# Image upload
file = request.files.get("file")
if file:
extension = file.filename.split(".")[-1]
if extension not in ('png', 'jpg', 'jpeg'):
return {"result" : 0, "message": "File Format Error"}
save_path = "my/save/path"
file.save(save_path)
The problem
This all works as expected, but I cannot create a valid Image object with pillow for processing. I even tried reloading the saved image using the save path but this did not work either.
Other attempts
The code below did not work. It caused an internal server error, though I am having trouble setting up more detailed Python debugging.
path = save_path + "/" + file.filename
image_data = open(path, "rb")
image = Image.open(image_data)
When logged manually, the path is a valid relative URL ("../domain-folder/images") and I have checked that I am definitely importing PIL (Pillow) correctly using PIL.PILLOW_VERSION.
I tried adapting this answer:
image = Image.frombytes('RGBA', (128,128), image_data, 'raw')
However, I won’t know the size until I have created the Image object. I also tried using io:
image = Image.open(io.BytesIO(image_data))
This did not work either. In each case, it is only the line trying to initialise the Image object that causes problems.
Summary
The Bottle documentation says the uploaded file is a file-like object, but I am not having much success in creating an Image object that I can process.
How should I go about this? I do not have a preference about processing before or after saving. I am comfortable with the processing, it is initialising the Image object that is causing the problem.
Edit - Solution
I got this to work by adapting the answer from eatmeimadanish. I had to use a io.BytesIO object to save the file from Bottle, then load it with Pillow from there. After processing, it could be saved in the usual way.
obj = io.BytesIO()
file.save(obj) # This saves the file retrieved by Bottle to the BytesIO object
path = save_path + "/" + file.filename
# Image processing
im = Image.open(obj) # Reopen the object with PIL
im = im.resize((700,400))
im.save(path, optimize=True)
I found this from the Pillow documentation about a different function that may also be of use.
PIL.Image.frombuffer(mode, size, data, decoder_name='raw', *args)
Note that this function decodes pixel data only, not entire images.
If you have an entire image file in a string, wrap it in a BytesIO object, and use open() to load it.
Use StringIO instead.
From PIL import Image
try:
import cStringIO as StringIO
except ImportError:
import StringIO
s = StringIO.StringIO()
#save your in memory file to this instead of a regular file
file = request.files.get("file")
if file:
extension = file.filename.split(".")[-1]
if extension not in ('png', 'jpg', 'jpeg'):
return {"result" : 0, "message": "File Format Error"}
file.save(s)
im = Image.open(s)
im.resize((700,400))
im.save(s, 'png', optimize=True)
s64 = base64.b64encode(s.getvalue())
From what I understand, you're trying to resize the image after it has been saved locally (note that you could try to do the resize before it is saved). If this is what you want to achieve here, you can open the image directly using Pillow, it does the job for you (you do not have to open(path, "rb"):
image = Image.open(path)
image.resize((700,400)).save(path)
I'm using the Falcon framework and Pillow to upload a profile picture of a contact to S3, then resizing that picture for a thumbnail, and then uploading that thumbnail.
I've looked at other answers but some of them require having bucket write access activated and some use django's default_storage feature which I don't have available.
client = boto3.client('s3',
aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID'),
aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY')
)
class UploadResource(object):
def on_post(self, req, res):
#gathering file from SPA
contact_id = req.get_param('id')
filename = req.get_param('file').filename
file = req.get_param('file').file
salt = ''.join(chr(random.randint(97, 122)) for i in range(20))
filename = salt + '-' + filename
filename_thumb = salt + '-thumb-' + filename
#uploading normal sized image
client.upload_fileobj(file, 'contacts-cloud-images', filename)
#pull down image again and resize
img = Image.open(requests.get(image_url, stream=True).raw)
img.thumbnail((50,50))
print(img.format, img.size)
#save it to BytesIO container
io = BytesIO()
img.save(io, img.format)
#upload value of BytesIO container
---> client.upload_fileobj(io.getvalue(), 'contacts-cloud-images', filename_thumb)
I get the following error from the line with the arrow (---->):
ValueError: Fileobj must implement read
The error means client.upload_fileobj is expecting a file-like object that implements a read method, but you're passing it the content of the file-like object (io.getvalue()) instead of the file-like object itself (io)
This is a link to the documentation of upload_fileobj
Important non-related note: An important thing to note is you're naming the variable that holds your buffer io. io is also the name of a module of the standard library, and you're overwriting it. That should be an absolute no-no. Despite the local scope of your variable, I recommend you rename it to something meaninful, like file_content or image_content.