Develop Reference

I have a problem uploading images to DO Spaces resized with PILL

I'm uploading images to Digital Ocean Spaces using boto3. It's working really good until I add PILL.
In django view I have this code when I get the image:
from digitalocean_spaces import DigitalOceanSpaces
from helpers import resize_maintain_its_aspect_ratio
def images_view(request):
if request.method == "POST":
images = request.FILES.getlist('images')
for index, image in enumerate(images):
size = image.size
content_type = image.content_type
file_name = image.name
# TODO: fix method
# image = resize_maintain_its_aspect_ratio(image, 500)
DigitalOceanSpaces().upload_file(
key=key,
file=image,
content_type=content_type,
acl='private'
)
I can see all the information of each image.
To upload the image I use this method that is working too:
class DigitalOceanSpaces:
def default_session_client(self):
session = boto3.session.Session()
client = session.client(
's3',
region_name=REGION_NAME,
endpoint_url=ENDPOINT_URL,
aws_access_key_id=ACCESS_KEY_ID,
aws_secret_access_key=ACCESS_SECRET_KEY
)
return client
def upload_file(self, key, file, content_type, acl='private'):
client = self.default_session_client()
client.put_object(
Bucket=BUCKET_NAME,
Key=key,
Body=file,
ACL=acl,
ContentType=content_type,
Metadata={
'x-amz-meta-my-key': '*****'
}
)
The problem start when I call this another method to resize the image
from PIL import Image
def resize_maintain_its_aspect_ratio(image, base_width):
pillow_image = Image.open(image)
width_percent = (base_width / float(pillow_image.size[0]))
height_size = int((float(pillow_image.size[1]) * float(width_percent)))
resized_image = pillow_image.resize((base_width, height_size), Image.ANTIALIAS)
return resized_image
I see the error even if resize_maintain_its_aspect_ratio method just have:
pillow_image = Image.open(image)
So, the error is:
An error occurred (BadDigest) when calling the PutObject operation
(reached max retries: 4): Unknown
Does anyone know what the problem is ?

Is there a faster way of uploading multiple files in Django?

I have a django project where the client can upload multiple files at once. My problem is that for each uploaded file I'm creating a model object - one at a time. Is there a way to do this with bulk create or some other method that is faster.
Views.py:
images = request.FILES.getlist('images')
xmls = request.FILES.getlist('xmls')
jsons = request.FILES.getlist('jsons')
for image in images:
img_name = (str(image).split('.')[0])
dp_name = dataset_name+'-'+img_name
if [xml for xml in xmls if img_name+'.' in str(xml)]:
xml = [xml for xml in xmls if img_name+'.' in str(xml)][0]
else:
xml = None
if [json for json in jsons if img_name+'.' in str(json)]:
json = [json for json in jsons if img_name+'.' in str(json)][0]
else:
json = None
dataset.create_datapoint(image, xml, json, username, dp_name)
Models.py:
def create_datapoint(self, image, xml, json, username, dp_name):
datapoint = Datapoint.objects.create(
xml = xml,
json = json,
name = dp_name,
uploaded_by = username,
img = image,
belongs_to = self,
)
self.num_datapoints += 1
datapoint.parse_xml()
self.datapoints.add(datapoint)
self.save()
return

#mnislam01 is right but there is a small mistake in the code provided.
Here it is fixed:
data_point_create_list = []
# First create a list of objects.
for image in images:
dp = Datapoint(xml = xml,
json = json,
name = dp_name,
uploaded_by = username,
img = image,
belongs_to = self)
data_point_create_list.append(dp)
# Then bulk create all the objects.
if data_point_create_list:
Datapoint.objects.bulk_create(data_point_create_list)
Just needed to assign the newly made datapoint before appending.

You can use .bulk_create() method. For example.
data_point_create_list = []
# First create a list of objects.
for image in images:
dp = Datapoint(xml = xml,
json = json,
name = dp_name,
uploaded_by = username,
img = image,
belongs_to = self)
data_point_create_list.append(dp)
# Then bulk create all the objects.
if data_point_create_list:
Datapoint.objects.bulk_create(data_point_create_list)

I cannot figure out how to share a session in a bottle request

I've been tinkering with this for hours now and I just can't seem to find a way to make this work. It seems like it should be simple, and I'm sure it is, but I'm stumpled.
I have one module called 'server.py' that handles all of the routing with bottle, this is the main point of execution. An example of a request handler is as such, I'm generalizing as my codebase is rather hefty and most of it is irrelevant to the question:
server.py
#route('home')
def home():
page = Page('home') # A template manager I made
objects = db.get_objects(10) # This is what I can't get to work
return page.render(objects=objects)
I would like the code to be that simple from the server side and all database interaction done in db.py using helper functions, however I would like to use the returned objects from queries which are still attached to the session and so it must be closed outside of db.get_objects. A session should be created and closed on each request. I could do that manually from home() like so:
server.py
#route('home')
def home():
session = Session()
page = Page('home') # A jinja template manager I made
objects = db.get_objects(session, 10)
document = page.render(objects=objects)
session.close()
return document
I don't mind opening and closing the session every time, that seems logical and unavoidable, whether directly or through another object/function, but I do not want to have to pass that session around (manually) to every db helper function, that just seems messy to me.
I feel this problem can be solved with some OOP, a session manager class or something that is shared between the two, but I cannot figure out how to design or share it. The best idea I have come up with so far is to wrap my entire db.py in a class and have the constructor create the session. That would work for the helper functions, but I also have a bunch of other objects in db.py that also need access to the session such as the following, this is an actual object from my codebase:
db.py
class Sticker(_Base):
__tablename__ = 'sticker'
sticker_id = Column(Integer, ForeignKey('product.product_id'), primary_key=True)
sticker_name = Column(String)
svg_location = Column(String, unique=True)
svg_width = Column(Numeric, nullable=False)
svg_height = Column(Numeric, nullable=False)
shaped = Column(Boolean) # Whether the cutpath countors the image
#reconstructor
def _get_related_properties(self, init=False):
'''Fetches and fills out all properties that are created by the
__init__ constructor that are not in the orm constructor.'''
if not init:
session = Session()
self._product = session.query(Product).filter(Product.product_id == self.sticker_id).first()
category_id = session.query(ProductCategory).filter(ProductCategory.product_id == self.sticker_id).first()
session.close()
self.sticker_id = self._product.product_id
self.product_type = self._product.product_type
self.date_added = self._product.date_added
self.sticker_name = self._product.product_name
def _get_svg_size(self):
"""Returns a tuple of the width, height of an svg"""
# Currently only works with pixels I think. I know it fails when saved as points.
# May want to improve this for future versions.
# May also consider moving this function to an external util file or something.
# Possible units: (~"em" | ~"ex" | ~"px" | ~"in" | ~"cm" | ~"mm" | ~"pt" | ~"pc")
# Also may use viewbox attribute to determine aspect ratio and set sizes algorithmically.
import xml.etree.ElementTree as ET
import decimal
# Set decimal precision
decimal.getcontext().prec=7
tree = ET.parse(self.svg_location)
root = tree.getroot()
width = None
height = None
width_attr = root.get('width')
height_attr = root.get('height')
# Get measurement units
units = width_attr[-2:]
if units[-1] == '%':
units = '%'
elif not units.isalpha():
# if units not set assume px
width = decimal.Decimal(width_attr)
height = decimal.Decimal(height_attr)
units = 'px'
if units != 'px':
width = decimal.Decimal(width_attr[:-2])
height = decimal.Decimal(height_attr[:-2])
# Convert to px if not already
# Currently only supports in, cm, mm, and pt
# Assumes DPI is 72
MMPI = 2.834645669291339
DPI = 72
if units == 'in':
width *= DPI
height *= DPI
elif units == 'pt':
width /= DPI
height /= DPI
elif units == 'mm':
width *= MMPI
height *= MMPI
elif units == 'cm':
width *= MMPI * 10
height *= MMPI * 10
else:
raise ValueError('Unsupported svg size unit:',units )
return width, height
def __init__(self, svg_location, name='', category='', metatags=[], sticker_sizes=None, active=False):
# If no name given use filename
if not name:
from os.path import basename
name = basename(svg_location).rstrip('.svg')
# Create parent product and save to db to generate primary key/product id
session = Session()
self._product = Product(product_name = name, product_type = 'sticker', active = active)
session.add(self._product)
session.commit()
# TODO: Handle category and metatags
# Categories should probably be created explicitly by the admin, and so should exist
# at the time of sticker creation. Metatags are more numerous and created on the fly
# and so should be created automatically by the sticker constructor.
# TODO: Expand the sticker table to reference these values from the product table maybe?
self.sticker_id = self._product.product_id
self.svg_location = svg_location
self.svg_width, self.svg_height = self._get_svg_size()
self._get_related_properties(init=True)
# Add to the Database
session.add(self)
session.commit()
# Get sticker sizes
self.sticker_sizes = []
# Check if a size tuple was added, default is empty
if sticker_sizes:
for size in sticker_sizes:
sticker_size = StickerSize(self.sticker_id, size[0], size[1])
session.add(sticker_size)
self.sticker_sizes.append(StickerSize)
session.commit()
session.close()
Most of that is unimportant, but as you can see in many cases I need to query the database from within my ORM mapped objects so they too need access to the session. So a simple question, I hope, how can I do that? Can it even be done or am I approaching this in the wrong way? If I am approaching it wrong how so, and could you offer a design pattern that would work?

I found a solution and that is attaching the session to the request object which is unique to, obviously, each request and can be shared between modules.

Django PIL : IOError Cannot identify image file

I'm learning Python and Django.
An image is provided by the user using forms.ImageField(). Then I have to process it in order to create two different sized images.
When I submit the form, Django returns the following error:
IOError at /add_event/
cannot identify image file
I call the resize function:
def create_event(owner_id, name, image):
image_thumb = image_resizer(image, name, '_t', 'events', 180, 120)
image_medium = image_resizer(image, name, '_m', 'events', 300, 200)
I get en error when image_resizer is called for the second time:
def image_resizer(image, name, size, app_name, length, height):
im = Image.open(image)
if im.mode != "RGB":
im = im.convert("RGB")
im = create_thumb(im, length, height)
posit = str(MEDIA_ROOT)+'/'+app_name+'/'
image_2 = im
image_name = name + size +'.jpg'
imageurl = posit + image_name
image_2.save(imageurl,'JPEG',quality=80)
url_image='/'+app_name+'/'+image_name
return url_image
Versions:
Django 1.3.1
Python 2.7.1
PIL 1.1.7
I'm trying to find the problem, but i don't know what to do. Thank you in advanced!
EDIT
I solved rewriting the function; now it creates the different images in batch:
I call the resize function:
url_array = image_resizer.resize_batch(image, image_name, [[180,120,'_t'], [300,200,'_m']], '/events/')
so:
image_thumb = url_array[0]
image_medium = url_array[1]
and the resize function:
def resize_batch(image, name, size_array, position):
im = Image.open(image)
if im.mode != "RGB":
im = im.convert("RGB")
url_array = []
for size in size_array:
new_im = create_thumb(im, size[0], size[1])
posit = str(MEDIA_ROOT) + position
image_name = name + size[2] +'.jpg'
imageurl = posit + image_name
new_im.save(imageurl,'JPEG',quality=90)
new_url_array = position + image_name
url_array.append(new_url_array)
return url_array
Thanks to all!

As ilvar asks in the comments, what kind of object is image? I'm going to assume for the purposes of this answer that it's the file property of a Django ImageField that comes from a file uploaded by a remote user.
After a file upload, the object you get in the ImageField.file property is a TemporaryUploadedFile object that might represent a file on disk or in memory, depending on how large the upload was. This object behaves much like a normal Python file object, so after you have read it once (to make the first thumbnail), you have reached the end of the file, so that when you try to read it again (to make the second thumbnail), there's nothing there, hence the IOError. To make a second thumbnail, you need to seek back to the beginning of the file. So you could add the line
image.seek(0)
to the start of your image_resizer function.
But this is unnecessary! You have this problem because you are asking the Python Imaging Library to re-read the image for each new thumbnail you want to create. This is a waste of time: better to read the image just once and then create all the thumbnails you want.

I'm guessing that is a TemporaryUploadedFile ... find this with type(image).
import cStringIO
if isinstance(image, TemporaryUploadedFile):
temp_file = open(image.temporary_file_path(), 'rb+')
content = cStringIO.StringIO(temp_file.read())
image = Image.open(content)
temp_file.close()
I'm not 100% sure of the code above ... comes from 2 classes I've got for image manipulation ... but give it a try.
If is a InMemoryUploadedFile your code should work!

Django Image upload and resize

I have a standard Django form with an image field. When the image is uploaded, I would like to make sure that the image is no larger than 300px by 300px. Here is my code:
def post(request):
if request.method == 'POST':
instance = Product(posted_by=request.user)
form = ProductModelForm(request.POST or None, request.FILES or None)
if form.is_valid():
new_product = form.save(commit=False)
if 'image' in request.FILES:
img = Image.open(form.cleaned_data['image'])
img.thumbnail((300, 300), Image.ANTIALIAS)
# this doesnt save the contents here...
img.save(new_product.image)
# ..because this prints the original width (2830px in my case)
print new_product.image.width
The problem I am facing is, it is not clear to me how I get the Image type converted to the type that ImageField type.

From the documentation on ImageField's save method:
Note that the content argument should be an instance of django.core.files.File, not Python's built-in file object.
This means you would need to convert the PIL.Image (img) to a Python file object, and then convert the Python object to a django.core.files.File object. Something like this (I have not tested this code) might work:
img.thumbnail((300, 300), Image.ANTIALIAS)
# Convert PIL.Image to a string, and then to a Django file
# object. We use ContentFile instead of File because the
# former can operate on strings.
from django.core.files.base import ContentFile
djangofile = ContentFile(img.tostring())
new_product.image.save(filename, djangofile)

There you go, just change a little bit to suit your need:
class PhotoField(forms.FileField, object):
def __init__(self, *args, **kwargs):
super(PhotoField, self).__init__(*args, **kwargs)
self.help_text = "Images over 500kb will be resized to keep under 500kb limit, which may result in some loss of quality"
def validate(self,image):
if not str(image).split('.')[-1].lower() in ["jpg","jpeg","png","gif"]:
raise ValidationError("File format not supported, please try again and upload a JPG/PNG/GIF file")
def to_python(self, image):
try:
limit = 500000
num_of_tries = 10
img = Image.open(image.file)
width, height = img.size
ratio = float(width) / float(height)
upload_dir = settings.FILE_UPLOAD_TEMP_DIR if settings.FILE_UPLOAD_TEMP_DIR else '/tmp'
tmp_file = open(os.path.join(upload_dir, str(uuid.uuid1())), "w")
tmp_file.write(image.file.read())
tmp_file.close()
while os.path.getsize(tmp_file.name) > limit:
num_of_tries -= 1
width = 900 if num_of_tries == 0 else width - 100
height = int(width / ratio)
img.thumbnail((width, height), Image.ANTIALIAS)
img.save(tmp_file.name, img.format)
image.file = open(tmp_file.name)
if num_of_tries == 0:
break
except:
pass
return image
Source: http://james.lin.net.nz/2012/11/19/django-snippet-reduce-image-size-during-upload/

How about using standard image field https://github.com/humanfromearth/django-stdimage

Here is an app that can take care of that: django-smartfields
from django.db import models
from smartfields import fields
from smartfields.dependencies import FileDependency
from smartfields.processors import ImageProcessor
class Product(models.Model):
image = fields.ImageField(dependencies=[
FileDependency(processor=ImageProcessor(
scale={'max_width': 300, 'max_height': 300}))
])

Try my solution here: https://stackoverflow.com/a/25222000/3731039
Highlight
Using Pillow for image processing (two packages required: libjpeg-dev, zlib1g-dev)
Using Model and ImageField as storage
Using HTTP POST or PUT with multipart/form
No need to save the file to disk manually.
Create multiple resolutions and stores their dimensions.

You can use my library django-sizedimagefield for this, it has no extra dependency and is very simple to use.