I am running a website on google app engine written in python with jinja2. I have gotten memcached to work for most of my content from the database and I am fuzzy on how I can increase the efficiency of images served from the blobstore. I don't think it will be much different on GAE than any other framework but I wanted to mention it just in case.
Anyway are there any recommended methods for caching images or preventing them from eating up my read and write quotas?
Blobstore is fine.
Just make sure you set the HTTP cache headers in your url handler. This allows your files to be either cached by the browser (in which case you pay nothing) or App Engine's Edge Cache, where you'll pay for bandwidth but not blobstore accesses.
Be very careful with edge caching though. If you set an overly long expiry, users will never see an updated version. Often the solution to this is to change the url when you change the version.
You can use google images api
https://developers.google.com/appengine/docs/python/images/functions
What I usually do is on upload, i store the url created by the images.get_serving_url(blob_key). Not sure if its cheaper but on my dev server each call to get_serving_url creates a datastore write.
My advice would be to use Google Cloud Storage for storing your images. It's better suited and recommended for serving static files. The good thing is that now you can use the same Images api for that:
Note: You can also serve images stored in Google Cloud Storage. To do this, you need to generate a Blob Key using the Blobstore API create_gs_key() function. You also need to set a default object ACL on the bucket being used that gives your app FULL_CONTROL permissions, so that the image service can add its own ACL to the objects. For information on ACLs and permissions, see the documentation for Google Cloud Storage.
PS. Another great feature I like here, is that you don't have to store different resolutions of your image if you need to serve them in different sizes. You can just add the parameters to the url which is returned by get_serving_url and that will do it. Also you only need to call get_serving_url once, store this url somewhere and use it whenever you need to serve the image. Plus you can reuse the same url for serving the same image in all different sizes.
URL Modifications:
=sXX To resize an image, append =sXX to the end of the image URL, where XX is an integer from 0–1600 representing the new image size in
pixels. The maximum size is defined in IMG_SERVING_SIZES_LIMIT. The
API resizes the image to the supplied value, applying the specified
size to the image's longest dimension and preserving the original
aspect ratio. For example, if you use =s32 to resize a 1200x1600
image, the resulting image is a 24x32. If that image were 1600x1200,
the resized image would be 32x24 pixels.
=sXX-c To crop and resize an image, append =sXX-c to the end of the image URL, where XX is an integer from 0–1600 representing the new
image size in pixels. The maximum size is defined in
IMG_SERVING_SIZES_LIMIT. The API resizes the image to the supplied
value, applying the specified size to the image's longest dimension
and preserving the original aspect ratio. If the image is portrait,
the API slices evenly from the top and bottom to make a square. If the
image is landscape, the API slices evenly from the left and right to
make a square. After cropping, the API resizes the image to the
specified size.
Related
I am trying to render a frame of a scene in Blender 2.8 using the Blender Python API, where a frame output is set via:
bpy.context.scene.render.filepath
It all works, but I was wondering can I render a frame to memory without saving an image to a disc.
I have not seen a function or property that I could use: RenderSettings API
Such in-memory rendering output would be preferred because the rendered frames are then serialized and sent over HTTP.
No option exists, so far.
There is an unreliable way that may suit your needs. The compositor can be setup with a viewer node which has its image contents accessible in bpy.data.images['Viewer Node'].pixels.
You will find a few questions on BSE, like this one that indicates that you can access the viewer node pixels, but there is a lack of updating that could break your workflow.
There is an old bug report about this but no fix.
As I commented in the above question, you could try creating a RAM backed tmpfs disk to save renders in which can eliminate diskIO.
I know about Google map static API which can return map as an image. But it returns static view and I need satellite view of google map without labels for my system. I've been searching for solution for so long but couldn't find any.
any help would really be appreciated.
If your intention is to get static maps satellite imagery via official service and hide all labels on it (except Google logo and copyrights) you can set map type to satellite and apply styles that hide all labels with Static Maps API. For example,
https://maps.googleapis.com/maps/api/staticmap?center=39.279117%2C-89.881745&zoom=18&size=600x400&maptype=satellite&style=feature%3Aall%7Celement%3Alabels%7Cvisibility%3Aoff&key=YOUR_API_KEY
Otherwise if you want download and store images for your needs, stitch them or remove copyright labels the solution might be not compliant with Google Maps API Terms of Service.
Look at the following sections of Terms of Service.
Paragraph 10.1 (a)
No access to APIs or Content except through the Service. You will not access the Maps API(s) or the Content except through the Service. For example, you must not access map tiles or imagery through interfaces or channels (including undocumented Google interfaces) other than the Maps API(s).
That means you cannot scrap satellite imagery for your needs and store it in your storage.
Paragraph 10.5 (g)
No removing, obscuring, or altering terms of service, links, or proprietary rights notices. You will not:
remove, obscure, or alter any Google terms of service or any links to or notices of those terms, or any copyright, trademark, or other proprietary rights notices; or
falsify or delete any author attributions, legal notices, or other labels of the origin or source of material.
Finally, paragraph 10.5 (b)
No derivative works. You will not modify or create a derivative work based on any Content unless expressly permitted to do so under these Terms. For example, the following are prohibited: (i) creating server-side modification of map tiles; (ii) stitching multiple static map images together to display a map that is larger than permitted in the Maps APIs Documentation; or (iii) tracing or copying the copyrightable elements of Google’s maps or building outlines and creating a new work, such as a new mapping or navigation dataset.
I hope my answer addresses your question.
I haven't found a similar question that I'm looking for Image Optimization.
I've tested how much Facebook can optimize the image uploaded:
980KB --> 77KB
846KB --> 62.1KB
From what I found out, Facebook is capable of optimizing the image up to 10 times while still pertaining some minimum image quality, as for the test above.
So, can anyone share what are the best ways that you have implemented to optimize image uploaded by user ?
When I searched in internet, I've seen some websites offer paid service for image optimization. However, we prefer not to subscribe for any paid service for image optimization at this stage.
I'm developing the project with Python language within Google App Engine environment. Any part where we can reuse from Python libraries or even Google App Engine libraries to achieve so ?
Probably you should star this issue to get pngcrush like functionality added to the AppEngine images API.
Basic optimization boils down to:
Choosing the appropriate format for the image (usually jpeg for
photographs; you can use jpeg across the board if you're not
concerned about image quality but otherwise png for screenshots etc.
may be wise)
Reducing the image to the smallest resolution appropriate for your
application
Increasing the compression level to the highest level possible while
maintaining your quality standards
You can also nitpick by stripping extraneous metadata, but that is usually unnecessary and not desirable.
If you want to do all of this in an automated fashion, you'll have to set a standard format and compression level across the board and accept that it won't be perfect in all cases, or else be able to determine what settings are appropriate for the image programmatically (which is quite difficult, unless you simply ask your users at upload time directly).
Normally I would use ImageMagick via the PythonMagick bindings for this task, but that may not be feasible on Google Apps Engine. In that case, maybe look at the Python Imaging Library.
Another solution is to use a 3rd party api, in this case you can use tinyPNG. There compression algorithm is probably one of the best out there. Check there developer guide here ~>
https://tinypng.com/developers
The first 500 photos per month are free & it's like $0.009 per image (> 500 && < 9500) or $0.002 > 10000 images.
You can't use PythonMagick unfortunately. But Python Imaging Library can be installed, and see Google Imaging Service on how to use it.
There is no magic bullet facebookesque optimization. You will have to try to develop your own that meets the standards you need. Most images these days are 5mp and up resizing them to 1280x720 or less is normal in web sites. The ability to crop extraneous image is also desirable before resizing.
I'm implementing my own link scraper to copy Facebook's technique as closely as possible (unless someone has a ready made lib for me...).
According to the many answers on SO, Facebook's process for determining the image to associate with a shared link involves searching for several recognized meta tags and then, if those are not found, stepping through the images on the page and returning a list of appropriately sized ones (at least 50px by 50px, have a maximum aspect ratio of 3:1, and in PNG, JPEG or GIF format according to this answer)
My question is, how does Facebook get the size information of the images? Is it loading all images for each shared link and inspecting them? Is there more efficient way to do this. (My backend is Python.)
(Side note: Would it make sense to use a client-side instead of server-side approach?)
Is there more efficient way to do this.
Most common “web” graphic formats – JPEG, GIF, PNG – contain info about the width & height in the header (or at least in the first block, for PNG).
So if the remote web server is accepting range requests it’d be possible to only request the first X bytes of an image resource instead of the whole thing to get the desired information.
(This is what Facebook’s scraper does for HTML pages, too – it’s quite common that you see in the debugger that the request was answered with HTTP status code 206 Partial Content – that meaning Facebook said they’re only interested in the first X (K)Bytes (for meta elements in head), and the web server was able to give them only that.
Note: I am cross-posting this from App Engine group because I got no answers there.
As part of my site about Japan, I have a feature where the user can
get a large PNG for use as desktop background that shows the user's
name in Japanese. After switching my site hosting entirely to App
Engine, I removed this particular feature because I could not find any
way to render text to a PNG using the image API.
In other words, how would you go about outputting an unicode string on
top of an image of known dimensions (1024x768 for example), so that
the text will be as large as possible horizontally, and centered
vertically? Is there a way to do this is App Engine, or is there some
external service besides App Engine that could make this easier for
me, that you could recommend (besides running ImageMagick on your own
server)?
Solution #1. Pure Python image library.
You can try to bundle PyPNG with your application. PyPNG is a pure Python library to create PNG images. It depends on zlib module, which is allowed on AppEngine, so PyPNG should work on AppEngine. Just use StringIO objects instead of files and write PNG data to them.
Shamelessly adapting PyPNG example how to make a bitmap PNG image:
import png
from StringIO import StringIO
# bitmap data
s = ['110010010011',
'101011010100',
'110010110101',
'100010010011']
s = map(lambda x: map(int, x), s)
f = StringIO()
w = png.Writer(len(s[0]), len(s), greyscale=True, bitdepth=1)
w.write(f, s)
# binary PNG data
print f.getvalue()
I suspect suboptimal performance, but as far as I know there is no other way to generate images on GAE.
And you still need to figure out how to rasterize text to produce bitmap data. The easiest way, probably, is just to keep bitmaps of all the symbols around (essentially, using a bitmap font).
To render ASCII text with PyPNG take a look at texttopng script.
So, limitations are:
Probably slow (needs to be checked)
Glyph rasterization is to be addressed
Solution #2. Off-site text-to-image rendering.
Google AppEngine does not provide tools to render text as raster images, but Google Charts does. With a proper choice of parameters, the outline text chart just renders simple text to PNG images.
For example, http://chart.apis.google.com/chart?chst=d_text_outline&chld=000000|32|h|FFFFFF|_|Render text to image|with Google Charts.|Some Unicode too:|Здра́вствуйте|こんにちは|नमस्ते|你好|שלו produces this:
Limitations:
You cannot generate images bigger than 300000 pixels
Style and font customizations are limited
Some Unicode scripts are not available
White background only
I ran into this same problem with writing text to an image. The issue at hand is that any imaging libraries used on google app engine must be pure python, which rules out PIL.
PyBMP
PyBMP is a pure-python library that can do simple text rendering. From there you can use google's imaging library to composite the resulting bitmap onto your other pictures. There's some sample code below. The downside is the library lacks nicer features like anti-aliasing and fine control over fonts so the text that it renders looks kind of crappy. It also may or may not handle unicode well.
# Create the image
text_img = bmp.BitMap(300,35,bmp.Color.WHITE)
# bmpfont_Tw_Cen_MT_30 is a generated file using PyBMP's tool
text_img.setFont(bmpfont_Tw_Cen_MT_30.font_data)
text_img.setPenColor( bmp.Color.BLACK )
text_img.drawText(name, 0, 0)
After this you can use google's composite function on text_img.getBitmap() as you would any other image.
External Image Processing
If the text isn't good enough (it wasn't for my project), an alternative solution is to set up an external server on a service like Rackspace purely for image processing. Set up an HTTP handler that does your image processing with PIL, and then returns the resulting image. From there you can either
upload the result straight to your static file hosting server (like s3) or
get the generated-text image result with app engine's urlfetch library and do the rest of your compositing in app engine
Not pretty, but it gets the job done.
It's a bit too late but I was looking for the same. I managed to draw unicode string (here Devanagari) onto an image and save it as a '.png' file by doing the following:
# -*- coding: utf-8 -*-
import Image, ImageDraw, ImageFont
img = Image.new('L', (16,16), 255)
draw = ImageDraw.Draw(img)
text_to_draw = unicode('क','utf-8')
font = ImageFont.truetype('Path/to/font/file',12)
draw.text((2,2), text_to_draw, font = font)
del draw
img.save('image.png')
P.S. got help from other posts on stackoverflow
[Stop press: As comment suggests - this answer doesn't work in Googe App Engine.]
The Python Imaging Library (PIL) can accomplish this.
You can load in the image, draw Unicode text on it with the ImageDraw.text() function.
You may need to call ImageDraw.textsize() a few times with different font sizes to find thelargest font that will fit.
Finally, you can save the .png image to a file (or serve it back directly).
Test with large images if you are running it from within the context of a web-server, to make sure you can allocate sufficient memory to processs large PNG files.
(Have I answered your question appropriately? I don't know if PIL is an option from within the Google App Engine.)