I'm making an app to upload text and images. I've readed a lot about blobstore and Google High Performance Image Serving and finally I got a way to implement it all together.
What I want to know is if all is well done or can be do in a better way, and also if it is better to save the serving_url in the model or must be calculated every time I want to print the images in the page.
There is a User and a Picture only.
This is the code (summarized, forget about my custom.PageHandler, that only have functions to render the pages easily, and the stuff for check forms values, etc.):
class User(ndb.Model):
""" A User """
username = ndb.StringProperty(required=True)
password = ndb.StringProperty(required=True)
email = ndb.StringProperty(required=True)
class Picture(ndb.Model):
""" All pictures that a User has uploaded """
title = ndb.StringProperty(required=True)
description = ndb.StringProperty(required=True)
blobKey = ndb.BlobKeyProperty(required=True)
servingUrl = ndb.StringProperty()
created = ndb.DateTimeProperty(auto_now_add=True)
user = ndb.KeyProperty(kind=User)
# This class shows the user pics
class List(custom.PageHandler):
def get(self):
# Get the actual user pics
pics = Picture.by_user(self.user.key)
for pic in pics:
pic.servingUrl = images.get_serving_url(pic.blobKey, size=90, crop=True)
self.render_page("myPictures.htm", data=pics)
# Get and post for the send page
class Send(custom.PageHandler, blobstore_handlers.BlobstoreUploadHandler):
def get(self):
uploadUrl = blobstore.create_upload_url('/addPic')
self.render_page("addPicture.htm", form_action=uploadUrl)
def post(self):
# Create a dictionary with the values, we will need in case of error
templateValues = self.template_from_request()
# Test if all data form is valid
testErrors = check_fields(self)
if testErrors[0]:
# No errors, save the object
try:
# Get the file and upload it
uploadFiles = self.get_uploads('picture')
# Get the key returned from blobstore, for the first element
blobInfo = uploadFiles[0]
# Add the key to the template
templateValues['blobKey'] = blobInfo.key()
# Save all
pic = Picture.save(self.user.key, **templateValues)
if pic is None:
logging.error('Picture save error.')
self.redirect("/myPics")
except:
self.render_page("customMessage.htm", custom_msg=_("Problems while uploading the picture."))
else:
# Errors, render the page again, with the values, and showing the errors
templateValues = custom.prepare_errors(templateValues, testErrors[1])
# The session for upload a file must be new every reload page
templateValues['form_action'] = blobstore.create_upload_url('/addPic')
self.render_page("addPicture.htm", **templateValues)
Basically, I list all the pics, showing the image in a jinja2 template with this line:
{% for line in data %}
<tr>
<td class="col-center-data"><img src="{{ line.servingUrl }}"></td>
So in the List class I calculate each serving url and add it temporarily to the Model. I don't know exactly if will be good to save it directly in the Model, because I don't know if the url can change with the time. Will be the url permanent for the image? In that case I can save it instead of calculate, true?
The Send class only shows a form to upload the image and saves the data to the Model. I always generate a new form_action link in the case of re-render the page, because docs talk about it. Is it right?
The code is working, but I want to know which is the better way to do it, in terms of performance and resource-saving.
You're right. You do want to save the get_serving_url() instead of calling it repeatedly. It stays the same.
Note that there's a delete_serving_url() when you're done with the url, like when you delete the blob.
Related
I have been practicing Django for a while now. Currently I am using it in a project where I'm fetching Facebook data via GET requests and then saving it to an sqlite database using Django models. I would like to know how can I improve the following code and save a list of Facebook posts and their metrics efficiently. In my current situation, I am using a for loop to iterate on a list containing several Facebook Posts and their respective metrics which is then associated to the specific Django model and finally saved.
def save_post(post_id, page_id):
facebook_post = Post(post_id=post_id,
access_token=fb_access_token)
post_db = PostsModel(page_id=page_id, post_id=post.post_id)
post_db.message = facebook_post.message
post_db.story = facebook_post.story
post_db.full_picture = facebook_post.full_picture
post_db.reactions_count = facebook_post.reactions_count
post_db.comments_count = facebook_post.comments_count
post_db.shares_count = facebook_post.shares_count
post_db.interactions_count = facebook_post.interactions_count
post_db.created_time = facebook_post.created_time
post_db.published = facebook_post.published
post_db.attachment_title = facebook_post.attachment_title
post_db.attachment_description = facebook_post.attachment_description
post_db.attachment_target_url = facebook_post.attachment_target_url
post_db.save()
post_db is a Django model object instantiated using PostsModel while Post is a normal Python Class which I wrote. The latter is simply a collection of GET requests which fetches data from Facebook's Graph API and returns JSON data whereby I associate relevant data to class attributes (message, 'shares_count`).
I read about the bulk_create function from Django's documentation but I don't know how to pass on the above. I also tried using multiprocessing and Pool but the above function does execute. Right now, I am just iterating sequentially on a list. As the list increases in length, it takes more time to save.
def create(self, request):
page_id = request.data['page_id']
page = get_object_or_404(PagesModel, pk=page_id)
post_list = get_list_or_404(PostsModel, page_id=page_id)
for post_id in post_list:
save_post(post_id=post_id, page_id=page)
The above function gets an already saved list from the database for a specific page based on the page_id. Then, the for loop iterates on each post in the list and its post_id and page instance are sent to the save_post function to fetch its data and save it.
Huge thanks if anyone can suggest a more effective way to tackle this. Thank you.
You are going in the right direction with the bulk_load. Generate a list of the PostsModel objects and then use bulk_create to upload them into the database. An important note here is that it won't work if the posts already exist in the database. For updating posts, try the bulk_update.
def save_post(post_id, page_id):
facebook_post = Post(post_id=post_id,
access_token=fb_access_token)
post_db = PostsModel(page_id=page_id, post_id=post.post_id)
post_db.message = facebook_post.message
post_db.story = facebook_post.story
post_db.full_picture = facebook_post.full_picture
post_db.reactions_count = facebook_post.reactions_count
post_db.comments_count = facebook_post.comments_count
post_db.shares_count = facebook_post.shares_count
post_db.interactions_count = facebook_post.interactions_count
post_db.created_time = facebook_post.created_time
post_db.published = facebook_post.published
post_db.attachment_title = facebook_post.attachment_title
post_db.attachment_description = facebook_post.attachment_description
post_db.attachment_target_url = facebook_post.attachment_target_url
return post_db
def create(self, request):
page_id = request.data['page_id']
page = get_object_or_404(PagesModel, pk=page_id)
post_list = get_list_or_404(PostsModel, page_id=page_id)
post_model_list = [save_post(post_id=post_id, page_id=page) for post_id in
post_list]
PostsModel.objects.bulk_create(post_model_list]
can anyone help me to display an image or pdf file stored in mongodb with a flask app to the browser? Currently, when I try, I get a 200 response but the browser is just blank. This is the function I use:
My mongo document is modeled like this:
class Users(db.Document):
_id = db.StringField()
name = db.StringField()
picture = db.ReferenceField('fs.files') #holds the reference for the file in the database
upload_picture = db.FileField() #used to load pics via flask into the database
email = db.StringField()
password = db.StringField()
meta = {'collection': 'Users'}
This is the function I had to retrieve and display the image. However with this function, I get a blank browser:
class RetrievePicture(Resource):
def get(self, id):
try:
user = Users.objects(_id=id).first()
doc = user.picture.read()
return send_file(io.BytesIO(doc), mimetype='image/png')
except Users.DoesNotExist:
return 'Picture not found', 401
The flask route I enter into the browser is supposed to return the picture from the specific user whose id I enter. The route looks like this http://127.0.0.1:5000/api/picture/(id) Does anyone know what I'm doing wrong or have another solution using MongoEngine?
I am using a custom python framework, (not django or webapp2), following this instructions: Create an upload URL, I have been available to upload the file, basically I submit my form to an URL provided by a script with this:
from google.appengine.ext import blobstore
upload_url = blobstore.create_upload_url('/upload')
By checking the blobstore viewer I notice that the file has been uploaded, this brings my first question.
How to filter, before uploading? for example, how to upload only if an image type is a jpg or png and discard anything else.
My second question is, how to properly handle the the uploaded file, what I found so far is to create a custom get_uploads method, this is what I am using but wondering if is the right way:
import cgi
from google.appengine.ext import blobstore
class APIResource(object):
def get_uploads(self, request, field_name=None):
if hasattr(request, '__uploads') == False:
request.environ['wsgi.input'].seek(0)
fields = cgi.FieldStorage(
request.environ['wsgi.input'],
environ=request.environ)
request.__uploads = {}
for key in fields.keys():
field = fields[key]
if isinstance(field, cgi.FieldStorage):
if 'blob-key' in field.type_options:
request.__uploads.setdefault(
key, []).append(
blobstore.parse_blob_info(field))
if field_name:
try:
return list(request.__uploads[field_name])
except KeyError:
return []
else:
results = []
for uploads in request.__uploads.itervalues():
results.extend(uploads)
return results
def dispatch(self, request, response):
blob_info = self.get_uploads(request, 'picture')[0]
print blob_info, blob_info.key(), request.__uploads
I would like to avoid uploading first, later check if type is valid and delete if not, so that I can not fill my storage with garbage in case I can't handle/get the upload key.
Any ideas ?
I have a function that gets an image from a form, and put's it into the database along with the username. So, here is my database:
class Imagedb(db.Model):
name = db.StringProperty(required = True)
image = db.BlobProperty()
And here is the code that writes to the database:
class Change_Profile_Image(MainHandler):
def get(self):
if self.user:
self.render('change_profile_image.html', username = self.user.name, firstname=self.user.first_name)
else:
self.render('change_profile_image.html')
def post(self):
imagedb = Imagedb(name = self.user.name)
imageupl = self.request.get("img")
imagedb.image = db.Blob(imageupl)
imagedb.put()
self.redirect('/profile')
Any who, it works awesome. Except for one thing. What i'm trying to accomplish is only storing ONE profile picture. What ends up happening is this:
Say I am the user admin. Ill upload a display pic, that pic shows in the profile. Ill upload another one, that one shows. Cool, except for the fact that I have 2 objects in my database that have the name = admin attribute. I would like to edit this...
def post(self):
imagedb = Imagedb(name = self.user.name)
imageupl = self.request.get("img")
imagedb.image = db.Blob(imageupl)
imagedb.put()
self.redirect('/profile')
so that I can post images to the database, but if one exists, it is overwritten. Could anyone help me with this please? I'm relatively new to python and app engine.
If something is unclear, please let me know.
You want to set the key of the Imagedb entity to "name". Essentially, you don't need the name field, but you'll instantiate it like
imagedb = Imagedb(key_name = self.user.name)
The key is a required field on all entities. By using your user name as the key it means every time you refere to a given key, it's the same entity.
Ok, first I want to state that this is Google App Engine via Python.
Any who,
These are my handlers / routing where the problem is occuring. Please read below for context and specifics:
class GetImage(MainHandler):
def get(self):
img = db.get(self.request.get("entity_id"))
self.response.out.write(img.image)
class Profile(MainHandler):
def get(self, profile_name):
current_user = str(self.user.name)
profile_name = current_user
if self.user:
key='ag5kZXZ-c3VpdGVnYW1lcnINCxIHSW1hZ2VkYhgxDA'
imgs = db.GqlQuery("select * from Imagedb WHERE name =:1", current_user)
for img in imgs:
key = img.key() # this is the key
self.render('profile.html', profile_name = self.user.name, current_user = self.user.name, profile_image = key ,username = self.user.name, email = self.user.email, first_name = self.user.first_name, last_name = self.user.last_name, country = self.user.country, prov_state = self.user.prov_state, city_town = self.user.city_town)
else:
self.redirect('/register')
class Change_Profile_Image(MainHandler):
def get(self):
if self.user:
self.render('change_profile_image.html', username = self.user.name, firstname=self.user.first_name, current_user = self.user.name)
else:
self.render('change_profile_image.html')
def post(self):
imagedb = Imagedb(name = self.user.name)
imageupl = images.resize(self.request.get("img"), 200, 200)
imagedb.image = db.Blob(imageupl)
imagedb.put()
self.redirect('/profile/'+self.user.name)
app = webapp2.WSGIApplication([('/', MainPage),
('/register', Register),
('/article', ArticlePage),
('/profile/([^/]+)', Profile),
('/login', Login),
('/logout', Logout),
('/welcome', Unit3Welcome),
('/games', Games),
('/forum', Forum),
('/media', Media),
('/rank', Rank),
('/review', Reviews),
('/events', Events),
('/alreadyloggedin', AlreadyLoggedIn),
('/change_profile_image', Change_Profile_Image),
('/img', GetImage)],
debug=True)
Alright so here is where stuff gets loopy. If I change the Profile class to take -- get(self) and remove my reg expression from the routing for the profile class, my images work perfectly. As soon as I route to unique profiles, i.e. pass profile_name into the Profile handler and map the URL to that profile, I lose all functionality of my GetImage handler. When I look at the source code, nothing has changed. The image is still being passed into the template as per usual.
Does anyone have any idea as to what is going on here? I would really appreciate it. Thank you very much in advance. Hopefully my knowledge will catch up to you guys and I'll be answering questions soon :p.
It's difficult to answer your question without seeing a (simplified) version of your template.
There are also a couple of weird elements in your code that make it hard to tell what's going on. It's hard to format this as a comment, so I'm putting it as an answer, just so you can at least see it.
In your get request in your profile handler, you try to get the current user before you check that the current user exists. You also just throw away the profile_name element completely when you assign profile_name to current_user, so you'll never get a profile image for anything but the current user on a profile page.
You take in profile_name here, but never use it:
def get(self, profile_name):
current_user = str(self.user.name)
profile_name = current_user
You loop over imgs but replace the key each time, which means that if you return more than 1 image you can't tell that this has occurred and you overwrite anything but the last image in the query. One thing you should do is add a check to see if imgs is even truthy, so you can tell if you got any results whatsoever, that might (though I can't imagine how) explain why your image handler is failing.
Finally, you might check your source to see which image url is actually being requested in the template.