I use this User class provided by Google App Engine: from webapp2_extras.appengine.auth.models import User
I want to make some change to an instance of User via Data Store in the Google cloud console: change the name attribute of a User to something else.
As far as I know the User object is cached in some way. If I make the change directly in data store, how can I locate it in the memcache and delete it? My goal is to force the cached User object to be refreshed.
The memcache key used can be found in google.appengine.ext.context.py: Context.get():
if use_memcache:
mkey = self._memcache_prefix + key.urlsafe()
The prefix (in my version at least) is 'NDB9:'. So to get the memcache key you can use something similar to:
from google.appengine.ext import ndb
key = ndb.Key('User', 5229916580741120)
mkey = 'NDB9:' + key.urlsafe()
print mkey
This results in something like: NDB9:ahpzfmR5bmFtaWMtdHJhdmVsbGVyLTItdGVzdHIRCxIEVXNlchiAgIDAzZKlCQw
To delete:
from google.appengine.api import memcache
value = memcache.get(mkey)
result = memcache.delete(mkey)
print "Memcache delete result={}".format(result)
if result == 2:
print "{} memcache value has been deleted".format(key)
elif result == 1:
print "{} memcache value not found".format(key)
else:
print "Failed to delete {} memcache value".format(key)
Caveat: without the memcache.get() call before memcache.delete() the delete result was always '1' (not found) using the interactive console. Not sure why that's the case.
Related
I am trying to add memcache to my webapp deployed on GAE, and to do this I am using memcache.Client() to prevent damage from any racing conditions:
from google.appengine.api import memcache
client = memcache.Client()
class BlogFront(BlogHandler):
def get(self):
global client
val = client.gets(FRONT_PAGE_KEY)
posts = list()
if val is not None:
posts = list(val)
else:
posts = db.GqlQuery("select * from Post order by created desc limit 10")
client.cas(FRONT_PAGE_KEY, list(posts))
self.render('front.html', posts = posts)
To test the problem I have a front page for a blog that displays the 10 most recent entries. If there is nothing in the cache, I hit the DB with a request, otherwise I just present the cached results to the user.
The problem is that no matter what I do, I always get val == None, thus meaning that I always hit the database with a useless request.
I have sifted through the documentation:
https://developers.google.com/appengine/docs/python/memcache/
https://developers.google.com/appengine/docs/python/memcache/clientclass
http://neopythonic.blogspot.pt/2011/08/compare-and-set-in-memcache.html
And it appears that I am doing everything correctly. What am I missing?
(PS: I am a python newb, if this is a retarded error, please bear with me xD )
from google.appengine.api import memcache
class BlogFront(BlogHandler):
def get(self):
client = memcache.Client()
client.gets(FRONT_PAGE_KEY)
client.cas(FRONT_PAGE_KEY, 'my content')
For a reason I cannot yet possible understand, the solution lies in having a gets right before having a cas call ...
I think I will stick with the memcache non-thread-safe version of the code for now ...
I suspect that the client.cas call is failing because there is no object. Perhaps client.cas only works to update and existing object (not to set a new object if there is none currently)? You might try client.add() (which will fail if an object already exists with the specified key, which I think is what you want?) instead of client.cas()
I've been using Django for a couple of days & setup a basic blog from a tutorial with django comments.
I've got a totally separate python script that generates screenshots and uploads them to Amazon S3, now I'd like my django app to display all the images in the bucket and use a comment system on the images. Preferably I'd do this by just storing the URLs in my sqlite db, which I've got hard-coded currently to display all images in the db and has comments enabled on these.
My model:
(Does this need a foreign key to the django comments or is that just part of the Django Magic?!)
class Image(models.Model):
imgUrl=models.CharField(max_length=200)
meta=models.CharField(max_length=300)
def __unicode__(self):
return self.imgUrl
My bucket structure:
https://s3-eu-west-1.amazonaws.com/bucket/revision/process/images.png
Almost all the tutorials and packages I'm finding are based on upload/download rather than a simple for keys in bucket type approach that I want.
One of my problems is understanding how I can integrate my Boto functions with Django if I'm using Base.html. In an earlier tutorial I had an index page which had a view and could call functions from there. But base doesn't need that so I'm starting to get a little lost.
haven't looked up if boto api changed, but this is how it worked last time i looked
from boto.s3.connection import S3Connection
from boto.s3.key import Key
import s3config
conn = S3Connection(s3config.passwd, s3config.secret)
bucket = conn.get_bucket(s3config.bucket)
s3_path = '/some/path/in/your/bucket'
keys = bucket.list(s3_path)
# or if you want all keys:
# keys = bucket.get_all_keys()
for key in keys:
print key
# here you can download or do other stuff
# with the keys like get some metadata
print key.name
print key.etag
print key.size
print key.last_modified
#s3config.py
passwd = 'BLABALBALABALA'
secret = 'xvdwv3efefefefefef'
bucket = 'name-of-your-bucket'
Update:
Amazon s3 is a key value store, where key is a string. So nothing prevents you from putting in keys like:
/this/string/key/looks/like/a/unix/path
/folder/images/fileA.jpg
/folder/images/fileB.jpg
/folder/images/folderX/fileX1.jpg
now bucket.list(prefix="/folder/images/") would yield the latter three.
Look here for further details:
http://readthedocs.org/docs/boto/en/latest/ref/s3.html#boto-s3-bucket
http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGET.html?r=8270
This is my code to store result from s3 to mysql by boto, django.
from demo.models import Movies
import boto
from boto.s3.key import Key
import string
from django.db import connection, transaction
def movietitle(b):
key = b.get_key('netflix/movie_titles.txt')
content = key.get_contents_as_string()
line = content.split('\n')
args = []
for imovie in line:
if len(imovie) > 0:
imovie = imovie.split(',')
movieid = imovie[0]
year = imovie[1]
title = imovie[2]
iargs = [string.atoi(movieid),title,year]
args.append(iargs)
cursor = connection.cursor()
sql = "insert into demo_movies(MovieID,MovieName,ReleaseYear) values(%s,%s,%s)"
cursor.executemany(sql,args)
transaction.commit_unless_managed()
cursor.close()
i have the following model in google app engine:
class CustomUsers(db.Model):
cuid = db.StringProperty()
name = db.StringProperty()
email = db.StringProperty()
bday = db.StringProperty()
now given a cuid i just want to check if he is present in the high replication data store.
in my views i write the following, but eventually get an 500 error:
usr_prev = CustomUsers.all().filter('cuid =', request.POST['id'])
if(not usr_prev):
logging.info("this user does not exist")
else:
logging.info("user exists")
but this gives me an error. How shall i do it?
(Do not consider any importing issues.)
"500 errors" don't exist in isolation. They actually report useful information, either on the debug page or in the logs. I'd guess that your logs show that request.POST has no element id, but that is just a guess.
I noticed that you have assigned a query object to usr_prev. You need to execute the query. I think Query.count(1) function should do it.
However, this may not be related to the 500 error.
you can go like this to see:
query = Users.all()
cnt = query.filter('id =', user_id).count()
logging.info(cnt)
return int(cnt)
once this is done, seems you are using django with app engine, as per the error you are not returning string response. you might want to return
return HttpResponse("1") instead of return HttpResponse(1)
i have this issue with app engine's datastore. In interactive console, I always get no entities when i ask if a url already exists in my db. When I execute the following code....
from google.appengine.ext import db
class shortURL(db.Model):
url = db.TextProperty()
date = db.DateTimeProperty(auto_now_add=True)
q = shortURL.all()
#q.filter("url =", "httphello")
print q.count()
for item in q:
print item.url
i get the this response, which is fine
6
httphello
www.boston.com
http://www.boston.com
httphello
www.boston.com
http://www.boston.com
But when I uncomment the line "q.filter("url =", "httphello")", i get no entities at all (a response of 0). I know its something ultra simple, but I just can't see it! help.
TextProperty values are not indexed, and cannot be used in filters or sort orders.
You might want to try with StringProperty if you don't need more than 500 characters.
I think .fetch() is missing . you can do a fetch before you do some manipulation on a model.
Also. I don't think you need db.TextProperty() for this, you can use db.StringProperty().
In Google App Engine, an entity has a Key. A key can be made from a path, in which case str(key) is an opaque hex string. Example:
from google.appengine.ext import db
foo = db.Key.from_path(u'foo', u'bar', _app=u'baz')
print foo
gives
agNiYXpyDAsSA2ZvbyIDYmFyDA
if you set up the right paths to run the code.
So, how can one take the hex string and get the path back? I thought the answer would be in Key or entity group docs, but I can't see it.
from google.appengine.ext import db
k = db.Key('agNiYXpyDAsSA2ZvbyIDYmFyDA')
_app = k.app()
path = []
while k is not None:
path.append(k.id_or_name())
path.append(k.kind())
k = k.parent()
path.reverse()
print 'app=%r, path=%r' % (_app, path)
when run in a Development Console, this outputs:
app=u'baz', path=[u'foo', u'bar']
as requested. A shorter alternative is to use the (unfortunately, I believe, undocumented) to_path method of Key instances:
k = db.Key('agNiYXpyDAsSA2ZvbyIDYmFyDA')
_app = k.app()
path = k.to_path()
print 'app=%r, path=%r' % (_app, path)
with the same results. But the first, longer version relies only on documented methods.
Once you have the Key object (which can be created by passing that opaque identifier to the constructor), use Key.to_path() to get the path of a Key as a list. For example:
from google.appengine.ext import db
opaque_id = 'agNiYXpyDAsSA2ZvbyIDYmFyDA'
path = db.Key(opaque_id).to_path()