I am trying to add memcache to my webapp deployed on GAE, and to do this I am using memcache.Client() to prevent damage from any racing conditions:
from google.appengine.api import memcache
client = memcache.Client()
class BlogFront(BlogHandler):
def get(self):
global client
val = client.gets(FRONT_PAGE_KEY)
posts = list()
if val is not None:
posts = list(val)
else:
posts = db.GqlQuery("select * from Post order by created desc limit 10")
client.cas(FRONT_PAGE_KEY, list(posts))
self.render('front.html', posts = posts)
To test the problem I have a front page for a blog that displays the 10 most recent entries. If there is nothing in the cache, I hit the DB with a request, otherwise I just present the cached results to the user.
The problem is that no matter what I do, I always get val == None, thus meaning that I always hit the database with a useless request.
I have sifted through the documentation:
https://developers.google.com/appengine/docs/python/memcache/
https://developers.google.com/appengine/docs/python/memcache/clientclass
http://neopythonic.blogspot.pt/2011/08/compare-and-set-in-memcache.html
And it appears that I am doing everything correctly. What am I missing?
(PS: I am a python newb, if this is a retarded error, please bear with me xD )
from google.appengine.api import memcache
class BlogFront(BlogHandler):
def get(self):
client = memcache.Client()
client.gets(FRONT_PAGE_KEY)
client.cas(FRONT_PAGE_KEY, 'my content')
For a reason I cannot yet possible understand, the solution lies in having a gets right before having a cas call ...
I think I will stick with the memcache non-thread-safe version of the code for now ...
I suspect that the client.cas call is failing because there is no object. Perhaps client.cas only works to update and existing object (not to set a new object if there is none currently)? You might try client.add() (which will fail if an object already exists with the specified key, which I think is what you want?) instead of client.cas()
Related
I have the following simple pytest case, which tests a delete operation:
def test_delete_club(client, systemAdmin, operationalCountry1, club1):
rv = loginTo(client, '/admin/clubs/1', '+00000000000','testpassword')
decodedRv = rv.data.decode('utf-8')
assert '<td>testClub1</td>' in decodedRv
rv = client.get('/admin/delete_club/1', follow_redirects = True)
decodedRv = rv.data.decode('utf-8')
assert '<td>testClub1</td>' not in decodedRv
#ensure club1 does not exist in the database either
c = Club.query.get(1)
assert c is None
# make sure club roles are deleted along with the club
clubRoles = Role.query.filter(Role.club_id == club1.id).all()
assert len(clubRoles)
Basically, it hits the delete URL (/admin/delete_club/1) and after following redirects, it asserts that there is neither such a club nor any roles associated with that club in the database.
I am using TDD (test driven development). So I wrote the above test case before the relevant routing code. My route method looks like this:
#flaskApp.route('/admin/delete_club/<int:id>', methods=['GET'])
#login_required
def delete_club(id):
'''Deletes the club identified by the id'''
if not isCurrentUserSysAdmin():
#TODO better error handling
return 'you can not touch this'
clubToDelete = Club.query.get_or_404(id)
logging.debug('About to delete club: {}'.format(clubToDelete))
opCountryId = clubToDelete.operationalcountry_id
try:
db.session.delete(clubToDelete)
logging.debug('Deleted club: {}'.format(clubToDelete))
except:
flash('Error deleting club')
logging.error('Error deleting club. Club Details: {}'.format(clubToDelete))
return redirect(url_for('clubs', countryid = opCountryId))
Well, so far so good. Test case passed without a glitch. I was happy. I wanted to give it a go on the real web page. Then I noticed that, although the delete operation had succeeded, on the redirected page, the club I was trying to delete would still be present.
Then I found the BUG : I simply forgot to add db.session.commit() after deleting the club instance entity. That change fixed the web page.
However, I am still puzzled why my test case does not complain about it and how come it does not fail at all? Again, test case works without the commit statement.
Any ideas from more seasoned Flask/Python developers?
In short, db.session.delete register transaction into memory to execute. But without db.commit that wouldn't execute a query on the database. This method will change only the local representation of databaes. What I recommend you is to use context manager with session.begin().
I use this User class provided by Google App Engine: from webapp2_extras.appengine.auth.models import User
I want to make some change to an instance of User via Data Store in the Google cloud console: change the name attribute of a User to something else.
As far as I know the User object is cached in some way. If I make the change directly in data store, how can I locate it in the memcache and delete it? My goal is to force the cached User object to be refreshed.
The memcache key used can be found in google.appengine.ext.context.py: Context.get():
if use_memcache:
mkey = self._memcache_prefix + key.urlsafe()
The prefix (in my version at least) is 'NDB9:'. So to get the memcache key you can use something similar to:
from google.appengine.ext import ndb
key = ndb.Key('User', 5229916580741120)
mkey = 'NDB9:' + key.urlsafe()
print mkey
This results in something like: NDB9:ahpzfmR5bmFtaWMtdHJhdmVsbGVyLTItdGVzdHIRCxIEVXNlchiAgIDAzZKlCQw
To delete:
from google.appengine.api import memcache
value = memcache.get(mkey)
result = memcache.delete(mkey)
print "Memcache delete result={}".format(result)
if result == 2:
print "{} memcache value has been deleted".format(key)
elif result == 1:
print "{} memcache value not found".format(key)
else:
print "Failed to delete {} memcache value".format(key)
Caveat: without the memcache.get() call before memcache.delete() the delete result was always '1' (not found) using the interactive console. Not sure why that's the case.
i have the following model in google app engine:
class CustomUsers(db.Model):
cuid = db.StringProperty()
name = db.StringProperty()
email = db.StringProperty()
bday = db.StringProperty()
now given a cuid i just want to check if he is present in the high replication data store.
in my views i write the following, but eventually get an 500 error:
usr_prev = CustomUsers.all().filter('cuid =', request.POST['id'])
if(not usr_prev):
logging.info("this user does not exist")
else:
logging.info("user exists")
but this gives me an error. How shall i do it?
(Do not consider any importing issues.)
"500 errors" don't exist in isolation. They actually report useful information, either on the debug page or in the logs. I'd guess that your logs show that request.POST has no element id, but that is just a guess.
I noticed that you have assigned a query object to usr_prev. You need to execute the query. I think Query.count(1) function should do it.
However, this may not be related to the 500 error.
you can go like this to see:
query = Users.all()
cnt = query.filter('id =', user_id).count()
logging.info(cnt)
return int(cnt)
once this is done, seems you are using django with app engine, as per the error you are not returning string response. you might want to return
return HttpResponse("1") instead of return HttpResponse(1)
i have this issue with app engine's datastore. In interactive console, I always get no entities when i ask if a url already exists in my db. When I execute the following code....
from google.appengine.ext import db
class shortURL(db.Model):
url = db.TextProperty()
date = db.DateTimeProperty(auto_now_add=True)
q = shortURL.all()
#q.filter("url =", "httphello")
print q.count()
for item in q:
print item.url
i get the this response, which is fine
6
httphello
www.boston.com
http://www.boston.com
httphello
www.boston.com
http://www.boston.com
But when I uncomment the line "q.filter("url =", "httphello")", i get no entities at all (a response of 0). I know its something ultra simple, but I just can't see it! help.
TextProperty values are not indexed, and cannot be used in filters or sort orders.
You might want to try with StringProperty if you don't need more than 500 characters.
I think .fetch() is missing . you can do a fetch before you do some manipulation on a model.
Also. I don't think you need db.TextProperty() for this, you can use db.StringProperty().
I'm brand new at Python and I'm trying to write an extension to an app that imports GA information and parses it into MySQL. There is a shamfully sparse amount of infomation on the topic. The Google Docs only seem to have examples in JS and Java...
...I have gotten to the point where my user can authenticate into GA using SubAuth. That code is here:
import gdata.service
import gdata.analytics
from django import http
from django import shortcuts
from django.shortcuts import render_to_response
def authorize(request):
next = 'http://localhost:8000/authconfirm'
scope = 'https://www.google.com/analytics/feeds'
secure = False # set secure=True to request secure AuthSub tokens
session = False
auth_sub_url = gdata.service.GenerateAuthSubRequestUrl(next, scope, secure=secure, session=session)
return http.HttpResponseRedirect(auth_sub_url)
So, step next is getting at the data. I have found this library: (beware, UI is offensive) http://gdata-python-client.googlecode.com/svn/trunk/pydocs/gdata.analytics.html
However, I have found it difficult to navigate. It seems like I should be gdata.analytics.AnalyticsDataEntry.getDataEntry(), but I'm not sure what it is asking me to pass it.
I would love a push in the right direction. I feel I've exhausted google looking for a working example.
Thank you!!
EDIT: I have gotten farther, but my problem still isn't solved. The below method returns data (I believe).... the error I get is: "'str' object has no attribute '_BecomeChildElement'" I believe I am returning a feed? However, I don't know how to drill into it. Is there a way for me to inspect this object?
def auth_confirm(request):
gdata_service = gdata.service.GDataService('iSample_acctSample_v1.0')
feedUri='https://www.google.com/analytics/feeds/accounts/default?max-results=50'
# request feed
feed = gdata.analytics.AnalyticsDataFeed(feedUri)
print str(feed)
Maybe this post can help out. Seems like there are not Analytics specific bindings yet, so you are working with the generic gdata.
I've been using GA for a little over a year now and since about April 2009, i have used python bindings supplied in a package called python-googleanalytics by Clint Ecker et al. So far, it works quite well.
Here's where to get it: http://github.com/clintecker/python-googleanalytics.
Install it the usual way.
To use it: First, so that you don't have to manually pass in your login credentials each time you access the API, put them in a config file like so:
[Credentials]
google_account_email = youraccount#gmail.com
google_account_password = yourpassword
Name this file '.pythongoogleanalytics' and put it in your home directory.
And from an interactive prompt type:
from googleanalytics import Connection
import datetime
connection = Connection() # pass in id & pw as strings **if** not in config file
account = connection.get_account(<*your GA profile ID goes here*>)
start_date = datetime.date(2009, 12, 01)
end_data = datetime.date(2009, 12, 13)
# account object does the work, specify what data you want w/
# 'metrics' & 'dimensions'; see 'USAGE.md' file for examples
account.get_data(start_date=start_date, end_date=end_date, metrics=['visits'])
The 'get_account' method will return a python list (in above instance, bound to the variable 'account'), which contains your data.
You need 3 files within the app. client_secrets.json, analytics.dat and google_auth.py.
Create a module Query.py within the app:
class Query(object):
def __init__(self, startdate, enddate, filter, metrics):
self.startdate = startdate.strftime('%Y-%m-%d')
self.enddate = enddate.strftime('%Y-%m-%d')
self.filter = "ga:medium=" + filter
self.metrics = metrics
Example models.py: #has the following function
import google_auth
service = googleauth.initialize_service()
def total_visit(self):
object = AnalyticsData.objects.get(utm_source=self.utm_source)
trial = Query(object.date.startdate, object.date.enddate, object.utm_source, ga:sessions")
result = service.data().ga().get(ids = 'ga:<your-profile-id>', start_date = trial.startdate, end_date = trial.enddate, filters= trial.filter, metrics = trial.metrics).execute()
total_visit = result.get('rows')
<yr save command, ColumnName.object.create(data=total_visit) goes here>