Develop Reference

Behavioral difference between pytest code and real flask session for a route

I have the following simple pytest case, which tests a delete operation:
def test_delete_club(client, systemAdmin, operationalCountry1, club1):
rv = loginTo(client, '/admin/clubs/1', '+00000000000','testpassword')
decodedRv = rv.data.decode('utf-8')
assert '<td>testClub1</td>' in decodedRv
rv = client.get('/admin/delete_club/1', follow_redirects = True)
decodedRv = rv.data.decode('utf-8')
assert '<td>testClub1</td>' not in decodedRv
#ensure club1 does not exist in the database either
c = Club.query.get(1)
assert c is None
# make sure club roles are deleted along with the club
clubRoles = Role.query.filter(Role.club_id == club1.id).all()
assert len(clubRoles)
Basically, it hits the delete URL (/admin/delete_club/1) and after following redirects, it asserts that there is neither such a club nor any roles associated with that club in the database.
I am using TDD (test driven development). So I wrote the above test case before the relevant routing code. My route method looks like this:
#flaskApp.route('/admin/delete_club/<int:id>', methods=['GET'])
#login_required
def delete_club(id):
'''Deletes the club identified by the id'''
if not isCurrentUserSysAdmin():
#TODO better error handling
return 'you can not touch this'
clubToDelete = Club.query.get_or_404(id)
logging.debug('About to delete club: {}'.format(clubToDelete))
opCountryId = clubToDelete.operationalcountry_id
try:
db.session.delete(clubToDelete)
logging.debug('Deleted club: {}'.format(clubToDelete))
except:
flash('Error deleting club')
logging.error('Error deleting club. Club Details: {}'.format(clubToDelete))
return redirect(url_for('clubs', countryid = opCountryId))
Well, so far so good. Test case passed without a glitch. I was happy. I wanted to give it a go on the real web page. Then I noticed that, although the delete operation had succeeded, on the redirected page, the club I was trying to delete would still be present.
Then I found the BUG : I simply forgot to add db.session.commit() after deleting the club instance entity. That change fixed the web page.
However, I am still puzzled why my test case does not complain about it and how come it does not fail at all? Again, test case works without the commit statement.
Any ideas from more seasoned Flask/Python developers?

In short, db.session.delete register transaction into memory to execute. But without db.commit that wouldn't execute a query on the database. This method will change only the local representation of databaes. What I recommend you is to use context manager with session.begin().

Create activities for an user with Stream-Framework

I'm trying to setup stream-framework the one here not the newer getstream. I've setup the Redis server and the environment properly, the issue I'm facing is in creating the activities for a user.
I've been trying to create activities, following the documentation to add an activity but it gives me an error message as follows:
...
File "/Users/.../stream_framework/activity.py", line 110, in serialization_id
if self.object_id >= 10 ** 10 or self.verb.id >= 10 ** 3:
AttributeError: 'int' object has no attribute 'id'
Here is the code
from stream_framework.activity import Activity
from stream_framework.feeds.redis import RedisFeed
class PinFeed(RedisFeed):
key_format = 'feed:normal:%(user_id)s'
class UserPinFeed(PinFeed):
key_format = 'feed:user:%(user_id)s'
feed = UserPinFeed(13)
print(feed)
activity = Activity(
actor=13, # Thierry's user id
verb=1, # The id associated with the Pin verb
object=1, # The id of the newly created Pin object
)
feed.add(activity) # Error at this line
I think there is something missing in the documentation or maybe I'm doing something wrong. I'll be very grateful if anyone helps me get the stream framework working properly.

The documentation is inconsistent. The verb you pass to the activity should be (an instance of?*) a subclass of stream_framework.verbs.base.Verb. Check out this documentation page on custom verbs and the tests for this class.
The following should fix the error you posted:
from stream_framework.activity import Activity
from stream_framework.feeds.redis import RedisFeed
from stream_framework.verbs import register
from stream_framework.verbs.base import Verb
class PinFeed(RedisFeed):
key_format = 'feed:normal:%(user_id)s'
class UserPinFeed(PinFeed):
key_format = 'feed:user:%(user_id)s'
class Pin(Verb):
id = 5
infinitive = 'pin'
past_tense = 'pinned'
register(Pin)
feed = UserPinFeed(13)
activity = Activity(
actor=13,
verb=Pin,
object=1,
)
feed.add(activity)
I quickly looked over the code for Activity and it looks like passing ints for actor and object should work. However, it is possible that these parameters are also outdated in the documentation.
* The tests pass in classes as verb. However, the Verb base class has the methods serialize and __str__ that can only be meaningfully invoked if you have an object of this class. So I'm still unsure which is required here. It seems like in the current state, the framework never calls these methods, so classes still work, but I feel like the author originally intended to pass instances.

With the help of great answer by #He3lixxx, I was able to solve it partially. As the package is no more maintained, the package installs the latest Redis client for python which was creating too many issues so by installation redis-2.10.5 if using stream-framework-1.3.7, should fix the issue.
I would also like to add a complete guide to properly add activity to a user feed.
Key points:
If you are not using feed manager, then make sure to first insert the activity before you add it to the user with feed.insert_activity(activity) method.
In case of getting feeds with feed[:] throws an error something like below:
File "/Users/.../stream_framework/activity.py", line 44, in get_hydrated
activity = activities[int(self.serialization_id)]
KeyError: 16223026351730000000001005L
then you need to clear data for that user using the key format for it in my case the key is feed:user:13 for user 13, delete it with DEL feed:user:13, In case if that doesn't fix the issue then you can FLUSHALL which will delete everything from Redis.
Sample code:
from stream_framework.activity import Activity
from stream_framework.feeds.redis import RedisFeed
from stream_framework.verbs import register
from stream_framework.verbs.base import Verb
class PinFeed(RedisFeed):
key_format = 'feed:normal:%(user_id)s'
class UserPinFeed(PinFeed):
key_format = 'feed:user:%(user_id)s'
class Pin(Verb):
id = 5
infinitive = 'pin'
past_tense = 'pinned'
register(Pin)
feed = UserPinFeed(13)
print(feed[:])
activity = Activity(
actor=13,
verb=Pin,
object=1)
feed.insert_activity(activity)
activity_id = feed.add(activity)
print(activity_id)
print(feed[:])

Django ORM access optimization with Filter

I've implemented a method on my django app that will check if a user has a specific permission or it's part of a group that contains this specific permission.
def user_has_perm(user, *permissions_to_check):
permission_check = True
permissions_not_found = []
user_groups = user.groups.all().prefetch_related('permissions')
for permission in permissions_to_check:
content_type, permission_codename = permission.split('.')
if not user.has_perm(permission) and not user_groups.filter(
permissions__content_type__model__icontains=content_type,
permissions__codename__icontains=permission_codename).exists(): # Goes down from Groups to the single user permission
permission_check = False
permissions_not_found.append(permission)
return permission_check, permissions_not_found
Now, everythings works like a charm, but, Django-Debug-Toolbar it's complaining about the query, that it's duplicated many times as the groups to check.
For me it's a bottleneck, because some users will have 50 groups associated, and really i don't know how to optimize this query...
Any suggestions?
Thanks

As mentioned by Klaus, this is a database killer. If there lots of permissions (even without any apps of your own, there will be quite a few), you could potentially find yourself executing hundreds of queries in your user_has_perm function. You could reduce that with caching
from django.core.cache import cache.
def user_has_perm(user, *permissions_to_check):
hash = # create a hash from user.id and permissions_to_check.
# maybe something as simple as "".join(permissions_to_check)
perms = cache.get(hash)
if perms:
return perms
permission_check = True
permissions_not_found = []
user_groups = user.groups.all().prefetch_related('permissions')
for permission in permissions_to_check:
content_type, permission_codename = permission.split('.')
if not user.has_perm(permission) and not user_groups.filter(
permissions__content_type__model__icontains=content_type,
permissions__codename__icontains=permission_codename).exists(): # Goes down from Groups to the single user permission
permission_check = False
permissions_not_found.append(permission)
cache.set(hash, [permisson_check, permissions_not_found], 10**5)
return permission_check, permissions_not_found

Accessing portal content with custom workflow and no 'View' permissions

I have a problem where I need to be able to make custom content accessible for search and retrieval via portal_catalog by anonymous users but not viewable by them.
I used custom content types and a custom workflow, what I'm getting is most likely a permission issue. I defined a custom workflow through ZMI -> portal_workflow and then exported it into source code as an XML definition. I set the permissions for anonymous users as 'Access Content Information' but not 'View'. Note that 'active' in the code snippet is a workflow state that has that permission enabled -- sales_workflow
Brain lookup works for 'Manager' role, but when the role is switched to 'Anonymous', catalog returns an empty list.
import unittest2 as unittest
from . import INTEGRATION_TESTING
from AccessControl import getSecurityManager
from plone.app.testing import setRoles, logout
from plone.app.testing import TEST_USER_ID
from Products.CMFCore.utils import getToolByName
def drop_to_anonymous(self):
"""
Drop site roles to anonymous user only.
Note this is a class method and not a function
assign this method as a class member and then call it
"""
logout()
setRoles(self.portal, TEST_USER_ID, ['Anonymous'])
user = getSecurityManager().getUser()
roles = user.getRolesInContext(self.portal)
self.assertListEqual(['Anonymous'], roles)
class TestSalesRepWorkflow(unittest.TestCase):
layer = INTEGRATION_TESTING
drop_to_anonymous = drop_to_anonymous
def setUp(self):
self.portal = self.layer['portal']
self.wftool = getToolByName(self.portal, 'portal_workflow')
self.catalog = getToolByName(self.portal, 'portal_catalog')
def test_workflow_lookup_anon(self):
setRoles(self.portal, TEST_USER_ID, ['Manager'])
self.portal.invokeFactory(
'CustomProduct',
'prod1',
title="Product 1"
)
prod1 = self.portal['prod1']
self.wftool.doActionFor(prod1, action='activate')
review_state = self.wftool.getInfoFor(prod1, 'review_state')
prod1.reindexObject()
self.assertEqual('active', review_state)
lookup = self.catalog(portal_type='CustomProduct', Title='Product 1',
review_state='active')
#This test passes with managerial permissions
self.assertEqual(len(lookup), 1)
#Repeat the same test in 'Anonymous' role
self.drop_to_anonymous()
lookup1 = self.catalog(portal_type='CustomProduct', Title='Product 1',
review_state='active')
#When dropped to anonymous role, the test fails,
#lookup returns an empty list
self.assertEqual(len(lookup1), 1)
Is there a way to fix this without drastically reworking permissions?
Using unrestrictedSearchResults seems to fix the search, but whenever I attempt to run 'getObject' on a brain, the following error gets raised:
Unauthorized: You are not allowed to access 'XXX' in this context

Your active state needs to give the View permission to Anonymous. Currently it is restricted to these roles:
<state state_id="active" title="">
<!-- other information elided here -->
<permission-map name="View" acquired="False">
<permission-role>Manager</permission-role>
<permission-role>Owner</permission-role>
<permission-role>Reviewer</permission-role>
<permission-role>SalesRep</permission-role>
<permission-role>Site Administrator</permission-role>
</permission-map>
Without the View permission anonymous cannot see your objects even when given the active state, nor can they be found in the catalog by that user.
You can override this behaviour of the catalog by using the .unrestrictedSearchResults() method of the catalog:
lookup1 = self.catalog.unrestrictedSearchResults(
portal_type='SalesProduct', Title='Product 1', review_state='active')
This method cannot be used from restricted code.
The returned brain objects are perfectly accessible by anonymous users, but you cannot use the getObject() method on them because that'll use the current user's permissions to traverse to it. If you need to get the actual object from the brain, there is again a special, private method to get to the actual object without those restrictions, called ._unrestrictedGetObject():
obj = brain._unrestrictedGetObject()
This method is, once again, only available to unrestricted code.

Google Analytics and Python

I'm brand new at Python and I'm trying to write an extension to an app that imports GA information and parses it into MySQL. There is a shamfully sparse amount of infomation on the topic. The Google Docs only seem to have examples in JS and Java...
...I have gotten to the point where my user can authenticate into GA using SubAuth. That code is here:
import gdata.service
import gdata.analytics
from django import http
from django import shortcuts
from django.shortcuts import render_to_response
def authorize(request):
next = 'http://localhost:8000/authconfirm'
scope = 'https://www.google.com/analytics/feeds'
secure = False # set secure=True to request secure AuthSub tokens
session = False
auth_sub_url = gdata.service.GenerateAuthSubRequestUrl(next, scope, secure=secure, session=session)
return http.HttpResponseRedirect(auth_sub_url)
So, step next is getting at the data. I have found this library: (beware, UI is offensive) http://gdata-python-client.googlecode.com/svn/trunk/pydocs/gdata.analytics.html
However, I have found it difficult to navigate. It seems like I should be gdata.analytics.AnalyticsDataEntry.getDataEntry(), but I'm not sure what it is asking me to pass it.
I would love a push in the right direction. I feel I've exhausted google looking for a working example.
Thank you!!
EDIT: I have gotten farther, but my problem still isn't solved. The below method returns data (I believe).... the error I get is: "'str' object has no attribute '_BecomeChildElement'" I believe I am returning a feed? However, I don't know how to drill into it. Is there a way for me to inspect this object?
def auth_confirm(request):
gdata_service = gdata.service.GDataService('iSample_acctSample_v1.0')
feedUri='https://www.google.com/analytics/feeds/accounts/default?max-results=50'
# request feed
feed = gdata.analytics.AnalyticsDataFeed(feedUri)
print str(feed)

Maybe this post can help out. Seems like there are not Analytics specific bindings yet, so you are working with the generic gdata.

I've been using GA for a little over a year now and since about April 2009, i have used python bindings supplied in a package called python-googleanalytics by Clint Ecker et al. So far, it works quite well.
Here's where to get it: http://github.com/clintecker/python-googleanalytics.
Install it the usual way.
To use it: First, so that you don't have to manually pass in your login credentials each time you access the API, put them in a config file like so:
[Credentials]
google_account_email = youraccount#gmail.com
google_account_password = yourpassword
Name this file '.pythongoogleanalytics' and put it in your home directory.
And from an interactive prompt type:
from googleanalytics import Connection
import datetime
connection = Connection() # pass in id & pw as strings **if** not in config file
account = connection.get_account(<*your GA profile ID goes here*>)
start_date = datetime.date(2009, 12, 01)
end_data = datetime.date(2009, 12, 13)
# account object does the work, specify what data you want w/
# 'metrics' & 'dimensions'; see 'USAGE.md' file for examples
account.get_data(start_date=start_date, end_date=end_date, metrics=['visits'])
The 'get_account' method will return a python list (in above instance, bound to the variable 'account'), which contains your data.

You need 3 files within the app. client_secrets.json, analytics.dat and google_auth.py.
Create a module Query.py within the app:
class Query(object):
def __init__(self, startdate, enddate, filter, metrics):
self.startdate = startdate.strftime('%Y-%m-%d')
self.enddate = enddate.strftime('%Y-%m-%d')
self.filter = "ga:medium=" + filter
self.metrics = metrics
Example models.py: #has the following function
import google_auth
service = googleauth.initialize_service()
def total_visit(self):
object = AnalyticsData.objects.get(utm_source=self.utm_source)
trial = Query(object.date.startdate, object.date.enddate, object.utm_source, ga:sessions")
result = service.data().ga().get(ids = 'ga:<your-profile-id>', start_date = trial.startdate, end_date = trial.enddate, filters= trial.filter, metrics = trial.metrics).execute()
total_visit = result.get('rows')
<yr save command, ColumnName.object.create(data=total_visit) goes here>