TL;DR
I am looking for a way to clear the cache after a request, or completely disable it when running tests. Django REST Framework seems to cache the results and I need a way around this.
Long version and code
Well this turned out to behave very weird as I kept testing it. In the end, I got it to work, but I really don't like my workaround and in the name of knowledge, I have to find out why this happens and how to solve this problem properly.
So, I have an APITestCase class declared like this:
class UserTests(APITestCase):
Inside this class, I have a test function for my user-list view, as I have a custom queryset depending on the permissions. To clear things up:
a superuser can get the whole users list (4 instances returned),
staff members cannot see superusers (3 instances returned),
normal users can only get 1 result, their own user (1 instance returned)
The test function version that works:
def test_user_querysets(self):
url = reverse('user-list')
# Creating a user
user = User(username='user', password=self.password)
user.set_password(self.password)
user.save()
# Creating a second user
user2 = User(username='user2', password=self.password)
user2.set_password(self.password)
user2.save()
# Creating a staff user
staff_user = User(username='staff_user', password=self.password, is_staff=True)
staff_user.set_password(self.password)
staff_user.save()
# Creating a superuser
superuser = User(username='superuser', password=self.password, is_staff=True, is_superuser=True)
superuser.set_password(self.password)
superuser.save()
# SUPERUSER
self.client.logout()
self.client.login(username=superuser.username, password=self.password)
response = self.client.get(url)
# HTTP_200_OK
self.assertEqual(response.status_code, status.HTTP_200_OK)
# All users contained in list
self.assertEqual(response.data['extras']['total_results'], 4)
# STAFF USER
self.client.logout()
self.client.login(username=staff_user.username, password=self.password)
response = self.client.get(url)
# HTTP_200_OK
self.assertEqual(response.status_code, status.HTTP_200_OK)
# Superuser cannot be contained in list
self.assertEqual(response.data['extras']['total_results'], 3)
# REGULAR USER
self.client.logout()
self.client.login(username=user2.username, password=self.password)
response = self.client.get(url)
# HTTP_200_OK
self.assertEqual(response.status_code, status.HTTP_200_OK)
# Only 1 user can be returned
self.assertEqual(response.data['extras']['total_results'], 1)
# User returned is current user
self.assertEqual(response.data['users'][0]['username'], user2.username)
As you see, I am testing user permissions in this order: superuser, staff, normal user. And this works, so...
Funny thing:
If I change the order of the tests, and start with normal user, staff, superuser, the tests fail. The response from the first request gets cached, and then I get the same response when I log in as staff user, so the number of results is again 1.
The version that doesn't work:
it's exactly the same as before, only the tests are made in reverse order
def test_user_querysets(self):
url = reverse('user-list')
# Creating a user
user = User(username='user', password=self.password)
user.set_password(self.password)
user.save()
# Creating a second user
user2 = User(username='user2', password=self.password)
user2.set_password(self.password)
user2.save()
# Creating a staff user
staff_user = User(username='staff_user', password=self.password, is_staff=True)
staff_user.set_password(self.password)
staff_user.save()
# Creating a superuser
superuser = User(username='superuser', password=self.password, is_staff=True, is_superuser=True)
superuser.set_password(self.password)
superuser.save()
# REGULAR USER
self.client.logout()
self.client.login(username=user2.username, password=self.password)
response = self.client.get(url)
# HTTP_200_OK
self.assertEqual(response.status_code, status.HTTP_200_OK)
# Only 1 user can be returned
self.assertEqual(response.data['extras']['total_results'], 1)
# User returned is current user
self.assertEqual(response.data['users'][0]['username'], user2.username)
# STAFF USER
self.client.logout()
self.client.login(username=staff_user.username, password=self.password)
response = self.client.get(url)
# HTTP_200_OK
self.assertEqual(response.status_code, status.HTTP_200_OK)
# Superuser cannot be contained in list
self.assertEqual(response.data['extras']['total_results'], 3)
# SUPERUSER
self.client.logout()
self.client.login(username=superuser.username, password=self.password)
response = self.client.get(url)
# HTTP_200_OK
self.assertEqual(response.status_code, status.HTTP_200_OK)
# All users contained in list
self.assertEqual(response.data['extras']['total_results'], 4)
I am working in python 2.7 with the following package versions:
Django==1.8.6
djangorestframework==3.3.1
Markdown==2.6.4
MySQL-python==1.2.5
wheel==0.24.0
UPDATE
I am using the default django cache, meaning I haven't put anything about cache in the django settings.
As suggested, I tried disabling the default Django cache:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
}
}
REST_FRAMEWORK = {
'DEFAULT_AUTHENTICATION_CLASSES': (
'rest_framework.authentication.SessionAuthentication',
)
}
The problem stands on.
Even though I don't think the problem is located here, this is my UserViewSet:
api.py (the important part)
class UserViewSet(
mixins.RetrieveModelMixin,
mixins.UpdateModelMixin,
mixins.ListModelMixin,
viewsets.GenericViewSet
):
queryset = User.objects.all()
serializer_class = UserExpenseSerializer
permission_classes = (IsAuthenticated, )
allowed_methods = ('GET', 'PATCH', 'OPTIONS', 'HEAD')
def get_serializer_class(self):
if self.action == 'retrieve':
return UserExpenseSerializer
return UserSerializer
def get_queryset(self):
if(self.action == 'list'):
return User.objects.all()
if self.request.user.is_superuser:
return User.objects.all()
if self.request.user.is_staff:
return User.objects.exclude(is_superuser=True)
return User.objects.filter(pk = self.request.user.id)
def list(self, request):
filter_obj = UsersFilter(self.request)
users = filter_obj.do_query()
extras = filter_obj.get_extras()
serializer = UserSerializer(users, context={'request' : request}, many=True)
return Response({'users' : serializer.data, 'extras' : extras}, views.status.HTTP_200_OK)
filters.py
class UsersFilter:
offset = 0
limit = 50
count = 0
total_pages = 0
filter_params = {}
def __init__(self, request):
if not request.user.is_superuser:
self.filter_params['is_superuser'] = False
if (not request.user.is_superuser and not request.user.is_staff):
self.filter_params['pk'] = request.user.id
# Read query params
rpp = request.query_params.get('rpp') or 50
page = request.query_params.get('page') or 1
search_string = request.query_params.get('search')
# Validate
self.rpp = int(rpp) or 50
self.page = int(page) or 1
# Set filter
set_if_not_none(self.filter_params, 'username__contains', search_string)
# Count total results
self.count = User.objects.filter(**self.filter_params).count()
self.total_pages = int(self.count / self.rpp) + 1
# Set limits
self.offset = (self.page - 1) * self.rpp
self.limit = self.page * self.rpp
def get_filter_params(self):
return self.filter_params
def get_offset(self):
return self.offset
def get_limit(self):
return self.limit
def do_query(self):
users = User.objects.filter(**self.filter_params)[self.offset:self.limit]
return users
def get_query_info(self):
query_info = {
'total_results' : self.count,
'results_per_page' : self.rpp,
'current_page' : self.page,
'total_pages' : self.total_pages
}
return query_info
UPDATE 2
As Linovia pointed out, the problem was not cache or any other DRF problem, but the filter. Here's the fixed filter class:
class UsersFilter:
def __init__(self, request):
self.filter_params = {}
self.offset = 0
self.limit = 50
self.count = 0
self.total_pages = 0
self.extras = {}
if not request.user.is_superuser:
# and so long...
Actually you create a new user which should make 2 users and you assert the length against 3. Not going to work even without caching.
Edit:
So you actually have you issue because the use of mutables objects at the class level.
Here's the evil code:
class UsersFilter:
filter_params = {}
def __init__(self, request):
if not request.user.is_superuser:
self.filter_params['is_superuser'] = False
Which should actually be:
class UsersFilter:
def __init__(self, request):
filter_params = {}
if not request.user.is_superuser:
self.filter_params['is_superuser'] = False
Otherwise UsersFilter.filter_params will be kept from one request to another and never resets. See http://www.toptal.com/python/python-class-attributes-an-overly-thorough-guide for more details about this.
You can disable caching in debug mode by adding this to your settings.py
if DEBUG:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
}
}
https://docs.djangoproject.com/en/dev/topics/cache/?from=olddocs/#dummy-caching-for-development
You can then disable caching by toggling DEBUG in settings.py or by having separate develop/test and /deploy settings.py files.
If you want to avoid separate files or toggling, then you can set DEBUG to true for certain tests with the override_settings decorator:
from django.test.utils import override_settings
from django.test import TestCase
from django.conf import settings
class MyTest(TestCase):
#override_settings(DEBUG=True)
def test_debug(self):
self.assertTrue(settings.DEBUG)
Related
I am trying for some time to test a view of mine that I have protected from access via LoginRequiredMixin and UserPassesTestMixin.
Unfortunately I do not manage to write the appropriate test.
here is the view. The special thing is, that the user must not only be logged in, but he must also belong to the group "Administrator" or "Supervisor".
With this combination I have not yet managed to write a test.
Please who can help me. Here is my View:
class FeatureListView(LoginRequiredMixin, UserPassesTestMixin, ListView):
model = FeatureFilm
template_name = "project/feature-list.html"
def test_func(self):
if self.request.user.groups.filter(name="Administrator").exists():
return True
elif self.request.user.groups.filter(name="Supervisor").exists():
return True
elif self.request.user.groups.filter(name="Operator").exists():
return True
else:
return False
def handle_no_permission(self):
return redirect("access-denied")
and here is a snippet of the url.py:
urlpatterns = [
path("feature/list/", FeatureListView.as_view(), name="feature-list"),
path(
"feature/<int:pk>/date",
FeatureDetailViewDate.as_view(),
name="feature-detail-date",
),
how would you test this FeatureListView and the template belongs to
thank you very much!
First of all it would be better if you would give those groups a permission. Because django offers a lot nice functionalities to check for permissions rather than checking for groups (see here). You would give Administrator, Supervisor and Operator all the same permission and then instead of checking if the user is in one of these groups, you would just check whether the user has that permission. But as that was not the question there you go:
from django.test import TestCase
from django.test import Client
class TestFeatureListView(TestCase):
def _create_administrator(self):
u = Users.objects.create( <input for administrator> )
u.groups.add(administrator)
return u
def _create_supervisor(self):
u = Users.objects.create( <input for supervisor> )
u.groups.add(supervisor)
return u
def _create_operator(self):
u = Users.objects.create( <input for operator> )
u.groups.add(operator)
return u
def _create_normal_user(self):
return Users.objects.create( <input for normal_user> )
def test_users_with_access_rights(self):
users = [
self._create_administrator(),
self._create_supervisor(),
self._create_operator(),
]
for u in users:
c = Client() # initialize every iteration a new client
c.login(username=<username>, password=<password>)
response = c.get(reverse("appwithfeaturelist:feature-list"))
self.assertTemplateUsed(response, "project/feature-list.html")
def test_users_withOUT_access_rights(self):
u = self._create_normal_user()
self.client.login(username=<username>, password=<password>)
response = self.client.get(reverse("appwithfeaturelist:feature-list"), follow=True)
self.assertTemplateNotUsed(response, "project/feature-list.html")
self.assertRedirects(response, "/access-denied/")
def test_without_any_user(self):
response = self.client.get(reverse("appwithfeaturelist:feature-list"), follow=True)
self.assertTemplateNotUsed(response, "project/feature-list.html")
self.assertRedirects(response, "/access-denied/") # this is probably wrong and should test whether the redirect goes to something like a login page
I am creating a REST-API with django and the django rest framework, therefore I am using ReadOnlyModelViewSet. In my unit test for testing create / POST method, I am expecting HTTP_405_METHOD_NOT_ALLOWED but actually it is HTTP_403_FORBIDDEN. For the update / PUT it is HTTP_405 as expected. Is this behavior the default behavior or do I have a dumb bug?
And yes the user is authenticated.
The Viewset:
class StudentSolutionReadOnlyModelViewSet(viewsets.ReadOnlyModelViewSet):
queryset = StudentSolution.objects.all()
serializer_class = StudentSolutionSerializer
def get_permissions(self):
if self.request.method == 'POST':
return (permissions.IsAuthenticated(), )
return (permissions.IsAuthenticated(), IsStudentSolutionOwnerOrAdmin(),)
def get_queryset(self):
if self.request.user.is_staff:
return StudentSolution.objects.all(
course_assignment=self.kwargs['course_assignment_pk']
).order_by('student__last_name', 'course_assignment__due_date')
return StudentSolution.objects.filter(
student=self.request.user,
course_assignment=self.kwargs['course_assignment_pk']
).order_by('course_assignment__due_date')
Edit 1:
class StudentSolutionReadOnlyModelViewSetTests(TestCase):
def setup(self):
#[..]
def test_create_user_not_allowed(self):
data = {
'course_assignment': self.course_assignment.id,
'student': self.user.id
}
url = self.generate_url(self.course_assignment.id)
self.csrf_client.credentials(HTTP_AUTHORIZATION='JWT ' + self.user_token)
resp = self.csrf_client.post(
self.url,
data=json.dumps(data),
content_type='application/json'
)
self.assertEqual(resp.status_code, status.HTTP_405_METHOD_NOT_ALLOWED)
I am using the same authentication and csrf stuff for all methods, as well as generate_url method.
Block particular user for some specific time to using Django REST Throttling.
I have seen Django REST Throttling.
I don't want to use third-party packages.
Thanks in advance
I have found the solution after customized Django REST Throttling,
Its Blocking particular user after 3 login attempts (Block user_id that presents in my application).
Block IP address after 6 login attempts for anonymous user.
prevent.py:-
#!/usr/bin/python
from collections import Counter
from rest_framework.throttling import SimpleRateThrottle
from django.contrib.auth.models import User
class UserLoginRateThrottle(SimpleRateThrottle):
scope = 'loginAttempts'
def get_cache_key(self, request, view):
user = User.objects.filter(username=request.data.get('username'))
ident = user[0].pk if user else self.get_ident(request)
return self.cache_format % {
'scope': self.scope,
'ident': ident
}
def allow_request(self, request, view):
"""
Implement the check to see if the request should be throttled.
On success calls `throttle_success`.
On failure calls `throttle_failure`.
"""
if self.rate is None:
return True
self.key = self.get_cache_key(request, view)
if self.key is None:
return True
self.history = self.cache.get(self.key, [])
self.now = self.timer()
while self.history and self.history[-1] <= self.now - self.duration:
self.history.pop()
if len(self.history) >= self.num_requests:
return self.throttle_failure()
if len(self.history) >= 3:
data = Counter(self.history)
for key, value in data.items():
if value == 2:
return self.throttle_failure()
return self.throttle_success(request)
def throttle_success(self, request):
"""
Inserts the current request's timestamp along with the key
into the cache.
"""
user = User.objects.filter(username=request.data.get('username'))
if user:
self.history.insert(0, user[0].id)
self.history.insert(0, self.now)
self.cache.set(self.key, self.history, self.duration)
return True
view.py:-
from .prevent import UserLoginRateThrottle
....
....
....
class ObtainAuthToken(auth_views.ObtainAuthToken):
throttle_classes = (UserLoginRateThrottle,)/use this method here your login view
def post(self, request, *args, **kwargs):
....
....
Add some parameters in setting file
settings.py:-
# Django-rest-framework
REST_FRAMEWORK = {
...
...
...
'DEFAULT_THROTTLE_CLASSES': (
'rest_framework.throttling.UserRateThrottle',
),
'DEFAULT_THROTTLE_RATES': {
'loginAttempts': '6/hr',
'user': '1000/min',
}
}
Docs says only mapping of GET
user_list = UserViewSet.as_view({'get': 'list'})
user_detail = UserViewSet.as_view({'get': 'retrieve'})
tests.py:
def test_admin_can_create_role(userprofiles, aoo_admin, bug_manager, note_admin):
aoo = User.objects.get(username='aoo')
factory = APIRequestFactory()
view = RoleViewSet.as_view()
url = reverse('api:role-list')
data = {
'name': 'FirstAdmin',
'type': Role.RoleType.admin,
'company': 1,
}
request = factory.post(url, data=data, format='json')
force_authenticate(request, user=aoo)
response = view(request)
assert 201 == response.data
viewsets.py
class RoleViewSetPermission(permissions.BasePermission):
message = 'Only Manager or Admin are allowed'
def has_permission(self, request, view):
user = request.user
return user.has_perm('roles.add_role') \
and user.has_perm('roles.change_role') \
and user.has_perm('roles.delete_role')
class RoleViewSet(viewsets.ModelViewSet):
permission_classes = (RoleViewSetPermission,)
queryset = Role.objects.all()
serializer_class = RoleSerializer
filter_backends = (filters.DjangoFilterBackend, SearchFilter)
filter_class = RoleFilter
search_fields = ('name', 'description', 'user__username', 'company__name', 'company__name_th')
def filter_queryset(self, queryset):
try:
company = self.request.user.user_profile.companyappid.company
except AttributeError:
logger.error(f'{self.request.user} has AttributeError')
return Role.objects.none()
else:
logger.info(f'{self.request.user} is {company} member')
return queryset.filter(company=company)
Trackback:
cls = <class 'poinkbackend.apps.roles.api.viewsets.RoleViewSet'>, actions = None, initkwargs = {}
#classonlymethod
def as_view(cls, actions=None, **initkwargs):
"""
Because of the way class based views create a closure around the
instantiated view, we need to totally reimplement `.as_view`,
and slightly modify the view function that is created and returned.
"""
# The suffix initkwarg is reserved for identifying the viewset type
# eg. 'List' or 'Instance'.
cls.suffix = None
# actions must not be empty
if not actions:
> raise TypeError("The `actions` argument must be provided when "
"calling `.as_view()` on a ViewSet. For example "
"`.as_view({'get': 'list'})`")
E TypeError: The `actions` argument must be provided when calling `.as_view()` on a ViewSet. For example `.as_view({'get': 'list'})`
../../.pyenv/versions/3.6.3/envs/poink/lib/python3.6/site-packages/rest_framework/viewsets.py:55: TypeError
Question:
How to do force_authenticate and request.post to the viewsets?
I have no problem with get. It has an answer already in the SO
References:
http://www.django-rest-framework.org/api-guide/viewsets/
I have to use APIClient not APIRequestFactory.
I though it has only one way to do testing.
Here is my example.
def test_admin_can_create_role(userprofiles, aoo_admin, bug_manager, note_admin):
aoo = User.objects.get(username='aoo')
client = APIClient()
client.force_authenticate(user=aoo)
url = reverse('api:role-list')
singh = Company.objects.get(name='Singh')
data = {
'name': 'HairCut',
'type': Role.RoleType.admin,
'company': singh.id, # Must be his companyid. Reason is in the RoleSerializer docstring
}
response = client.post(url, data, format='json')
assert 201 == response.status_code
I have a profile page for my users where they should be able to update their information. For now they can update their names but I also want phonenumbers, addresses, etc.
The code for updating the name of my user is
class AccountPage(BaseRequestHandler):
def get(self):
self.render('accountpage.html', {'request': self.request, 'user': self.current_user,'loggedin': self.logged_in, 'session': self.auth.get_user_by_session(),})
def post(self):
user = self.current_user
user.name = self.request.POST['name']
user.put()
self.auth.set_session(
self.auth.store.user_to_dict(user))
self.render('accountpage.html', {'request': self.request, 'loggedin': self.logged_in,'user': self.current_user})
But how can I use extra variables such as phonenumbers, address variable etc? The webapp2 User model is an expando model. It did not work to just add the variables to the model:
class User(model.Expando):
"""Stores user authentication credentials or authorization ids."""
#: The model used to ensure uniqueness.
unique_model = Unique
#: The model used to store tokens.
token_model = UserToken
created = model.DateTimeProperty(auto_now_add=True)
updated = model.DateTimeProperty(auto_now=True)
# ID for third party authentication, e.g. 'google:username'. UNIQUE.
auth_ids = model.StringProperty(repeated=True)
# Hashed password. Not required because third party authentication
# doesn't use password.
password = model.StringProperty()
phonenumber = model.StringProperty()
address = model.StringProperty()
I use simpleauth and I get this error msg from simpleauth:
INFO 2015-07-20 06:09:34,426 authhandlers.py:78] user_dict | {'name': u'DAC', 'user_id': 5620703441190912, 'token': u'c9BbE72EmrgTDpG1Dl4tlo', 'token_ts': 1437371676, 'cache_ts': 1437371676, 'remember': 0}
ERROR 2015-07-20 06:09:34,437 authhandlers.py:42] 'phonenumber'
INFO 2015-07-20 06:09:34,445 module.py:812] default: "POST /account/ HTTP/1.1" 404 -
INFO 2015-07-20 06:09:34,501 module.py:812] default: "GET /favicon.ico HTTP/1.1" 200 450
In my BaseRequestHandler I have this cached_property that creates an object.
#webapp2.cached_property
def current_user(self):
"""Returns currently logged in user"""
user_dict = self.auth.get_user_by_session()
logging.info('user_dict | %s ' % user_dict)
if user_dict:
return self.auth.store.user_model.get_by_id(user_dict['user_id'])
else:
return api.users.get_current_user()
Then I tried changing the user model but I still get the ERR phone_number when making these changes.
class BaseRequestHandler(webapp2.RequestHandler):
class User(auth_models.User):
address = ndb.StringProperty(indexed=False)
phone_number = ndb.IntegerProperty(indexed=False)
def dispatch(self):
# Get a session store for this request.
self.session_store = sessions.get_store(request=self.request)
if self.request.host.find('.br') > 0:
i18n.get_i18n().set_locale('pt-br')
elif self.request.host.find('klok') > 0:
i18n.get_i18n().set_locale('sv')
elif self.request.host.find('business') > 0:
i18n.get_i18n().set_locale('en')
else:
lang_code_get = self.request.get('hl', None)
if lang_code_get is None:
lang_code = self.session.get('HTTP_ACCEPT_LANGUAGE', None)
lang_code_browser = os.environ.get('HTTP_ACCEPT_LANGUAGE')
if lang_code:
i18n.get_i18n().set_locale(lang_code)
if lang_code_browser and lang_code is None:
self.session['HTTP_ACCEPT_LANGUAGE'] = lang_code_browser
i18n.get_i18n().set_locale(lang_code_browser)
else:
i18n.get_i18n().set_locale(lang_code_get)
try:
# Dispatch the request.
logging.info('trying to dispatch')
webapp2.RequestHandler.dispatch(self)
except Exception, ex:
logging.error(ex)
self.error(404)
finally:
# Save all sessions.
self.session_store.save_sessions(self.response)
#webapp2.cached_property
def jinja2(self):
"""Returns a Jinja2 renderer cached in the app registry"""
return jinja2.get_jinja2(app=self.app)
#webapp2.cached_property
def session(self):
"""Returns a session using the default cookie key"""
return self.session_store.get_session()
#webapp2.cached_property
def auth(self):
return auth.get_auth()
#webapp2.cached_property
def session_store(self):
return sessions.get_store(request=self.request)
#webapp2.cached_property
def auth_config(self):
"""
..........Dict to hold urls for login/logout
......"""
return {'login_url': self.uri_for('login'),
'logout_url': self.uri_for('logout')}
#webapp2.cached_property
def current_user(self):
"""Returns currently logged in user"""
user_dict = self.auth.get_user_by_session()
logging.info('user_dict | %s ' % user_dict)
if user_dict:
return self.auth.store.user_model.get_by_id(user_dict['user_id'])
else:
return api.users.get_current_user()
As mentioned in the comment above - you should NOT be making any changes in any of the built-in libraries, instead, you can extend them and then add any additional code/properties you need.
So first, you'd need to define your own User model, which would look simmilar to this:
from google.appengine.ext import ndb
import webapp2_extras.appengine.auth.models as auth_models
class User(auth_models.User):
address = ndb.StringProperty(indexed=False)
phone_number = ndb.IntegerProperty(indexed=False)
You are only adding the new properties you need or the ones you need to override, so no created / updated / etc as they're inherited from the model you were referring to.
You then need to work with this model inside your BaseRequestHandler class (I'm not sure what the line self.current_user does, you might need to include the code for that as well).
You can also read this article to get some more ideas: http://gosurob.com/post/20024043690/gaewebapp2accounts