I'm writing a Django application in which we will have multiple client libraries that make calls to third party APIs. We want to be able to determine which client library to load at runtime when calling different endpoints. I'm trying to determine the most Django/Pythonic way to do this
The current idea is to store class name in a model field, query the model in the View, and pass the class name to a service factory that instantiates the relevant service and makes the query. I've also considered writing a model method that uses reflection to query the class name
For example lets say we have a model called Exchange that has an id, a name, and stores the client name.
class Exchange(models.Model):
exchange_name = models.CharField(max_length=255)
client_name = models.CharField(max_length=255)
def __str__(self):
return self.exchange_name
Then in the View we might have something like:
from rest_framework.views import APIView
from MyProject.trades.models import Exchange
from django.http import Http404
from MyProject.services import *
class GetLatestPrice(APIView):
def get_object(self, pk):
try:
exchange = Exchange.objects.get(pk=pk)
except Exchange.DoesNotExist:
raise Http404
def get(self, request, pk, format=None):
exchange = self.get_objectt(pk)
service = service_factory.get_service(exchange.client_name)
latest_price = service.get_latest_price()
serializer = PriceSerializer(latest_price)
return Response(serializer.data)
This syntax may not be perfect but it hasn't been tested. But the idea is that maybe we have
ExchangeA - client1
ExchangeB - client2
ExchangeC - client3
And for each client we have a service class that all inherit from the same Abstract Base. Each service has an override that are all the same. That way when you call /api/get_latest_price you pass the Exchange id as a query param and the code loads whatever service and client library is relevant. This would be so we could easily add new types to the system, have consistent and small Views, and decouple the business logic. Is this an acceptable, scalable pythonic method? Or is there a better solution for it? Essentially the problem is implementing Polymorphism in django and seperating the domain model from the data model. This has to be a solved problem so I'm curious on the way other people might be doing this. I know it's unusual to be calling a third party API like this but since we are forced to I want to do it in the cleanest most scalable way possible.
TL;DR
There's no silver bullet.
More useful answer:
There's no universal and convenient approach to solve your problem. solution may vary greatly depending on the implementation of client libraries and third party APIs. If they all are similar, you can use your approach and you'll be fine. If APIs have some differences, for instance, one API don't have endpoint to get latest price, you'll need to find some workarounds. If API's have some big differences in workflow you will need to find completely different approach, for instance, if one API can return a report right away, and some other will return report via callback after 5 minutes, you'll need to find workaround for this.
So the one real advise i can give you is to examine differences between third party APIs and focus more on the design of the highest abstractions of your own project, that will use low level client libraries. Try to minimize the impact of differences between third parties on your project.
We have a similar situation here (and some other that are closely related), and use exactly this pattern: storing the name of the "service" class in the model and using some registry (your "service_factory") to get the class. It JustWorks(tm), no one (from the various devs that joined - and eventually left) ever complained about this design nor suggested any better solution, and the only small complication we've had so far is handling "discontinued" services that are still referenced from the database (and this really was no rocket science either).
FWIW, this is mostly a variant on the "strategy" design pattern, and I really fail to imagine a better (for "better" = "simpler, more effective and easier to maintain") solution for this need.
EDIT
Note that my above answer assumes that Exchange has other responsabilites not mentionned in your question - else, as Alexandr Zayets states, this model seems rather useless ;-)
Related
Say I have a simple application with users, groups and users are members of the groups (a M2M relationship). So I create a viewset which shall do the following:
GET /group lists all groups
GET /group/1 gives detail of a group
POST /group/1/member adds a member specified in the request to group 1
GET /group/1/member lists all members of group 1
and some other CRUD operations on groups/memberships which are not relevant for my question.
To make the code as readable as possible, I came up with this solution:
class GroupViewSet(ModelViewSet):
...
#action(detail=True, methods=["get", "post"])
def member(self, request, *args, **kwargs):
return MembershipView.as_view()(request, *args, **kwargs)
class MembershipView(ListCreateAPIView):
# handle the membership somehow here
However, this approach ends with
The `request` argument must be an instance of `django.http.HttpRequest`, not `rest_framework.request.Request`.
Although manually instantiating a view inside another view and passing the request would most probably work in pure Django, the REST framework refuses to do this, as each DRF view expects a pure Django request and cannot just take an existing DRF request as it is.
I have two questions:
Is my design idea bad? And if yes, how shall I redesign?
Do you think it will be worth creating a pull request to DRF? Allowing this is very easy.
Firstly, I think your URL structure may not really be conventionally RESTful. You should be using plurals to access your resources, i.e.
POST /groups/
returns groups.
POST /groups/1
fetches an item from the groups list with an id of 1. It makes things a little clearer.
Secondly, I personally would redesign your flow as I think the current design overcomplicates your REST structure.
Are Members and Groups distinct resources? I would say yes, these are distinct. So they should be kept in different endpoints. Have /members/ and /groups/ and use django's built in filters to get the desired behavior, i.e.:
GET /members/?group_id=1
This is cleaner and more RESTful. You are fetching members who have a corresponding group_id of 1.
If you really want to keep this embedded structure, You can probably do some magic with some of rest_framework.decorators list_route and detail_route to probably achieve your current desired behavior.
However, in my experience, I've found embedding like that causes issues with some django mechanics and makes for some gnarly URL routing. I really would advise keeping distinct endpionts.
You're basically doing this:
/list-view/detail-view/list-view/
I can understand use cases where this design could make sense on paper, but I don't think it's ideal.
I have a fairly complex django application that has been in production for over a year.
The application holds data from different customers. The data is obviously in the same table, separated by customer_id.
Recently the client has started to ask questions about data segregation. Since the app is sold on a per user basis and holds sensitive information, customers have been asking if and how we maintain data segregation per customer, and are there any security measures that we take to prevent data leakages (ie. data from one customer being accessed by another customer).
We do our filters in the view endpoints, but eventually a developer in the team might forget to include a filter in his ORM query, and cause a data leakage
So we came up with the idea to implement default filters on our models. Basically whenever a developer writes:
Some_Model.objects.all()
essentially they will execute:
Some_Model.objects.filter(customer_id = request.user.customer_id)
We plan to achieve this by overriding the objects property on each model to point to a manager with a filtered queryset. Something like this:
class Allowed_Some_Model_Manager(models.Manager):
def get_queryset(self):
return super(Allowed_Some_Model_Manager, self).get_queryset().filter(
customer_id = request.user.customer_id
# the problem is that request.user is not available in models.py
)
class Some_Model(models.Model):
name = models.CharField(max_length=50)
customer = models.ForeignKey(Customer)
objects = Allowed_Some_Model_Manager()
all_objects = models.Manager() # use this if we want all objects
However our problem is that request.user is not available in models.py.
I have found several ways to solve this.
Option 1 includes passing the request.user to the manager each time. However since I am dealing with thousands of lines of old code, I don't want to go and change all of our ORM queries.
Option 2, included using threading.local() to set the request.user in the thread local data.
Something like this: https://djangosnippets.org/snippets/2179/
There is a module that seems to be doing this: https://github.com/Alir3z4/django-crequest
However, a lot of people seem to be against this idea... Namely these two discussions:
django get_current_user() middleware - strange error message which goes away if source code is "changed" , which leads to an automatic server restart
Django custom managers - how do I return only objects created by the logged-in user?
So that brings me to Option 3 which I came up with, and I can not find anybody else using it. Use the python builtins module to pass the user from the middleware to the model.
#middleware.py
import builtins
def process_request(self, request):
if request.user.id:
builtins.django_user = request.user
#models.py
import builtins
class Allowed_Some_Model_Manager(models.Manager):
def get_queryset(self):
if 'django_user' in vars(builtins):
return super(Allowed_Some_Model_Manager, self).get_queryset().filter(
customer_id = django_user.customer_id
)
else:
return super(Allowed_Some_Model_Manager, self).get_queryset()
I have tested the code and it is working on my local django server and on Apache with mod_wsgi. But I really want to hear if there are any pitfalls of this approach. I have never used builtins module before, and I am not sure if I understand how it works, and what is the use-case for it.
What do you prefer when you want to "RESTify" your Django project in Django?
I came to the conclusion that there are really three options to do that:
django-piston http://bitbucket.org/jespern/django-piston/wiki/Home
django-rest-interface http://code.google.com/p/django-rest-interface/
django-restful-resources http://watchitlater.com/blog/2010/02/django-restful-resources/
Right way to do this for me would be to try all of'em and pick the one that is best for me, so meanwhile I'd like to hear yours...
Thanks.
I'm most familiar with django-piston, so I would naturally steer you in that direction.
A quick glance at the other two, though, indicates that django-rest-interface does nothing more than expose models as resources, and that django-restful-resources is some guy's one-off attempt at the same.
Piston, if I recall correctly, grew out of bitbucket.org's own site development, and allows a lot of flexibility - you can return almost any object from your resource's access methods, not just model instances, and it will be properly encoded. It also has built-in support for some nice features, like form validation (if you can get it to work right, anyway) and request throttling, among other things.
With the new class-based generic views in django 1.3 it will be super-easy to implement your own rest interface, with custom serializers and deserializers, replicating the almost complete piston's implementation using just stock code. I made a View(1.3)-based rest module in 500 lines of code, with generic RESTful resource class and sub resources, natural key support for associations, json and XML serialization and more. the module is really tailored upon my app's requirements
I did it to overcome a couple of limitations in piston's code, like having a query set modified (e.g. With .values(...)) before the handler calling .get() on it, or not being able to use a model's method in serialization.
If you do it as you need it, in a couple of days you'll have a working set of classes and mixins, that you will fully understand and be in control of.
As the "some guy" who wrote django-restful-resources I would like to clarify why it exists. It is NOT an attempt to expose models as resources, rather it is a means of allowing a single URL to be mapped to a number of different handler methods, one per HTTP verb. That's all. It can be used to expose model objects, but it can also be used to expose services as resources or anything else that you want to interact with via a single URL and HTTP verbs. If you are looking for a more full-featured solution then by all means go with Piston.
As mentioned by eternicode, django-piston is excellent. It's mature, well featured and has a good community behind it. It does seem to be lacking much ongoing development at the moment, although there is talk of a community driven fork, so that may change.
django-tastypie is also well worth a look, and seems to have a lot of impetus behind it at the moment.
I've also just released another option that's worth considering: django-rest-framework. There's a couple of really nice features behind it such as the API auto-documentation.
It uses Django 1.3's class based views, as mentioned by saverio, which means you can just drop in some of the MixIn classes it provides, without having to use the framework outright. (For example adding HTTP content negotiation for serializing output to multiple types)
Trying to figure out the best way to do this:
Should I do something like:
def index(self):
if request.POST:
u = User(id)
u.setCookie() #All session logic in def setCookie()
Or set the cookie in the controller like:
def index(self):
if request.POST:
u = User(id)
response.set_cookie('session_key', u.session_key, max_age=3600)
Why do it one way or the other? Thank you.
I think traditionally you would want the model to be concerned with data persistence and validation but not http related stuff like cookies. Which leaves the controller to be the more appropriate place in my opinion.
A reason(not the only one) I can think of for this is that it might be required someday that you need to run applications against your model logic that have nothing to do with web stuff.
given your implementation above I suspect the user object would have to get access to the response from the stacked proxy/globals stuff in pylons(could be wrong). So, if you ever needed to use the same model classes in a program that consumes messages from a message queue for example, the pylons machinery that makes the response available would not be available. This could be a problem that is easily avoided.
I share also Tom's opinion, you should try to avoid to much dependencies in different classes. So the controller should to all the http (request, response) related stuff. Also for testing it is much easier.
I'm currently learning Python and coming from a strong C# background. I keep hearing about doing things in a Pythonic way to take advantage of the dynamic nature of the language and some of it I get and some I don't.
I'm creating a site with Django and my approach to views is to use classes. My current thinking is to have a base class that has some stuff about the template and the model to use. This will have a default funky 404 type page with site search and stuff on then base all the other pages off this. So each area of the site will have its own EG News and all the model related functions and filtering will be in that class with a further class on top of that for HTML or AJAX requests. So you would have something like this:
\site\common\ViewBase
--\news\NewsBase(ViewBase)
--\news\HtmlView(NewsBase)
--\news\AJAXView(NewsBase)
URLs would be mapped like http://tld/news/latest maps to site.news.htmlview and http://tld/news//to/ will be also be mapped site.news.htmlview but the class will figure out what to do with the extra params.
This is pretty much what I would do in C# but the Django tutorial only shows using methods for views, making me wonder if this is not a very pythonic solution?
Thoughts?
Edit: After S.Lott comment about thread safety, Is it better to leave the functions as they are and have them create an instance of a class and call a method on it?
What I am looking for is a place to put common code for each section of the site for filtering the model, authentication for the site, etc
Certainly there's nothing wrong with using a class for a view, provided you route the URL to an actual instance of a class and not just a class directly.
The Django admin does exactly this - look at the source code in django/contrib/admin.
The advantage of classes is that they are much easier to customize, for example you can add hooks for permission checking.
There is a proposal to move all existing generic views over to classes, it was supposed to get into 1.2 but failed to meet the deadline.
As the above poster points out, be very careful about handling instance variables - if you look at the admin classes, you see the request being passed to the various methods instead of relying on "self".
Setting aside other concerns (such as thread-safety issues), it feels like there's a real possible danger here to cross the bright lines between Model / View / Template.
Or maybe it feels like a replacement for url dispatching (not that there's anything wrong with that :-). I'm not sure, but it just feels slightly off.
While class-based views are useful, inheritance may not be the right tool for this particular job. Helper functions and decorators are two great ways to factor out common code from your views. They also tend to be be more familiar/natural to other (python) coders who might work on your code.
I'm not sure what the best approach is in your case as I don't know how much you ultimately want to factor, just keep in mind that there are other ways to factor in python besides inheritance.
p.s. kudos for seeking out a pythonic solution.