Develop Reference

How do I build a Django model that retrieves some fields from an API?

Question
How can I build a Model that that stores one field in the database, and then retrieves other fields from an API behind-the-scenes when necessary?
Details:
I'm trying to build a Model called Interviewer that stores an ID in the database, and then retrieves name from an external API. I want to avoid storing a copy of name in my app's database. I also want the fields to be retrieved in bulk rather than per model instance because these will be displayed in a paginated list.
My first attempt was to create a custom Model Manager called InterviewManager that overrides get_queryset() in order to set name on the results like so:
class InterviewerManager(models.Manager):
def get_queryset(self):
query_set = super().get_queryset()
for result in query_set:
result.name = 'Mary'
return query_set
class Interviewer(models.Model):
# ID provided by API, stored in database
id = models.IntegerField(primary_key=True, null=False)
# Fields provided by API, not in database
name = 'UNSET'
# Custom model manager
interviewers = InterviewerManager()
However, it seems like the hardcoded value of Mary is only present if the QuerySet is not chained with subsequent calls. I'm not sure why. For example, in the django shell:
>>> list(Interviewer.interviewers.all())[0].name
'Mary' # Good :)
>>> Interviewer.interviewers.all().filter(id=1).first().name
'UNSET' # Bad :(
My current workaround is to build a cache layer inside of InterviewManager that the model accesses like so:
class InterviewerManager(models.Manager):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.api_cache = {}
def get_queryset(self):
query_set = super().get_queryset()
for result in query_set:
# Mock querying a remote API
self.api_cache[result.id] = {
'name': 'Mary',
}
return query_set
class Interviewer(models.Model):
# ID provided by API, stored in database
id = models.IntegerField(primary_key=True, null=False)
# Custom model
interviewers = InterviewerManager()
# Fields provided by API, not in database
#property
def name(self):
return Interviewer.interviewers.api_cache[self.id]['name']
However this doesn't feel like idiomatic Django. Is there a better solution for this situation?
Thanks

why not just make the API call in the name property?
#property
def name(self):
name = get_name_from_api(self.id)
return name

If that isnt possible by manipulating a get request where you can add a list of names and recieve the data. The easy way is to do it is in a loop.
I would recommand you to build a so called proxy where you load the articles in a dataframe/dict, save this varible data ( with for example pickle ) and use it when nessary. It reduces loadtimes and is near efficient.

How to map an existing python class to a Django model

I'm writing a web scraper to get information about customers and appointment times to visit them. I have a class called Job that stores all the details about a specific job. (Some of its attributes are custom classes too e.g Client).
class Job:
def __init__(self, id_=None, client=Client(None), appointment=Appointment(address=Address(None)), folder=None,
notes=None, specific_reqs=None, system_notes=None):
self.id = id_
self.client = client
self.appointment = appointment
self.notes = notes
self.folder = folder
self.specific_reqs = specific_reqs
self.system_notes = system_notes
def set_appointment_date(self, time, time_format):
pass
def set_appointment_address(self, address, postcode):
pass
def __str__(self):
pass
My scraper works great as a stand alone app producing one instance of Job for each page of data scraped.
I now want to save these instances to a Django database.
I know I need to create a model to map the Job class onto but that's where I get lost.
From the Django docs (https://docs.djangoproject.com/en2.1/howto/custom-model-fields/) it says in order to use my Job class in the Django model I don't have to change it at all. That's great - just what I want. but I can't follow how to create a model that maps to my Job class.
Should it be something like
from django.db import models
import Job ,Client
class JobField(models.Field):
description = "Job details"
def __init__(self, *args, **kwargs):
kwargs['id_'] = Job.id_
kwargs['client'] = Client(name=name)
...
super().__init__(*args, **kwargs)
class Job(models.Model):
job = JobField()
And then I'd create a job using something like
Job.objects.create(id_=10101, name="Joe bloggs")
What I really want to know is am I on the right lines? Or (more likely) how wrong is this approach?
I know there must be a big chunk of something missing here but I can't work out what.

By mapping I'm assuming you want to automatically generate a Django model that can be migrated in the database, and theoretically that is possible if you know what field types you have, and from that code you don't really have that information.
What you need to do is to define a Django model like exemplified in https://docs.djangoproject.com/en/2.1/topics/db/models/.
Basically you have to create in a project app's models.py the following class:
from django import models
class Job(models.Model):
client = models.ForeignKey(to=SomeClientModel)
appointment = models.DateTimeField()
notes = models.CharField(max_length=250)
folder = models.CharField(max_length=250)
specific_reqs = models.CharField(max_length=250)
system_notes = models.CharField(max_length=250)
I don't know what data types you actually have there, you'll have to figure that out yourself and cross-reference it to https://docs.djangoproject.com/en/2.1/ref/models/fields/#model-field-types. This was just an example for you to understand how to define it.
After you have these figured out you can do the Job.objects.create(...yourdata).
You don't need to add an id field, because Django creates one by default for all models.

HTTP requests for nested objects

Is it possible to use the library 'requests' (HTTP library for python) to post and update nested objects in django rest framework?
I made a new create method in serializers, but I can't post outside the shell, nor with the requests library or in the api webview.
My Serializers:
class QualityParameterSerializer(serializers.ModelSerializer):
class Meta:
model = QualityParameter
fields = ("id","name", "value")
class ProductQualityMonitorSerializer(serializers.ModelSerializer):
parameters = QualityParameterSerializer(many=True)
class Meta:
model = ProductQualityMonitor
fields = ("id","product_name", "area", "timeslot", "processing_line",
"updated_on",'parameters')
def create(self, validated_data):
params_data = validated_data.pop('parameters')
product = ProductQualityMonitor.objects.create(**validated_data)
for param_data in params_data:
QualityParameter.objects.create(product=product, **param_data)
return product
POST HTTP request

If I may suggest the following form for your serializer:
from django.db import transaction
class ProductQualityMonitorSerializer(serializers.ModelSerializer):
parameters = QualityParameterSerializer(many=True)
class Meta:
model = ProductQualityMonitor
fields = (
"id",
"updated_on",
"product_name",
"area",
"timeslot",
"processing_line",
"parameters",
)
def create(self, validated_data):
# we will use transactions, so that if one of the Paramater objects isn't valid
# that we will rollback even the parent ProductQualityMonitor object creation,
# leaving no dangling objects in the database
params_data = validated_data.pop('parameters')
with transaction.atomic():
product = ProductQualityMonitor.objects.create(**validated_data)
# you can create the objects in a batch, hitting the dB only once
params = [QualityParameter(product=product, **param) for param in params_data]
QualityParameter.objects.bulk_create(params)
return product
About using python requests library: you will have to pay attention to the following aspects when posting to a django back-end:
you must provide a valid CSRF token in your request; the way this is done is via csrf-token cookie;
you must provide the proper authentication headers / tokens / cookies; this is your choice, depends how you're implementing this on the DRF back-end
if this is a request from one domain to another domain, then you have to care for CORS setup.
More to the point: what have you tried already and didn't worked ?

Django custom registration fields

I'm becoming increasingly bewildered by the range of answers on offer to the seemingly simple problem of adding custom fields to the django-registration register form/flow. This should be a default, documented aspect of the package (not to sound ungrateful, just that it is such a well-equipped package), but solutions to the problem are dizzying.
Can anyone give me the most simple solution to getting UserProfile model data included in the default registration register page?
Update:
I eventually used Django Registration's own signals to give me this hacky fix. It is particularly ugly because, I had to use try on the POST attribute dealing with my Boolean since I found that the checkbox returned nothing if left empty.
Would appreciate any advice on improving this, or best practice.
My app / models.py
from registration.signals import user_registered
from django.dispatch import receiver
class UserProfile(models.Model):
user = models.OneToOneField(User)
event_commitments = models.ManyToManyField(Event, null=True, blank=True)
receive_email = models.BooleanField(default=True)
#receiver(user_registered)
def registration_active_receive_email(sender, user, request, **kwargs):
user_id = user.userprofile.id
user = UserProfile.objects.get(pk=user_id)
try:
if request.POST['receive_email']:
pass
except:
user.receive_email = False
user.save()
Registration app / forms.py
class RegistrationForm(forms.Form):
# default fields here, followed by my custom field below
receive_email = forms.BooleanField(initial=True, required=False)
Thanks

What you have looks like a workable approach.
I've looked through the django-registration code, and based on the following comments in the register view I've come up with another solution. I'm not totally sure this is cleaner, but if you aren't a fan of signals this is good. This also provides a much easier avenue if you intend to make more customizations.
# from registration.views.register:
"""
...
2. The form to use for account registration will be obtained by
calling the backend's ``get_form_class()`` method, passing the
``HttpRequest``. To override this, see the list of optional
arguments for this view (below).
3. If valid, the form's ``cleaned_data`` will be passed (as
keyword arguments, and along with the ``HttpRequest``) to the
backend's ``register()`` method, which should return the new
``User`` object.
...
"""
You could create a custom backend and override those mentioned methods:
# extend the provided form to get those fields and the validation for free
class CustomRegistrationForm(registration.forms.RegistrationForm):
receive_email = forms.BooleanField(initial=True, required=False)
# again, extend the default backend to get most of the functionality for free
class RegistrationBackend(registration.backends.default.DefaultBackend):
# provide your custom form to the registration view
def get_form_class(self, request):
return CustomRegistrationForm
# replace what you're doing in the signal handler here
def register(self, request, **kwargs):
new_user = super(RegistrationBackend, self).register(request, **kwargs)
# do your profile stuff here
# the form's cleaned_data is available as kwargs to this method
profile = new_user.userprofile
# use .get as a more concise alternative to try/except around [] access
profile.receive_email = kwargs.get('receive_email', False)
profile.save()
return new_user
To use the custom backend, you can then provide separate urls. Before including the default urls, write 2 confs that point at your custom backend. Urls are tested in the order defined, so if you define these two before including the defaults, these two will capture before the default ones are tested.
url(r'^accounts/activate/(?P<activation_key>\w+)/$',
activate,
{'backend': 'my.app.RegistrationBackend'},
name='registration_activate'),
url(r'^accounts/register/$',
register,
{'backend': 'my.app.RegistrationBackend'},
name='registration_register'),
url(r'^accounts/', include('registration.backends.default.urls')),
The docs actually describe all this, but they aren't particularly accessible (no readthedocs). They are all included in the project, and I was browsing them here.

I eventually used Django Registration's own signals to give me this fix.
I will clean up the try/except flow at some point. dokkaebi also points out above that I might be able to assess the request.GET parameters for when a checkbox is left empty.
My app / models.py
from registration.signals import user_registered
from django.dispatch import receiver
class UserProfile(models.Model):
user = models.OneToOneField(User)
event_commitments = models.ManyToManyField(Event, null=True, blank=True)
receive_email = models.BooleanField(default=True)
#receiver(user_registered)
def registration_active_receive_email(sender, user, request, **kwargs):
user_id = user.userprofile.id
user = UserProfile.objects.get(pk=user_id)
try:
if request.POST['receive_email']:
pass
except:
user.receive_email = False
user.save()
Registration app / forms.py
class RegistrationForm(forms.Form):
# default fields here, followed by my custom field below
receive_email = forms.BooleanField(initial=True, required=False)

Django: form values not updating when model updates

I am creating a form that uses MultipleChoiceField. The values for this field are derived from another model. This method works fine, however, I am noticing (on the production server) that when I add a new item to the model in question (NoticeType), the form does not dynamically update. I have to restart the server for the new item to show up on my MultipleChoiceField.
Any changes to the NoticeType model (editing items or creating new ones) do not propagate to the form. After I restart the production server, the updates appear.
Any ideas why this might be ? The relevant portion of the form is below. Thanks.
from django import forms
from django.contrib.auth.models import User
from notification.models import NoticeType
class EditUserProfileForm(forms.Form):
CHOICES = []
for notice in NoticeType.objects.all():
CHOICES.append( (notice.label,notice.display) )
notifications = forms.MultipleChoiceField(
label="Email Notifications",
required=False,
choices=( CHOICES ),
widget=forms.CheckboxSelectMultiple,)

Although mherren is right that you can fix this problem by defining your choices in the __init__ method, there is an easier way: use the ModelMultipleChoiceField which is specifically designed to take a queryset, and updates dynamically.
class EditUserProfileForm(forms.Form):
notifications = forms. ModelMultipleChoiceField(
label="Email Notifications",
required=False,
queryset = NoticeType.objects.all(),
widget=forms.CheckboxSelectMultiple)

My hunch is that the class definition is only being processed once on load rather than for each instantiation. Try adding the CHOICES computation to the init method like so:
def __init__(self, *args, **kwargs):
super(self.__class__, self).__init__(*args, **kwargs)
CHOICES = []
for notice in NoticeType.objects.all():
CHOICES.append( (notice.label, notice.display) )
self.fields['notifications'].choices = CHOICES