Develop Reference

Django model manager queryset not updating until server restarts

I have a program that lets users upload data files and lookup tables (both which are ID'd to a specific company) and map them together. One page lets users choose which company they want to map data for by looking at which companies have both data files and lookup tables, which I use a queryset/model manager for. The problem is if I load a new data file and hierarchy the queryset doesn't pick them up until the server restarts. The queryset returns all the companies that have a data file and hierarchies at the time the server starts, but not anything that's added afterwards. I think this must be because the queryset is defined at startup, but I'm not sure. Is there a way to work around this?
forms.py
class CompanySelectionForm(forms.Form):
companies = RawData.objects.get_companyNames(source="inRDandH")
companiesTuple = makeTuple(companies)
print(companiesTuple)
company = forms.ChoiceField(widget=forms.Select(attrs={'class': 'form-select'}), choices=companiesTuple)
managers.py
class RawDataManager(models.Manager):
def get_queryset(self):
return RawDataQuerySet(self.model, using=self._db)
def get_companyNames(self, source):
return self.get_queryset().get_companyNames(source)
class RawDataQuerySet(models.QuerySet):
def get_companyNames(self, source):
if (source == 'inRDandH'):
distinct_companiesRD = self.filter(fileType=0).values_list('companyName', flat=True).distinct()
distinct_companiesH = self.filter(fileType=1).values_list('companyName', flat=True).distinct()
distinct_companies = set(distinct_companiesRD).intersection(set(distinct_companiesH))
else:
distinct_companies = self.values_list('companyName', flat=True).distinct()
return distinct_companies

The problem is that this code runs only once, when the code is initialised on server start, because it is part of your form class definition:
companies = RawData.objects.get_companyNames(source="inRDandH")
The solution is to make choices a callable, which is run every time the form is instantiated. define that field dynamically, in the __init__ method of the form:
def get_companies_tuple():
companies = RawData.objects.get_companyNames(source="inRDandH")
return makeTuple(companies)
class CompanySelectionForm(forms.Form):
company = forms.ChoiceField(
widget=forms.Select(attrs={'class': 'form-select'}),
choices=get_companies_tuple
)
This will now fetch the data from the database every time the form is initialised, rather than only once during startup.

Stop affecting other objects of Django Many to Many model

I'm trying to replicate Blood Group as Model as defined in this picture.
.
In my models.py file I had my code to replicate the blood groups like this
class BloodGroup(models.Model):
name = models.CharField(
max_length=3
)
gives = models.ManyToManyField("self")
receives = models.ManyToManyField("self")
def __str__(self):
return self.name
And in my admin.py file I had registered the model as follows
class BloodGroupAdmin(admin.ModelAdmin):
model = BloodGroup
list_display = ['name', 'get_gives', 'get_receives']
def get_gives(self, obj):
return ", ".join([item.name for item in obj.gives.all()])
def get_receives(self, obj):
return ", ".join([item.name for item in obj.receives.all()])
admin.site.register(BloodGroup, BloodGroupAdmin)
Initially I created plain BloodGroup objects without their gives and receives attribute by providing just their names alone. Thus I create an object for all 8 types. Then as I added relationships to each object I found that adding gives or receives for one object affects other objects gives and receives too, making it impossible to replicate the structure in image.
How do I define relationships, without affecting other objects?
In my admin site, I see field names as "get_gives" and "get_receives". How would i make the admin page show field names as "gives" and "receives" but still displaying objects as strings like the image below?

For first question, probably it is better to have only one relation gives. receives can be found from the reverse query. Like this:
class BloodGroup(models.Model):
name = models.CharField(
max_length=3
)
gives = models.ManyToManyField("self", related_name="receives", symmetrical=False)
Then you only need to add objects to gives. receives will be generated automatically.
For second question, add short_description attribute to function(reference to docs). Like this:
get_gives.short_description = 'Gives'
get_receives.short_description = 'Receives'

How to map an existing python class to a Django model

I'm writing a web scraper to get information about customers and appointment times to visit them. I have a class called Job that stores all the details about a specific job. (Some of its attributes are custom classes too e.g Client).
class Job:
def __init__(self, id_=None, client=Client(None), appointment=Appointment(address=Address(None)), folder=None,
notes=None, specific_reqs=None, system_notes=None):
self.id = id_
self.client = client
self.appointment = appointment
self.notes = notes
self.folder = folder
self.specific_reqs = specific_reqs
self.system_notes = system_notes
def set_appointment_date(self, time, time_format):
pass
def set_appointment_address(self, address, postcode):
pass
def __str__(self):
pass
My scraper works great as a stand alone app producing one instance of Job for each page of data scraped.
I now want to save these instances to a Django database.
I know I need to create a model to map the Job class onto but that's where I get lost.
From the Django docs (https://docs.djangoproject.com/en2.1/howto/custom-model-fields/) it says in order to use my Job class in the Django model I don't have to change it at all. That's great - just what I want. but I can't follow how to create a model that maps to my Job class.
Should it be something like
from django.db import models
import Job ,Client
class JobField(models.Field):
description = "Job details"
def __init__(self, *args, **kwargs):
kwargs['id_'] = Job.id_
kwargs['client'] = Client(name=name)
...
super().__init__(*args, **kwargs)
class Job(models.Model):
job = JobField()
And then I'd create a job using something like
Job.objects.create(id_=10101, name="Joe bloggs")
What I really want to know is am I on the right lines? Or (more likely) how wrong is this approach?
I know there must be a big chunk of something missing here but I can't work out what.

By mapping I'm assuming you want to automatically generate a Django model that can be migrated in the database, and theoretically that is possible if you know what field types you have, and from that code you don't really have that information.
What you need to do is to define a Django model like exemplified in https://docs.djangoproject.com/en/2.1/topics/db/models/.
Basically you have to create in a project app's models.py the following class:
from django import models
class Job(models.Model):
client = models.ForeignKey(to=SomeClientModel)
appointment = models.DateTimeField()
notes = models.CharField(max_length=250)
folder = models.CharField(max_length=250)
specific_reqs = models.CharField(max_length=250)
system_notes = models.CharField(max_length=250)
I don't know what data types you actually have there, you'll have to figure that out yourself and cross-reference it to https://docs.djangoproject.com/en/2.1/ref/models/fields/#model-field-types. This was just an example for you to understand how to define it.
After you have these figured out you can do the Job.objects.create(...yourdata).
You don't need to add an id field, because Django creates one by default for all models.

How to populate choice form from db in Django?

I can't figure out how to populate choice form from db. I know about ModelChoiceForm but the problem seems to be slightly different.
I want user to choose which sector does he work in. For example: 'Finance','Electronics' etc. which I would do simple:
SECTOR_CHOICES = (('finance',_('Finance'),
'electronics',_('Electronics')...
))
But the problem is that I want admin of the web to be able to add new choices, remove choice etc.
What came to my mind is to create a simple Model called Sector:
class Sector(models.Model):
name = models.CharField(max_length=40)
and User would have new attribute sector = models.ModelChoice(Sector).
But I'm scared what would happend when admin changes or removes a sector which is already used, and more, what if he removes it and the sector attribute is required?
How to solve this problem?

I would just override the delete_model as custom action and there check if the selected sector object is in use.
def delete_model(modeladmin, request, queryset):
for obj in queryset:
if UserModel.objects.filter(sector=obj).exists():
# do not delete, just add some message warning the admin about it
else:
obj.delete()
class UserModelAdmin(admin.ModelAdmin):
actions = [delete_model]
# ...

Django admin list_display weirdly slow with foreign keys

Django 1.2.5
Python: 2.5.5
My admin list of a sports model has just gone really slow (5 minutes for 400 records). It was returning in a second or so until we got 400 games, 50 odd teams and 2 sports.
I have fixed it in an awful way so I'd like to see if anyone has seen this before. My app looks like this:
models:
Sport( models.Model )
name
Venue( models.Model )
name
Team( models.Model )
name
Fixture( models.Model )
date
sport = models.ForeignKey(Sport)
venue = models.ForeignKey(Venue)
TeamFixture( Fixture )
team1 = models.ForeignKey(Team, related_name="Team 1")
team2 = models.ForeignKey(Team, related_name="Team 2")
admin:
TeamFixture_ModelAdmin (ModelAdmin)
list_display = ('date','sport','venue','team1','team2',)
If I remove any foreign keys from list_display then it's quick. As soon as I add any foreign key then slow.
I fixed it by using non foreign keys but calculating them in the model init so this works:
models:
TeamFixture( Fixture )
team1 = models.ForeignKey(Team, related_name="Team 1")
team2 = models.ForeignKey(Team, related_name="Team 2")
sport_name = ""
venue_name = ""
team1_name = ""
team2_name = ""
def __init__(self, *args, **kwargs):
super(TeamFixture, self).__init__(*args, **kwargs)
self.sport_name = self.sport.name
self.venue_name = self.venue.name
self.team1_name = self.team1.name
self.team2_name = self.team2.name
admin:
TeamFixture_ModelAdmin (ModelAdmin)
list_display = ('date','sport_name','venue_name','team1_name','team2_name',)
Administration for all other models are fine with several thousand records at the moment and all views in the actual site is functioning fine.

It's driving me crazy. list_select_related is set to True, however adding a foreign key to User in the list_display generates one query per row in the admin, which makes the listing slow. Select_related is True, so the Django admin shouldn't call this query on each row.
What is going on ?

The first thing I would look for, are the database calls. If you shouldn't have done that already, install django-debug-toolbar. That awesome tool lets you inspect all sql queries done for the current request. I assume there are lots of them. If you look at them, you will know where to look for the problem.
One problem I myself have run into: When the __unicode__ method of a model uses a foreign key, that leads to one database hit per instance. I know of two ways to overcome this problem:
use select_related, which usually is your best bet.
make your __unicode__ return a static string and override the save method to update this string accordingly.

This is a very old problem with django admin and foreign keys. What happens here is that whenever you try to load an object it tries to get all the objects of that foreign key. So lets say you are trying to load a fixture with a some teams (say the number of teams is about 100), its going to keep on including all the 100 teams in one go. You can try to optimize them by using something called as raw_fields. What this would do is instead of having to calling everything at once, it will limit the number of calls and make sure that the call is only made when an event is triggered (i.e. when you are selecting a team).
If that seems a bit like a UI mess you can try using this class:
"""
For Raw_id_field to optimize django performance for many to many fields
"""
class RawIdWidget(ManyToManyRawIdWidget):
def label_for_value(self, value):
values = value.split(',')
str_values = []
key = self.rel.get_related_field().name
for v in values:
try:
obj = self.rel.to._default_manager.using(self.db).get(**{key: v})
x = smart_unicode(obj)
change_url = reverse(
"admin:%s_%s_change" % (obj._meta.app_label, obj._meta.object_name.lower()),
args=(obj.pk,)
)
str_values += ['<strong>%s</strong>' % (change_url, escape(x))]
except self.rel.to.DoesNotExist:
str_values += [u'No input or index in the db']
return u', '.join(str_values)
class ImproveRawId(admin.ModelAdmin):
raw_id_fields = ('created_by', 'updated_by')
def formfield_for_dbfield(self, db_field, **kwargs):
if db_field.name in self.raw_id_fields:
kwargs.pop("request", None)
type = db_field.rel.__class__.__name__
kwargs['widget'] = RawIdWidget(db_field.rel, site)
return db_field.formfield(**kwargs)
return super(ImproveRawId, self).formfield_for_dbfield(db_field, **kwargs)
Just make sure that you inherit the class properly. I am guessing something like TeamFixture_ModelAdmin (ImproveRawIdFieldsForm). This will most likely give you a pretty cool performance boost in your django admin.

I fixed my problem by setting list_select_related to the list of related model fields instead of just True