Develop Reference

Trying to get two random samples to have the same matching foreignkey value

I am working on a django app that creates random fantasy character names that pull from the following models:
class VillagerFirstNames(models.Model):
first_name=models.CharField(max_length=30, unique=True)
race = models.ForeignKey(Race, on_delete=models.CASCADE)
def __str__(self):
return self.first_name
class VillagerLastNames(models.Model):
last_name = models.CharField(max_length=30, unique=True)
race = models.ForeignKey(Race, on_delete=models.CASCADE)
def __str__(self):
return self.last_name
My issue is arising in my Views. In order to pull a random.sample I have to convert my query to a list like so:
foreign_first = list(VillagerFirstNames.objects.all()
foreign_first_random = random.sample(foreign_first, 3)
context["foreign_first"] = foreign_first_random
foreign_last = list(VillagerLastNames.objects.filter(race__race=foreign_first_random.race))
context["foreign_last"] = random.sample(foreign_last, 3)
Basically, I want the last names pulled to be of the same race as the ones pulled in the first random sample. I'm having trouble figuring this one out, since the way I'm doing it above takes away the "race" attribute from foreign_first_random.

You can do random selection in Race, from there you can select random VillagerFirstNames and VillagerLastNames. For example:
race = Race.objects.all().order_by('?').first()
race_firstname = race.villagerfirstname_set.all().order_by('?').first()
race_lastname = race.villagerlastname_set.all().order_by('?').first()
Here order_by('?') makes the queryset random.
Update
To pass the values to template, you can try like this:
context["foreign_first"] = race.villagerfirstname_set.order_by('?')[:5]
context["foreign_last"] = race.villagerlastname_set.order_by('?')[:5]

Storing and retrieving default values for fields in a related model instance

I would like to store default values for a model instance in a related object; for example, given this code:
class Contract(models.Model):
user = models.ForeignKey(User)
product = models.ForeignKey(Product)
duration = models.IntegerField(null=True, help_text='Contract validity (days)')
template = models.ForeignKey(ContractTemplate)
class ContractTemplate(models.Model):
name = models.CharField(max_length=100)
duration = models.IntegerField(help_text='Contract validity (days)')
I would like to store objects representing different common durations like:
yearly_contract = ContractTemplate.object.create(name='yearly', duration=365)
monthly_contract = ContractTemplate.object.create(name='monthly', duration=30)
and return the default value from the linked template when the object contract does not specify the value:
contract1 = Contract.objects.create(user=foo_user, foo_product, template=monthly_contract)
# contract1.duration should return 365
contract2 = Contract.objects.create(user=foo_user, foo_product, duration=45, template=monthly_contract)
# contract2.duration should return 45
So, what is the best way to achieve something like this?

You can use a callable object as default. Which seems to be what you want:
Have a look here:
https://docs.djangoproject.com/en/3.1/ref/models/fields/#default

How to update queryset value in django?

I have written a python script in my project. I want to update the value of a field.
Here are my modes
class News_Channel(models.Model):
name = models.TextField(blank=False)
info = models.TextField(blank=False)
image = models.FileField()
website = models.TextField()
total_star = models.PositiveIntegerField(default=0)
total_user = models.IntegerField()
class Meta:
ordering = ["-id"]
def __str__(self):
return self.name
class Count(models.Model):
userId = models.ForeignKey(User, on_delete=models.CASCADE)
channelId = models.ForeignKey(News_Channel, on_delete=models.CASCADE)
rate = models.PositiveIntegerField(default=0)
def __str__(self):
return self.channelId.name
class Meta:
ordering = ["-id"]
This is my python script:
from feed.models import Count, News_Channel
def run():
for i in range(1, 11):
news_channel = Count.objects.filter(channelId=i)
total_rate = 0
for rate in news_channel:
total_rate += rate.rate
print(total_rate)
object = News_Channel.objects.filter(id=i)
print(total_rate)
print("before",object[0].total_star,total_rate)
object[0].total_star = total_rate
print("after", object[0].total_star)
object.update()
After counting the total_rate from the Count table I want to update the total star value in News_Channel table. I am failing to do so and get the data before the update and after the update as zero. Although total_rate has value.

The problem
The reason why this fails is because here object is a QuerySet of News_Channels, yeah that QuerySet might contain exactly one News_Channel, but that is irrelevant.
If you then use object[0] you make a query to the database to fetch the first element and deserialize it into a News_Channel object. Then you set the total_star of that object, but you never save that object. You only call .update() on the entire queryset, resulting in another independent query.
You can fix this with:
objects = News_Channel.objects.filter(id=i)
object = objects[0]
object.total_star = total_rate
object.save()
Or given you do not need any validation, you can boost performance with:
News_Channel.objects.filter(id=i).update(total_star=total_rate)
Updating all News_Channels
If you want to update all News_Channels, you actually better use a Subquery here:
from django.db.models import OuterRef, Sum, Subquery
subq = Subquery(
Count.objects.filter(
channelId=OuterRef('id')
).annotate(
total_rate=Sum('rate')
).order_by('channelId').values('total_rate')[:1]
)
News_Channel.objects.update(total_star=subq)

The reason is that object in your case is a queryset, and after you attempt to update object[0], you don't store the results in the db, and don't refresh the queryset. To get it to work you should pass the field you want to update into the update method.
So, try this:
def run():
for i in range(1, 11):
news_channel = Count.objects.filter(channelId=i)
total_rate = 0
for rate in news_channel:
total_rate += rate.rate
print(total_rate)
object = News_Channel.objects.filter(id=i)
print(total_rate)
print("before",object[0].total_star,total_rate)
object.update(total_star=total_rate)
print("after", object[0].total_star)

News_Channel.total_star can be calculated by using aggregation
news_channel_obj.count_set.aggregate(total_star=Sum('rate'))['total_star']
You can then either use this in your script:
object.total_star = object.count_set.aggregate(total_star=Sum('rate'))['total_star']
Or if you do not need to cache this value because performance is not an issue, you can remove the total_star field and add it as a property on the News_Channel model
#property
def total_star(self):
return self.count_set.aggregate(total_star=Sum('rate'))['total_star']

Iterating with Django ORM through large datasets is slow

I'm using Django ORM to get data out of a database with a few million items. However, computation takes a while (40 minutes+), and I'm not sure how to pin point where the issue is located.
Models I've used:
class user_chartConfigurationData(models.Model):
username_chartNum = models.ForeignKey(user_chartConfiguration, related_name='user_chartConfigurationData_username_chartNum')
openedConfig = models.ForeignKey(user_chartConfigurationChartID, related_name='user_chartConfigurationData_user_chartConfigurationChartID')
username_selects = models.CharField(max_length=200)
blockName = models.CharField(max_length=200)
stage = models.CharField(max_length=200)
variable = models.CharField(max_length=200)
condition = models.CharField(max_length=200)
value = models.CharField(max_length=200)
type = models.CharField(max_length=200)
order = models.IntegerField()
def __unicode__(self):
return str(self.username_chartNum)
order = models.IntegerField()
class data_parsed(models.Model):
setid = models.ForeignKey(sett, related_name='data_parsed_setid', primary_key=True)
setid_hash = models.CharField(max_length=100, db_index = True)
block = models.CharField(max_length=2000, db_index = True)
username = models.CharField(max_length=2000, db_index = True)
time = models.IntegerField(db_index = True)
time_string = models.CharField(max_length=200, db_index = True)
def __unicode__(self):
return str(self.setid)
class unique_variables(models.Model):
setid = models.ForeignKey(sett, related_name='unique_variables_setid')
setid_hash = models.CharField(max_length=100, db_index = True)
block = models.CharField(max_length=200, db_index = True)
stage = models.CharField(max_length=200, db_index = True)
variable = models.CharField(max_length=200, db_index = True)
value = models.CharField(max_length=2000, db_index = True)
class Meta:
unique_together = (("setid", "block", "variable", "stage", "value"),)
The code I'm running is looping through data_parsed, with relevant data that matches between user_chartConfigurationData and unique_variables.
#After we get the tab, we will get the configuration data from the config button. We will need the tab ID, which is chartNum, and the actual chart
#That is opened, which is the chartID.
chartIDKey = user_chartConfigurationChartID.objects.get(chartID = chartID)
for i in user_chartConfigurationData.objects.filter(username_chartNum = chartNum, openedConfig = chartIDKey).order_by('order').iterator():
iterator = data_parsed.objects.all().iterator()
#We will loop through parsed objects, and at the same time using the setid (unique for all blocks), which contains multiple
#variables. Using the condition, we can set the variable gte (greater than equal), or lte (less than equal), so that the condition match
#the setid for the data_parsed object, and variable condition
for contents in iterator:
#These are two flags, found is when we already have an entry inside a dictionary that already
#matches the same setid. Meaning they are the same blocks. For example FlowBranch and FlowPure can belong
#to the same block. Hence when we find an entry that matches the same id, we will put it in the same dictionary.
#Added is used when the current item does not map to a previous setid entry in the dictionary. Then we will need
#to add this new entry to the array of dictionary (set_of_pk_values). Otherwise, we will be adding a lot
#of entries that doesn't have any values for variables (because the value was added to another entry inside a dictionary)
found = False
added = False
storeItem = {}
#Initial information for the row
storeItem['block'] = contents.block
storeItem['username'] = contents.username
storeItem['setid'] = contents.setid
storeItem['setid_hash'] = contents.setid_hash
if (i.variable != ""):
for findPrevious in set_of_pk_values:
if(str(contents.setid) == str(findPrevious['setid'])):
try:
items = unique_variables.objects.get(setid = contents.setid, variable = i.variable)
findPrevious[variableName] = items.value
found = True
break
except:
pass
if(found == False):
try:
items = unique_variables.objects.get(setid = contents.setid, variable = i.variable)
storeItem[variableName] = items.value
added = True
except:
pass
if(found == False and added == True):
storeItem['time_string'] = contents.time_string
set_of_pk_values.append(storeItem)
I've tried to use select_related() or prefetch_related(), since it needs to go to unique_variables object and get some data, however, it still takes a long time.
Is there a better way to approach this problem?

Definitely, have a look at django_debug_toolbar. It will tell you how many queries you execute, and how long they last. Can't really live without this package when I have to optimize something =).
PS: Execution will be even slower.
edit: You may also want to enable db_index for the fields you use to filter with or index_together for more than one field. Ofc, measure the times between your changes so you make sure which option is better.

Filter Generic Foreign Key

Is there a more "Python/Django" way to query/filter objects by generic foreign key? I'm trying to get all FullCitation objects for a particular software, where is_primary is True.
I know I can't do this but I want to do something like this:
ct_supported = ContentType.objects.get(app_label="supportedprogram", model="software")
primary_citations = FullCitation.objects.filter(content_type__name=ct_supported, object_id__in='', is_primary=True)
models.py
class FullCitation(models.Model)
# the software to which this citation belongs
# either a supported software program or a non-supported software program
limit = models.Q(app_label = 'myprograms', model = 'supportedprogram') | models.Q(app_label = 'myprograms', model = 'nonsupportedprogram')
content_type = models.ForeignKey(ContentType), limit_choices_to = limit, )
object_id = models.PositiveIntegerField()
content_object = generic.GenericForeignKey('content_type', 'object_id')
is_primary = models.BooleanField(help_text="Is this the Primary Citation for the software program?")
class NonSupportedProgram(models.Model):
title = models.CharField(max_length=256, blank = True)
full_citation = generic.GenericRelation('FullCitation')
class SupportedProgram(models.Model):
title = models.CharField(max_length=256, blank = True)
full_citation = generic.GenericRelation('FullCitation')
# and a bunch of other fields.....
views.py # My current attempt
primary_citations = []
sw_citations = sw.full_citations.all()
for x in sw_citations:
if x.is_primary:
primary_citations.append(x)

Comprehensions should be a last resort for filtering QuerySets. Far better to let them remain as QuerySets as long as you can. I think this is what you're looking for:
ct_supported = ContentType.objects.get_for_model(SupportedProgram))
primary_citations = FullCitation.objects.filter(content_type=ct_supported, is_primary=True)
Updated:
If you want to filter for a specific SupportedProgram instance, do this:
my_supported = SupportedProgram.objects.get(id=instance_id_goes_here)
ct_supported = ContentType.objects.get_for_model(SupportedProgram))
primary_citations = FullCitation.objects.filter(content_object=my_supported, content_type=ct_supported, is_primary=True)