I have the following code that iterates the tags queryset, and for each item, creates a Department object and adds it to the departments list:
departments: List[Department] = []
tags = Tag.objects.filter(type="department")
for tag in tags:
dept_id = tag.reference_id
dept_name = tag.name
parent_tag = Tag.objects.get(type="department", reference_id=tag.parent_reference_id)
dept_parent_id = parent_tag.reference_id
departments.append(Department(dept_id, dept_name, dept_parent_id))
However, as you can see, it is making multiple DB calls via Tag.objects.get(), which seems highly inefficient. Is there an efficient way to populate that departments list without making so many DB calls?
TIA.
What you need to use is "in" in your query.
check querysets
Entry.objects.filter(id__in=[1, 3, 4])
Entry.objects.filter(headline__in='abc')
so in your case you can use the the following example :
tags = Tag.objects.filter(id=some_id, type="department").values('id')
tags_list = [tag['id'] for tag in tags]
parent_tag = Tag.objects.get(id__in=tags_list, type="department")
I have used parts of the answer from #Vanda to write the following solution, and this solves my problem.
departments: List[Department] = []
tags = Tag.objects.filter(type="department")
parents_set = {tag.parent_reference_id for tag in tags}
for tag in tags:
dept_id = tag.reference_id
dept_name = tag.name
dept_parent_id = tag.parent_reference_id
if(dept_parent_id not in parents_set):
dept_parent_id = None
departments.append(Department(dept_id, dept_name, dept_parent_id))
Related
I am trying to get all my data out of the database in a array or list format.
Such that I can make an for loop in the html to loop through all the names in the database.
I tried with:
all_name = name.objects.all()
the output will be shown as
<QuerySet [<allName: name1>, < allName: name2>, < allName: name3>]>
However I want to get something like:
name1;
name2; name3
What I did next is to use the .get function:
all_name = name.objects.get(id=1)
my name model:
class name(models.Model):
firstname = models.CharField(max_length=100)
def __str__(self):
return (self.firstname)
This gives me only one object with the given id.
Is there a way to get what I am looking for and display my array/list with a forloop in a html file?
You can simply try for Name model with consider allName as a attribute/column
list(Name.objects.all().values_list('allName', flat=True))
or
Name.objects.all().values_list('allName')
You can join the strings that originate from the database with:
'; '.join(map(str, name.objects.all()))
Many:
name_list = Name.objects.all().values_list('firstname', flat=True))
One:
query = name.objects.get(id=1)
name = query.firstname
I am doing some data changes in a django app with a large amount of data and would like to know if there is a way to make this more efficient. It's currently taking a really long time.
I have a model that used to look like this (simplified and changed names):
class Thing(models.Model):
... some fields...
stuff = models.JSONField(encoder=DjangoJSONEncoder, default=list, blank=True)
I need to split the list up based on a new model.
class Tag(models.Model):
name = models.CharField(max_length=200)
class Thing(models.Model):
.... some fields ...
stuff = models.JSONField(encoder=DjangoJSONEncoder, default=list, blank=True)
other_stuff = models.JSONField(encoder=DjangoJSONEncoder, default=list, blank=True)
tags = models.Many2ManyField(Tag)
What I need to do is take the list that is currently in stuff, and split it up. For items that have a tag in the Tag model, add it to the Many2Many. For things that don't have a Tag, I add it to other_stuff. Then in the end, the stuff field should contain of the items that were saved in tags.
I start by looping through the Tags to make a dict that maps the string version that would be in the stuff list to the tag object so I don't have to keep querying the Tag model.
Then I loop through the Thing model, get the stuff field, loop through that and add each Tag item to the many2many while keeping lists for each item that is or isn't in Tags. Then put those in the stuff and other stuff fields at the end.
tags = Tag.objects.all()
tag_dict = {tag.name.lower():Tag for tag in tags}
things = Thing.objects.all()
for thing in things:
stuff_list = thing.stuff
stuff_in_tags = []
stuff_not_in_tags = []
for item in stuff_list:
if item.lower() in tag_dict.keys():
stuff_in_tags.append(item)
thing.tags.add(tag_dict[item.lower()])
else:
stuff_not_in_tags.append(item)
thing.stuff = stuff_in_tags
thing.other_stuff = stuff_not_in_tags
thing.save()
(Ignore any typos. This code works in my actual code)
That seems pretty efficient to me, but its taking hours to run as our database is pretty big (about 500k+ records). Are there any other ways to make this more efficient?
Unless you move some work to the database level with bulk operations, it won't run faster. You are making at least N (500k+) UPDATE queries.
If the parsing cannot be done on the DB level, chunked bulk_update is the next option.
Also, you can use iterator() to avoid loading all the objects to memory and only() to load only relevant columns.
There is a typo in tag_dict - it should be : tag (instance) instead of : Tag (model).
EDIT: I've originally missed the thing.tags.add - this will need additional handling. You have to bulk_create m2m table rows.
chunk_size = 10000
TagsToThing = Thing.tags.through
tag_dict = {tag.name.lower():tag for tag in Tag.objects.all()}
for_update = []
tags_for_create = []
for thing in Thing.objects.only('pk', 'stuff').iterator(chunk_size):
stuff_in_tags = []
stuff_not_in_tags = []
for item in thing.stuff:
if item.lower() in tag_dict.keys():
stuff_in_tags.append(item)
tags_for_create.append(
TagsToThing(thing=thing, tag=tag_dict[item.lower()])
)
else:
stuff_not_in_tags.append(item)
thing.stuff = stuff_in_tags
thing.other_stuff = stuff_not_in_tags
for_update.append(thing)
if len(for_update) == chunk_size:
Thing.objects.bulk_update(for_update, ['stuff', 'other_stuff'], chunk_size)
TagsToThing.objects.bulk_create(tags_for_create, ignore_conflicts=True) # in case the tag is already assigned
for_update = []
tags_for_create = []
# Save remaining objects
Thing.objects.bulk_update(for_update, ['stuff', 'other_stuff'], chunk_size)
TagsToThing.objects.bulk_create(tags_for_create, ignore_conflicts=True) # in case the tag is already assigned
The main issue is with me trying to use the append function, and getting an SQL integrity error when I call session.commit(), even when the object is not being re-created. Here's my case:
I have a one to many relationship in a Flask SQLAlchemy database.
It's a Tag-Category relationship, tags belong to only one category, and by definition a category has many tags. For the sake of explanation, tags are being used on X objects.
In my Flask models file, I did this:
class Category(db.Model):
__tablename__ = 'category'
id = db.Column(db.Integer, primary_key = True)
category = db.Column(db.Text, unique = True)
tags = db.relationship('Tag', backref = 'category', lazy = 'dynamic')
class Tag(db.Model):
__tablename__ = 'tags'
id = db.Column(db.Integer, primary_key = True)
tag = db.Column(db.Text, unique = True)
category_id = db.Column(db.Integer, db.ForeignKey('category.id'))
This should define a Category class to which I can "append" tags using the session.append(Tag) function.
Categories and tags are unique, I wanna take input from the user, and add the category/tag as the user specifies. In some cases, the user might want to just add a tag to an existing category.
The way I'm taking input is by using a form, from which I receive the category and tag, and I create the objects and add them to the database if they don't exist already, and if they do, I just want to append the tags to the X's or to the categories.
Here's my code:
category = Category(category = request.form['categories']) #Category related to its respective list of tags
tag_list = request.form.getlist('tags') # tags are received as a list
for tag_entity in tag_list:
tag = Tag(tag = tag_entity)
X.tags.append(tag)
# Checking if the category exists or not in the database
list1 = []
if len(db.session.query(Category.tags).filter(Category.category == request.form['categories']).all()) != 0:
list1 = db.session.query(Category.tags).filter(Category.category == request.form['categories']).all()
list1.append(tag)
for entry in list1:
print entry.tag
list1 = [entry.tag for entry in list1]
# Here I should check if the tag already exists as well:
db.session.add(tag)
# Only add category if it doesn't already exist
if len(db.session.query(Category.id).filter(Category.category == request.form['categories']).all()) == 0:
print "I'm here"
db.session.add(Category(category = request.form['categories'], tags = list1))
else:
print "woops im here"
newCat = Category(category=request.form['categories'])
newCat.tags.extend(list1)
db.session.merge(newCat)
db.session.add(X)
db.session.commit()
I know my code is really messy and it looks like it doesn't do what it's supposed to, that's a result of me just failing over and over again.
My main issue was with me trying to call category.tags.append(Tag), then getting an integrity error when session.commit() was being called even when the category was not even re-created.
I hope what I wrote makes sense, any help would be appreciated.
I have the following code and i'm trying to update an embedded document in a listfield.
store = store_service.get_store_from_product_id(product_id)
got_product, idx = get_product_from_store(store, product_id)
product = Product()
product.pid = got_product.pid
product.display_name = display_name
product.description = description
product.rank = rank
product.price = price
product.categories = categories
product.properties = properties
store.catalog.products[idx] = product
print store.catalog.products[idx].__unicode__()
store.save()
When I print out my product, it has the correct values, but when I save it, its not persisting. There are no errors being thrown. Any thoughts one what I could be doing wrong?
store.catalog.products[idx] = product can be applied for DictField(). For ListField(). You can try:
store.catalog.products = [product]
or
store.catalog.products.append(product)
And you need to call save on the object:
store.save()
There is the possibility of atomic updates which can help in other cases:
Store.objects(id='123400000').update_one(push__catalog__products=product)
I'm trying to retrieve a list of tags that share the same name as a django Country. (i will be throwing it into my autocomplete search). What I have isn't working:
View:
from django_countries.countries import COUNTRIES
...
#login_required
def country_tags(request):
result = {}
tags = Tags.objects.all()
countries = list(COUNTRIES)
for tag in tags:
for country in countries:
if country.name == tag.name:
result[tag.name] = tag.name.title()
return HttpResponse(json.dumps(result))
Can't quite figure out why this isn't working. Am I wrong to reference country.name?
Here is a version that should work. COUNTRIES is a 2-tuple tuple.
countries_only = [x[1] for x in COUNTRIES]
tags = Tag.objects.filter(tag.name__in=countries_only)
results = {}
for t in tags:
results[t.name] = t.name.title()
COUNTRIES is just a list of 2 elements tuples - there is no name property. You should do something like country[1] == tag.name.