I have a table in a django app where one of the fields is called Order (as in sort order) and is an integer. Every time a new record is entered the field auto increments itself to the next number. My issue is when a record is deleted I would like the other records to shift a number up and cant find anything that would recalculate all the records in the table and shift them a number up if a record is deleted.
For instance there are 5 records in the table where order numbers are 1, 2, 3, 4, and 5. Someone deleted record number 2 and now I would like numbers 3, 4, and 5 to move up to take the deleted number 2's place so the order numbers would now be 1, 2, 3, and 4. Is it possible with python, postgres and django?
Thanks in Advance!
You are going to have to implement that feature yourself, I doubt very much that a relational db will do that for you, and for good reason: it means updating a potentially large number of rows when one row is deleted.
Are you sure you need this? It could become expensive.
Here what I ended up using:
item.delete()
items = table.objects.order_by('order')
count =0
for element in items:
element.order = count
element.save()
count=count+1
You're probably better off leaving the values in the table alone and using a query to generate the numbering. You can use window functions to do this if you're up to writing some SQL.
SELECT
output_column,
...,
row_number() over (
order by
order_column)
FROM
TheTable;
Instead of deleting orders - you should create a field which is a boolean (call it whatever you like - for example, deleted) and set this field to 1 for "deleted" orders.
Messing with a serial field (which is what your auto-increment field is called in postgres) will lead to problems later; especially if you have foreign keys and relationships with tables.
Not only will it impact your database server's performance; it also will impact on your business as eventually you will have two orders floating around that have the same order number; even though you have "deleted" one from the database, the order number may already be referenced somewhere else - like in a receipt your printed for your customer.
You could try using signals post_save and post_delete to query the appropriate objects, sort them, and then look for missing numbers and reassign/save as necessary. This might be pretty heavy for a lot of data, but for only a few items that change rarely, it would be ok.
from django.db.models.signals import post_delete
from django.dispatch import receiver
def fix_order(sorted_objects):
#ensures that the given objects have sequential order values from 1 upwards
i = 1
for item in sorted_objects
if item.order != i:
item.order = i
item.save()
i += 1
#receiver(post_delete, sender=YourSortedModel)
def update_order_post_delete(sender, kwargs):
#get the sorted items you need
sort_items = YourSortedModel.objects.filter(....).order_by('order')
fix_order(sort_items)
I came across this looking for something else and wanted to point something out:
By storing the order in a field in the same table as your data, you lose data integrity, or if you index it things will get very complicated if you hit a conflict. In other words, it's very easy to have a bug (or something else) give you two 3's, a missing 4, and other weird things can happen. I inherited a project with a manual sort order that was critical to the application (there were other issues as well) and this was constantly an issue, with just 200-300 items.
The right way to handle a manual sort order is to have a separate table to manage it and sort with a join. This way your Order table will have exactly 10 entries with just it's PK (the order number) and a foreign key relationship to the ID of the items you want to sort. Deleted items just won't have a reference anymore.
You can continue to sort on delete similar to how you're doing it now, you'll just be updating the Order model's FK to list instead of iterating through and re-writing all your items. Much more efficient.
This will scale up to millions of manually sorted items easily. But rather than using auto-incremented ints, you would want to give each item a random order id in between the two items you want to place it between and keep plenty of space (few hundred thousand should do it) so you can arbitrarily re-sort them.
I see you mentioned that you've only got 10 rows here, but designing your architecture to scale well the first time, as a practice, will save you headaches down the road, and once you're in the habit of it, it won't really take you any more time.
Try to set the value with type sequence in postgres using pgadmin.
Related
I am working on designing a relational database for a meal scheduler web application.
I have it 99% set up, but I am wondering if to use a separate table for the "meal type" entries.
To sum it up, users can add their own meal type(breakfast, snack, dinner) arbitrarily, in any order, and I am currently storing them in a simple list (ordered with javascript in the frontend for convenience).
It won't have more than half a dozen elements at worst (who even plans more than 6 meals a day anyway), so I am saving it all in the database's settings table, which contains rows as key:value pairs.
In this case, it's 'meals': [json string representing the python list]
The problem is that every scheduled recipe needs to be qualified by meal type.
id_scheduled_meal
id_recipe
meal_type
Right now, I'd have to use the exact string saved in the key:value pair in order to associate it with a specific meal type, so meal_type would be "Breakfast" or "Snack", rather than an id. It feels like too much redundant data.
At the same time, I am not sure it would be good to create a separate object (Meal) with a separate table (meal), only to add 4-6 entries and 1-3 columns (id, name, position).
Any suggestions? I am happy to clarify, I realize the explanation might not be as clear as it could.
Thanks in advance
I feel like this is pretty opinion based, and the answer will depend on how you want to interact with the data. If you plan on writing queries that include the meal type, then you might save yourself some pain and just do the extra table, though managing/saving items will be more complex. If it's just a list that you plan on doing everything with in python (or whatever), then serialising a list and saving the text might be the better choice. Whether the extra redundant space will adversely affect you will depend on your application and requirements.
Using Python and the built-in Sqlite:
When I use a prepared statement, not knowing what columns the user will want to update:
UPDATE_AUTHOR = """UPDATE lastName=?, firstName=?, age=?, nationality=? FROM authors
WHERE _id = ?
"""
How can I replace the '?' with some value that will keep the current value for some of the columns?
From the user perspective, I will tell him for example to press 'Enter' if he wishes to keep the current value. So for instance, he presses 'Enter' on lastName, updates firstName, updates age, and presses 'Enter' on nationality. And I replace his 'Enter' with, hopefully, a value that will keep the current value.
Is that possible? If not, how can I solve this problem differently but efficiently?
I thought about building the prepared statement dynamically, in the above example: adding firstName=?, and age=?, after "UPDATE, and then the rest of the statement FROM authors WHERE _id = ?". But this seems less comfortable and less organized.
There are 2 ways of handling this question. One is to build a specific UPDATE query containing only the fields that will change. As you have said it is less comfortable because the query and the parameter list have to be tweaked.
Another way it to consistently update all the parameters, but keep the saved values for those which should not change. This is a common design in user interfaces:
the user is presented all the values for an object and can change some of them
if they confirm their choice, the application retrieves all the values, either changed or not and uses them in an UPDATE query.
Anyway, it is common the read the all the values before changing some, so it is not necessarily expensive. And at the database level, changing one or more values in an update has generally almost the same cost: a record is loaded from disk (or cache), some values are updated which is the cheapest operation, and it is then written back to disk. Even with the database caches, the most expensive part in the databases I know is to load and save the record.
What is the fastest way to query one record from database that is satisfy my filter query.
mydb.objects.filter(start__gte='2017-1-1', status='yes').order_by('?')[:1]
This statement will first query thousands of records and then select one, and it is very slow, but I only need one, a random one. what is the fastest one to get?
Well, I'm not sure you will be able to do exactly what you want. I was running into a similar issue a few months ago and I ended up redesigning my implementation of my backend to make it work.
Essentially, you want to make the query time shorter by having it choose a random record that fulfills both requirements (start__gte='2017-1-1', status='yes'), but like you say in order for the query to do so, it needs to filter your entire database. This means that you can cannot get a "true" random record from the database that also fulfills the filter requirements, because filtering inherently needs to look through all of your records (otherwise it wouldn't be really random, it would just be the first one it finds that fulfills your requirements).
Instead, consider putting all records that have a status='yes' in a separate relation, so that you can pull a random record from there and join with the larger relation. That would make the query time considerably faster (and it's the type of solution I implemented to get my code to work).
If you really want a random record with the correct filter information, you might need to employ some convoluted means.
You could use a custom manager in Django to have it find only one random record, something like this:
class UsersManager(models.Manager):
def random(self):
count = self.aggregate(count=Count('id'))['count']
random_index = randint(0, count - 1)
return self.all()[random_index]
class User(models.Model):
objects = UsersManager()
#Your fields here (whatever they are, it seems start__gte and status are some)!
objects = UserManager()
Which you can invoke then just by using:
User.objects.random()
This could be repeated with a check in your code until it returns a random record that fulfills your requirements. I don't think this is necessarily the cleanest or programmatically correct way of implementing this, but I don't think a faster solution exists for your specific issue.
I used this site as a source for this answer, and it has a lot more solid information about using this custom random method! You'll likely have to change the custom manager to serve your own needs, but if you add the random() method to your existing custom manager it should be able to do what you need of it!
Hope it helps!
Using order_by('?') will cause you a great performance issue. A better way is to use something like this: Getting a random row from a relational database.
count = mydb.objects.filter(start__gte='2017-1-1', status='yes').aggregate(count=Count('id'))['count']
random_index = randint(0, count - 1)
result= mydb.objects.filter(start__gte='2017-1-1', status='yes')[random_index]
I want to save Entries in my database such that I can delete them later. To identify them, i put the key attribute in my class Entry in models.py as shown below:
class Entry(models.Model):
name = models.CharField(max_length=200)
key = models.IntegerField(unique=True,default=0)
Every time I start the server, I will find the biggest key that is in Entry.objects.all().
Every time I create the first new Entry after starting the server, I want to take the key I found in the very beginning, increase it by 1 and then set this as the key for my new Entry. For every subsequent Entry, I will just take the key of the previous element, increase it by 1 and. set it as the key for my new Entry
However, before I do this, I want to know what Django considers as unique. For example, if i added three entries initially with keys 1, 2 and 3, and then I delete the last element with key 3. If I then restart the server, the biggest key I will find is 2 and the next Entry I will add will have key 2+1 which is equal to 3 again.
Is this key unique?
It was entered before but I deleted that element right? So is uniqueness determined by whatever I entered in the past or just depending on the elements currently in the database?
Also, does Django keep a track of all the instances of Entry that I added, so that it would somehow know if I added an Entry with key 3 before?
Note that my data is stored in a file called db.sqlite3
Thank you.
Seems like you are looking for something that already exists; models have an id field by default which is unique and monotonic (newer entries have bigger id).
None of this has anything to do with Django at all. This is pure database stuff: unique constraints are handled exclusively by the database, in your case SQLite, although the functionality is exactly the same for other DBs as well.
Of course, a unique constraint only takes into account rows that actually exist: what would be the point otherwise?
There are other things to bear in mind with your proposal as well. In particular, there is the issue of race conditions: if you do a query to get the maximum key value, and then create a new row with key+1, you run the risk of another user in the meantime adding their own row at key+1. Much better, as Iris and ryanageles suggest, to use the built-in primary key which is already automatically defined as the id field.
try to add primary_key=True in your key field.
So I have an application in which a user creates a list. The user also orders the items in the list and can add and remove items from the list. If the user logs out and then logs in from another device, the list needs to be presented in the same order it was in before.
So far I have approached this problem by just adding a field called "order" to the records in the table. Let's say I have a list of 800 items. If the user deletes item 4, I cannot simply remove the record from the table -- I also have to update 796 records to reflect the new order of those items. If the user then adds an item, I have to not only add a record to the table, I have to update every item with an order count higher than the position the new item was added.
My approach seems expensive and naive to me. Is there some clever and efficient way to approach this problem? Any ideas? Thanks.
You should implement the equivalent of a doubly linked list where each node has a pointer to its previous and next node.
Inserting a node is only updating previous/next pointers so no need to update anything else.
Removing a node is only updating previous/next pointers (by having them point to each other) so no need to update anything else.
So instead of one order field you need two fields previous and next that indicate the previous and next node in the ordered list.