I have a django model to which I would like to add a UUIDField :
uuid = models.UUIDField(
default = uuid.uuid4,
unique = True,
editable = False
)
I'd like this uuid to be unique, however at migration of an existing database, I have the following error :
django.db.utils.IntegrityError: (1062, "Duplicate entry '3ba46cb5f7f340ffa43e348cb789901a' for key 'carbon.uuid'")
How can I provide to default an other uuid if this one is already used ?
You have lot of problems in here, there are 3 steps to achieve your goal.
If you have data already in your model you have to define it as null=True and remove the default=uuid.uuid4. Why? Because if you put a default value, it will try to add the same default address to all the models when you run your migration so it will of course give you the error django.db.utils.IntegrityError: (1062, "Duplicate entry '3ba46cb5f7f340ffa43e348cb789901a' for key 'carbon.uuid'") because use the same uuid for each row.
Create a migration to update your existing rows in the model and put them a uuid as following:
phython src/manage.py makemigrations app --name set_uuid_mymodel --empty
# Generated by Django A.B on YYYY-MM-DD HH:MM
from django.db import migrations
import uuid
def gen_uuid(apps, schema_editor):
MyModel = apps.get_model('myapp', 'MyModel')
for row in MyModel.objects.all():
row.uuid = uuid.uuid4()
row.save(update_fields=['uuid'])
class Migration(migrations.Migration):
dependencies = [
('myapp', '0004_add_uuid_field'),
]
operations = [
# omit reverse_code=... if you don't want the migration to be reversible.
migrations.RunPython(gen_uuid, reverse_code=migrations.RunPython.noop),
]
Look at the official documentation: https://docs.djangoproject.com/en/3.0/howto/writing-migrations/#migrations-that-add-unique-fields
Then you can manage new rows generation uuid in the save method of the model:
def save(self, *args, **kwargs):
''' On save, update uuid and ovoid clash with existing code '''
if not self.uuid:
self.uuid = uuid.uuid4()
return super(YourModel, self).save(*args, **kwargs)
To add a new required field (null=False) for a model, you should include a default value. This value will be applied to all existing records.
For UUIDField, during migrations, Django does not create a new UUID value for each record. A single value is generated through uuid.uuid4() and applied to all records using a single SQL. So, knowing that unique=True is applied at database-level, if you have existing records on your database, during migration an IntegrityError will be raised for this reason.
If you need the existing records with a valid uuid value (you cannot recreate the entire model migration with the uuid field or just drop the existing records before run the migration) is possible force a new value for each record using a RunPython operation inside your migration.
Edit your migration to do the following steps:
Add a nullable uuid field, without unique and default settings, so Django can create the column without integrity errors.
Add a custom Python code to force the uuid for existing records
Alter the uuid field, recovering the original settings
import uuid
from django.db import migrations
def force_uuid(apps, schema_editor):
# We get the model from the versioned app registry;
# if we directly import it, it'll be the wrong version
MyModel = apps.get_model('myapp', 'MyModel')
db_alias = schema_editor.connection.alias
for instance in MyModel.objects.using(db_alias).all():
instance.uuid = uuid.uuid4()
instance.save()
class Migration(migrations.Migration):
dependencies = []
operations = [
# Pre generate the uuid field (nullable)
migrations.AddField(
model_name='mymodel',
name='uuid',
field=models.UUIDField(null=True, unique=False, editable=False),
),
# Force uuid on existing records
migrations.RunPython(force_uuid),
# Recovering original settings
migrations.AlterField(
model_name='mymodel',
name='uuid',
field=models.UUIDField(
default=uuid.uuid4, null=False, unique=True, editable=False
),
),
]
Notes
If you have a lot of records to generate a unique uuid, you can benefit from bulk_create() instead loop over all records and adds a lot of single inserts
Related
I've modified the foreign key calendar as nullable in my Django model CalendarAssign. \
# ---------------------------------------------------------------------------- #
class Calendars(models.Model):
id = models.CharField(primary_key=True, max_length=100)
cms_id = models.CharField(max_length=100)
default_program = models.ForeignKey(ControlPrograms, models.CASCADE, blank=True, null=True)
timestamp = models.DateTimeField(auto_now_add=True)
class Meta:
managed = True
db_table = 'calendars'
# ---------------------------------------------------------------------------- #
class CalendarAssign(models.Model):
device_mac = models.ForeignKey(Device, models.CASCADE)
calendar = models.ForeignKey(Calendars, models.CASCADE, null=True)
timestamp = models.DateTimeField(auto_now_add=True)
class Meta:
managed = True
db_table = 'calendar_assign'
When applying the migration generated by Django it gives me an error.
operations = [
migrations.AlterField(
model_name='calendarassign',
name='calendar',
field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.CASCADE, to='smartbridge.Calendars'),
)
Generated sql code uses unsupported feature 'WITH ORDINALITY'.
It's because Django doesn't support the Postrges version we are using.
WITH ORDINALITY appears in psql 9.4 but we use version 9.1.
Both Postgres and Django cannot be upgraded right now. So I need to write the migration manually (without 'WITH ORDINALITY' feature).
migrations.RunSQL("DO $$DECLARE r record;\
BEGIN\
FOR r IN SELECT table_name,constraint_name \
FROM information_schema.constraint_table_usage \
WHERE table_name IN ('calendars') AND constraint_name like '%calendar_assign_calendar_id%'\
LOOP\
EXECUTE 'ALTER TABLE calendar_assign DROP CONSTRAINT '|| quote_ident(r.constraint_name) || ';';\
END LOOP;\
ALTER TABLE calendar_assign ALTER COLUMN calendar_id DROP NOT NULL; \
ALTER TABLE calendar_assign \
ADD CONSTRAINT calendar_assign_calendar_id_fk_calendars_id FOREIGN KEY (calendar_id) REFERENCES calendars(id);\
END$$;")
Migration seems to work fine.
calendar is now nullable but Django still detect some difference.
If a ask Django to generate the migration corresponding to the difference it generates the same as before my manual migration.
I would like Django to see no difference after my migration.
Thanks
I think you will have to set managed = False for the time being, otherwise the makemigrations command will each time think it has not been made nullable yet.
The migration construction command looks to the previous migration files, and thus constructs a conceptual model how a table will look like in the database if all the previous migrations took place. Based on that model it will look for differences with your Django model that you constructed, and thus create a migration file for that.
As long as you thus do not migrate with the AlterField command, Django will think you did not make the field nullable. It can not parse SQL so even if you made it nullable over there, it will still assume that that the field is non-NULLable.
By setting it to managed=False [Django-doc], Django will no longer manage the migrations of that file. You can create an empty migration [Django-doc] with:
python3 manage.py makemigrations --empty
and use this to define SQL queries to perform on the table.
I want to add a new HASH column to my existing django table. I referred How to generate HASH for Django model. Here is how I am doing it:
def _create_uuid():
return uuid.uuid4().hex[:20]
class users(models.Model):
user_id = models.CharField(max_length=100, primary_key=True)
uuid= models.CharField(max_length=20, null=False, default=_create_uuid)
While this works completely fine for all the new users created after the addition of the uuid column. But all the existing users get assigned with the same uuid in the uuid column when I migrate. I want the existing users to have unique uuid as well. How can I do this?
You could run a Python script during the migration to generate the hashes. Generate a migrations file first with an empty default value. Then add a RunPython function and alter the field to use your _create_uuid default function for new values.
The migration script could look like this:
from django.db import migrations, models
import YOURPROJECT.YOURAPP.models
import uuid
def generate_hash(apps, schema_editor):
User = apps.get_model("YOURAPP", "users")
users = User.objects.all()
for u in users:
u.uuid = uuid.uuid4().hex[:20]
u.save()
def reverse_generate_hash(apps, schema_editor):
pass
class Migration(migrations.Migration):
dependencies = [
('YOUTAPP', 'PREVIOUS_MIGRATIONS_FILE'),
]
operations = [
migrations.AddField(
model_name='users',
name='uuid',
field=models.CharField(default='', max_length=20),
),
migrations.RunPython(generate_hash, reverse_generate_hash),
migrations.AlterField(
model_name='users',
name='uuid',
field=models.CharField(default=jYOURPROJECT.YOURAPP.models._create_uuid, max_length=20),
),
]
I've the following Django model:
class Apple(models.Model):
text = models.TextField()
I've already many records, and I'd like to add a subject field to the model, so it'll look like:
class Apple(models.Model):
text = models.TextField()
subject = models.CharField(max_length = 128)
. In this case I run a makemigrations, but since subject can be empty, I need to set a default value either in the model, or in the migration file.
What would be the correct procedure if I'd like to take the subject from the text for the already existing database lines (for instance: text[:64])?
My solution would be to create a migration with a default value, run a management command to update the values, and with a new migration remove the default value for the subject. Is there a better solution? What is it? Can I somehow combine / do this in the migration itself?
Python: 3.4.5
Django: 1.9.2
For some databases including postgresql, it can be quicker to add a nullable field, therefore I would change your approach to:
schema migration creates the field with null=True (no need to set a default)
data migration populates the field
schema migration removes null=True from field
You can combine the three operations in one migration file. However the Django docs for data migrations recommend that you keep them separate.
You can do it in migration itself, create a migration file with blank=True, null=True in subject field.
class Apple(models.Model):
text = models.TextField()
subject = models.CharField(max_length=128, blank=True, null=True)
Then create another empty migration file.
python manage.py makemigrations --empty yourappname
Paste below code in that file.
from django.db import migrations
def set_subject(apps, schema_editor):
Apple = apps.get_model('yourappname', 'Apple')
for a in Apple.objects.all():
a.subject = a.text
a.save()
class Migration(migrations.Migration):
dependencies = [
('yourappname', 'name of above migration file'),
]
operations = [
migrations.RunPython(set_subject),
]
I have django 1.10 project. There are I have a model Feedback:
class Feedback(FirstMixin, SecondMixin, models.Model):
company = models.OneToOneField(
verbose_name='Company',
to=Company,
related_name='feedback'
)
This model exists and DB table's column Company is filled by keys to Company's items.
Now I need to add some new field to the model:
custom_name = models.CharField(
verbose_name='Company Custom Name',
null=False,
max_length=settings.DATABASE_STRING_LENGTH
)
This field should store custom names of Companies.
What should I do to make values of this field the same as relevant Companies names during migration?
Should I change migration's code or is there are some way to define it in model?
Yes you want to change a migration file that are created.
Try to use following solution
from django.db import migrations, models
from django.db.models import F
def migrate_custome_name(apps, schema_editor):
Feedback = apps.get_model("app_name","Feedback")
Feedback.objects.all().update(
custom_name=F('company'))
class Migration(migrations.Migration):
dependencies = [
------
]
operations = [
migrations.AddField(
model_name='feedback',
name='custom_name',
-- your code --
),
migrations.RunPython(migrate_custome_name), # Add this function to migrate data
]
Hope this will help you.
You can use a data migration, see the Django docs here: https://docs.djangoproject.com/en/1.10/topics/migrations/#data-migrations . You will need to run the operation that sets Feedback.custom_name = Feedback.company after applying the changes to the table.
It is stated in the Django documentation that each model requires a Primary Key. In my MySQL schema it is specified that for my table the primary key is made up of multiple columns. Using inspectdb to auto-generate the model creates the unique_together Meta information.
class Connections(models.Model):
router = models.ForeignKey('Routers', models.CASCADE)
src_mac = models.CharField(db_column='src_MAC', max_length=17) # Field name made lowercase.
src_ip = models.CharField(max_length=45, blank=True, null=True)
src_port = models.CharField(max_length=45)
dest_mac = models.CharField(db_column='dest_MAC', max_length=17, blank=True, null=True) # Field name made lowercase.
dest_ip = models.CharField(max_length=45)
dest_port = models.CharField(max_length=45)
first_activity = models.DateTimeField(blank=True, null=True)
last_activity = models.DateTimeField(blank=True, null=True)
hits_counter = models.IntegerField(blank=True, null=True)
def __unicode__(self):
return self.src_ip
class Meta:
db_table = 'Connections'
unique_together = (('router', 'src_mac', 'src_port', 'dest_ip', 'dest_port'),)
When I go to run
> python manage.py shell
>>> Connections.objects.all()
it gives me the OperationalError: (1054, "Unknown column 'Connections.id' in 'field list'") to my understanding this means that Django is trying to auto-create the primary key column for Connections.
How do I get around this?
tl;dr: Recreate the table or add the "id" field to the database manually.
I don't believe it's possible to create a primary key in all databases.
For example when using SQLite, if I create a simplified version of your table, altering it to add a PK returns an error:
> CREATE TABLE connections(src_MAC CHAR(17) NOT NULL, first_activity DATETIME NULL);
> INSERT INTO connections VALUES ('abcd1234',date('now'));
> ALTER TABLE connections add column id integer NOT NULL PRIMARY KEY AUTOINCREMENT;
Error: Cannot add a PRIMARY KEY column
Similarly, running a Django migration to create the field with SQLite simply fails to add the field silently (possibly a Django issue).
If you're unlucky and use a database that does not support adding a primary key (if you're not sure, check by running the ALTER command), your best bet would be to dump the table data, drop the table, create a new table with a primary key and finally reload the table data. This seems also to be the safest way to do it generally.
On the other hand if you do use a database that supports adding a primary key, you can either try to make migrations or alter the database manually.
Creating the id field manually in Postgres is as simple as:
ALTER TABLE connections ADD COLUMN id SERIAL PRIMARY KEY;
This here is sufficient to make the Model usable in Django. Now for the hard way.
It seems that makemigrations assumes an ID field already exists, and thus if you opt to use migrations, the initial migration would need a fix (remove the id field). You can create the CreateField migration manually thus:
migrations/0002_connections_id.py
# -*- coding: utf-8 -*-
# Generated by Django 1.9.7 on 2016-07-08 08:56
from __future__ import unicode_literals
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('testapp', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='connections',
name='id',
field=models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
),
]
Unfortunately, this still doesn't seem to be enough, since the created column does not auto-increment (possibly another Django issue). The solution now would be to alter the created field like so:
> ALTER TABLE connections ALTER id SET default nextval('connections_id_seq');
Finally the values make sense to Django and you can now query and create "Connections" instances as usual.
Just in case anyone needs this later, I've pushed the code I used to test this to this GitHub repo.