If one is using Django, what happens with changes made directly to the database (in my case postgres) through either pgadmin or psql?
How are such changes handled by migrations? Do they take precedence over what the ORM thinks the state of affairs is, or does Django override them and impose it's own sense of change history?
Finally, how are any of these issues effected, or avoided, by git, if at all?
Thanks.
You can exclude a model completely from the django migrations, and then you are responsible to adjust the schema to the django code (or the django code to the existing schema):
class SomeModel(models.Model):
class Meta:
managed = False
db_table = "some_table_name"
name = models.Fields....
Note that you can't have it both ways, so migrations are preferred when possible. You can always define a custom SQL migration, that will save the need for external changes. However, sometimes you do need to handle the schema elsewhere instead of migrations, and then use managed=False
The migrations system does not look at your current schema at all. It builds up its picture from the graph of previous migrations and the current state of models.py. That means that if you make changes to the schema from outside this system, it will be out of sync; if you then make the equivalent change in models.py and create migrations, when you run them you will probably get an error.
For that reason, you should avoid doing this. If it's done already, you could apply the conflicting migration in fake mode, which simply marks it as done without actually running the code against the database. But it's simpler to do everything via migrations in the first place.
git has no impact on this at all, other than to reiterate that migrations are code, and should be added to your git repo.
Related
I have a big Django application, currently trying to upgrade from 1.6 to 1.7 (there's been attempts to upgrade straight to 1.11, but it was too much trouble, so my plan is to do it one minor at a time).
I'm following the Upgrade from South instructions, and deleted all previous migrations, but I can't get makemigrations to work. The current problem is that the auth.User model has been patched to include two new fields:
User.add_to_class('profile',
models.ForeignKey('access.Profile', null=True, blank=True,
related_name='user_foreignkey'))
User.add_to_class('profiles',
models.ManyToManyField('access.Profile', null=True,
blank=True))
This patch was made in a separate app. If I just leave it where it is, I get the following error when running python manage.py makemigrations:
ValueError: Lookup failed for model referenced by field auth.User.profiles:
access.Profiles
I tried moving the add_to_class calls to the same file where Profile is defined, (after the definition), but got the same error. I also tried changing the syntax from 'access.Profile' to Profile, to no effect. Is there something else that could make this work?
If not, since I'm adding fields to the model, I figure the correct approach would be extend the AbstractUser model, as suggested by this guide. The problem with this is that the new initial migration will create a table access_user instead of using the existing auth_user. Would it be safe to simply rename auth_user to access_user and fake the migration?
Any other suggestions on how to overcome this with the least refactoring possible (management always thinks there are more urgent things than the upgrade) are welcome.
tl, dr; I removed auth migrations and ran makemigrations again, that seems to have solved the problem.
I had decided to give option 2 a try. I removed the add_to_class patches, created a new class User(AbstractUser) with the new fields, removed all the existing migrations and ran makemigrations again. I got the exact same error, which made no sense because there was no longer any code relating auth.User to Profile, so I decided to investigate auth migrations. I ran
$ python migrate --list settings=settings.mysettings
and it showed a suspicious migration for auth, with yesterday's date, probably when I first tried to make migrations.
Tutorials on "how to reset all migrations" do not mention third-party installed apps, but thanks to this thread I knew where Django stored the auth migrations, so I removed both of them (the initial and the suspicious one) and reordered the INSTALLED_APPS so that access is now before django.contrib.auth. Then ran makemigrations again, and it got through access and auth migrations successfully.
I'm now getting other problems with migrations in other apps, but they don't seem to be related with this.
It might be nice to have a more intuitive way to reset all existing migrations, not only those from your repo. But then again, I don't think the add_to_class patches are the proper way to do what we wanted.
I am working on an existing Django project that contains migration code like:
someFluffyModel.objects.all().delete()
or
someModel = SomeModel(....)
someModel.save()
Now wherever there is such code and newer migrations change the schema that reflect the current version of the model, there is an issue in applying migrations from scratch.
As I understand the reason is that the model used in the migration doesn't reflect the model used in the migration at that point in time. As I have found fixtures can help in loading data but how about deletion?
Is the preferred way to manually delete data from the database?
Sorry forgot to answer my own old question it's been years now but just in case anybody is curious in your migrations you should always be using historical models. The reason is that you get the model at a point in time(reconstructed incrementally base on schema migrations).
def forwards_func(apps, schema_editor):
# We get the model from the versioned app registry;
# if we directly import it, it'll be the wrong version
Country = apps.get_model("myapp", "Country")
This way if you try to re-run your migrations at some point in the future with model schema x'' old migrations will be run using x' and x depending on their dependencies.
Avoid importing models to your migrations directly at any cost.
It is always important to be able to rer-run you migrations since that allows you to migrate forwards backwards between your environments roll back faulty deployments and above all run your tests. If you are doing it wrong your tests will fail at some point when you'll get to schema x' and you'll have to follow the correct approach mentioned above to fix them.
Thanks to #Daniel Roseman for giving me some pointers that day.
I am having trouble running the django migrations on a new database because of the get_or_create() function in a form.
form.py
class EmployeeForm(...):
# Make sure that we have a directory before initializing the form
default_upload_directory = Directory.objects.get_or_create(name='Employee', parent=None)[0]
This is the important part of a custom form I wrote. It will link the Employee's uploaded files to a default directory. Everything works fine except for the migration. When I run it on a new database it will complain with django.db.utils.OperationalError: no such table: filemanager_directory. It also happens if I use a standard objects.get(..) to access the Directory object.
I am not sure how to solve this. Obviously the table does not exist, yet. On an existing database with the tables already in place everything works fine.
EDIT: The migrations I am trying to run are not related to this app. Furthermore, as a workaround, you can comment out the line, run the migrations and reactivate it... But this is not a neat solution...
This is, quite simply, not a thing you should do. You shouldn't run any kind of database access in any code that runs at import time, which this does.
There should never be a need to do this in any case. If your goal is to ensure an object exists when the form is instantiated, put it in the __init__ method.
But you should also reflect that ensuring that database objects exist is the job of migrations in the first place. Write a data migration with a RunPython method to do this.
I'm running into quite interesting situation.
I need to extend default django's Group model with some fields.
I tried to use inheritance first, e.g. inherit from Group model and change some references, but seems I can't change all needed references, so, this way completely breaks django permission system.
Then I found this answer: How do I extend the Django Group model? where guy suggested to use field.contribute_to_class() method.
I have put this adjustment right above the model definition in < myapp >. (don't ask me why do I need roles for group, it's not my idea, I just need them :D)
if not hasattr(Group, 'roles'):
field = models.ManyToManyField(
Role, verbose_name=_('roles'), blank=True,
help_text=_('List of roles attached to this group'),
related_name='groups')
field.contribute_to_class(Group, 'roles')
class MyGroup(Group):
class Meta:
proxy = True
def _sync_permissions(self):
"""
This method will sync group permissions with all attached Roles.
"""
self.permissions.clear()
for role in self.roles.all():
self.permissions.add(role.permissions)
self.save()
This part seems to be working (it really modifies django.contrib.auth.models.Group model)
But what I need next is to generate a migration for the Group model.
If I simply run ./manage.py makemigrations <myapp> it generates a migration for Group model, but tries to put it inside django.contrib.auth application, that is definitely not what I need.
So, my question here is:
Is there a way to tell django to generate a migration for Group model, but not to create a migration file under python libs directory, but rather create it inside < myapp > or just output the migration code?
the location where django looks for migrations cn be custominzed using MIGRATION_MODULES in your settings.py, anyway this means that ALL the migrations (not only the new) must be there.
You need to copy the original migrations and manually update them when you upgrade Django
You can create a dedicated package so to not clash with your migrations
Es.
MIGRATION_MODULES = {
'django.contrib.auth' : 'myapp.auth_migrations',
'myapp': 'myapp.migrations' # this line is only to clarify. IT'S NOT NEEDED AT ALL
}
I have not seen the field.contribute_to_class() to class solution to this problem before.
For me it looks a bit dirty, like monkey patching.
You would go back and use a 1:1 relation to Group, or inherit from Group.
You say that you can't change all needed references to Group? What is missing?
I know this is not an answer to your question, but maybe a solution to your problem :-)
I am trying to add two additional profile fields and have the native authentication work like normal.
I am trying to fallow the documentation here
and the SO here
In my settings file
#settings.py
AUTH_USER_MODEL = 'users.User'
in my users.user model
#users/models.py
from django.contrib.auth.models import AbstractUser
from django.db import models
class User(AbstractUser):
foo = models.CharField(max_length=32, default='Blue')
bar = models.CharField(max_length=32, default='Blue')
print "user.user"
i have created 3 superusers non can log into admin. i have tried syncing the DB after adding a user. i have tried restating the dev server between adding a user.
the only time i see the output of print "user.user" is when i run the createsuperuser command.
i think i cant log in because the user is not really being created. it runs my User class and then skips actually creating the user. but i am kinda new to this so i could be way off and way out of my league.
why cant i log in and how do i add the two fields?
Have you read the warning in Django's documentation?
Changing AUTH_USER_MODEL has a big effect on your database structure. It changes the tables that are available, and it will affect the construction of foreign keys and many-to-many relationships. If you intend to set AUTH_USER_MODEL, you should set it before running manage.py syncdb for the first time.
If you have an existing project and you want to migrate to using a custom User model, you may need to look into using a migration tool like South to ease the transition.
Given this warning, are you working on a fresh database, or are you migrating using South? If you have an existing database and made these changes, then simply running syncdb will most likely no be sufficient.
If this is a development server without any important data, I would recreate your database, and then run ./manage.py syncdb. If you are using a SQLite database, then you can simply copy it to somewhere else (if you would like to keep the data), and run syncdb again to create a new database.
Here is the relevant documentation.
It would also be helpful to know exactly what error you are receiving. Do you attempt to login and admin tells you that your user/pass combination is not correct, or is there an actual error thrown? Your question doesn't quite make this clear.