As in this question, I set up a dumpdata-based backup system for my database. The setup is akin to running a cron script that calls dumpdata and moves the backup to a remote server, with the aim of simply using loaddata to recover the database. However, I'm not sure this plays well with migrations. loaddata now has an ignorenonexistent switch to deal with deleted models/fields, but it is not able to resolve cases where columns were added with one-off defaults or apply RunPython code.
The way I see it, there are two sub-problems to address:
Tag each dumpdata output file with the current version of each app
Splice the fixtures into the migration path
I'm stumped about how to tackle the first problem without introducing a ton of overhead. Would it be enough to save an extra file per backup that contained an {app_name: migration_number} mapping?
The second problem I think is easier once the first one is solved, since the process is roughly:
Create a new database
Run migrations forward to the appropriate point for each app
Call loaddata with the given fixture file
Run the rest of the migrations
There's some code in this question (linked from the bug report) that I think could be adapted for this purpose.
Since these are fairly regular/large snapshots of the database, I don't want to keep them as data migrations cluttering up the migrations directory.
I am taking the following steps to backup, restore or transfer my postgresql database between any instance of my project:
The idea is to keep the least possible migrations as if manage.py makemigrations was run for the first time on an empty database.
Let's assume that we have a working database to our development environment. This database is a current copy of the production database that should not be open to any changes. We have added models, altered attributes etc and those actions have generated additional migrations.
Now the database is ready to be migrated to production which -as stated before- is not open to public so it is not altered in any way. In order to achieve this:
I perform the normal procedure in the development environment.
I copy the project to the production environment.
I perform the normal procedure in the production environment
We make the changes in our development environment. No changes should happen in the production database because they will be overridden.
Normal Procedure
Before anything else, I have a backup of the project directory (which includes a requirements.txt file), a backup of the database and -of course- git is a friend of mine.
I take a dumpdata backup in case I need it. However, dumpdata has some serious limitations regarding content types, permissions or other cases where a natural foreignkey should be used:
./manage.py dumpdata --exclude auth.permission --exclude contenttypes --exclude admin.LogEntry --exclude sessions --indent 2 > db.json
I take a pg_dump backup to use:
pg_dump -U $user -Fc $database --exclude-table=django_migrations > path/to/backup-dir/db.dump
Only if I want to merge existing migrations in one, I delete all migrations from every application.
In my case the migrations folder is a symlink, so I use the following script:
#!/bin/bash
for dir in $(find -L -name "migrations")
do
rm -Rf $dir/*
done
I delete and recreate the database:
For example, a bash script can include the following commands:
su -l postgres -c "PGPASSWORD=$password psql -c 'drop database $database ;'"
su -l postgres -c "createdb --owner $username $database"
su -l postgres -c "PGPASSWORD=$password psql $database -U $username -c 'CREATE EXTENSION $extension ;'"
I restore the database from the dump:
pg_restore -Fc -U $username -d $database path/to/backup-dir/db.dump
If migrations were deleted in step 3, I recreate them in the following way:
./manage.py makemigrations <app1> <app2> ... <appn>
... by using the following script:
#!/bin/bash
apps=()
for app in $(find ./ -maxdepth 1 -type d ! -path "./<project-folder> ! -path "./.*" ! -path "./")
do
apps+=(${app#??})
done
all_apps=$(printf "%s " "${apps[#]}")
./manage.py makemigrations $all_apps
I migrate using a fake migration:
./manage.py migrate --fake
In case something has gone completely wrong and everything is ***, (this can happen, indeed), I can use the backup to revert everything to its previous working state. If I would like to use the db.json file from step one, it goes like this:
When pg_dump or pg_restore fails
I perform the steps:
3 (delete migrations)
4 (delete and recreate the database)
6 (makemigrations)
and then:
Apply the migrations:
./manage.py migrate
Load the data from db.json:
./manage.py loaddata path/to/db.json
Then I try to find out why my previous effort was not successful.
When the steps are performed successfully, I copy the project to the server and perform the same ones to that box.
This way, I always keep the least number of migrations and I am able to use pg_dump and pg_restore to any box that shares the same project.
Related
I need to perform a migration using orator (postgres), but when I change my migration file and run the command to perform it, it shows me the message "Nothing to migrate"
orator migrate -p "./src/migrations/2022_03_10_142913_create_systems_table.py" -c "./src/core/utils/database/Orator.py"
My migration code
De orator.py (Infos censured)
I upload my first Django-project into DigitalOcean. After command python manage.py loaddata initial_data.json, I have received this message:
django.db.utils.IntegrityError: Problem installing fixture
'/webapps/django_shop/shop/initial_data.json': Could not load
contenttypes.ContentType(pk=3): duplicate key value violates unique
constraint "django_content_type_app_label_76bd3d3b_uniq" DETAIL: Key
(app_label, model)=(auth, permission) already exists.
How can I fix it?
I had the same problem and I solved this way
DB with data to export from
python manage.py dumpdata --exclude auth.permission --exclude contenttypes > db.json
New DB to import to
python manage.py flush
// Important! Disable all signals on models pre_save and post_save
python manage.py loaddata db.json
// Do not forget to enable all signals that you disabled
It looks like you've generated fixtures that include Django's default data set, i.e. the built-in entries that are inserted normally as part of the first migrate run for some of Django's plumbing data types.
You should review your fixture process, because content type entries will be created automatically when your (and Django's) apps' migrations are run, so they should not be present in fixtures. It's possible there are other tables that will have this same problem, so now would be a good time to make sure you're not including any other data that would result in this situation.
I am running Django unit tests against a multithreaded app. Often a thread hasn't terminated by the time the unit test finishes, so the test database cannot be deleted. When I next run the tests, I get the message:
Type 'yes' if you would like to try deleting the test database 'test_appname', or 'no' to cancel`
The create_test_db autoclobber option is the functionality I want, but how can I use that? I can't find any examples or clues. I'm working in the PyCharm IDE, which is pretty configurable. I just want to delete the test database silently every time.
I'm putting tests in Transaction TestCase classes, running setup_test_environment() then Client().post(reverse(etc..))..
In Pycharm django tests, you can enable the option input and enter --noinput
see screenshot below
If you're using PyCharm and want to run a single test using the green arrow but you keep getting this error, you can modify the default django tests configuration template, so that you don't have to keep setting the --noinput option on each.
If you are using flush in some other way, (eg, on the pre-existing development database like here), the --noinput option supresses the user prompt, eg:
from django.core.management import call_command
call_command('flush', '--noinput')
My answer was to create a script like this:
export PGPASSWORD=the_password
if [[ `psql -h 127.0.0.1 -d postgres -U username -p 5432 -tAc "SELECT 1 FROM pg_database WHERE datname='test_djangoprojectname'"` == "1" ]]
then
psql -h 127.0.0.1 -d postgres -U username -p 5432 -a -w -c "SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'test_djangoprojectname' AND pid <> pg_backend_pid();"
psql -h 127.0.0.1 -d postgres -U username -p 5432 -a -w -c "DROP DATABASE test_djangoprojectname;"
fi
The -d setting is database name - it can be any database your user has access to, except the one you are deleting.
The default username is postgres.
The -p setting is the port your database is on - 5432 is the default.
Save as (for example) del_test_db.sh (Windows users see below), then
chmod +x del_test_db.sh
Then in PyCharm:
Run, Edit Configurations...
Unfold Defaults, click Django tests
In the Before launch window click +, External tools, click +
Under Program, select your file del_test_db.sh, give the command a name (eg, 'del test db') and click OK.
Select your tool in the list and click OK
You may need to unfold Django tests in the left and delete existing test configurations
Then the script force deletes the test database before every run.
This works on Mac OS X and Ubuntu etc. For Windows the process is the same, except instead of export use SET, save the commands as a .bat file instead of .sh, and you don't need to chmod +x, and use the following syntax for the IF statement in the batch file:
if 'command' == '1' (
...
)
Apologies I'm unable to check this as I don't have a Windows machine.
Thanks to this answer for the code checking whether the database exists.
Trying to create a super user for my database:
manage.py createsuperuser
Getting a sad recursive message:
Superuser creation skipped due to not running in a TTY. You can run manage.py createsuperuser in your project to create one manually.
Seriously Django? Seriously?
The only information I found for this was the one listed above but it didn't work:
Unable to create superuser in django due to not working in TTY
And this other one here, which is basically the same:
Can't Create Super User Django
If you run $ python manage.py createsuperuser
Superuser creation skipped due to not running in a TTY. You can run manage.py createsuperuser in your project to create one manually. from Git Bash and face the above error message try to append winpty i.e. for example:
$ winpty python manage.py createsuperuser
Username (leave blank to use '...'):
To be able to run python commands as usual on windows as well what I normally do is appending an alias line to the ~/.profile file i.e.
MINGW64 ~$ cat ~/.profile
alias python='winpty python'
After doing so, either source the ~/.profile file or simply restart the terminal and the initial command python manage.py createsuperuser should work as expected!
I had same problem when trying to create superuser in the docker container with command:
sudo docker exec -i <container_name> sh. Adding option -t solved the problem:
sudo docker exec -it <container_name> sh
In virtualenv, for creating super-user for Django project related to git-bash use the command:
winpty python manage.py createsuperuser.
Since Django 3.0 you can create a superuser without TTY in two ways
Way 1: Pass values and secrets as ENV in the command line
DJANGO_SUPERUSER_USERNAME=admin2 DJANGO_SUPERUSER_PASSWORD=psw \
python manage.py createsuperuser --email=admin#admin.com --noinput
Way 2: set DJANGO_SUPERUSER_PASSWORD as the environment variable
# .admin.env
DJANGO_SUPERUSER_PASSWORD=psw
# bash
source '.admin.env' && python manage.py createsuperuser --username=admin --email=admin#admin.com --noinput
The output should say: Superuser created successfully.
To create an admin username and password, you must first use the command:
python manage.py migrate
Then after use the command:
python manage.py createsuperuser
Once these steps are complete, the program will ask you to enter:
username
email
password
With the password, it will not show as you are typing so it will appear as though you are not typing, but ignore it as it will ask you to renter the password.
When you complete these steps, use the command:
python manage.py runserver
In the browser add "/admin", which will take you to the admin site, and then type in your new username and password.
Check your docker-compose.yml file and make sure your django application is labeled by web under services.
I tried creating superuser from Stash [ App: Pythonista on iOS ]
[ Make sure migrations are already made ]
$ django-admin createsuperuser
I figured out how to do so. What I did was I went to VIEWS.py. Next, I imported the module os. Then I created a function called createSuperUser(request):. Then, I then created a variable called admin and set it equal to os.system("python manage.py createsuperuser"). Then after that, return admin. Finally, I restarted the Django site, then it will prompt you in the terminal.
import os
def createSuperUser(request):
admin = os.system("python manage.py createsuperuser")
return
I want to generate a basic DB schema for my django project to display all my Apps with Models and Model Fields with boundary conditions etc. Is there already any DB schema generator for django in python? Or otherwise how should i go about doing it.
If your talking about needing to see the SQL schema, run ./manage.py sqlall <appname>
If you want a visualisation of the schema you can get django-extensions and run ./manage.py graph_models -a -g -o my_project.png. This will produce a pretty schema graph for you, but generally omits border conditions. you may want to check the options to add more data. http://readthedocs.org/docs/django-extensions/en/latest/graph_models.html
manage.py sql <appname appname ...> (docs)
Using Your DB
As mentioned in the tutorial, you can use your database's command line client to get the schema.
Example using sqlite:
python manage.py dbshell
> .schema
You may need to install sqlite3 for this to work.
Using Django
You used to be able to use python manage.py sql ..., but it has been deprecated in 1.9 in favor of migrations. You can check out the initial migration scripts using:
python manage.py sqlmigrate myapp 0001_initial
(From Answer: Equivalent of sqlall in Django 1.9?)