Develop Reference

Database config for Django testing

I am building an app using Django and Postgres. I managed to do migrations and I want to test it. When I test with sqlite everything works fine, but when I run tests with postgres I'm getting this error:
Creating test database for alias 'default'...
Got an error creating the test database: permission denied to create database
I've checked user's permissions and I'm sure that this user have permission to create database.
My database config looks like this:
# DATABASES = {
# 'default': {
# 'ENGINE': 'django.db.backends.sqlite3',
# 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
# }
# }
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': '***',
'USER': '***',
'PASSWORD': '***',
'HOST': '****',
'PORT': '****',
}
}
My postgres db is on a server.
My questions are:
What is the right way to config my db and run tests?
Should I be using sqlite for testing?
If so how my code should look like, so I don't have to comment configs?

It looks like your DB user doesn't have permission to create a new database. Please, take a look here. This command-line utility allows you to create a user and set their permissions.
Example:
createuser my_user --createdb -W --username postgres
Note: you are creating user "my_user" on behalf of PostgreSQL admin role which is postgres by default.
Answering your questions:
You may have several configs for different stages, e.g development, testing, production.
You could use both SQLite and Postgres databases for testing purposes to some extent. You should be awarded, though, if your app relies on some specific features available only in Postgres, then using SQLite for testing doesn't make sense. I personally prefer using the same database for all stages. You could also use docker if you don't want to install DB server on your machine.

Use multiple databases in Django with only one table "django_migrations"

For a project in Django I have to use two databases: default and remote. I have created routers.py and everything works fine.
There was a requirement to create a table on the remote database and I created migration, run it and the table django_migrations was created. I want to have only one table django_migrations, in the default database.
The relevant part of routers.py is here:
class MyRouter(object):
# ...
def allow_migrate(self, db, app_label, model_name=None, **hints):
if app_label == 'my_app':
return db == 'remote'
return None
I run the migration like this:
python manage.py migrate my_app --database=remote
Now when I do:
python manage.py runserver
I get the following warning:
You have 1 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): my_app.
Run 'python manage.py migrate' to apply them.
The tables for my_app are created in the remote database, and in django_migrations inside the remote database the migrations are marked as applied.
EDIT:
How to force Django to use only one table django_migrations, but still apply the migrations into different databases?
How to apply the migrations in different databases so that no warnings are raised?

Thanks to the comments on my question I did some research and came up with the following findings.
Using multiple databases results in creating a table django_migrationswhen migrations are used. There is no option to record the migrations in only one table django_migrations, as the comment from Kamil Niski explains. This is clear after reading the file django/db/migrations/recorder.py.
I will illustrate an example with a project foo and an app bar inside the project. The app bar has only one model Baz.
We create the project:
django-admin startproject foo
Now we have these contents inside the main project directory:
- foo
- manage.py
I have a habit to group all apps inside the project directory:
mkdir foo/bar
python manage.py bar foo/bar
In the file foo/settings.py we adjust the settings to use two different databases, for the purposes of this example we use sqlite3:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db1.sqlite3'),
},
'remote': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db2.sqlite3'),
}
}
Now we run the migrations:
python manage.py migrate --database=default
This runs all migrations, the part --database=default is optional, because if not specified Django uses the default database.
Operations to perform:
Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying sessions.0001_initial... OK
Django has applied all migrations to the default database:
1 contenttypes 0001_initial 2019-11-13 16:51:04.767382
2 auth 0001_initial 2019-11-13 16:51:04.792245
3 admin 0001_initial 2019-11-13 16:51:04.827454
4 admin 0002_logentr 2019-11-13 16:51:04.846627
5 admin 0003_logentr 2019-11-13 16:51:04.864458
6 contenttypes 0002_remove_ 2019-11-13 16:51:04.892220
7 auth 0002_alter_p 2019-11-13 16:51:04.906449
8 auth 0003_alter_u 2019-11-13 16:51:04.923902
9 auth 0004_alter_u 2019-11-13 16:51:04.941707
10 auth 0005_alter_u 2019-11-13 16:51:04.958371
11 auth 0006_require 2019-11-13 16:51:04.965527
12 auth 0007_alter_v 2019-11-13 16:51:04.981532
13 auth 0008_alter_u 2019-11-13 16:51:05.004149
14 auth 0009_alter_u 2019-11-13 16:51:05.019705
15 auth 0010_alter_g 2019-11-13 16:51:05.037023
16 auth 0011_update_ 2019-11-13 16:51:05.054449
17 sessions 0001_initial 2019-11-13 16:51:05.063868
Now we create the model Baz:
models.py:
from django.db import models
class Baz(models.Model):
name = models.CharField(max_length=255, unique=True)
register the app bar into INSTALLED_APPS (foo/settings.py) and create the migrations:
python manage.py makemigrations bar
Before we run the migrations we create routers.py inside the bar app:
class BarRouter(object):
def db_for_read(self, model, **hints):
if model._meta.app_label == 'bar':
return 'remote'
return None
def db_for_write(self, model, **hints):
if model._meta.app_label == 'bar':
return 'remote'
return None
def allow_relation(self, obj1, obj2, **hints):
return None
def allow_migrate(self, db, app_label, model_name=None, **hints):
if app_label == 'bar':
return db == 'remote'
if db == 'remote':
return False
return None
and register it in foo/settings.py:
DATABASE_ROUTERS = ['foo.bar.routers.BarRouter']
Now the naive approach would be to run the migrations for bar into the remote database:
python manage.py migrate bar --database=remote
Operations to perform:
Apply all migrations: bar
Running migrations:
Applying bar.0001_initial... OK
The migrations has been applied to the remote database:
1 bar 0001_initial 2019-11-13 17:32:39.701784
When we run:
python manage.py runserver
the following warning will be raised:
You have 1 unapplied migration(s). Your project may not work properly
until you apply the migrations for app(s): bar.
Run 'python manage.py migrate' to apply them.
Everything seems to work fine though. However it isn't satisfying having this warning.
The proper way would be to run all migrations for each database as suggested in this answer.
It would look like this:
python manage.py migrate --database=default
python manage.py migrate --database=remote
and after creating the migrations for bar:
python manage.py migrate bar --database=default
python manage.py migrate bar --database=remote
The router will take care that the table bar_baz is created only in the remote database, but Django will mark the migrations as applied in both databases. Also the tables for auth, admin, sessions, etc. will be created only in the default database, as specified in routers.py. The table django_migrations in the remote database will have records for these migrations too.
It is a long reading, but I hope it sheds some light on this, in my opinion, not thoroughly explained issue in the official documentation.

django postgresql tests.py RuntimeWarning error

I have a Django app that adds and displays stuff to my postgresql database online at elephantsql.com. My project file setup looks like this:
website/
website/
music/
playlist/
__pycache__/
migrations/
static/
templates/
__init__.py
admin.py
apps.py
models.py
tests.py
urls.py
views.py
.coverage
db.sqlite3
manage.py
My project works right now where when I run the server and go to /playlist/ it displays stuff correctly and connects to my postgresql database fine.
My settings.py DATABASES object looks like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'kxvmghva',
'USER': 'kxvmghva',
'PASSWORD': '--an actual password---',
'HOST': 'raja.db.elephantsql.com',
'PORT': '5432',
}
}
Now I'm trying to write test cases in my playlist/tests.py file, but when I try to run these tests I'm getting errors.
my testing file I'm trying to run /playlist/tests.py
from django.test import TestCase
from .models import plays
from django.utils import timezone
class AnimalTestCase(TestCase):
def setUp(self):
print("setup")
#Animal.objects.create(name="lion", sound="roar")
#Animal.objects.create(name="cat", sound="meow")
def test_animals_can_speak(self):
"""Animals that can speak are correctly identified"""
print("test")
#lion = Animal.objects.get(name="lion")
#cat = Animal.objects.get(name="cat")
#self.assertEqual(lion.speak(), 'The lion says "roar"')
#self.assertEqual(cat.speak(), 'The cat says "meow"')
When I run the command "python manage.py test playlist" I get these errors:
C:\Users\marti\Documents\kexp\website>python manage.py test playlist
Creating test database for alias 'default'...
C:\Python36-32\lib\site-packages\django\db\backends\postgresql\base.py:267: RuntimeWarning: Normally Django will use a connection to the 'postgres' database to avoid running initialization queries against the production database when it's not needed (for example, when running tests). Django was unable to create a connection to the 'postgres' database and will use the default database instead.
RuntimeWarning
Got an error creating the test database: permission denied to create database
Type 'yes' if you would like to try deleting the test database 'test_kxvmghva', or 'no' to cancel:
if I type 'yes' it leads to this error:
Destroying old test database for alias 'default'...
Got an error recreating the test database: database "test_kxvmghva" does not exist
I've been trying to solve this error by searching it online and have tried stuff like giving my user 'kxvmghva' CREATEDB permissions as well as running this line in my elephantsql db:
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO kxvmghva;
But I'm still getting these errors when trying to run the tests.py file for my playlist/ app. This is my first time setting up test cases for a postgresql database in django and I would really appreciate any help or guidance. Thanks.

I suppose the plan you are using for database is free/shared, in case which you do not have rights to create additional databases.
This is not a Django problem but restrictions of the service you are using.

As pointed in the first answer, ElephantSQL free tier (I suppose you're using this one) is shared. In that server, they have several DBs, so you have access to one and can't create another one.
I've tried to create an instance just for tests, but it also fails because "On PostgreSQL, USER will also need read access to the built-in postgres database." (from django docs).
So, my solution, and understand it is not the best of practices (because you'll have a different environment for testing) is replacing the engine for testing:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
},
}
This configuration works because (from the django docs):
When using SQLite, the tests will use an in-memory database by default (i.e., the database will be created in memory, bypassing the filesystem entirely!)
Two notes:
Use GitHub actions or a similar solution to test against PostgreSQL in each commit (this way, you reduce the danger of using different test and prod databases)
The easiest way to have a different setting config is to use separate files. I follow the logic in cookiecutter-django

Got an error creating the test database: Django unittest

When I run python manage.py test, django is asking a strange question each time:
$ python manage.py test
Creating test database for alias 'default'...
Got an error creating the test database: (1044, "Access denied for user 'nyble'#'localhost' to database 'test_nybledb'")
Type 'yes' if you would like to try deleting the test database 'test_nybledb', or 'no' to cancel:
I was expecting it to just delete and remake a basic sqlite3 DB, and I don't want this behavior.
Whether I say yes or no it just exits tests:
Destroying old test database for alias 'default'...
Got an error recreating the test database: (1044, "Access denied for user 'nyble'#'localhost' to database 'test_nybledb'")
In settings I have
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'nybledb',
'USER': 'nyble',
'PASSWORD': 'password',
'HOST': 'localhost', # Or an IP Address that your DB is hosted on
'PORT': '3306',
}
}
How can I stop this and make it use normal sqlite3 db during running tests?

You can set this up in settings.py. It says that if there is 'test' in the command line input, use sqlite as db engine.
import sys
if 'test' in sys.argv or 'test_coverage' in sys.argv: #Covers regular testing and django-coverage
DATABASES['default']['ENGINE'] = 'django.db.backends.sqlite3'

Adding LOGGING field to settings.py causes "Not Found: /" every page load

c:\Dev>django-admin startproject mysite
Add the following (copied from https://docs.djangoproject.com/en/1.8/topics/logging/) to c:\Dev\mysite\mysite\settings.py:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': 'c:/Dev/mysite/debug.log',
},
},
'loggers': {
'django.request': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
},
}
Migrate, run the dev server, and visit http://127.0.0.1:8000/ in the browser:
c:\Dev>cd mysite
c:\Dev\mysite>python manage.py migrate
Operations to perform:
Synchronize unmigrated apps: staticfiles, messages
Apply all migrations: admin, contenttypes, auth, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying sessions.0001_initial... OK
c:\Dev\mysite>python manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
September 21, 2015 - 15:52:25
Django version 1.8.4, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
Not Found: /
[21/Sep/2015 15:52:27] "GET / HTTP/1.1" 200 1767
Why am I getting Not Found: / every page load?
The log file at c:\Dev\mysite\debug.log contains Not Found: / as well.
Removing the LOGGING lines from settings.py resolves the issue.
The standard "It worked!" page itself loads fine with a proper GET 200.

The "Not Found: /" is being logged by code in django.core.handlers.base.py (in the BaseHandler.get_response method). You wouldn't see if you didn't configure logging, but will see it if you do configure logging and don't have a handler for the '/' URL in your urlconfs.
The 200 you see in the log output is because Django handled the 404 case with a default response.