I want to keep my unit test database completely separate from other environments including using different user credentials. This is mostly to prevent anyone from unintentionally running unit tests against the development database and mangling the dev data or wiping it out entirely if the --keepdb option isn't specified. The code below detects the "test" in the sys args and this seems to work but is very clunky. If I'm missing a better way to do this please advise.
I have separate settings files for each environment so this will only be on the development server where the unit tests are run automatically and won't end up on any production servers.
Environment:
Django 1.11
Python 3.4.x
MariaDB
# this works but is clunky
import sys
if 'test' in sys.argv:
DATABASES = { # test db and user
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'dev_db_test',
'USER': 'test_user',
'PASSWORD': 'secretpassword',
'HOST': 'the-db-host',
'PORT': '3306',
'TEST': { # redundant but explicit!
'NAME':'dev_db_test',
},
}
}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'dev_db',
'USER': 'dev_db_user',
'PASSWORD': 'dev_password',
'HOST': 'the-db-host',
'PORT': '3306',
'TEST': {
'NAME':'dev_db_test', # redundant but explicit!
},
}
}
I'd like to do this but unfortunately Django doesn't look at the TEST credentials
# cleaner approach but doesn't work - don't do this!
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'dev_db',
'USER': 'dev_db_user',
'PASSWORD': 'dev_password',
'HOST': 'the-db-host',
'PORT': '3306',
'TEST': {
'NAME':'dev_db_test', # Django uses the test db NAME
'USER':'test_user_ignored', # but ignores the USER and PASSWORD
'PASSWORD':'ignoredpassword',
},
}
}
Would something like this work for your situation?
import sys
if 'test' in sys.argv:
NAME = 'dev_db_test'
USER = 'test_user'
PASSWORD ='secretpassword'
else:
NAME = 'dev_db'
USER = 'dev_db_user'
PASSWORD ='dev_password'
DATABASES ={ # test db and user
'default':
{
'ENGINE': 'django.db.backends.mysql',
'NAME': NAME,
'USER': USER,
'PASSWORD': PASSWORD,
'HOST': 'the-db-host',
'PORT': '3306',
'TEST':
{ # redundant but explicit!
'NAME':'dev_db_test',
},
}
}
print(DATABASES)
AFAIK, you don't need to create a separate test database if you want to run unit tests over it. Here is the documentation link of test database.
It states that:
Tests that require a database (namely, model tests) will not use your “real” (production) database.
Separate, blank databases are created for the tests.
You can follow this link to write your tests and it will not affect your production or development database.
Related
Question:
How can I add multiple databases for testing in Django?
When I ran my test suits I got this Error:
AssertionError: Database queries to 'mig' are not allowed in this test. Add 'mig' to path_to_test_suit.MigrationServiceTest.databases to ensure proper test isolation and silence this failure.
Here are my PostgreSQL settings:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'db_1_name',
'USER': 'db_1_user',
'PASSWORD': 'db_1_passwd',
'HOST': 'db_1_host',
'PORT': 'db_1_port',
},
'mig': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': db_2_name,
'USER': db_2_user,
'PASSWORD': 'db_2_passwd',
'HOST': 'db_2_host',
'PORT': 'db_2_port',
},
}
I use Django-nose for running test suits and use Django 2.2
You could use the database parameter in your tests.
For example:
class TestYourClass(TestCase):
databases = {'default', 'mig'}
def test_some_method(self):
call_some_method()
For more information please see Multi-database support.
P.S. In earliest Django version than 2.2 please use multi_db = True
class TestYourClass(TestCase):
multi_db = True
You need to grant DJANGO user the priviledge to write.
Then, to make the tests and preserve your data, you have to make a copy of your schema with test_ as prefix and use the keywork --keepdb.
I have two databases that my Django application needs access. One is a shared database owned by a separate app with the Django app only having read access. The second is entirely owned by the Django app.
For local development I am ok but I'm not sure how to configure things so that Heroku uses the second database.
Currently I have the shared database promoted to DATABASE_URL and the secondary database is at HEROKU_POSTGRESQL_BLUE_URL.
In my settings I have:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'main_database_name',
'USER': 'username',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '5432',
}, 'secondary': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'secondary_database_name',
'USER': 'username',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '5432',
}
}
Please ask any more questions if you need me to clarify.
Thanks!
In summary, my specific problem is: I don't know how to have Heroku use the HEROKU_POSTGRESQL_BLUE_URL as the "secondary" database.
---Edit----
At the bottom of settings.py:
# Configure Django App for Heroku.
import django_heroku
django_heroku.settings(locals())
This is where the connection is made between my app's default database and Heroku's DATABASE_URL. I still haven't solved the issue but after some troubleshooting help in the comments, I believe the answer will be found in there.
Here is my working solution.
I have a local_settings.py file not tracked by version control. This contains the database settings for the developer's postgres instance.
local_settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'main_database_name',
'USER': 'username',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '5432',
}, 'secondary': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'secondary_database_name',
'USER': 'username',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '5432',
}
}
Then in settings.py I try to import the local_settings.py file. If it doesn't exist, I use the django_heroku package to configure everything for Heroku. In my case, I need to pass the keyword argument databases=False since I want to do a more custom configuration for the databases.
settings.py (only the relevant parts)
# Database
# https://docs.djangoproject.com/en/2.0/ref/settings/#databases
# These database settings are used for heroku deployments. They are overwritten with local_settings.py in development.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ.get('DATABASE_URL'),
'USER': '',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '5432',
}, 'secondary': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ.get('SECONDARY_DATABASE_URL'),
'USER': '',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '5432',
}
}
try:
from local_settings import *
except ImportError as e:
# Configure Django App for Heroku.
import django_heroku
django_heroku.settings(locals(), databases=False)
In order for this to work you would need to set the Heroku env variable SECONDARY_DATABASE_NAME to the url of your database.
To find out the URL of your secondary database run heroku config -a your_app_name. Copy the url and then set a new environment param in heroku by running
heroku config:set SECONDARY_DATABASE_URL=postgres://xxxxxxx -a your_app_Name
There are a lot of choices with how you would handle the ENV variables, I liked this because I can just set it in my staging and production environments and the same settings work for both.
In my project I am using multiple databases, 3 to be precise.
One default and two reference databases (another host), which are powered and updated by our data team. I am using 4 tables from 2 external databases, which were inspected with ./manage.py inspectdb. They are set to managed=False
Everything works fine, my application works as expected and planned.
Now the fun begins, I need to write tests. I do not want to delete my databases.
I need to read from them, I do not want to create fixtures.
Every time when I try to use pytest or custom django tests, it is trying to delete my databases, recreate, run migrations and stuff.
How can I preserve that ? Either in pytest or django tests ? I've tried to --re-usedb, and couple of another params.
This is my database settings (parts which can be uploaded)
DATABASES = {
'default': {
'ENGINE': 'django.contrib.gis.db.backends.postgis',
'NAME': 'default_db',
'USER': 'default_usr',
'PASSWORD': 'default_pass',
'HOST': 'postgres.somehost.com',
'SUPPORTS_TRANSACTIONS': True,
},
'geolocations': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'OPTIONS': {
'options': '-c search_path=some_schema'
},
'NAME': 'geo_name',
'USER': 'geo_user',
'PASSWORD': 'geo_password',
'HOST': 'some-host.domain.com',
'SUPPORTS_TRANSACTIONS': True,
},
'eoc': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'OPTIONS': {
'options': '-c search_path=another_schema'
},
'NAME': 'eco_name',
'USER': 'eoc_user',
'PASSWORD': 'eoc_',
'HOST': 'some-another-host.domain.com',
'SUPPORTS_TRANSACTIONS': True,
}
}
So inside of pytest.ini you should be able to add a '--no-migrations' flag in order to ensure that migrations are not being made. Using a pytest marker pytest documentation should enable you to have access to your database.
The mark you need is:
pytestmark = pytest.mark.django_db
Does anyone kno how to use a specific database for a single test in Django?
EDIT: At the moment, i'm using sqlite for testing, but, i want to use a mysql database for a specific test (def test_benchmark...), which will benchmark some sql statements.
Thanks
I don't know of any builtin way to do it - but I didn't look for it neither honestly. One possible (dirty) hack is to check what's in sys.argv in your settings (which is just another Python module) and have an alternate database settings for your particular test, ie:
# settings.py
import os, sys
DATABASES = {
'default': {
'NAME': 'db_name',
'ENGINE': 'your.engine',
'HOST': 'server',
'USER': 'django',
'PASSWORD': 'django',
}
}
if "test" in sys.argv and "your-test-name" in sys.argv:
DATABASES = {
'default': {
'NAME': 'other_db_name',
'ENGINE': 'your.engine',
'HOST': 'server',
'USER': 'django',
'PASSWORD': 'django',
}
}
In Django you set the database in your settings.py file. The configuration should look something like this:
DATABASES = {
'default': {
'NAME': 'db_name',
'ENGINE': 'sql_server.pyodbc',
'HOST': 'server',
'USER': 'django',
'PASSWORD': 'django',
'OPTIONS' : { "use_mars" : False,
"provider" : "SQLNCLI10",
"extra_params" : "MARS Connection=True"},
}
}
I want to have a single settings.py file that will behave differently when running the application
./manage.py runserver
and when testing
./manage.py test myapp
So, I can change the test db to sqlite for example, with something like:
if IS_TESTING:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': 'test_db',
}
}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': DATABASE_NAME,
'USER': DATABASE_USER,
'PASSWORD': DATABASE_PASS,
'HOST': 'localhost',
'PORT': '5432',
}
}
I can get this behaviour by changing the manage.py script like this:
if __name__ == "__main__":
os.environ.setdefault('IS_TESTING', 'false')
print 'off'
if sys.argv[1] == 'test':
print 'on'
os.environ['IS_TESTING'] = 'true'
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "frespo.settings")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
But I think this is still not good enough, because when I run the tests inside an IDE (PyCharm) it won't use my custom manage.py file. There has to be a variable for this already inside Django. Do you know where it is?
If you need this condition just for Django unit test purposes, the following line in the settings.py file should work:
if 'test' in sys.argv:
DATABASES['default']['ENGINE'] = 'django.db.backends.sqlite3'
SOUTH_TESTS_MIGRATE = False # if you're using south
This assumes the other standard DATABASES setting is always declared anyway. The above line simply sets the database to sqlite in case of a unit test run.
Afaik Django does not have such a variable. Assuming you can edit the exact command run by PyCharm, you could just add the test argument there, and then look for it in the settings.py file.
if "test-argument" in sys.argv:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': 'test_db',
}
}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': DATABASE_NAME,
'USER': DATABASE_USER,
'PASSWORD': DATABASE_PASS,
'HOST': 'localhost',
'PORT': '5432',
}
}