I'm working on my first-ever Heroku/Django app. I just want to be sure I'm setting my DATABASE_URL and DATABASES variables correctly. Here's what's in my code:
import dj_database_url
DATABASE_URL = 'postgresql:///my_app'
# Parse database configuration from $DATABASE_URL
DATABASES = {
'default': dj_database_url.config(default=DATABASE_URL)
}
When I just have DATABASES['default'] = dj_database_url.config() and I try to use Django commands like run server or migrate I get the following error: NameError: name 'DATABASES' is not defined. I set the DATABASE_URL since doing so appears to solve this issue (after I create the my_app database).
Everything appears to be working fine as I code and test, but I've also seen a half-dozen different ways to set the database variables on the internet. If this isn't correct, I'd like to fix it now. The thing that really confuses me is, when I push my app to Heroku, how will the data get pushed to the web, when the database is /usr/local/var/postgres? Or will this not happen at all? Am I just too confused/tired at this point?
This is documented on Heroku Devecenter
# Parse database configuration from $DATABASE_URL
import dj_database_url
# DATABASES['default'] = dj_database_url.config()
#updated
DATABASES = {'default': dj_database_url.config(default='postgres://user:pass#localhost/dbname')}
If you need Database connection pooling add this bits too. More details
# Enable Connection Pooling
DATABASES['default']['ENGINE'] = 'django_postgrespool'
This is a simple matter of logic. You can't set the "default" key of the DATABASES dictionary before you have defined the dictionary itself.
Whether or not you set the default parameter to dj_database_url inside the call or as a separate DATABASE_URL variable is irrelevant, especially as that won't even be used on Heroku as it will be overridden by environment variables.
This allows you to use any database settings during development,
but at production (on Heroku), DATABASES['default'].update(db_from_env) changes the database settings to the one created by Heroku.
import dj_database_url
DATABASES = {
'"default": {
"ENGINE": "django.db.backends.sqlite3",
"NAME": os.path.join(BASE_DIR, "db.sqlite3"),
}
}
db_from_env = dj_database_url.config(conn_max_age=500)
DATABASES['default'].update(db_from_env)
You could use any database settings, but the last two lines allow heroku to create its own database for you.
Related
I have a Django app that adds and displays stuff to my postgresql database online at elephantsql.com. My project file setup looks like this:
website/
website/
music/
playlist/
__pycache__/
migrations/
static/
templates/
__init__.py
admin.py
apps.py
models.py
tests.py
urls.py
views.py
.coverage
db.sqlite3
manage.py
My project works right now where when I run the server and go to /playlist/ it displays stuff correctly and connects to my postgresql database fine.
My settings.py DATABASES object looks like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'kxvmghva',
'USER': 'kxvmghva',
'PASSWORD': '--an actual password---',
'HOST': 'raja.db.elephantsql.com',
'PORT': '5432',
}
}
Now I'm trying to write test cases in my playlist/tests.py file, but when I try to run these tests I'm getting errors.
my testing file I'm trying to run /playlist/tests.py
from django.test import TestCase
from .models import plays
from django.utils import timezone
class AnimalTestCase(TestCase):
def setUp(self):
print("setup")
#Animal.objects.create(name="lion", sound="roar")
#Animal.objects.create(name="cat", sound="meow")
def test_animals_can_speak(self):
"""Animals that can speak are correctly identified"""
print("test")
#lion = Animal.objects.get(name="lion")
#cat = Animal.objects.get(name="cat")
#self.assertEqual(lion.speak(), 'The lion says "roar"')
#self.assertEqual(cat.speak(), 'The cat says "meow"')
When I run the command "python manage.py test playlist" I get these errors:
C:\Users\marti\Documents\kexp\website>python manage.py test playlist
Creating test database for alias 'default'...
C:\Python36-32\lib\site-packages\django\db\backends\postgresql\base.py:267: RuntimeWarning: Normally Django will use a connection to the 'postgres' database to avoid running initialization queries against the production database when it's not needed (for example, when running tests). Django was unable to create a connection to the 'postgres' database and will use the default database instead.
RuntimeWarning
Got an error creating the test database: permission denied to create database
Type 'yes' if you would like to try deleting the test database 'test_kxvmghva', or 'no' to cancel:
if I type 'yes' it leads to this error:
Destroying old test database for alias 'default'...
Got an error recreating the test database: database "test_kxvmghva" does not exist
I've been trying to solve this error by searching it online and have tried stuff like giving my user 'kxvmghva' CREATEDB permissions as well as running this line in my elephantsql db:
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO kxvmghva;
But I'm still getting these errors when trying to run the tests.py file for my playlist/ app. This is my first time setting up test cases for a postgresql database in django and I would really appreciate any help or guidance. Thanks.
I suppose the plan you are using for database is free/shared, in case which you do not have rights to create additional databases.
This is not a Django problem but restrictions of the service you are using.
As pointed in the first answer, ElephantSQL free tier (I suppose you're using this one) is shared. In that server, they have several DBs, so you have access to one and can't create another one.
I've tried to create an instance just for tests, but it also fails because "On PostgreSQL, USER will also need read access to the built-in postgres database." (from django docs).
So, my solution, and understand it is not the best of practices (because you'll have a different environment for testing) is replacing the engine for testing:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
},
}
This configuration works because (from the django docs):
When using SQLite, the tests will use an in-memory database by default (i.e., the database will be created in memory, bypassing the filesystem entirely!)
Two notes:
Use GitHub actions or a similar solution to test against PostgreSQL in each commit (this way, you reduce the danger of using different test and prod databases)
The easiest way to have a different setting config is to use separate files. I follow the logic in cookiecutter-django
Some people use just one settings file for their project. A good alternative is writing modules, replacing the one settings file with a directory with configurations for each specific need:
...
|
settings
|
|-base.py
|-local.py
|-local_alex.py
|-production.py
|-staging.py
|-test.py
I guess it's normal to have one single settings file for production.
But what happens if I have more staging instances/versions?
Let's suppose I have a postgresql DB for each staging instance/environment. Is it ok to have more staging files? Or should I treat the differences between staging versions in another way?
So I could wether have two settings file, one for each staging version or use the same settings file but specify these differences in another way.
What is the recommended approach?
DjangoTwoScoops recommends having more local.py settings files, but no one mentions about several staging files.
Example, I have the production, local and test files. But two staging versions, each with an URL and different database.
DATABASES = {
'default': {
...
'NAME': 'dbname1',
'USER': 'username1',
...
}
}
and the second one:
DATABASES = {
'default': {
...
'NAME': 'dbname2',
'USER': 'username2',
...
}
}
It's a good idea to use a base settings file to define common settings which can be overridden by environment variables. So your database settings might look like;
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST': os.environ.get('DATABASE_HOST', 'localhost'),
'PORT': os.environ.get('DATABASE_PORT', '3306'),
'USER': os.environ.get('DATABASE_USER', 'root'),
'PASSWORD': os.environ.get('DATABASE_PASSWORD', ''),
'NAME': os.environ.get('DATABASE_NAME'),
},
}
The example you've provided is good practice, but I'd avoid specific files for a person, or if you can, specific local settings. Instead look to have environment variables to account for what would otherwise get it's own settings file. That way, it's less code to maintain & easily configured on each server/environment. Each environment your project runs in would define DJANGO_SETTINGS_MODULE in the environment variables to tell django which settings to use.
Use environment specific settings files to inherit the base and then include things like logging and error notifications, if you subscribe to a service like Sentry, which you wouldn't want to be integrated outside of production. So production.py might include additional security settings which you'd want when running in that environment;
DEBUG = False
CSRF_COOKIE_SECURE = True
SESSION_COOKIE_SECURE = True
SECURE_BROWSER_XSS_FILTER = True
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_HSTS_SECONDS = 31536000
SECURE_HSTS_INCLUDE_SUBDOMAINS = False
And if you find that the local settings approach works for you, exclude them from git so that a specific person's settings aren't checked in and if they've got a local database server, they don't need to worry that there password is forever in version control etc.
If you want to have those files tracked via git you don't really have a choice - you put all common settings in one tracked file (settings.py), connection settings for each environment in a different file (settings_prod.py) and then access keys in one untracked (settings_local.py).
But honestly it's not that difficult to make connections work the same way in all environments, so you can put those settings in common file and then only swap untracked file with keys for oauth, etc.
I Want to add Redis Database in Django-1.9, so i followed this documents for integration
https://niwinz.github.io/django-redis/latest/
But i didn't find any clue on how to mention Database name in the settings, Here i want to mention Redis as a database on behalf Sqlite3, If uncommented this line django is throwing an Error of DATABASE not found
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'sqlite3'),
}
}
Thanks in Advance for your solution
What's django-redis?
django-redis is a BSD Licensed, full featured Redis cache/session
backend for Django.
What's redis
Redis is an open source (BSD licensed), in-memory data structure
store, used as a database, cache and message broker
Essentially that means django-redis is a django package that allows you to replace the default memcache as django's cache backend and also allows you to replace the DB as the default session storage. However django-redis does not implement features required to use it as a replacement for sqlite3 or any other database.
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": "redis://127.0.0.1:6379/0",
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient"
}
}
}
To use the redis database in django you need to add this code to your settings file,based on your requirement you can change the value of the database at the end of LOCATION value like ("redis://127.0.0.1:6379/1") for database '1'.
You can also check here: https://niwinz.github.io/django-redis/latest/#_configure_as_cache_backend
By default Django provides no support for non-relational database backends. However, if you intend to use Redis as your primary database, you can take a look at Django non-rel.
In this tutorial, it never says to edit the settings.py, in the previous docs, which supported django 1.5 you needed to edit settings.py.
So do you need to edit that file or not? Did the author skip that part because it was kinda obvious?
I actually don't see the name settings.py in the new docs, and there aren't too many mongo db django tutorials on the web. And the questions here, if any are outdated. Thereby I'm sorry if this turns out to be a naive question.
And if you want to use pymongo, AFAIK you don't connect from settings.py so I just had to ask.
Personally, I like to have all my database configs in settings, so I have mongo database configuration in settings.py along with my relational database configs:
MONGO_DBS = {
'default': {
'alias': 'default', # the alias that Documents refer to
'name': 'default', # the name of the database to connect to
'host': 'localhost', # the host
'port': 27017, # the port
'username': '', # not implemented
'password': '', # not implemented
'enabled': False, # whether or not we connect to this database
},
}
Then, I have a little snippet of code that runs in settings.py (cue some grumbling) and connects to all of the relevant mongo instances:
from mongoengine import connect
import sys
if not (len(sys.argv) > 1 and sys.argv[1] == 'test'):
# Don't run this if we're running in unit tests. The test runner will spin
# up the appropriate databases and spin them down appropriately.
for db_name in MONGO_DBS:
db_meta = MONGO_DBS[db_name]
if db_meta['enabled'] and 'alias' in db_meta:
connect(db_meta['name'], alias=db_meta['alias'], host=db_meta['host'], port=db_meta['port'],
lazy_connect=db_meta.get('lazy', True))
Obviously, this code is still somewhat incomplete in so far as authentication doesn't happen. But it should be a reasonable launching point for you.
I should add that I just dug up references to settings.py in the django documentation page for mongoengine. Currently, it's located at http://docs.mongoengine.org/en/latest/django.html.
Lastly, I should add that this advice applies through mongoengine 0.8.7 (latest as of this answer). YMMV with future versions.
I have created a production_settings.py in which I am putting all the production env variables and values eg:
import dj_database_url
DATABASES['default'] = dj_database_url.config()
I thought I'd declare an env variable like
MYPROJECT_PRODUCTION
and set this like
heroku config:add MYPROJECT_PRODUCTION=True or export MYPROJECT_PRODUCTION=True
In the settings.py (which is the default created by django) ,I thought I'd add at the end of the file
import os
if os.environ.has_key('MYPROJECT_PRODUCTION') and os.environ.get('MYPROJECT_PRODUCTION')=='True':
from production_settings import *
Is this the correct way of doing this?
I am getting an import error when I try python manage shell
export DJANGO_SETTINGS_MODULE='myproject.settings'
export MYPROJECT_PRODUCTION=True
me#ubuntu:~/dev/python/django/myproject$ python manage.py shell
Error: Can't find the file 'settings.py' in the directory containing 'manage.py'. It appears you've customized things.
You'll have to run django-admin.py, passing it your settings module.
(If the file settings.py does indeed exist, it's causing an ImportError somehow.)
the manage.py exists in the same folder as settings.py ..still the error occurs.
I checked echo $MYPROJECT_PRODUCTION which outputs True
Personally, I keep my production settings in settings.py then include a local_settings.py file (that's excluded from revision control with .hgignore).
I add the following to the end of settings.py
try:
from local_settings import *
except ImportError, e:
pass
Then in my local_settings.py I override the appropriate settings -
DEBUG = True
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'tag',
'USER': 'tag',
'PASSWORD': 'tag',
'HOST': 'localhost',
'PORT': '5432',
}
}
I'm under the impression that this is a fairly well used method (I picked it up from colleagues but I've seen it blogged about too)
EDIT
In response to balazs very good point you might include a variation on this method, in order to keep sensitive data private. Perhaps include the following after the local_settings import -
try:
from production_settings import *
except ImportError, e:
pass
Then exclude production_settings.py from version control. I guess you may need to use a different method to deploy production_settings.py, but I don't suppose that's too big a deal.
I advise against using different settings files for different environments in favour of customising your settings with environment variables. This allows you by default to use your local development settings and override it in production.
For example database and static / media root settings
# Default database URL is a local SQLite instance
DATABASE_URL = 'sqlite:///%s' % os.path.join(os.path.dirname(__file__), 'db.sqlite')
DATABASES = {
'default': dj_database_url.config('DATABASE_URL', default=DATABASE_URL),
}
MEDIA_ROOT = os.environ.get('MEDIA_ROOT',
os.path.join(os.path.dirname(__file__), 'media'))
MEDIA_URL = os.environ.get('MEDIA_URL', '/media/')
STATIC_ROOT = os.environ.get('STATIC_ROOT',
os.path.join(os.path.dirname(__file__), 'static'))
STATIC_URL = os.environ.get('STATIC_URL', '/static/')
This allows you to set any setting on Heroku via heroku config:set and removes the complexity of dealing with multiple settings files, for example:
heroku config:set MEDIA_URL=http://custom-media-server.com/media/
heroku config:set STATIC_URL=http://custom-media-server.com/static/
I have also created a custom Django project template that can take most settings from environment variables.
You should also look at the 12 Factor Application website and specifically at how to store configuration.
Have you defined DATABASES as a dictionary? by:
DATABASES = {}
also show your heroku logs