I have a basic django rest API. I want to separate some of my settings for dev and prod just for organizational purposes. I'm also just learning about separating environments. I've read a few things, but I can't seem to get it working the way I want it to.
The hierarchy looks like this:
- djangorest
- api
- __init__.py
- models.py
- views.py
- urls.py
- etc..
- djangorest
- __init__.py
- settings.py
- urls.py
- wsgi.py
Right now, when I run the server, I do a simple:
python3 manage.py runserver
That command reads the settings from settings.py and runs it appropriately but I've looked around on how to separate the settings into prod vs dev and it doesn't work right.
I want to be able to have:
commonsettings.py
dev.py
prod.py
In commonsettings, it'll just have whatever's in both dev and prod. I've tried running:
python3 manage.py runserver --settings=settings.dev
But it gives me an error saying there is no module named 'setting'.
Please help. Thank you!
For example I open SQL log in dev settings
My file structure:
create settings folder and mv original settings.py to settings/defaults.py
load defaults in init.py
# proj/proj/settings/__init__.py
from .defaults import *
Edit dev.py
# proj/proj/settings/dev.py
from .defaults import *
DEBUG = True
# print sql to the console
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'console': {
'class': 'logging.StreamHandler',
}
},
'loggers': {
'django.db.backends': {
'handlers': ['console'],
'propagate': True,
'level': 'DEBUG',
}
},
}
run in the python env
./manage.py runserver --settings proj.settings.dev
Django 2.1.7 | Mac
Create a folder called config
config/
commonsettings.py
dev.py
prod.py
make sure that in dev.py and prod.py you import everything from commonsettings.py like this:
from .commonsettings import *
then if you want to run the dev.py settings:
python manage.py runserver --settings=config.dev
if you want to run prod.py:
python manage.py runserver --settings=config.prod
NOTE:
For more readable files many developers call their settings files: local.py (for the local settings) production.py (for the production settings) and base.py (the common settings for both)
Personally I place my setting files in:
config/
settings/
base.py
local.py
production.py
test.py (For tests)
This solution works without extra arguments, and without having to replace/delete files across environments.
Make settings.py decide which file to load
Let's assume your Django project is named MyDjangoApp.
Create the config/ dir in your Django project's root dir, and one .py file per environment, e.g. like this:
config/
local.py
dev.py
prod.py
MyDjangoApp/
settings.py
There is no logical limit to the number of environments, I just added the three I usually have.
Then, inside MyDjangoApp/settings.py we can add the logic to choose which settings file to load.
"""
Django settings for MyDjangoApp project.
"""
import os
# Begin: Custom per-env settings
import socket
# Use the appropriate logic to understand "where" we're running
HOST = socket.gethostname()
configs = {
'localhost': 'local', # replace 'localhost' with whatever your machine name is
'dev.mysite.com': 'dev',
'www.mysite.com': 'production',
}
config = 'config.{}'.format(configs[HOST])
settings_module = __import__(config, globals(), locals(), ['MyDjangoApp', ])
try:
for setting in dir(settings_module):
# Only fully-uppercase variables are supposed to be settings
if setting == setting.upper():
locals()[setting] = getattr(settings_module, setting)
except Exception:
# you may ignore this exception, the print() is for debugging purposes
print(Exception)
# End: Custom per-env settings
# ... the rest of the common settings go here
Please note: this logic chooses the settings file to load based on the hostname of the server which is running the Django app. It's just an example- you can implement any other logic that works for you.
Change only what's needed
The good news is, settings.py can still hold the vast majority of your settings, and the files inside config/ just need to contain the settings that change across your environments, e.g.:
DATABASES
DEBUG
ADMINS
LOGGING
Avoid --settings in the commandline
You can set the Django settings as a shell environment variable, like this:
export DJANGO_SETTINGS_MODULE=MyDjangoApp.settings
The best place to do this is in a .sh file; usually I create a file named env.sh file in the root dir of the Django project, and run it (once) just after activating the virtualenv.
File /env.sh contents
#!/usr/bin/env bash
export DJANGO_SETTINGS_MODULE=MyDjangoApp.settings
To run it from Terminal (Bash, or git-scm Bash on Windows):
. env.sh
Then you can use ./manage.py runserver to start the Django server wherever you are.
Reference: https://code.djangoproject.com/wiki/SplitSettings#DevelopmentMachineDependantSettingsConfiguration
Put all your common configurations in the commonsettings.py file.
add following the line into the __init__.py file in the settings directory
from commonsettings import *
This makes every configuration in the commonsettings file available in all other files in the settings folder
start your server with
python3 manage.py runserver --settings=djangorest.settings.dev
Project Hierarchy will be
- djangorest
- api
- __init__.py
- models.py
- views.py
- urls.py
- etc..
- djangorest
- settings (folder)
- commonsettings.py
- dev.py
- prod.py
- __init__.py
- settings.py
- urls.py
- wsgi.py
After This run following commands before python manage.py runserver
export PYTHONPATH=$PWD
export DJANGO_SETTINGS_MODULE=djangorest.settings.dev
and Finally python manage.py runserver
Replace settings.py with dev.py on dev server and prod.py on production server.
now in init.py files write code
try:
print("Trying import production.py settings...")
from .prod import *
except ImportError:
print("Trying import development.py settings...")
from .dev import *
it find prod.py in production server and import it. On dev server it not find and prod.py and import dev.py
then you don't need to specify configuration file name while running runserver command.
What I've always done is on the local env create a file called local_settings.py that can have it's own database settings, static path and so on and at the end of the settings file include this
try:
from local_settings import *#
except:
pass
so on the server I always make sure not to import the local_settings and it separates the two nicely.
Related
I'm new to Python and Django. I'm trying to create a user authentication system with Djoser following an online tutorial. I am getting this error "no module named "auth_system"" when I try to run the server to test the connection with Postman.
Is it because of the way that I organized the folders that my "auth_system" file is not recognized?
Also, is the way that I put all the files into the virtual environment folder correct?
Have you added auth_system to installed app in settings file:
INSTALLED_APPS = [
'auth_system',
]
I will give you a quick reference for how to start a django project and review the arranged files.
Be sure that you have properly installed django.
$ python -m django --version
2)Move to the folder where you will create your project and command:
$ django-admin startproject mysite
Where mysite is the name of your project.
The created files of the project mysite will look like the following:
mysite/
manage.py
mysite/
init.py
settings.py
urls.py
asgi.py
wsgi.py
Now you can test if everything is ok
$ python manage.py runserver
July 08, 2021 - 10:28:12 Django version 3.2, using settings
'mysite.settings' Starting development server at
http://127.0.0.1:8000/ Quit the server with CONTROL-C.
4)You can add a project with the next instruction:
$ python manage.py startapp polls
The created directory polls, looks like this:
polls/
init.py
admin.py
apps.py
migrations/
init.py
models.py
tests.py
views.py
Then you can create a superuser like this:
$ python manage.py createsuperuser --username=admin --email=admin#example.com
Or normal users like this (in python prompt):
from django.contrib.auth.models import User
user = User.objects.create_user('user01', 'user01#example.com', 'user01password')
Hope this useful for you!
More information:
https://docs.djangoproject.com/en/3.2/intro/tutorial01/
I have a GitHub workflow for Django and when it gets to migrating the database it gives the error
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
the secret key is stored in a .env file and loaded with
from dotenv import load_dotenv
load_dotenv()
from pathlib import Path
env_path = Path('.') / '.env'
load_dotenv(dotenv_path=env_path)
SECRET_KEY = os.getenv("secret_key")
Here is the file tree
C:.
| db.sqlite3
| manage.py
|
\---djangosite
| .env
| asgi.py
| settings.py
| urls.py
| wsgi.py
| __init__.py
|
\---__pycache__
...
This is the manage.py, it is the regular django one with the load .env code from settings.py
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
from dotenv import load_dotenv
load_dotenv()
from pathlib import Path
env_path = Path('.') / '.env'
load_dotenv(dotenv_path=env_path)
import os
import sys
def main():
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'djangosite.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()
when I run manage.py on my PC it loads the key and runs the server, but GitHub gives the error above.
How do I stop this error from happening?
If you have stored the SECRET_KEY in your system's environment variable, then for GitHub workflow, you can add a dummy environment variable in the YAML file.
The settings.py should look like this
import os
...
SECRET_KEY = os.environ.get('SECRET_KEY') # Or the name by which you stored environment variable
...
The steps are given below:
Step 1: Generate a dummy SECRET_KEY. You can create it yourself by
import secrets
print(secrets.token_hex(25))
Or generate from a site like this.
Step 2: In your .github/workflows YAML file (e.g., django.yml), add this
steps:
...
- name: Run Tests
env:
SECRET_KEY: your-genereated-secret_key
run: |
python manage.py test
Then everything will work fine with the same version of code in your local environment, production environment, and GitHub workflow.
Adding to #PhysicistSouravDas's answer. You can alternatively do:
- name: Run Tests
env:
SECRET_KEY: ${{ secrets.SECRET_KEY }}
run: |
python manage.py test
Now, go to the settings of your GitHub repository. Under the secrets menu, click Actions and then click New Repository Secret.
Add a new secret with the name SECRET_KEY and value as the dummy SECRET_KEY generated by the method suggested by #PhysicistSouravDas.
GitHub Actions would pick up the SECRET_KEY from there.
When you run python manage.py runserver 8000 you are using manage.py which is settings your DJANGO_SETTINGS_MODULE to settings. It doesn't appear that you have a settings.py in your root directory, so this line in manage.py:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
should become:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "Your-App-Name.settings")
as you have in your wsgi.py file.
You can also check here for some other possible solutions.
I have different settings for different environments for my application, such as local, dev, stage, production. So, when I run my apps, say, locally, I pass the settings as the parameter to manage.py, e.g. python3 manage.py runserver 0.0.0.0:8080 --settings=myapp.settings.local, and indeed all my settings are correctly initialised, e.g. DEBUG is True as it is set in my settings file, and not False as it is in defaultsettings.py.
However, when I try to run tests with python3 manage.py test --settings=myapp.settings.local, the value of DEBUG is set to false, that is it is loaded from defaultsettings.py. Why does that happen and how can I fix this?
Make sure that in the test you import the settings like this:
from django.conf import settings
Turns out that when you run your tests, all your settings are just like they are in your settings file except for DEBUG. DEBUG is always set to False to make the set up as close to production as possible.
I ended up following this post: Detect django testing mode
import sys
TESTING = sys.argv[1:2] == ['test']
I located my client project folder and django project folder under the root folder separately. I configured django settings.py to contain client's app folder to serve in development and dist folder to be gathered by collectstatic
STATICFILES_DIRS = [os.path.join(BASE_DIR, "..", "Client/app"), # in development
os.path.join(BASE_DIR, "..", "Client/dist"), # to be gathered by collectstatic
]
Client/app contains original js/css/html files for development and
Client/dist contains concatenated and uglified files for production.
Because Client/app folder is only for development, I want to exclude the folder when I use collectstaic command.
However, collectstatic -i app does not exclude Client/app folder. I tried
collectstatic -i Client/app
collectstatic -i ../Client/app
collectstatic -i app*
but, nothing did work.
How can I exclude folder outside django directory?
You would not do that normally. You would define a different STATICFILES_DIR depending on what environment you run.
Very basic idea:
if DEBUG:
STATICFILES_DIRS = [os.path.join(BASE_DIR, "..", "Client/app")]
else:
STATICFILES_DIRS = [os.path.join(BASE_DIR, "..", "Client/dist")]
Instead of relying on the DEBUG setting though, I'd recommend you use a separate config file for each. You then choose which to run when you invoke Django.
For instance, assuming this file tree:
settings/
├ __init__.py # empty
├ dev.py
└ prod.py
…you'd start Django this way:
export DJANGO_SETTINGS_MODULE="settings.dev"
./manage.py runserver
To avoid repeating shared configuration, create a common.py and import it from both dev.py and prod.py, using from settings.common import * (probably the only use case where it's permissible to import *).
Though it does not technically answer your question, I think this is a cleaner approach to the wider problem of handling environment-specific configuration.
I am trying to follow the advice of the book "Two Scoops of Django" and although it is a really good book, I think it this section is unclear.
So, I split my settings file and created a folder like this:
settings/
__init__.py
base.py (allmost everything there)
local.py (dev. specific settings)
production.py (settings for Heroku)
most of the settings are in the base.py file
in local.py I have this:
# settings/local.py
from .base import *
DEBUG = True
TEMPLATE_DEBUG = DEBUG
INSTALLED_APPS += ("debug_toolbar", "django_extensions", "south",)
in production.py I have this:
from .base import *
INSTALLED_APPS += ("gunicorn",)
When I run locally:
python manage.py runserver 7000 --settings=appname.settings.local
python manage.py runserver 7000 --settings=appname.settings.production
everything works fine.
But when I push changes to Heroku, I get the log:
File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
So, I guess Heroku is not finding my settings files, I don't know how to fix this (it might be very simple).
Two Scoops of Django is kind of ironic here, it writes "Platform as Service - See section 25.2" and then in that section it just writes "read Platform Documentation" : /
After you have logged into heroku with heroku login you can check your configs by running:
heroku config.
If you dont see a SECRET_KEY and DJANGO_SETTINGS_MODULE you can set them by running:
heroku config:set SECRET_KEY='secret_key_goes_here'
and
heroku config:set DJANGO_SETTINGS_MODULE=mysite.settings.production
Finally, make sure that you have the following syntax inside of your production setting file:
SECRET_KEY = os.environ['SECRET_KEY']
The above intstructions are for the following project structure
-myproject
-app1
-app2
-mysite
-settings
__init__.py
base.py
dev.py
production.py
-manage.py
-Pipfile
-Procfile
-requirements.txt
You can use the environment variable DJANGO_SETTINGS_MODULE to specify a default settings module:
https://docs.djangoproject.com/en/dev/topics/settings/#envvar-DJANGO_SETTINGS_MODULE
On local Linux machine:
export DJANGO_SETTINGS_MODULE=settings.local
On Heroku:
heroku config:set DJANGO_SETTINGS_MODULE=settings.production