Develop Reference

How to load environment variables in a config.ini file?

I have a config.ini file which contains some properties but I want to read the environment variables inside the config file.
[section1]
prop1:(from envrinment variable) or value1
Is this possible or do I have to write a method to take care of that?

I know this is late to the party, but someone might find this handy.
What you are asking after is value interpolation combined with os.environ.
Pyramid?
The logic of doing so is unusual and generally this happens if one has a pyramid server, (which is read by a config file out of the box) while wanting to imitate a django one (which is always set up to read os.environ).
If you are using pyramid, then pyramid.paster.get_app has the argument options: if you pass os.environ as the dictionary you can use %(variable)s in the ini. Not that this is not specific to pyramid.paster.get_app as the next section shows (but my guess is get_app is the case).
app.py
from pyramid.paster import get_app
from waitress import serve
import os
app = get_app('production.ini', 'main', options=os.environ)
serve(app, host='0.0.0.0', port=8000, threads=50)
production.ini:
[app:main]
sqlalchemy.url = %(SQL_URL)s
...
Configparse?
The above is using configparser basic interpolation.
Say I have a file called config.ini with the line
[System]
conda_location = %(CONDA_PYTHON_EXE)
The following will work...
import configparser, os
cfg = configparser.ConfigParser()
cfg.read('config.ini')
print(cfg.get('System', 'conda_location', vars=os.environ))

I think :thinking_face, use .env and in config.py from dotenv import load_dotenv() and in next line load_dotenv() and it will load envs from .env file

How can put files in folder in django for settings

I have this folder structure for django
settings/dev.py
settings/prod.py
settings/test.py
Then i have common settings in settings/common.py in which i check the ENV variable like
if PROD:
from settings.prod import *
Based on ENV variable each one of them will be active
I want to use something like this in my code
from myapp import settings
rather than
from myapp.settings import dev

This is the method which I follow. Learnt this from the book Two Scoops of Django.
Have a file, such as, settings/common.py which will contain the properties/settings which are common in dev, prod and test environment. (You already have this.)
The other 3 files should:
Import the common settings from the settings/common.py by adding the line from .common import *
And should contain settings for its own corresponding environment.
The manage.py file decides which settings file to import depending on the OS environment variable, DJANGO_SETTINGS_MODULE. So, for test environment the value of DJANGO_SETTINGS_MODULE should be mysite.settings.test
Links for reference:
Django documentation for django-admin utility - Link
Two Scoops of Django sample project - Link

Preserve your settings folder structure and create __init__.py there.
Please, use code below in your settings/__init__.py:
import os
# DJANGO_SERVER_TYPE
# if 1: Production Server
# else if 2: Test Server
# else: Development Server
server_type = os.getenv('DJANGO_SERVER_TYPE')
if server_type==1:
from prod import *
elif server_type==2:
from test import *
else:
from dev import *
Now you can set environment variable called DJANGO_SERVER_TYPE to choose between Production, Test or Development Server and import settings using:
import settings

Way to setup settings_staging, settings_production and settings_local files in django

I have three systems for my django project for which i need three different settings file for each respective system i.e local,staging and production.
After doing some research I though of a way i.e to set an environmental variable corresponding to the system. For local i am setting env variable as 'localserver', for staging server i am setting 'staging' and so on for production.
Settings.py
server_environment = os.environ.get('XYZ_ENV')
if server_environment == 'staging':
try:
from rest_apis.settings_staging import *
except ImportError:
pass
elif server_environment == 'production':
try:
from settings_production import *
except ImportError:
pass
elif server_environment == 'localserver':
try:
from settings_local import *
except ImportError:
pass
local.py
from rest_apis.settings import *
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'xyz',
'USER': 'postgres',
'PASSWORD': 'postgres123',
'HOST': '127.0.0.1',
'PORT': 5432,
}
}
BROKER_URL = 'amqp://myuser:mypassword#127.0.0.1:5672//'
Same type of config for different systems.
Here what is happening, inspite of cross verifying that it goes to correct if-else block, it doesn't apply the proper settings. It always connect to DB Host specify in Settings.py file.
Also if i stop my instance (i have private IP, so on restart IP doesn't change) and I start again, then it again create the same issue.
I did stackoverflow for it and tried various solutions for it but none of them helped. What am i doing wrong ? Also, what is the correct way for this type of situation.
I don't want to make hardcore changes to the system by logging(ssh) into each and every system.

You can use different environments as suggested in other answers, though I recommend using separate settings files for different working environments.
The ideal project layout would be something like -
project_folder -
settings -
__init__.py
common.py
development.py
staging.py
production.py
test.py
main.py
Common staging file needs to contain all settings common to all environments. All settings file import from 'common.py'
main.py import all settings from staging.py in staging environment, development.py in development environment and production.py in production environment.
Thus, main.py would be the main settings file which needs to added to DJANGO_SETTINGS_MODULE. As main.py is different for different environments, it should be excluded from git.
Sample Code:-
common.py
# All common configurations
development.py
from .common import *
# Add settings for different connections like db, cache, smtp etc.
production.py
from .common import *
# Add connection settings for production environment
main.py # In development environment
from .development import *
try:
from local import *
except ImportError:
pass
main.py # In staging environment
from .staging import *
try:
from local import *
except ImportError:
pass
main.py # In production environment
from .production import *
try:
from local import *
except ImportError:
pass
manage.py
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
settings_file = 'project_folder.settings.test' if 'test' in sys.argv else 'project_folder.settings.main'
os.environ.setdefault("DJANGO_SETTINGS_MODULE", settings_file)
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)

What am I doing wrong?
It's hard to tell what exactly is wrong, but you can:
Remove from rest_apis.settings import * from local settings.
Your local settings should override settings.py, not vice versa.
Make sure that your imports are correct.
Remove try/except and check if ImportError is raised.
Place your imports at the end of settings.py.
Thus imported settings will override existing.
Also, what is the correct way for this type of situation?
I think that using one local settings file for each environment (that's why it is called local) would be a more elegant solution:
Create a template for local settings, e.g. local_settings.template.py:
"""
Local Django settings template.
Usage:
1. Copy ``local_settings.template.py`` to ``local_settings.py``.
2. Modify ``local_settings.py`` according to your needs.
Note that settings from ``local_settings.py``
will override any existing Django settings.
"""
SECRET_KEY = ''
ALLOWED_HOSTS = []
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST': '',
'PORT': '',
'NAME': '',
'USER': '',
'PASSWORD': ''
}
}
Add local_settings.template.py to your VCS.
Add local_settings.py to your VCS ignore.
Modify settings.py:
"""
Core Django settings.
Consider using local settings (see ``local_settings.template.py``)
for environment specific Django settings and sensitive data.
All involved settings, however, should be listed
here with their default values and description.
For more information on this file, visit
https://docs.djangoproject.com/en/1.8/topics/settings/
For the full list of settings and their values, visit
https://docs.djangoproject.com/en/1.8/ref/settings/
"""
# Settings here.
# WARNING: local settings import should remain at the end of this file.
try:
from local_settings import *
except ImportError:
pass

First of all, that settings file is assuming that all your additional settings are inside PYTHONPATH. If they're just inside same (sub)module as your main settings file is, use relative import like this:
server_environment = os.environ.get('XYZ_ENV')
if server_environment == 'staging':
try:
from rest_apis.settings_staging import *
except ImportError:
pass
elif server_environment == 'production':
try:
from .settings_production import *
except ImportError:
pass
elif server_environment == 'localserver':
try:
from .settings_local import *
except ImportError:
pass
Secondly, that imports should be at the end of your settings file (unless you want to have some settings that can't be ovveriden. In that case, put them below your imports).
And last but not least, make sure that your importing succeeds. Remove try - except blocks, leaving only import in your file. You can also remove conditions, just import one file to test if it's working.

Launching Scrapyd with multiple configurations

I'm trying to develop my Scrapy application using multiple configurations depending on my environment (e.g. development, production). My problem is that there are some settings that I'm not sure how to set them. For example, if I have to setup my database, in development should be "localhost" and in production has to be another one.
How can I specify these settings when I'm doing scrapy deploy ? Can I set them with a variable in command-line?

You should set the deploy options in your scrapy.cfg file. For example:
[deploy:dev]
url = http://dev_url/
[deploy:production]
url = http://production_url/
With that, you could do:
scrapyd-deploy def
or
scrapyd-deploy production

You can refer to the answer in the following link :
https://alanbuxton.wordpress.com/2018/10/09/using-local-settings-in-a-scrapy-project/
I copy here for quick reference:
Edit the settings.py file so it would read from additional settings files depending on a SCRAPY_ENV environment variable
Move all the settings files to a separate config directory (and change scrapy.cfg so it knew where to look
The magic happens at the end of settings.py：
from importlib import import_module
from scrapy.utils.log import configure_logging
import logging
import os
SCRAPY_ENV=os.environ.get('SCRAPY_ENV',None)
if SCRAPY_ENV == None:
raise ValueError("Must set SCRAPY_ENV environment var")
logger = logging.getLogger(__name__)
configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
# Load if file exists; incorporate any names started with an
# uppercase letter into globals()
def load_extra_settings(fname):
if not os.path.isfile("config/%s.py" % fname):
logger.warning("Couldn't find %s, skipping" % fname)
return
mdl=import_module("config.%s" % fname)
names = [x for x in mdl.__dict__ if x[0].isupper()]
globals().update({k: getattr(mdl,k) for k in names})
load_extra_settings("secrets")
load_extra_settings("secrets_%s" % SCRAPY_ENV)
load_extra_settings("settings_%s" % SCRAPY_ENV)
Then in the python file you want to get the variables defined in the setting, use the following code
from scrapy.utils.project import get_project_settings
settings = get_project_settings()
env_variable = settings.get('ENV_VARIABLE')

How do I stop getting ImportError: Could not import settings 'mofin.settings' when using django with wsgi?

I can't get wsgi to import my settings file for my project 'mofin'.
The list of errors from the apache error log are as follows
mod_wsgi (pid=4001): Exception occurred within WSGI script '/var/www/wsgi-scripts/django.wsgi'.
Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/django/core/handlers/wsgi.py", line 228, in __call__
self.load_middleware()
File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py", line 31, in load_middleware
for middleware_path in settings.MIDDLEWARE_CLASSES:
File "/usr/lib/python2.5/site-packages/django/conf/__init__.py", line 28, in __getattr__
self._import_settings()
File "/usr/lib/python2.5/site-packages/django/conf/__init__.py", line 59, in _import_settings
self._target = Settings(settings_module)
File "/usr/lib/python2.5/site-packages/django/conf/__init__.py", line 94, in __init__
raise ImportError, "Could not import settings '%s' (Is it on sys.path? Does it have syntax errors?): %s" % (self.SETTINGS_MODULE, e)
ImportError: Could not import settings 'mofin.settings' (Is it on sys.path? Does it have syntax errors?): No module named mofin.settings
I got the "hello world!" wsgi app listed here(http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide) to work fine.
The settings.py file loads fine with python manage.py (runserver|shell|syncdb|test store)
as does the application.
Here is my wsgi file:
import os
import sys
sys.path.append('/home/django/mofin/trunk')
sys.path.append('/home/django/mofin/trunk/mofin')
print >> sys.stderr, sys.path
os.environ['DJANGO_SETTINGS_MODULE'] = 'mofin.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
the sys.path as printed in the error log is
['/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/gtk-2.0', '/home/django/mofin/trunk', '/home/django/mofin/trunk/mofin']
if I open an interactive shell with manage.py, sys.path is
['/home/django/mofin/trunk/mofin', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/gtk-2.0']
My django settings file looks like this:
# Django settings for mofin project.
DEBUG = True
TEMPLATE_DEBUG = DEBUG
ADMINS = (
# ('Dan xxxx', 'xxxx#yyyyyyyyyy.com'),
)
MANAGERS = ADMINS
DATABASE_ENGINE = 'mysql' # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
DATABASE_NAME = 'mofin' # Or path to database file if using sqlite3.
DATABASE_USER = 'aaaaaa' # Not used with sqlite3.
DATABASE_PASSWORD = 'bbbbbb' # Not used with sqlite3.
DATABASE_HOST = '' # Set to empty string for localhost. Not used with sqlite3.
DATABASE_PORT = '' # Set to empty string for default. Not used with sqlite3.
# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'Europe/London'
# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-GB'
SITE_ID = 1
# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True
# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = '/home/django/media/'
# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash if there is a path component (optional in other cases).
# Examples: "http://media.lawrence.com", "http://example.com/media/"
MEDIA_URL = 'http://mofin.mywebsite.co.uk/media/'
# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
ADMIN_MEDIA_PREFIX = '/admin_media/'
# Make this unique, and don't share it with anybody.
SECRET_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.load_template_source',
'django.template.loaders.app_directories.load_template_source',
# 'django.template.loaders.eggs.load_template_source',
)
MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
)
ROOT_URLCONF = 'mofin.urls'
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
)
INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.admin',
'mofin.store'
)

This can also happen if you have an application (subdirectory to the project with an init file in it) named the same thing as the project. Your settings.py file may be in your project folder, but it seems that a part of the django system looks first for a module inside the project by the same name as the project and when it can't find a settings.py in there, it fails with a misleading message.
-uniquename1
---settings.py
---manage.py
---application1
-----file.py
-----file2.py
---uniquename1 (problem, rename this to some other unique name)
-----file.py
-----file2.py
Just something else to check for anyone else having this problem. Applies to Django 1.3 and probably others.

I had a similar permissions problem, and although my settings.py had the right permissions, the .pyc's did not!!! So watch out for this.

Hey, just adding an additional answer to this problem. I had the exact same issue, but it wasn't file permissions. I was appending "path/to/project", but not also appending "path/to". Linked is mod_wsgi's Django integration explanation that showed me the answer.

I found the answer... file permissions. /home/django was set to 700. i.e. only django can view the contents. apache runs as Apache and so can't get past /home/django.

I think you need to have a trailing forward slash on that its what I have to do in my wsgi script in apache before I load up django.
import os
import sys
sys.path.append('/home/django/mofin/trunk/')
sys.path.append('/home/django/mofin/trunk/mofin/')
print >> sys.stderr, sys.path
os.environ['DJANGO_SETTINGS_MODULE'] = 'mofin.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
In my case
import os
import sys
if os.uname()[1] == 'vivien':
sys.path.append('/home/www/sitebuilder.blacknight.ie/web/')
os.environ['DJANGO_SETTINGS_MODULE'] = 'gibo.dev_settings'
elif os.uname()[1] == 'thingy':
sys.path.append('/home/www/sitebuilder.blacknight.ie/web/')
os.environ['DJANGO_SETTINGS_MODULE'] = 'gibo.dev_settings'
else:
sys.path.append('/home/www/sitebuilder.blacknight.ie/web/')
os.environ['DJANGO_SETTINGS_MODULE'] = 'gibo.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

Another cause of this problem is that you can't name your application the same as another python module. For example I called mine site, little realising that site is already a python module.
You can check this by starting python, and running import site, help(site), and it will show you it isn't using your module. This of course gives you errors when django tries to import site.settings which doesn't exist.

Possible problem:
you forgot the __init__.py file, which must be in your project and in all directories which you consider a python module for import.
Other thing you could try is to add the path directly into the manage.py file, like :
import sys
...
...
sys.path.insert(0, '/home/django/mofin/trunk')
I hope it helps

I had the same problem but another solution :
My project folder was named exactly as one of my application.
I had :
/home/myApp
/home/myApp/settings.py
/home/myApp/manage.py
/home/myApp/rights.py
/home/myApp/static/
/home/myApp/static/
/home/myApp/myApp/model.py
/home/myApp/myApp/admin.py
/home/myApp/myApp/views.py
This kind of tree doesn't seems to be possible easily.
I changed the name of my project root folder and the problem was solved!

(I wrote up this same answer for Django deployment problem in Apache/mod_wsgi. ImportError: Could not import settings 'site.settings' in case someone only finds this question.)
This doesn't appear to be the problem in your case, but I ran smack into the same ImportError when I used the WSGIPythonPath directive (instead of the .wsgi file) to set up sys.path. That worked fine until I switched to running WSGI in daemon mode. Once you do that, you have to use the python-path argument to the WSGIDaemonProcess directive instead.

In my case, I had a circular import that was causing this error. From settings.py I was importing one function in another module, and from that module I was importing a settings variable. To fix it, instead of directly importing from settings, I did this:
from django.conf import settings

Let me add and my experience for that issue. After head banging for few hours and try all from the above answers I found that few lines in settings.py file cause the problem:
from south.modelsinspector import add_introspection_rules
add_introspection_rules([], ["^dynamicsites.fields.FolderNameField"])
add_introspection_rules([], ["^dynamicsites.fields.SubdomainListField"])
After that I made copy of the settings.py, named scripts_settings.py whithout that lines, and used that file and everything is ok now.

At first look I'd say the python path is wrong but compared to interactive shell it looks ok.
So maybe try this:
from django.core.management import setup_environ
from mofin import settings
setup_environ(settings)

I was going to say that you can just insert/append your project directory to your sys.path in your wsgi file but if your settings file is at
/home/django/mofin/trunk/mofin/settings.py
Then you should be good there.
Is it on sys.path? Does it have syntax errors?
That pretty much sums up what you are looking for.
Interesting that the error propagates though:
for middleware_path in settings.MIDDLEWARE_CLASSES:
but you have what appears to be the exact default.
You might want to check which python interpreter is pointed to by wsgi. Are you intending to use a virtualenv but wsgi is looking at your system install?
You can also set the user and group that wsgi is running under. I use something like:
WSGIDaemonProcess mysite.com user=skyl group=skyl processes=n threads=N python-path=/home/skyl/pinax/pinax-env2/lib/python2.6/site-packages

I had a similar problem, solved it with the following snippet in my python:
ALLDIRS = ['/var/www/MarkerDB/']
import sys
import site
# Remember original sys.path.
prev_sys_path = list(sys.path)
# Add each new site-packages directory.
for directory in ALLDIRS:
site.addsitedir(directory)
# Reorder sys.path so new directories at the front.
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_pat
Source: http://code.google.com/p/modwsgi/wiki/VirtualEnvironments#Application_Environments

I just had this error and the solution was to enable my virtual environment via myvenv/source/activate.