So I have my Django project, and am trying to deploy it via heroku server. I realized that I need to created secrets.json file (in the same directory as manage.py), and then add it to .gitignore, for sake of security. But after deploying the project, I got an error saying it couldn't find secrets.json file. Below is my code.
settings.py
secret_file = os.path.join(BASE_DIR, 'secrets.json')
with open(secret_file) as f:
secrets = json.loads(f.read())
def get_secret(setting, secrets=secrets):
try:
return secrets[setting]
except KeyError:
error_msg = "Set the {} environment variable".format(setting)
raise ImproperlyConfigured(error_msg)
SECRET_KEY = get_secret("SECRET_KEY")
secrets.json
{
"SECRET_KEY" : xxx,
"EMAIL_HOST_USER" : xxx,
"EMAIL_HOST_PASSWORD" : xxx
}
After saving these, I added secrets.json to .gitignore.
But an error occurs at line 2 (with open(secret_file) as f:), saying it can't find secrets.json file. What do you think is the problem?
*FYI, The BASE_DIR doesn't seem to be the problem, since it works fine else where. Such as STATICFILES_DIRS.
You are correct to ignore your secrets file. It shouldn't be part of your repository, and it shouldn't be shipped to Heroku. Since that file is not included in your source code, Heroku can't open it.
As far as I know, having a secrets.json is not a common pattern in the Django / Python world¹.
I'm a bit confused by this:
def get_secret(setting, secrets=secrets):
try:
return secrets[setting]
except KeyError:
error_msg = "Set the {} environment variable".format(setting)
raise ImproperlyConfigured(error_msg)
Your KeyError handler refers to environment variables. That's on the right track! Configuring your application with environment variables is a common pattern.
But a JSON file called secrets.json isn't an environment variable. A more common approach would be to load values in your settings file directly from the environment:
import os
SECRET_KEY = os.getenv("SECRET_KEY")
# You can even provide a default in case the environment variable isn't set:
ANOTHER_SETTING = os.getenv("ANOTHER_SETTING", default="some-default-value")
On Heroku, you can set config vars that will be automatically injected into your environment:
heroku config:set SECRET_KEY=some-value
So that leaves local development. There are several ways to set environment variables on your local machine, but a common one is to create a file called .env:
SECRET_KEY=xxx
EMAIL_HOST_USER=xxx
EMAIL_HOST_PASSWORD=xxx
If you run your code locally via heroku local it will automatically pick up your .env file. So will Pipenv, if you use that. There's also python-dotenv and direnv and others.
.env files have become something of a de facto standard, and you're much more likely to find tools supporting them than ones supporting a secrets.json file.
¹There is python-secrets, which might help you with secrets.json files. You appear to be opening your secrets file manually, though.
If you plan to keep using your secrets file, I suggest you find something like this that automatically reads the file and merges settings into the environment. I've never used this library and only came across it in a search while answering this question.
Related
How do you dynamically access the domain name URL in the settings.py file of Django? (ie "http://localhost:8000")
I am trying to overwrite a package CDN while the internet is unavailable during development, and want to point to the local file in the static files directory. While os.path.join(BASE_DIR, "path/to/local.file") should work, it is context-dependent as to which app/url (ie "http://localhost:8000/app/static/css/bootstrap.min.css
"), and not just the main domain with the static file location appended to the starting server with ./manage.py runserver 0:8000 (ie " http://localhost:8000/static/css/bootstrap.min.css").
Notes:
Because this is in the settings.py, I can't load any apps or reverse because of the error *** django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
I am not in a template, so I can't use the static url
statically defining it won't allow for different port loadings when starting via ./manage.py runserver 0:8000
settings.py is basically a python module, but how can you get the domain within it?
Basically in the settings.py file:
# If in local dev
if "RDS_DB_NAME" not in os.environ:
# the setting for the package I am pointing to a local version
BOOTSTRAP5 = {
"css_url": {
### dynamically get domain here ###
# "href": os.path.join(LOCAL_DIR, "static/css/bootstrap.min.css"),
"href": "static/css/bootstrap.min.css",
}
You can't access the domain in settings.py. When you run ./manage.py runserver 0:8000 you are telling Django to listen on port 8000 of whatever localhost is. But you can't tell from this machine which requests are going to come to this machine. For instance, you could have DNS configured to send www.domain1.com and www.domain2.com to come to this machine which looks like localhost from your perspective. So you can't know "domain" until a requests comes in.
Your machine does however have a local machine name which you could figure out from settings.py. Different OSes do this differently but on Macs you can get it with scutil --get LocalHostName and on other Unixes you can cat /etc/hostname.
Chris Curvey is right that the canonical way to differentiate between environments is with different settings files. Most devs differentiate between local and production environments but you are proposing a third: local with no internet access.
There are 2 steps to making this work.
Detect when to use the "local with no internet" environment.
The du jour way (and easiest way, IMO) to do this is with an environment variable. There are lots of ways to do this. You could, for instance, set LOCAL_NO_INTERNET=True in the environment and then check that variable anytime you need to do something special in your code.
If you really need to detect this automatically you could:
Query for the local machine's name (machine name, not domain, as described above)
Check if the name matches your local dev machine (so that you don't do this thing in prod, etc.)
Check if you don't have internet (ping google.com or something and see if it comes back successfully or times out)
I do not recommend this approach; there are many edge cases. Plus you probably can determine for yourself if you would like to be in "local with no internet" environment and set the environment variable manually.
Serve cached files usually provided by CDN
I think the easiest way, although a little verbose, is to put an if statement in your templates to either target the CDN or your local version (note that you will have to pass LOCAL_NO_INTERNET in the context of the view):
{% if LOCAL_NO_INTERNET %}
<script src="{% static "cdncache/bootstrap.min.js" %}"></script>
{% else %}
<script src="https://cdn.jsdelivr.net/npm/bootstrap#5.1.3/dist/js/bootstrap.min.js" integrity="..." crossorigin="anonymous"></script>
{% endif %}
Or you could do something way more complicated here like a middleware to somehow replace references to CDNs.
I would suggest that instead of defining this new "local with no internet" environment you could change your local setup to always assume "local with no internet". The CDN should speed up requests in production but doesn't do you much good in local development compared with always using your own version of those files.
I've never found a way to find out details about the Django listener.
The canonical way to have different settings in different environments is to have different settings files, and then set the DJANGO_SETTINGS_MODULE environment variable to control which one is used.
I am creating a Python script which reads a spreadsheet and issues Rest requests to an external service. A token must be obtained to issue requests to the Rest service, so the script needs a username and password to obtain the oauth2 token.
What's the best or standard way to hide or not have visible information in the Python script?
I recommend using a config file. Let's create a config file and name it config.cfg. The file structure should look more or less like this:
[whatever]
key=qwerertyertywert2345
secret=sadfgwertgrtujdfgh
Then in python you can load it this way:
from configparser import ConfigParser
config = ConfigParser()
config.read('config.cfg')
my_key = config['whatever']['key']
my_secret = config['whatever']['secret']
In general, the most standard way to handle secrets in Python is by putting them in runtime configuration.
You can do that by reading explicitly from external files or using os.getenv to read from environment variables.
Another way is to use a tool like python-decouple, which lets you use the environment (variables), a .env file, and an .ini file in a cascade, so that developers and operations can control the environment in local, dev, and production systems.
I'm trying to configure Celery using an external file, to separate config from code. Both Celery and Flask have config.from_envvar() / config_from_envvar() methods except they behave a bit differently and what I am doing only works with Flask.
Flask
Basically, in Flask, I do
app.config.from_object(config_class)
app.config.from_envvar('SETTINGS_FILE', silent=True)
which loads a default configuration stored in application code, then loads a settings file stored anywhere in the filesystem with a few customized settings overriding those in the default file. I just need to write that file and pass its path through an environment variable.
(More on this in Flask docs or in this answer. I find it a bit ambiguous that Flask treats the file as a Python file while the example uses settings.cfg (no .py extension) but it works fine).
Celery
When doing the same with Celery, I get this error:
ImportError: No module named '/absolute/path/to/settings'
I named the file settings.py (.py extension, in case it matters).
I don't know where to put that file. The examples I've seen, for instance in answers to this question, put the config file in the code, which is precisely what I would like to avoid.
since you neither want to manipulate PYTHONPATH nor put config file under application directory, the only option left will be load config from filepath:
def load_config_from_file(celery_app, filepath):
conf = {}
with open(filepath) as fp:
exec(compile(fp.read(), filepath, 'exec'), {}, d)
celery_app.config_from_object(conf)
BTW, to my knowledge most (if not all) celery config option names have no conflict with flask's, you could write them together into one file, let flask load it, then celery could just read config from flask:
celery_app.conf.update(flask_app.config)
At present my application related variables like external server IP / default pattern etc.. These variables are specific to my application. It includes my external server username and password.
Now how can I have a single common place so that I can externalise this from my application.
The options which I thought are below:
Have one conf.ini file and use configparser and read this during the start of the django app
But I am not sure from where I should have method to read this so that it will be done in the startup.
Other option is to save this in a py file itself and import this file in my modules
Please suggests me the good and standard approach for above issue.
Save the important secret details in an ini file and place it in etc/project_name/my_settings.ini. Read these settings from settings.py. This way it will be secure. Can be read directly in the settings file itself
Or better way is to set them in bashrc, read the env vars from it.
Check this: Setting envs
In pyramid, I need to render my templates according to different runtime environments -- enable google analytics, use minified code, etc. (when in production). Is there an easy way to find out the current environment -- perhaps an existing flag to find out which ini file was used?
Pyramid INI files can hold arbitrary configuration entries, so why not include a flag in your files that distinguishes between production and development deployments?
I'd do it like this; in your production .ini file:
[app:main]
production_deployment = True # Set to False in your development .ini file
Pass this value on to the Pyramid Configurator:
def main(global_config, **settings):
# ...
from pyramid.settings import asbool
production_deployment = asbool(settings.get(
'production_deployment', 'false'))
settings['production_deployment'] = production_deployment
config = Configurator(settings=settings)
You can now access the settings from just about anywhere in your Pyramid code. For example, in a request handler:
settings = request.registry.settings
if settings['production_deployment']:
# Enable some production code here.
However, I'd also use more finegrained settings in this case; a flag for enabling Google Analytics, one for minifying resources, etc. That way you can test each individual setting in your development environment, write unit tests for these switches, etc.
I set this sort of thing as an environmental variable named something like PYRAMID_ENV which can be viewed via os.environ. For example in your code:
import os
pyramid_env = os.environ.get('PYRAMID_ENV', 'debug')
if pyramid_env == 'debug':
# Setup debug things...
else:
# Setup production things...
Then you can set the variable in the init script or when starting the server:
PYRAMID_ENV=production python server.py
Docs on access to environmental variables: http://docs.python.org/library/os.html#os.environ