I am using Django, python, virtualenv, virtualenvwrapper and Vagrant.
So far I have simply left my secret_key inside of the settings.py file.
This works file for local files. However I have already placed my files in Git. I know this is not acceptable for production(Apache).
What is the correct way to go about hiding my secret_key?
Should I use virtualenv to hide it?
There's a lot of different methods to hide secrets.
Use another, non-versioned file.
Create a new file secrets.py or what have you and put your secrets in that. Place it alongside your settings file and place everything secret in there; then in your settings file put from secrets import * at the top. Then, like Rahul said, add a .gitignore file and add secrets.py to this file so that it won't be committed.
The disadvantage of this approach is that there is no source control at all on that file; if you lose it you're SOL.
Use environment variables.
Use the Apache SetEnv or PassEnv directives to pass environment variables to your process, then retrieve them with os.environ() in your settings file. This has the advantage in that in development, you can set new variables (as simply as VAR1=whatever VAR2=whatever ... ./manage.py runserver ...) or set them from whatever mechanism you use to launch your development project.
The disadvantage is much the same; if you lose your Apache configs you're boned.
Use a second repository in combination with method 1.
Personally, I like the idea of having a dedicated secrets repository that you put all your secrets into and keep that repo under lock and key. Then as part of your deployment process, you can use git archive or another similar command to extract the proper keys for the place you're deploying to, and you can keep your secrets backed up and under version control easily. You can also add the appropriate files in the secrets repo to the .gitingore file of your site repository so that they don't accidentally get committed.
The downside of this is that you have another extra repository and another deployment step. I think that's worth it, personally, but it's really up to you.
In general, the more secure you want it, the more inconvenient it's going to be to access those secrets. That's really a rule in general, though.
You can create a file named secret_settings.py and place your SECRET_KEY inside this file. Then add this file to .gitignore.
Then in your settings, you can remove the secret key variable and import it from there. This should ensure that SECRET_KEY variable remains out of version control.
Create a file named secret_settings and then place your SECRET_KEY and other secret settings in it.
SECRET_KEY = .. # add your setting here
Then in your settings.py file, import these settings.
from secret_settings import *
Finally, add secret_settings.py to your .gitignore file.
Note:
If you already have committed some sensitive data to your repo, then change it!
As per Github website in the removing sensitive data article:
If you committed a password, change it! If you committed a key,
generate a new one.
Check this link on how to purge a file from your repository's history.
Common approach, if you'd like to configure region, but did not want to store sensitive information in repo, is to pass it through environment variables. When you need it just call os.environ('SECRET') (even in your settings.py). Better with some fallback value.
Virtualenv does not helps you to hide anything, it just prevent you system-wide Python installation from littering by one-project-required-packages.
The solution I use is to create a file sec.py and place it next to my settings.py file. Then in at line 1 of settings.py call from .sec import *. Be sure to include the period in front of the file name. Be sure to list sec.py in your .gitignore file.
Related
I have a settings.py file for my Django project that reads in environment variables for some fields. This works great for local development. I'm not sure of how to make this work for my Jenkins build (Freestyle project). An extra layer of complexity is that some of these variables contain sensitive data (passwords, etc.) and so need to be secured.
So far, I have been able to use Credentials Binding Plugin to set up a secret .env file and can successfully access that as an environment variable.
How do I go about setting all the 'actual' variables from that file? Or is this bad practice?
If it's not bad, wouldn't those variables continue living on the server after the build has finished?
If it is bad, what is the better way to do this? I can update the settings.py file easily enough if needed to make this work.
Maybe using something like django-environ is the right approach?
Some background first:
i am currently testing a class that sends a GET request with a configurable url, which is built like this
url = f"{os.environ["TARGET_URL"]}/api/etc"
For normal operation, my TARGET_URL environment variable is set at project startup from a .env file and everything works. When testing locally, everything is still fine, tests passes and everyone is happy. My issue arose when I discovered that my Drone CI server failed to complete the project's build because the TARGET_URL environment variable wasn't found.
After some digging I found out that I had the wrong (dumb) idea that environment variables were reset at every project/test startup, and I basically was using my production environment variable all this time (even during tests) because it was set at first project startup.
From this story comes my question: given that environment variables are kept between executions, would storing configurations in them result in a bad practice? Is there an equally convenient alternative (no global objects and access from everywhere in the code) that can be used instead?
Thanks everyone for the quick responses, here's a bit of what-happened-next:
environment variables stay loaded after the first initialization, so I needed a way to test my code after loading only the variables I needed, with values that were expected. This would allow me to keep using environment variables loaded from a .env file and keep building my project remotely, where no .env files are present.
The solution was to add a pytest plugin called pytest-dotenv, which when properly configured would allow me to overwrite every variable in my .env files with a custom variable from another file (.env.test in my case). I filled the .env.test file with all the variables found in the .env file, and assigned empty values to each of them.
This allowed my tests to run ensuring no weird edge cases are missed because something had the wrong value.
example .env file
TARGET_URL="http://my.api.dev
example .env.test file
TARGET_URL=
pytest.ini configuration
[pytest]
env_override_existing_values = 1
env_files =
.env.test
Environment variables stored in config files or .env files is not a bad practice.
However, it is recommended that you use a key vault such as Azure Key Vault or AWS Key Management System for production deployments.
This way you further remove the keys away from your server (if env files) as well as code (if it is in config files)
I had implemented my project along with daily commits to Github, which is a website implemented using Flask and Python, while daily commits I used to git add --a, thus it has added all folders like __pycache__ and flask_session. I think it is safe to delete those from my Github, Is it so? If not why?
If you look at gitignore.io/python, you will find in the generated .gitignore file
### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
However, you won't find flask_session, not even in flask-common/.gitignore.
The Flask documentation mentions:
SESSION_FILE_DIR The directory where session files are stored.
Default to use flask_session directory under current working directory.
If that is something private to the user and the flask execution, then add to your .gitignore:
/*_session/
But considering a Flask session stores secrets, the best practice would be to configure Flask to store its session outside the Git repository: that way, no risk to add by mistake anything, .gitignore or not.
Both these folders contain temporary objects that are created in runtime, and there's no point in committing them to git.
I'd remove those folders from your repository and then add them to .gitignore to prevent them from being re-added there by mistake.
In layman language, YES because these things can be generated again by Flask thus it's good practise to not push it to a repo and generate locally individually.
I'm trying to deploy a Django app on Heroku. As recommended for security, I don't want to commit and push my secret key from the settings.py file, so I put the key in a separate file and imported it into settings.py. I put my file containing the secret key into .gitignore so that it doesn't get committed.
Now the problem is when I'm pushing to Heroku, I get a ModuleNotFoundError - which makes sense because this file was never committed. Is there a workaround or best practice to use secret keys without exposing them?
It seems you noticed what happened when you excluded the file: it's not there and this is exactly how the security is achieved. The obvious problem now is how does the program get access to the data anyway.
On Heroku, this in handled by setting environment variables on your app that you will then read out when initializing your program -- how this is done differs between programming languages.
So either go https://dashboard.heroku.com/apps/<your-app>/settings and set the variables there. Or utilize the command-line tool like this:
heroku config:set KEY=value -a your-app
The next question then is how you handle this in for local development. A simple solution could be to check for the existence of the git-ignored file, and if it exists use it, otherwise read the values from the environment.
I'm using AWS for the first time and have just installed boto for python. I'm stuck at the step where it advices to:
"You can place this file either at /etc/boto.cfg for system-wide use or in the home directory of the user executing the commands as ~/.boto."
Honestly, I have no idea what to do. First, I can't find the boto.cfg and second I'm not sure which command to execute for the second option.
Also, when I deploy the application to my server, I'm assuming I need to do the same thing there too...
"You can place this file either at /etc/boto.cfg for system-wide use
or in the home directory of the user executing the commands as
~/.boto."
The former simply means that you might create a configuration file named boto.cfg within directory /etc (i.e. it won't necessarily be there already, depending on how boto has been installed on your particular system).
The latter is indeed phrased a bit unfortunate - ~/.boto means that boto will look for a configuration file named .boto within the home directory of the user executing the commands (i.e. Python scripts) which are facilitating the boto library.
You can read more about this in the boto wiki article BotoConfig, e.g. regarding the question at hand:
A boto config file is simply a .ini format configuration file that
specifies values for options that control the behavior of the boto
library. Upon startup, the boto library looks for configuration files
in the following locations and in the following order:
/etc/boto.cfg - for site-wide settings that all users on this machine
will use
~/.boto - for user-specific settings
You'll indeed need to prepare a respective configuration file on the server your application is deployed to as well.
Good luck!
For those who want to configure the credentials in Windows:
1-Create your file with the name you want(e.g boto_config.cfg) and place it in a location of your choice(e.g C:\Users\\configs).
2- Create an environment variable with the Name='BOTO_CONFIG' and Value= file_location/file_name
3- Boto is now ready to work with credentials automatically configured!
To create environment variables in Windows follow this tutorial: http://www.onlinehowto.net/Tutorials/Windows-7/Creating-System-Environment-Variables-in-Windows-7/1705
For anyone looking for information on the now-current boto3, it does not use a separate configuration file but rather respects the default one created by the aws cli when running aws configure (Ie, it will look at ~/.aws/config)