Some background first:
i am currently testing a class that sends a GET request with a configurable url, which is built like this
url = f"{os.environ["TARGET_URL"]}/api/etc"
For normal operation, my TARGET_URL environment variable is set at project startup from a .env file and everything works. When testing locally, everything is still fine, tests passes and everyone is happy. My issue arose when I discovered that my Drone CI server failed to complete the project's build because the TARGET_URL environment variable wasn't found.
After some digging I found out that I had the wrong (dumb) idea that environment variables were reset at every project/test startup, and I basically was using my production environment variable all this time (even during tests) because it was set at first project startup.
From this story comes my question: given that environment variables are kept between executions, would storing configurations in them result in a bad practice? Is there an equally convenient alternative (no global objects and access from everywhere in the code) that can be used instead?
Thanks everyone for the quick responses, here's a bit of what-happened-next:
environment variables stay loaded after the first initialization, so I needed a way to test my code after loading only the variables I needed, with values that were expected. This would allow me to keep using environment variables loaded from a .env file and keep building my project remotely, where no .env files are present.
The solution was to add a pytest plugin called pytest-dotenv, which when properly configured would allow me to overwrite every variable in my .env files with a custom variable from another file (.env.test in my case). I filled the .env.test file with all the variables found in the .env file, and assigned empty values to each of them.
This allowed my tests to run ensuring no weird edge cases are missed because something had the wrong value.
example .env file
TARGET_URL="http://my.api.dev
example .env.test file
TARGET_URL=
pytest.ini configuration
[pytest]
env_override_existing_values = 1
env_files =
.env.test
Environment variables stored in config files or .env files is not a bad practice.
However, it is recommended that you use a key vault such as Azure Key Vault or AWS Key Management System for production deployments.
This way you further remove the keys away from your server (if env files) as well as code (if it is in config files)
Related
I have a settings.py file for my Django project that reads in environment variables for some fields. This works great for local development. I'm not sure of how to make this work for my Jenkins build (Freestyle project). An extra layer of complexity is that some of these variables contain sensitive data (passwords, etc.) and so need to be secured.
So far, I have been able to use Credentials Binding Plugin to set up a secret .env file and can successfully access that as an environment variable.
How do I go about setting all the 'actual' variables from that file? Or is this bad practice?
If it's not bad, wouldn't those variables continue living on the server after the build has finished?
If it is bad, what is the better way to do this? I can update the settings.py file easily enough if needed to make this work.
Maybe using something like django-environ is the right approach?
I'm following the guide here and can't seem to get my Python app (which is deployed fine on GCP) to read the environment variables I've created in Cloud Functions.
The REST endpoint for the function returns the environment variables fine as I've coded up the Python method in the function to just do os.environ.get() on a request parameter that is passed in. However, in my actual deployed application, I don't want to do a REST GET call every time I need an environment variable. I would expect using os.environ.get() in my application would be enough, but that returns blank.
How does one go about retrieving environment variables on GCP with just a simple os.environ.get() or do I really have to make a call to an endpoint every time?
I have been struggling with this for some time. The only solution I have found to set environment variables for the whole app is to define them in app.yaml. See the env_variables section here.
But then you cannot commit app.yaml to any version control repository if you don't want people to see the environment variables. You could add it to .gitignore. There are more secure ways to handle secrets storage if these variables contain sensitive data. If you need more robust security, you might find some inspiration here.
I'm trying to deploy a Django app on Heroku. As recommended for security, I don't want to commit and push my secret key from the settings.py file, so I put the key in a separate file and imported it into settings.py. I put my file containing the secret key into .gitignore so that it doesn't get committed.
Now the problem is when I'm pushing to Heroku, I get a ModuleNotFoundError - which makes sense because this file was never committed. Is there a workaround or best practice to use secret keys without exposing them?
It seems you noticed what happened when you excluded the file: it's not there and this is exactly how the security is achieved. The obvious problem now is how does the program get access to the data anyway.
On Heroku, this in handled by setting environment variables on your app that you will then read out when initializing your program -- how this is done differs between programming languages.
So either go https://dashboard.heroku.com/apps/<your-app>/settings and set the variables there. Or utilize the command-line tool like this:
heroku config:set KEY=value -a your-app
The next question then is how you handle this in for local development. A simple solution could be to check for the existence of the git-ignored file, and if it exists use it, otherwise read the values from the environment.
Let's say I have some code running on a Heroku dyno (such as this autoscaling script), that needs access to the Platform API. To access the API, I have to authenticate using my app's API Key.
What's the right way to do this?
That script I referenced hardcoded the API Key in the script itself.
A better practice generally seems to put secrets in environment variables, which is what Heroku normally recommends. However, they say they say:
Setting the HEROKU_API_KEY environment variable on your machine will
interfere with normal functioning of auth commands from Toolbelt.
Clearly I could store the API key with under a different key name.
What's the right way? I couldn't find this in the documentation, but seems like a common issue.
Yes, storing this token into a config var is the right way to go.
As for HEROKU_API_KEY, this will happen because locally, the toolbelt will look for the environment variable as one solution to try to fetch your token.
This won't impact your production environment (the heroku toolbelt isn't available within dynos).
Locally, you can also set it easily with a tool like python-dotenv, which will allow you to have a local .env file (don't check it into source control, or your token could be corrupted), with all of it's values available as env vars in your dev app.
I am using Django, python, virtualenv, virtualenvwrapper and Vagrant.
So far I have simply left my secret_key inside of the settings.py file.
This works file for local files. However I have already placed my files in Git. I know this is not acceptable for production(Apache).
What is the correct way to go about hiding my secret_key?
Should I use virtualenv to hide it?
There's a lot of different methods to hide secrets.
Use another, non-versioned file.
Create a new file secrets.py or what have you and put your secrets in that. Place it alongside your settings file and place everything secret in there; then in your settings file put from secrets import * at the top. Then, like Rahul said, add a .gitignore file and add secrets.py to this file so that it won't be committed.
The disadvantage of this approach is that there is no source control at all on that file; if you lose it you're SOL.
Use environment variables.
Use the Apache SetEnv or PassEnv directives to pass environment variables to your process, then retrieve them with os.environ() in your settings file. This has the advantage in that in development, you can set new variables (as simply as VAR1=whatever VAR2=whatever ... ./manage.py runserver ...) or set them from whatever mechanism you use to launch your development project.
The disadvantage is much the same; if you lose your Apache configs you're boned.
Use a second repository in combination with method 1.
Personally, I like the idea of having a dedicated secrets repository that you put all your secrets into and keep that repo under lock and key. Then as part of your deployment process, you can use git archive or another similar command to extract the proper keys for the place you're deploying to, and you can keep your secrets backed up and under version control easily. You can also add the appropriate files in the secrets repo to the .gitingore file of your site repository so that they don't accidentally get committed.
The downside of this is that you have another extra repository and another deployment step. I think that's worth it, personally, but it's really up to you.
In general, the more secure you want it, the more inconvenient it's going to be to access those secrets. That's really a rule in general, though.
You can create a file named secret_settings.py and place your SECRET_KEY inside this file. Then add this file to .gitignore.
Then in your settings, you can remove the secret key variable and import it from there. This should ensure that SECRET_KEY variable remains out of version control.
Create a file named secret_settings and then place your SECRET_KEY and other secret settings in it.
SECRET_KEY = .. # add your setting here
Then in your settings.py file, import these settings.
from secret_settings import *
Finally, add secret_settings.py to your .gitignore file.
Note:
If you already have committed some sensitive data to your repo, then change it!
As per Github website in the removing sensitive data article:
If you committed a password, change it! If you committed a key,
generate a new one.
Check this link on how to purge a file from your repository's history.
Common approach, if you'd like to configure region, but did not want to store sensitive information in repo, is to pass it through environment variables. When you need it just call os.environ('SECRET') (even in your settings.py). Better with some fallback value.
Virtualenv does not helps you to hide anything, it just prevent you system-wide Python installation from littering by one-project-required-packages.
The solution I use is to create a file sec.py and place it next to my settings.py file. Then in at line 1 of settings.py call from .sec import *. Be sure to include the period in front of the file name. Be sure to list sec.py in your .gitignore file.