I am trying to deploy my django app on Heroku, but the issue is that I cannot access my config vars from python code.
Here are my config vars (obfuscated for obvious reasons)
$ heroku config --remote production
=== myapp Config Vars
DATABASE_URL: postgres://<removed>
DJANGO_DEBUG: 0
GOOGLE_APPLICATION_CREDENTIALS: <removed>
SECRET_KEY: <removed>
Now, in my django settings.py, the code
import os
os.environ["DJANGO_DEBUG"]
for example results in an error because the key "DJANGO_DEBUG" is nowhere to be found in the dictionary os.environ.
My question is: how can I access Heroku config variables in production?
I have tried the package python-decouple, but it is not able to access the config vars neither. (Locally, my variables are put in a .env file and are accessible with the help of the package python-decouple)
Edit: I actually realized that the config variables are accessible from my app and from the command if I run bash from within the app page on heroku, but there are not accessible if I run python on an ssh session, i.e., if I run python like this:
$heroku ps:exec
$python
os.environ does not contain my config variables.
Thank you for your help!
I am the original OP. Turns out the problem was the way I was accessing the remote terminal session. The correct way to do it to have all the config vars accessible is
heroku run bash
as opposed to
heroku ps:exec
You can set the config vars for a Heroku application in the settings of the App. Click the Settings Tab then navigate to the Config Vars section. Click Reveal Config Vars and enter the key pair values for your variables.
These will get injected into your application.
Related
I tried everything but couldn't import app.py. It gives me the following error
set FLASK_APP = "C:\Users\Hp\PycharmProjects\sakshi.py\app.py"
(env) C:\Users\Hp\Documents\flask_app>flask run
Serving Flask app " app.py"
Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
Debug mode: off
Usage: flask run [OPTIONS]
Error: Could not import " app".
Your help would be much appreciated!
Change the sever environment to development env.
export FLASK_APP=yourapp
export FLASK_ENV=development
and then start the server with:
flask run
If you still get error try this way:
set FLASK_ENV=development
flask run
And then run your python file.
At least one problem: spaces in the set command. From the syntax in the microsoft doc:
set [<Variable>=[<String>]]
set [/p] <Variable>=[<PromptString>]
set /a <Variable>=<Expression>
There is no space on either side of the = in the set command.
And, the path name "C:\Users\Hp\PycharmProjects\sakshi.py\app.py" is questionable. It is, at the very least, a bad idea to name a directory sakshi.py.
I could see the problem is with space between the variable and value in set command,
In windows,
set var=value
In *nix family,
export var=value
Your script name is app.py which by default flask recognizes as entry point for the app.
You only have to explicitly set the FLASK_APP variable when you are running from another directory (which you are) or the name of the script is something other than app.py. You can directly run the flask run command from the project base directory itself.
I'm setting up flask with docker. I've two ways to set environment variables, one in flask .cfg files and another in docker .env files.
I'm wondering which one is the better practice. Evaluating the pros and cons, if I move my environment variables to docker .env files, I would have to do os.environ.get at all the places in my application code, including handling of defaults which brings extra dependency inside application from os environment variables. On the other hand, adding environment variables like DB password, secret keys, etc inside flask config might not be a right idea, although all my environment variables and defaults would be at one single place.
You don't have to settle for one or the other.
Is not an unusual practice to use environment variables for configuring every aspect of your application, including DB passwords. This is staring to be an often practice in container environments such as Kubernetes or using docker secrets as they give the ability of keeping critical information encrypted and mount it as an environment variable to your container.
You could directly check in your application for the values of the environment variables or another option is to have an entrypoint in docker that checks for those values and ends up creating a configuration file that you use in your application.
This last option allows you to use environment variables to configure your application or if you don't want to you could directly mount a configuration file to your container skipping the envs completely.
This is used for example on the logstash docker image.
Just to add to the great answer by Esteban Garcia, another good way is to use both. Have a configuration file, class-based configs are great here because you can take advantage of config inheritance, and have all non-sensitive options in there. For the sensitive stuff like secrets, database passwords, etc - define them right in the config with os.environ.get to extract these values from the environment. So it ends up looking like this:
class DefaultConfig(Config):
TESTING = False
DEBUG = False
SECRET_KEY = os.getenv('APP_SECRET_KEY')
SQLALCHEMY_DATABASE_URI = os.getenv('APP_DATABASE_URI')
This way you can keep using app.config and not have to do os.environ.get all over your app.
This is my 2 cents to the subject.
The problem
I was working on porting a flask app to docker and wanted to move my settings.py config vars to env vars so they will be picked up by the container's env. This way, it's easier to deploy it on an external service like ECS just by setting env vars and the app will still the same.
Because I had a lot of config vars on settings.py and didn't want to set each of them by hand on the flask app creation, I came up with the following solution.
Also, I will be using docker-compose to execute all the containers.
Migrate from .py to .env
First thing is to move from python code to a syntax that is compatible with docker-compose. So, if you have this in settings.py:
# *****************************
# Environment specific settings
# *****************************
# DO NOT use "DEBUG = True" in production environments
DEBUG = True
# DO NOT use Unsecure Secrets in production environments
SECRET_KEY = 'This is an UNSECURE Secret. CHANGE THIS for production environments.'
# SQLAlchemy settings
SQLALCHEMY_DATABASE_URI = 'postgresql:///../app.postgresql'
Rename that file to settings.env and change the content to:
# *****************************
# Environment specific settings
# *****************************
# DO NOT use "DEBUG=True" in production environments
DEBUG=True
# DO NOT use Unsecure Secrets in production environments
SECRET_KEY=This is an UNSECURE Secret. CHANGE THIS for production environments.
# SQLAlchemy settings
SQLALCHEMY_DATABASE_URI=postgresql:///../app.postgresql
Note that you have to remove all white spaces and quotes (doubles and singles)
After that, you need to load that file on the docker-compose.yaml file:
version: '3.1'
services:
web:
image: web-server
env_file: path_to/settings.env
Then, when you create Flask app, do the following:
# Instantiate Flask
app = Flask(__name__)
for variable, value in os.environ.items():
app.config[env_name] = value
If you don't want to load all available env vars on flask
You can add a prefix to the vars on settings.env like this:
YOURAPP_DEBUG=True
And then on the app creation:
# Instantiate Flask
app = Flask(__name__)
for variable, value in os.environ.items():
if variable.startswith("YOURAPP_"):
env_name = variable.split("YOURAPP_")[1]
app.config[env_name] = value
Every time I start up my flask app the environment variable is set to production. I want to have it set to development mode by default. Otherwise every time I start my app i have to run ..
export FLASK_ENV=development
How can I set environment's default value as development in every startup?
EDIT: I am using flask in a virtual environment on a raspberry pi.
You can edit your main flask app file and add these lines:
if __name__ == '__main__':
app.run(debug=True)
Using this method you have to run your flask app with Python interpreter like this => python app.py
Best Practice:
Install python-dotenv package inside your working environment =>pip install python-dotenv
Create a file named .env, put your environment variables in it, for your case it's FLASK_ENV=development
Then add this code to your config.py or some file that will get loaded before Flask main App
from dotenv import load_dotenv
dotenv_path = join(dirname(__file__), '.env') # Path to .env file
load_dotenv(dotenv_path)
Note that: If you are using flask command to run your application, you don't need to do the third step, flask will find .env files in the project directory by itself.
Using this method, it will only set Environment variable for the project that you have added this code to.
On Linux distro, like "Raspberry pi o.s", specify the environment on the terminal with the code below.
Unless you specify the environment, flask will assume production.
export FLASK_ENV=development
flask run
Like the first answer and instead of adding the variable to a .env file which can be forgotten, do this instead.
This way, if you try to run the file in production, you'll get an assertion error to remind you to actually use a dedicated web server (which "imports" the app). If you run locally, not only will you be reminded to use a .env file, but in the case no environment file is needed, the flask env is set to development to avoid any production conflicts.
import os
app = Flask(__name__)
IS_DEV = app.env == 'development' # FLASK_ENV env. variable
# code
if __name__ == '__main__':
# guaranteed to not be run on a production server
assert os.path.exists('.env') # for other environment variables...
os.environ['FLASK_ENV'] = 'development' # HARD CODE since default is production
app.run(debug=True)
I don't understand why if I want to run Flask application I need
(venv) $ export FLASK_APP=microblog.py
(venv) $ flask run
But if I want to run Django application I only
(venv) $ python manage.py runserver
without export DJANGO_APP=microblog.py
Why? Why I need the export app in the first case, but in the second case I don't need?
Firstly, Django and Flask are different frameworks. There's no reason why the commands to start them should be the same.
You need to export FLASK_APP to tell flask which app to run.
Doing export FLASK_APP=microblog.py sets an environment variable FLASK_APP. The flask application can then read this variable from the environment and use it to run the application.
In Python you can access environment variables from os.environ, or use the os.getenv method:
import os
flask_app = os.getenv('FLASK_APP')
If you use the django-admin command, you need to export DJANGO_SETTINGS_MODULE in a similar way:
$ export DJANGO_SETTINGS_MODULE=yourproject.settings
$ django-admin runserver
However with Django, you usually use runserver with manage.py instead of django-admin. The manage.py is specific to your project and sets the DJANGO_SETTINGS_MODULE environment variable if it hasn't been set already:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "yourproject.settings")
Therefore you don't need to export DJANGO_SETTINGS_MODULE when using manage.py.
From a source code prospective, seems that the FLASK_APP variabile is used for understand which flask app run in a 'multi-flask-app' environment.
Is necessary only if the app_name from the source is not present.
The FLASK_APP env variable is used only into the find_best_app method of the cli.py file of the flask framework.
Simply in your project, you don't need something like export FLASK_APP=microblog.py unless you want environment variables
from flask import jsonify,Flask
app = Flask(__name__)
#app.route("/<Your Route>/<string:<Your Param>>")
def main(<Your Param>):
//DO LOGIC HERE
data =[{'TestData1' : "" ,<YOUR OUTPUT>}]
return jsonify(data), 200
app.run(debug=False,host="0.0.0.0",port=<PORT YOU WANT TO HOST>)
Problem: When you write the command flask run in the console, how does flask know which file to run?
Solution: Thats why we use export FLASK_APP=microblog.py
It is setting the FLASK_APP (an internal flask variable) value to microblog.py
It tells flask to use microblog.py as start-up file for the application when you run flask via flask run command.
If you decide not to do so, then when you run flask run, then there is no way for flask to know which file to run. Now you could run the application using python filename.py instead of flask run
So python microblog.py in your case.
I have been developing on Heroku, using the config variables to store sensitive and other environmental variables. While developing locally, I have mirrored these variables in the .env file.
What I am finding now is the variables from the .env file do not load during migrations. They ARE loading while running the local web server with heroku local, but NOT loading for migrations. This was not a problem while my local app was still using the default sqlite3 database, because the sqlite default DB was "hard-coded" in the settings file. However recently I want to use my local Postgresql DB for local dev. OK so I added the DATABASE_URL variable to my .env file.
I cannot get my local app to migrate to the DB. I have found out that this is because the .env file contents are not added to the os.environ mapping duing migrations.
To test, I added a test variable to the .env file:
TEST="teeeest"
Then in settings.py:
import os
import dj_database_url
if "TEST" not in os.environ:
raise Exception("No .env vars found.")
I tried python manage.py migrate:
File "/Users/apple/heroku/b/b/settings.py", line 16, in
raise Exception("No .env vars found.") Exception: No .env vars found.
However I can run heroku local and there is no error. I have also done further testing to ensure that the .env variables ARE available during heroku local.
For various reason I want to set my local DB connection string in the .env file, but doesn't seem possible at the moment. Is this the correct behavior for django on heroku? .env file variables are only accessible when you run the server, and not for migrations?
I finally figured out that simply running python manage.py migrate did nothing to load the .env file variables. You need to do run the commands in the heroku local environment:
heroku local:run python manage.py migrate
https://devcenter.heroku.com/articles/heroku-local#run-your-app-locally-using-the-heroku-local-command-line-tool-run-a-one-off-command-locally
manage.py doesn't know anything about your .env file. You'll need to run the command under something that does; either Foreman, which is what Heroku itself uses, or Honcho, which is a Python implementation.