The flask documentation recommends to use an instance folder to hide sensitive configuration variables (e.g. api keys, secret keys) from version control. Is it acceptable to keep sensitive configuration variables in an instance folder when deploying to an Azure App Service for production or should the variables be stored using a different method (e.g. enviromental variables, Azure key vaults)?
keeping sensiitive data in GIT is not a good practice.
Azure keyvalut is not free, but its safer than anything else.
you can just use the azure app configuration
I'd say depends on your definition of acceptable :) it would be more secure to use Environment variable or Azure Key Vault, but its certainly possible to just store those in git
As the title says, I have a Python script I wrote that I would like to allow others to use.
The script is an API aggregator, and it requires a client_id and secret to access the API. As of now I have an env file which stores these values and I'm able to get the values from the env file.
My question is now that I have finished the script locally, how do I deploy with the environment variables it so others can use it given that the environment variables are required?
Sorry if this is a simple question - new to writing scripts for others to use.
The only thing I could think of was including the .env when I push to github, but not sure if that's great practice since my client_id and secret are stored there
Similar question was asked here, however the solution does not give the shell access to the same environment as the deployment. If I inspect os.environ from within the shell, none of the environment variables appear.
Is there a way to run the manage.py shell with the environment?
PS: As a little side question, I know the mantra for EBS is to stop using eb ssh, but then how would you run one-off management scripts (that you don't want to run on every deploy)?
One of the cases you have to run something once is db schema migrations. Usually you store information about that in the db... So you can use db to sync / ensure that something was triggered only once.
Personally I have nothing against using eb ssh, I see problems with it however. If you want to have CI/CD, that manual operation is against the rules.
Looks like you are referring to WWW/API part of Beanstalk. If you need something that is quite frequent... maybe worker is more suitable? Problem here is that if API goes deployed first you would have wrong schema.
In general you are using EC2, so it's user data stores information that spins up you service. So there you can put your "stuff". Still you need to sync / ensure. Here are docs for beanstalk - for more information how to do that.
Edit
Beanstalk is kind of instrumentation on top of EC2. So there must be a way to work with it, since you have access to user data of that EC2s. No worries you don't need to dig that deep. There is good way of instrumenting your server. It is called ebextensions. It can be used to put files on the server, trigger commands, instrument cron. What ever you want.
You can create ebextension, with container_commands this time Python Configuration Namespaces section. That commands are executed on each deployment. Still, problem is that you need to sync since more then one deployment can go at the same time. Good part is that you can set env in the way you want.
I have no problem to access to the environment variables. How did you get the problem? Try do prepare page with the map.
Let's say I have some code running on a Heroku dyno (such as this autoscaling script), that needs access to the Platform API. To access the API, I have to authenticate using my app's API Key.
What's the right way to do this?
That script I referenced hardcoded the API Key in the script itself.
A better practice generally seems to put secrets in environment variables, which is what Heroku normally recommends. However, they say they say:
Setting the HEROKU_API_KEY environment variable on your machine will
interfere with normal functioning of auth commands from Toolbelt.
Clearly I could store the API key with under a different key name.
What's the right way? I couldn't find this in the documentation, but seems like a common issue.
Yes, storing this token into a config var is the right way to go.
As for HEROKU_API_KEY, this will happen because locally, the toolbelt will look for the environment variable as one solution to try to fetch your token.
This won't impact your production environment (the heroku toolbelt isn't available within dynos).
Locally, you can also set it easily with a tool like python-dotenv, which will allow you to have a local .env file (don't check it into source control, or your token could be corrupted), with all of it's values available as env vars in your dev app.
Having properly configured a Development server and a Production server, I would like to set up a Staging environment on Google App Engine useful to test new developed versions live before deploying them to production.
I know two different approaches:
A. The first option is by modifying the app.yaml version parameter.
version: app-staging
What I don't like of this approach is that Production data is polluted with my staging tests because (correct me if I'm wrong):
Staging version and Production version share the same Datastore
Staging version and Production version share the same logs
Regarding the first point, I don't know if it could be "fixed" using the new namespaces python API.
B. The second option is by modifying the app.yaml application parameter
application: foonamestaging
with this approach, I would create a second application totally independent from the Production version.
The only drawback I see is that I'm forced to configure a second application (administrators set up).
With a backup\restore tool like Gaebar this solution works well too.
What kind of approach are you using to set up a staging environment for your web application?
Also, do you have any automated script to change the yaml before deploying?
If separate datastore is required, option B looks cleaner solution for me because:
You can keep versions feature for real versioning of production applications.
You can keep versions feature for traffic splitting.
You can keep namespaces feature for multi-tenancy.
You can easily copy entities from one app to another. It's not so easy between namespaces.
Few APIs still don't support namespaces.
For teams with multiple developers, you can grant upload to production permission for a single person.
I chose the second option in my set-up, because it was the quickest solution, and I didn't make any script to change the application-parameter on deployment yet.
But the way I see it now, option A is a cleaner solution. You can with a couple of code lines switch the datastore namespace based on the version, which you can get dynamically from the environmental variable CURRENT_VERSION_ID as documented here: http://code.google.com/appengine/docs/python/runtime.html#The_Environment
We went with the option B. And I think it is better in general as it isolates the projects completely. So for example playing around with some of the configurations on the staging server will not affect and wont compromise security or cause any other butterfly effect in your production environment.
As for the deployment script, you can have any application name you want in your app.yaml. Some dummy/dev name and when you deploy, just use an -A parameter:
appcfg.py -A your-app-name update .
That will simplify your deploy script quite much, no need to string replace or anything similar in your app.yaml
We use option B.
In addition to Zygmantas suggestions about the benefits of separating dev from prod at application level, we also use our dev application to test performance.
Normally the dev instance runs without much available in the way of resources, this helps to see where the application "feels" slow. We can then also independently tweak the performance settings to see what makes a difference (e.g. front-end instance class).
Of course sometimes we need to bite the bullet and tweak & watch on live. But it's nice to have the other application to play with.
Still use namespaces and versions, just dev is dirty and experimental.
Here is what the Google documentation says :
A general recommendation is to have one project per application per
environment. For example, if you have two applications, "app1" and
"app2", each with a development and production environment, you would
have four projects: app1-dev, app1-prod, app2-dev, app2-prod. This
isolates the environments from each other, so changes to the
development project do not accidentally impact production, and gives
you better access control, since you can (for example) grant all
developers access to development projects but restrict production
access to your CI/CD pipeline
With this in mind, add a dispatch.yaml file at the root directory, and in each directory or repository that represents a single service and contain that service, add a app.yaml file along with the associated source code, as explained here : Structuring web services in App Engine
Edit, check out the equivalent link in the python section if you're using python.
No need to create a separate project. You can use dispatch.yaml to route your staging URL to another service (staging) in the same project.
Create a custom domain staging.yourdomain.com
Create a separate app-staging.yaml, that specifies staging service.
...
service: staging
...
Create distpatch.yaml that contains something like
...
url: "*staging.mydomain.com/"
service: staging
url: "*mydomain.com/"
service: default
...
gloud app deploy app-staging.yaml dispatch.yaml
use of application in app.yaml has been shut down.
Instead Google recommends
gcloud app deploy --project [YOUR_PROJECT_ID]
Please see https://cloud.google.com/appengine/docs/standard/python/config/appref