I'm trying to configure Celery using an external file, to separate config from code. Both Celery and Flask have config.from_envvar() / config_from_envvar() methods except they behave a bit differently and what I am doing only works with Flask.
Flask
Basically, in Flask, I do
app.config.from_object(config_class)
app.config.from_envvar('SETTINGS_FILE', silent=True)
which loads a default configuration stored in application code, then loads a settings file stored anywhere in the filesystem with a few customized settings overriding those in the default file. I just need to write that file and pass its path through an environment variable.
(More on this in Flask docs or in this answer. I find it a bit ambiguous that Flask treats the file as a Python file while the example uses settings.cfg (no .py extension) but it works fine).
Celery
When doing the same with Celery, I get this error:
ImportError: No module named '/absolute/path/to/settings'
I named the file settings.py (.py extension, in case it matters).
I don't know where to put that file. The examples I've seen, for instance in answers to this question, put the config file in the code, which is precisely what I would like to avoid.
since you neither want to manipulate PYTHONPATH nor put config file under application directory, the only option left will be load config from filepath:
def load_config_from_file(celery_app, filepath):
conf = {}
with open(filepath) as fp:
exec(compile(fp.read(), filepath, 'exec'), {}, d)
celery_app.config_from_object(conf)
BTW, to my knowledge most (if not all) celery config option names have no conflict with flask's, you could write them together into one file, let flask load it, then celery could just read config from flask:
celery_app.conf.update(flask_app.config)
Related
So I have my Django project, and am trying to deploy it via heroku server. I realized that I need to created secrets.json file (in the same directory as manage.py), and then add it to .gitignore, for sake of security. But after deploying the project, I got an error saying it couldn't find secrets.json file. Below is my code.
settings.py
secret_file = os.path.join(BASE_DIR, 'secrets.json')
with open(secret_file) as f:
secrets = json.loads(f.read())
def get_secret(setting, secrets=secrets):
try:
return secrets[setting]
except KeyError:
error_msg = "Set the {} environment variable".format(setting)
raise ImproperlyConfigured(error_msg)
SECRET_KEY = get_secret("SECRET_KEY")
secrets.json
{
"SECRET_KEY" : xxx,
"EMAIL_HOST_USER" : xxx,
"EMAIL_HOST_PASSWORD" : xxx
}
After saving these, I added secrets.json to .gitignore.
But an error occurs at line 2 (with open(secret_file) as f:), saying it can't find secrets.json file. What do you think is the problem?
*FYI, The BASE_DIR doesn't seem to be the problem, since it works fine else where. Such as STATICFILES_DIRS.
You are correct to ignore your secrets file. It shouldn't be part of your repository, and it shouldn't be shipped to Heroku. Since that file is not included in your source code, Heroku can't open it.
As far as I know, having a secrets.json is not a common pattern in the Django / Python world¹.
I'm a bit confused by this:
def get_secret(setting, secrets=secrets):
try:
return secrets[setting]
except KeyError:
error_msg = "Set the {} environment variable".format(setting)
raise ImproperlyConfigured(error_msg)
Your KeyError handler refers to environment variables. That's on the right track! Configuring your application with environment variables is a common pattern.
But a JSON file called secrets.json isn't an environment variable. A more common approach would be to load values in your settings file directly from the environment:
import os
SECRET_KEY = os.getenv("SECRET_KEY")
# You can even provide a default in case the environment variable isn't set:
ANOTHER_SETTING = os.getenv("ANOTHER_SETTING", default="some-default-value")
On Heroku, you can set config vars that will be automatically injected into your environment:
heroku config:set SECRET_KEY=some-value
So that leaves local development. There are several ways to set environment variables on your local machine, but a common one is to create a file called .env:
SECRET_KEY=xxx
EMAIL_HOST_USER=xxx
EMAIL_HOST_PASSWORD=xxx
If you run your code locally via heroku local it will automatically pick up your .env file. So will Pipenv, if you use that. There's also python-dotenv and direnv and others.
.env files have become something of a de facto standard, and you're much more likely to find tools supporting them than ones supporting a secrets.json file.
¹There is python-secrets, which might help you with secrets.json files. You appear to be opening your secrets file manually, though.
If you plan to keep using your secrets file, I suggest you find something like this that automatically reads the file and merges settings into the environment. I've never used this library and only came across it in a search while answering this question.
I am creating a Python script which reads a spreadsheet and issues Rest requests to an external service. A token must be obtained to issue requests to the Rest service, so the script needs a username and password to obtain the oauth2 token.
What's the best or standard way to hide or not have visible information in the Python script?
I recommend using a config file. Let's create a config file and name it config.cfg. The file structure should look more or less like this:
[whatever]
key=qwerertyertywert2345
secret=sadfgwertgrtujdfgh
Then in python you can load it this way:
from configparser import ConfigParser
config = ConfigParser()
config.read('config.cfg')
my_key = config['whatever']['key']
my_secret = config['whatever']['secret']
In general, the most standard way to handle secrets in Python is by putting them in runtime configuration.
You can do that by reading explicitly from external files or using os.getenv to read from environment variables.
Another way is to use a tool like python-decouple, which lets you use the environment (variables), a .env file, and an .ini file in a cascade, so that developers and operations can control the environment in local, dev, and production systems.
At present my application related variables like external server IP / default pattern etc.. These variables are specific to my application. It includes my external server username and password.
Now how can I have a single common place so that I can externalise this from my application.
The options which I thought are below:
Have one conf.ini file and use configparser and read this during the start of the django app
But I am not sure from where I should have method to read this so that it will be done in the startup.
Other option is to save this in a py file itself and import this file in my modules
Please suggests me the good and standard approach for above issue.
Save the important secret details in an ini file and place it in etc/project_name/my_settings.ini. Read these settings from settings.py. This way it will be secure. Can be read directly in the settings file itself
Or better way is to set them in bashrc, read the env vars from it.
Check this: Setting envs
I'm adding some static files to my Python app on Google app engine. I do it like this, as described here.
app.yaml:
handlers:
- url: /stylesheets
static_dir: stylesheets
I put my files in the subdirectory stylesheets within the my_project_app folder.
How can I get the last modified date of a file in stylesheets?
When you indicate the a file or directory is static by default you can not access it with your application, which means you can't get the modification date. You can think of the static files as being uploaded to a different machine that is configured to to serve static files.
You can upload the file both as a static file and an application resource (or use a symlink), but this means it counts twice against the quota. Recently this has been made easier with the addition of the application_readable option. Setting it to true in app.yaml essentially does the same thing. (see: https://cloud.google.com/appengine/docs/python/config/appconfig#Static_Directory_Handlers ).
Once your application can read the file you can use the standard os library to read the whatever information you need.
Answer is simple make static files accessible that is all.
https://cloud.google.com/appengine/docs/python/config/appconfig#Python_app_yaml_Static_file_handlers
application_readable
Optional. By default, files declared in static file handlers are
uploaded as static data and are only served to end users, they cannot
be read by an application. If this field is set to true, the files are
also uploaded as code data so your application can read them. Both
uploads are charged against your code and static data storage resource
quotas.
use such code to locate file relative to python file - sometimes you need .. to go up.
os.path.join(os.path.dirname(__file__), 'template', 'list_tbody_part.html')
Per suggestions on SO/SF and other sites, I am using CherryPy as the WSGI server to launch multiple instances of a Python web server I built with Flask. Each instance runs on its own port and sits behind Nginx. I should note that the below does work for me, but I'm troubled that I have gone about things the wrong way and it works "by accident".
Here is my current cherrypy.conf file:
[global]
server.socket_host = '0.0.0.0'
server.socket_port = 8891
request.dispatch: cherrypy.dispatch.MethodDispatcher()
tree.mount = {'/':my_flask_server.app}
Without diving too far into my Flask server, here's how it starts:
import flask
app = flask.Flask(__name__)
#app.route('/')
def hello_world():
return "hello"
And here is the command I issue on the command line to launch with Cherryd:
cherryd -c cherrypy.conf -i my_flask_server
Questions are:
Is wrapping Flask inside CherryPy still the preferred method of using Flask in production? https://stackoverflow.com/questions/4884541/cherrypy-vs-flask-werkzeug
Is this the proper way to use a .conf file to launch CherryPy and import the Flask app? I have scoured the CherryPy documentation, but I cannot find any use cases that match what I am trying to do here specifically.
Is the proper way to launch multiple CherryPy/Flask instances on a single machine to execute multiple cherryd commands (daemonizing with -d, etc) with unique .conf files for each port to be used (8891, 8892, etc)? Or is there a better "CherryPy" way to accomplish this?
Thanks for any help and insight.
I can't speak for Flask, but I can for CherryPy. That looks like the "proper way"...mostly. That line about a MethodDispatcher is a no-op since it only affects CherryPy Applications, and you don't appear to have mounted any (just a single Flask app instead).
Regarding point 3, you have it right. CherryPy allows you to run multiple Server objects in the same process in order to listen on multiple ports (or protocols), but it doesn't have any sugar for starting up multiple processes. As you say, multiple cherryd commands with varying config files is how to do it (unless you want to use a more integrated cluster/config management tool like eggmonster).
Terminology: Mounting vs Grafting
In principle this is a proper way to serve a flask app through cherrypy, just a quick note on your naming:
It is worth noting here that tree.mount is not a configuration key by itself - tree will lead to cherrypy._cpconfig._tree_config_handler(k, v) being called with the arguments 'mount', {'/': my_flask_server.app}.
The key parameter is not used at all by the _tree_config_handler so in your config "mount" is just an arbitrary label for that specific dict of path mappings. It also does not "mount" the application (it's not a CherryPy app after all). By that I mean, it does not cherrypy.tree.mount(…) it but rather cherrypy.tree.grafts an arbitrary WSGI handler onto your "script-name" (paths, but in CherryPy terminology) namespace.
Cherrypy's log message somewhat misleadingly says "Mounted <app as string> on /"]
This is a somewhat important point since with graft, unlike mount, you cannot specify further options such as static file service for your app or streaming responses on that path.
So I would recommend changing the tree.mount config key to something descriptive that does not invite reading too much semantics about what happens within CherryPy (since there is the cherrypy.tree.mount method) due to that config. E.g., tree.flask_app_name if you're just mapping that one app in that dict (there can be many tree directives, all of them just getting merged into the paths namespace) or tree.wsgi_delegates if you map many apps in that dict.
Using CherryPy to serve additional content without making an app of it
Another side note, if you want cherrypy to e.g. provide static file service for your app, you don't have to create a boilerplate cherrypy app to hold that configuration. You just have to mount None with the appropriate additional config. The following files would suffice to have CherryPy to serve static content from the subdirectory 'static' if they are put into the directory where you launch cherryd to serve static content (invoke cherryd as cherryd -c cherrypy.conf -i my_flask_server -i static:
static.py
import cherrypy
# next line could also have config as an inline dict, but
# file config is often easier to handle
cherrypy.tree.mount(None, '/static-path', 'static.conf')
static.conf
# static.conf
[/]
tools.staticdir.on = True
tools.staticdir.root = os.getcwd()
tools.staticdir.dir = 'static'
tools.staticdir.index = 'index.html'