I have a Pyramid application which I can start using pserve some.ini. The ini file contains the usual paste configuration and everything works fine. In production, I use uwsgi, having a paste = config:/path/to/some.ini entry, which works fine too.
But instead of reading my configuration from a static ini file, I want to retrieve it from some external key value store. Reading the paste documentation and source code, I figured out, that there is a call scheme, which calls a python function to retrieve the "settings".
I implemented some get_conf method and try to start my application using pserve call:my.module:get_conf. If the module/method do not exist, I get an appropriate error, so the method seems to be used. But whatever I return from the method, I end up with this error message:
AssertionError: Protocol None unknown
I have no idea what return value of the method is expected and how to implement it. I tried to find documentation or examples, but without success. How do I have to implement this method?
While not the answer to your exact question, I think this is the answer to what you want to do. When pyramid starts, your ini file vars from the ini file just get parsed into the settings object that is set on your registry, and you access them through the registry from the rest of your app. So if you want to get settings somewhere else (say env vars, or some other third party source), all you need to do is build yourself a factory component for getting them, and use that in the server start up method that is typically in your base _ _ init _ _.py file. You don't need to get anything from the ini file if that's not convenenient, and if you don't, it doesn't matter how you deploy it. The rest of your app doesn't need to know where they came from. Here's an example of how I do this for getting settings from env vars because I have a distributed app with three separate processes and I don't want to be mucking about with three sets of ini files (instead I have one file of env vars that doesn't go in git and gets sourced before anything gets turned on):
# the method that runs on server startup, no matter
# how you deploy.
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application."""
# settings has your values from the ini file
# go ahead and stick things it from any source you'd like
settings['db_url'] = os.environ.get('DB_URL')
config = Configurator(
settings=settings,
# your other configurator args
)
# you can also just stick things directly on the registry
# for other components to use, as everyone has access to
# request.registry.
# here we look in an env var and fall back to the ini file
amqp_url = os.environ.get('AMQP_URL', settings['amqp.url'] )
config.registry.bus = MessageClient( amqp_url=amqp_url )
# rest of your server start up code.... below
Related
For example, I might try the following config:
class Defaults(Enum):
a = 1
b = 2
Then from my main file, I can refer to it with:
import myconfig
windowSize = Defaults.a
This would allow me to change the enum values whenever I want to vary how my program runs. Is this a common way to use Enums in python configuration?
I think you're asking whether it's common to hold the configuration settings as members of an enumeration. As a more explicit example:
class Defaults(Enum):
window_width = 600
window_height = 480
font_size = 14
Technically, I think that would work, but what benefit is the enumeration providing? Enum is useful for providing choices to pick from. If you really want to do this, I think a plain class, a data class, or just module-level variables would be less confusing. Django's settings.py configuration file seems to be the closest thing to your idea that I've seen in common use.
Your broader question is how to read configuration values for a Python program. Personally, I like the style recommended by The Twelve-Factor App.
The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.
The most flexible way I've found is to use the argparse module, and use the environment variables as the defaults. That way, you can override the environment variables on the command line. Be careful about putting passwords on the command line, though, because other users can probably see your command line arguments in the process list.
Here's an example that uses argparse and environment variables:
def parse_args(argv=None):
parser = ArgumentParser(description='Watch the raw data folder for new runs.',
formatter_class=ArgumentDefaultsHelpFormatter)
parser.add_argument(
'--kive_server',
default=os.environ.get('MICALL_KIVE_SERVER', 'http://localhost:8000'),
help='server to send runs to')
parser.add_argument(
'--kive_user',
default=os.environ.get('MICALL_KIVE_USER', 'kive'),
help='user name for Kive server')
parser.add_argument(
'--kive_password',
default=SUPPRESS,
help='password for Kive server (default not shown)')
args = parser.parse_args(argv)
if not hasattr(args, 'kive_password'):
args.kive_password = os.environ.get('MICALL_KIVE_PASSWORD', 'kive')
return args
Setting those environment variables can be a bit confusing, particularly for system services. If you're using systemd, look at the service unit, and be careful to use EnvironmentFile instead of Environment for any secrets. Environment values can be viewed by any user with systemctl show.
I usually make the default values useful for a developer running on their workstation, so they can start development without changing any configuration.
If you do want to put the configuration settings in a settings.py file, just be careful not to commit that file to source control. I have often committed a settings_template.py file that users can copy.
Some background (not mandatory, but might be nice to know): I am writing a Python command-line module which is a wrapper around latexdiff. It basically replaces all \cite{ref1, ref2, ...} commands in LaTeX files with written-out and properly formatted references before passing the files to latexdiff, so that latexdiff will properly mark changes to references in the text (otherwise, it treats the whole \cite{...} command as a single "word"). All the code is currently in a single file which can be run with python -m latexdiff-cite, and I have not yet decided how to package or distribute it. To make the script useful for anybody else, the citation formatting needs to be configurable. I have implemented an optional command-line argument -c CONFIGFILE to allow the user to point to their own JSON config file (a default file resides in the module folder and is loaded if the argument is not used).
Current implementation: My single-file command-line Python module currently parses command-line arguments in if __name__ == '__main__', and loads the config file (specified by the user in -c CONFIGFILE) here before running the main function of the program. The config variable is thus available in the entire module and all is well. However, I'm considering publishing to PyPI by following this guide which seems to require me to put the command-line parsing in a main() function, which means the config variable will not be available to the other functions unless passed down as arguments to where it's needed. This "passing down by arguments" method seems a little cluttered to me.
Question: Is there a more pythonic way to set some configuration globals in a module or otherwise accomplish what I'm trying to? (I don't want to rely on 3rd party modules.) Am I perhaps completely off the tracks in some fundamental way?
One way to do it is to have the configurations defined in a class or a simple dict:
class Config(object):
setting1 = "default_value"
setting2 = "default_value"
#staticmethod
def load_config(json_file):
""" load settings from config file """
with open(json_file) as f:
config = json.load(f)
for k, v in config.iteritems():
setattr(Config, k, v)
Then your application can access the settings via this class: Config.setting1 ...
I was reading about using YAML to store settings in a GAE application and I think I want to go this way. I'm talking about my own constants like API keys and temp variables, NOT about standard GAE config files like app.yaml.
So far I used a simple settings.py file for it (which included separate configs for production/testing/etc environment), but it doesn't seem to do the job well enough.
I even had some serious problems when a git merge overwrote some settings (hard to control it).
Eventually I want to store as much data as I can in the data store but as for now I'm looking for ideas.
So does anybody have any ideas or examples for simple storing and access of, sometimes protected, config data?
Storing the API keys and configuration variables in static files is usually always a bad idea for a few reasons:
Hard to update: you need to deploy a new app in order to update the values
Not secure: you might commit it to the version control system and there is no way back
Don't scale: very often you have different keys for different domains
So why not storing all these values right from the very beginning securely in the datastore, especially when it's actually so much easier and here is how:
All you have to do is to create one new model for your configuration values and have only one record in it. By using NDB, you have caching out of the box and because of the nature of this configuration file you can even cache it per running instance to avoid regular reads to the datastore.
class Config(ndb.Model):
analytics_id = ndb.StringProperty(default='')
brand_name = ndb.StringProperty(default='my-awesome-app')
facebook_app_id = ndb.StringProperty(default='')
facebook_app_secret = ndb.StringProperty(default='')
#classmethod
def get_master_db(cls):
return cls.get_or_insert('master')
I'm also guessing that you already have one file that most likely is called config.py where you have some constants like these ones:
PRODUCTION = os.environ.get('SERVER_SOFTWARE', '').startswith('Google App Engine')
DEVELOPMENT = not PRODUCTION
If you don't create one or if you do just add this to that file as well:
import model # Your module that contains the models
CONFIG_DB = model.Config.get_master_db()
and finally to be able to read your config values you can simply do that, wherever in your app (after importing the config.py of course):
config.CONFIG_DB.brand_name
From now on you don't even have to create any special forms to update these values (it is recommended though), because you can do that from the admin console locally or the Dashboard on the production.
Just remember that if you're going to store that record in a local variable you will have to restart the instances after updating the values to see the changes.
All of the above you can see it in action, in one of my open sourced projects called gae-init.
You should not store config values in source code, for the reasons outlined in the other answers. In my latest project, I put config data in the datastore using this class:
from google.appengine.ext import ndb
class Settings(ndb.Model):
name = ndb.StringProperty()
value = ndb.StringProperty()
#staticmethod
def get(name):
NOT_SET_VALUE = "NOT SET"
retval = Settings.query(Settings.name == name).get()
if not retval:
retval = Settings()
retval.name = name
retval.value = NOT_SET_VALUE
retval.put()
if retval.value == NOT_SET_VALUE:
raise Exception(('Setting %s not found in the database. A placeholder ' +
'record has been created. Go to the Developers Console for your app ' +
'in App Engine, look up the Settings record with name=%s and enter ' +
'its value in that record\'s value field.') % (name, name))
return retval.value
Your application would do this to get a value:
AMAZON_KEY = Settings.get('AMAZON_KEY')
If there is a value for that key in the datastore, you will get it. If there isn't, a placeholder record will be created and an exception will be thrown. The exception will remind you to go to the Developers Console and update the placeholder record.
I find this takes the guessing out of setting config values. If you are unsure of what config values to set, just run the code and it will tell you!
I have some python scripts that configure linux computers. One of the tasks is to modify a configuration file for subversion. This file, ~/.subversion/servers is very simple and looks like this:
# store-passwords = no
# store-plaintext-passwords = no
# store-ssl-client-cert-pp = no
# store-ssl-client-cert-pp-plaintext = no
... lots of other options ...
The task of my script is to find a required option, for example store-plaintext-passwords and to set it to specified value, for example yes. The problem is: the script can run multiple times on same machine, so if it is run first time this option can be just commented, if it is run second time it can be uncommented and set to yes, third run can point out that it is uncommented - but set to no etc. Currently i have a rather complex code that search file for the string, splits it for comment/name/value, uncomments it if needed, changes value if needed and replaces it. Maybe it's an easier way possible?
The ~/.subversion/servers file is in INI format.
So you can use the ConfigParser for implementing whatever you need.
http://docs.python.org/library/configparser.html
I am working on a Pylons app that runs on top of Apache with mod_wsgi. I would like to send logging messages that my app generates to files in my app's directory, instead of to Apache's logs. Further, I would like to specify the location of logfiles via a relative path so that it'll be easier to deploy my app on other people's servers. Right now I can log to files, but only via a fragile absolute path.
Here is the relevant part of my development.ini file:
# Logging configuration
[loggers]
keys = root, routes, myapp, sqlalchemy, debugging-logger
[handlers]
keys = console, debugging-logger-file
[formatters]
keys = generic
[logger_debugging-logger]
level = DEBUG
handlers = debugging-logger-file
qualname = myapp.controllers.logging-test-controller.debugging-logger
[handler_debugging-logger-file]
class = FileHandler
args = ('/var/pylons/myapp/logs/myapp-debugging-errors.log', 'a')
level = DEBUG
formatter = generic
Although the .ini helpfully advises using %(here)s to refer to the current path, using %(here)s in the "args = ('foo')" line of the error handler does not behave the way that I expect it to. The syntax of this ini file is documented on the Paste Deploy site, but does not specify how %(here)s can be used in relation to quoted strings.
What syntax should I use in the "args = ('foo')" line to specify the current path?
The problem is that Paste Deploy creates one ConfigParser object to store the 'here' tag in it's set of defaults, and logging.config.fileConfig() is never passed that set of defaults. Therefore, when fileConfig() reads the .ini file, it doesn't have access to the 'here' tag, and the ConfigParser's interpolation can't find it.
You could do something like this:
[DEFAULT]
my_log_dir = '/var/pylons/myapp/logs'
...
[handler_debugging-logger-file]
args = (%(my_log_dir)s + '/myapp-debugging-errors.log', 'a')
Not exactly what you're looking for, but a tiny bit more configurable.
Another possibility is:
args = (os.getcwd() + '/myapp-debugging-errors.log', 'a')
(This works because 'os' is a valid variable in the logging module's namespace when it calls eval() on the args value. But this is an implementation detail of the logging package that may not be reliable long term.) But this most likely won't give you what you want-- it will most likely use the Apache process's working directory.
You could even set an environment variable outside the program, and use it like:
args = (os.environ['MY_LOG_DIR'] + '/myapp-debugging-errors.log', 'a')
And yet another possibility is overriding the behavior of some of the functions or class methods in the logging module or paste package.
Hope those give you some ideas.
Configuration files for Paste Deploy allow a 'here' tag to indicate directory where configuration file is. You can then work relative to that.