I was trying to read a boolean value using config parser, It finds the section fine but it fails finding the option for some reason.
configparser.NoOptionError: No option 'firsttime' in section: 'DEFAULT'
In the INI i have:
[DEFAULT]
"firsttime" = "true"
And in my main folder i have:
import configparser
config = configparser.ConfigParser()
print(config.getboolean('DEFAULT', 'firsttime'))
ConfigParser() constructs the configuration parsing object, but doesn't read in any configuration file. To do that you have to invoke its read method, e.g. config.read(<your_config_file>). So if your config file was named example.ini, your code would look like:
import configparser
config = configparser.ConfigParser()
config.read('example.ini')
print(config.getboolean('DEFAULT', 'firsttime'))
As a side note putting quotations in the init file will screw up the parsing of it. So you'll want your init file to look like:
[DEFAULT]
firsttime = true
Related
I am using Argparse for fetching the necessary value from config file.
For example:
python arg.py --event_conf=/opt/open-stack-tools/track_events.conf --openstack_conf=/etc/nova/nova.conf
I need to fetch value from two different files.
I can be able to get the results as needed for one local config file.
But In case of fetching the necessary values from nova.conf file, it results in following error:
Traceback (most recent call last):
File "arg.py", line 36, in <module>
oslo_messaging_rabbit= dict(config.items("oslo_messaging_rabbit"))
File "/usr/lib/python2.7/ConfigParser.py", line 655, in items
for option in options]
File "/usr/lib/python2.7/ConfigParser.py", line 691, in _interpolate
self._interpolate_some(option, L, rawval, section, vars, 1)
File "/usr/lib/python2.7/ConfigParser.py", line 723, in _interpolate_some
option, section, rest, var)
ConfigParser.InterpolationMissingOptionError: Bad value substitution:
section: [oslo_messaging_rabbit]
option : logging_exception_prefix
key : color
rawval : %(asctime)s.%(msecs)03d TRACE %(name)s %(instance)s
Is there any way to fix the same.
I have copied the necessary contents and created a new local file, I can see that it is working fine.
When I am using the nova.conf file it results in error.
I can't change the file which I am using.
So I need a fix for the particular error.
Note:
Adding more details as needed:
parser.add_argument("-c", "--event_conf",
help="Specify config file 1", metavar="FILE")
args1, remaining_argv1 = parser.parse_known_args()
parser.add_argument("-o", "--openstack_conf",
help="Specify config file 2", metavar="FILE")
args2, remaining_argv2 = parser.parse_known_args()
if args1.event_conf:
config = ConfigParser.SafeConfigParser()
print config.read([args1.event_conf])
config.read([args1.event_conf])
configdetails_section1 = dict(config.items("configdetails_section1"))
I found the solution for the same.
Actually issue was with the configparser which I have used.
Instead of SafeConfigParser I changed it to RawConfigParser.
Then I can be able to see that it is working fine.
Interpolation allows you to refer to values of the configuration while defining others. For example, in this config file:
[bug_tracker]
protocol = http
server = localhost
port = 8080
url = %(protocol)s://%(server)s:%(port)s/bugs/
username = dhellmann
password = SECRET
if you read like this:
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('interpolation.ini')
print 'value =', parser.get('bug_tracker', 'url')
you will get:
value = http://localhost:8080/bugs/
that can be very useful, the problem seems that that you pass some values in run time. I mean you need the format string to actually substitute the values by yourself.
You can use RawConfigParser instead of SafeConfigParser but then you loose all the interpolations.
Instead you can suppress the interpolation for one specific value:
print 'value =', parser.get('bug_tracker', 'url', raw=True)
and the result would be:
value = %(protocol)s://%(server)s:%(port)s/bugs/
There is also the possibility that you need to combine interpolated values with some of them given in evaluation time. For example if you want to give the user in evaluation time, you can also include it into the config expression:
[bug_tracker]
protocol = http
server = localhost
port = 8080
url = %(protocol)s://%(user)s#%(server)s:%(port)s/bugs/
username = dhellmann
password = SECRET
and then you need to make something like this:
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('interpolation.ini')
parser.set('bug_tracker', 'user', 'admin')
print 'value =', parser.get('bug_tracker', 'url')
and you'll get:
value = http://admin#localhost:8080/bugs/
Sorry that I didn't use your example. I did take the one in the documentation of another project. See the section: Combining Values with Interpolation
it's a little bit I'm out of python syntax and I have a problem in reading a .ini file with interpolated values.
this is my ini file:
[DEFAULT]
home=$HOME
test_home=$home
[test]
test_1=$test_home/foo.csv
test_2=$test_home/bar.csv
Those lines
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('config.ini')
print parser.get('test', 'test_1')
does output
$test_home/foo.csv
while I'm expecting
/Users/nkint/foo.csv
EDIT:
I supposed that the $ syntax was implicitly included in the so called string interpolation (referring to the manual):
On top of the core functionality, SafeConfigParser supports
interpolation. This means values can contain format strings which
refer to other values in the same section, or values in a special
DEFAULT section.
But I'm wrong. How to handle this case?
First of all according to the documentation you should use %(test_home)s to interpolate test_home. Moreover the key are case insensitive and you can't use both HOME and home keys. Finally you can use SafeConfigParser(os.environ) to take in account of you environment.
from ConfigParser import SafeConfigParser
import os
parser = SafeConfigParser(os.environ)
parser.read('config.ini')
Where config.ini is
[DEFAULT]
test_home=%(HOME)s
[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv
You can write custom interpolation in case of Python 3:
import configparser
import os
class EnvInterpolation(configparser.BasicInterpolation):
"""Interpolation which expands environment variables in values."""
def before_get(self, parser, section, option, value, defaults):
value = super().before_get(parser, section, option, value, defaults)
return os.path.expandvars(value)
cfg = """
[section1]
key = value
my_path = $PATH
"""
config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])
If you want to expand some environment variables, you can do so using os.path.expandvars before parsing a StringIO stream:
import ConfigParser
import os
import StringIO
with open('config.ini', 'r') as cfg_file:
cfg_txt = os.path.expandvars(cfg_file.read())
config = ConfigParser.ConfigParser()
config.readfp(StringIO.StringIO(cfg_txt))
the trick for proper variable substitution from environment is to use the ${} syntax for the environment variables:
[DEFAULT]
test_home=${HOME}
[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv
ConfigParser.get values are strings, even if you set values as integer or True. But ConfigParser has getint, getfloat and getboolean.
settings.ini
[default]
home=/home/user/app
tmp=%(home)s/tmp
log=%(home)s/log
sleep=10
debug=True
config reader
>>> from ConfigParser import SafeConfigParser
>>> parser = SafeConfigParser()
>>> parser.read('/home/user/app/settings.ini')
>>> parser.get('defaut', 'home')
'/home/user/app'
>>> parser.get('defaut', 'tmp')
'/home/user/app/tmp'
>>> parser.getint('defaut', 'sleep')
10
>>> parser.getboolean('defaut', 'debug')
True
Edit
Indeed you could get name values as environ var if you initialize SafeConfigParser with os.environ. Thanks for the Michele's answer.
Quite late, but maybe it can help someone else looking for the same answers that I had recently. Also, one of the comments was how to fetch Environment variables and values from other sections. Here is how I deal with both converting environment variables and multi-section tags when reading in from an INI file.
INI FILE:
[PKG]
# <VARIABLE_NAME>=<VAR/PATH>
PKG_TAG = Q1_RC1
[DELIVERY_DIRS]
# <DIR_VARIABLE>=<PATH>
NEW_DELIVERY_DIR=${DEL_PATH}\ProjectName_${PKG:PKG_TAG}_DELIVERY
Python Class that uses the ExtendedInterpolation so that you can use the ${PKG:PKG_TAG} type formatting. I add the ability to convert the windows environment vars when I read in INI to a string using the builtin os.path.expandvars() function such as ${DEL_PATH} above.
import os
from configparser import ConfigParser, ExtendedInterpolation
class ConfigParser(object):
def __init__(self):
"""
initialize the file parser with
ExtendedInterpolation to use ${Section:option} format
[Section]
option=variable
"""
self.config_parser = ConfigParser(interpolation=ExtendedInterpolation())
def read_ini_file(self, file='./config.ini'):
"""
Parses in the passed in INI file and converts any Windows environ vars.
:param file: INI file to parse
:return: void
"""
# Expands Windows environment variable paths
with open(file, 'r') as cfg_file:
cfg_txt = os.path.expandvars(cfg_file.read())
# Parses the expanded config string
self.config_parser.read_string(cfg_txt)
def get_config_items_by_section(self, section):
"""
Retrieves the configurations for a particular section
:param section: INI file section
:return: a list of name, value pairs for the options in the section
"""
return self.config_parser.items(section)
def get_config_val(self, section, option):
"""
Get an option value for the named section.
:param section: INI section
:param option: option tag for desired value
:return: Value of option tag
"""
return self.config_parser.get(section, option)
#staticmethod
def get_date():
"""
Sets up a date formatted string.
:return: Date string
"""
return datetime.now().strftime("%Y%b%d")
def prepend_date_to_var(self, sect, option):
"""
Function that allows the ability to prepend a
date to a section variable.
:param sect: INI section to look for variable
:param option: INI search variable under INI section
:return: Void - Date is prepended to variable string in INI
"""
if self.config_parser.get(sect, option):
var = self.config_parser.get(sect, option)
var_with_date = var + '_' + self.get_date()
self.config_parser.set(sect, option, var_with_date)
Based on #alex-markov answer (and code) and #srand9 comment, the following solution works with environment variables and cross-section references.
Note that the interpolation is now based on ExtendedInterpolation to allow cross-sections references and on before_read instead of before_get.
#!/usr/bin/env python3
import configparser
import os
class EnvInterpolation(configparser.ExtendedInterpolation):
"""Interpolation which expands environment variables in values."""
def before_read(self, parser, section, option, value):
value = super().before_read(parser, section, option, value)
return os.path.expandvars(value)
cfg = """
[paths]
foo : ${HOME}
[section1]
key = value
my_path = ${paths:foo}/path
"""
config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])
It seems in the last version 3.5.0, ConfigParser was not reading the env variables, so I end up providing a custom Interpolation based on the BasicInterpolation one.
class EnvInterpolation(BasicInterpolation):
"""Interpolation as implemented in the classic ConfigParser,
plus it checks if the variable is provided as an environment one in uppercase.
"""
def _interpolate_some(self, parser, option, accum, rest, section, map,
depth):
rawval = parser.get(section, option, raw=True, fallback=rest)
if depth > MAX_INTERPOLATION_DEPTH:
raise InterpolationDepthError(option, section, rawval)
while rest:
p = rest.find("%")
if p < 0:
accum.append(rest)
return
if p > 0:
accum.append(rest[:p])
rest = rest[p:]
# p is no longer used
c = rest[1:2]
if c == "%":
accum.append("%")
rest = rest[2:]
elif c == "(":
m = self._KEYCRE.match(rest)
if m is None:
raise InterpolationSyntaxError(option, section,
"bad interpolation variable reference %r" % rest)
var = parser.optionxform(m.group(1))
rest = rest[m.end():]
try:
v = os.environ.get(var.upper())
if v is None:
v = map[var]
except KeyError:
raise InterpolationMissingOptionError(option, section, rawval, var) from None
if "%" in v:
self._interpolate_some(parser, option, accum, v,
section, map, depth + 1)
else:
accum.append(v)
else:
raise InterpolationSyntaxError(
option, section,
"'%%' must be followed by '%%' or '(', "
"found: %r" % (rest,))
The difference between the BasicInterpolation and the EnvInterpolation is in:
v = os.environ.get(var.upper())
if v is None:
v = map[var]
where I'm trying to find the var in the enviornment before checking in the map.
Below is a simple solution that
Can use default value if no environment variable is provided
Overrides variables with environment variables (if found)
needs no custom interpolation implementation
Example:
my_config.ini
[DEFAULT]
HOST=http://www.example.com
CONTEXT=${HOST}/auth/
token_url=${CONTEXT}/oauth2/token
ConfigParser:
import os
import configparser
config = configparser.ConfigParser(interpolation=configparser.ExtendedInterpolation())
ini_file = os.path.join(os.path.dirname(__file__), 'my_config.ini')
# replace variables with environment variables(if exists) before loading ini file
with open(ini_file, 'r') as cfg_file:
cfg_env_txt = os.path.expandvars(cfg_file.read())
config.read_string(cfg_env_txt)
print(config['DEFAULT']['token_url'])
Output:
If no environtment variable $HOST or $CONTEXT is present this config will take the default value
user can override the default value by creating $HOST, $CONTEXT environment variable
works well with docker container
I'm having trouble using a config file, because the option starts with #, thus python treats it as a comment (like it should).
The part of the config file that is not working:
[channels]
#channel
As you may see, it's an IRC channel, that is why it needs the #. Now I could use some ugly method of adding the # everytime I need it, but I'd prefer to keep it clean.
So is there any way to ignore this? So that when I were to print the option, it would start with
If your setting that in a python file you can escape the # with \
Otherwise I think that should be in a config file with other syntax that doesn't treat # as a commented line
You are probably using ConfigParser - which you should mention btw - then you have to pre-/postprocess the configfile before feeding it to the parser, because ConfigParser ignores the comment-parts.
I can think of two ways, both of them make use of the readfp, instead of the read-method of the ConfigParser-class:
1) subclass StreamWriter and StreamReader from the codecs-module and use them to wrap the opening-process in a transparent recoding.
2) use StringIO from the io module like:
from io import StringIO
...
s = configfile.read()
s.replace("#","_")
f = StringIO(unicode(s))
configparser.readfp(f)
And if you don't have to use an "ini"-file syntax take a look at the json module. I use it more often then the ini-file for configuration, especially if the config-files shouldn't be manually edited by simple users.
my_config={
"channels":["#mychannel", "#yourchannel"],
"user"="bob",
"buddy-list":["alice","eve"],
}
import json
with open(configfile, 'rw') as cfg:
cfg.write(json.dumps(my_config))
ConfigParser has no way to not ignore lines beginning with '#'.
ConfigParser.py, line 476:
# comment or blank line?
if line.strip() == '' or line[0] in '#;':
continue
No way to turn it off.
In your defense ConfigParser is letting you make this mistake:
import sys
import ConfigParser
config = ConfigParser.RawConfigParser()
config.add_section('channels')
config.set('channels', '#channel', 'true')
config.write(sys.stdout)
Produces this output:
[channels]
#channel = true
However you can give section names that start with a # like so:
import sys
import ConfigParser
config = ConfigParser.RawConfigParser()
config.add_section('#channels')
config.set('#channels', 'channel', 'true')
config.write(sys.stdout)
with open('q15123871.cfg', 'wb') as configfile:
config.write(configfile)
config = ConfigParser.RawConfigParser()
config.read('q15123871.cfg')
print config.get('#channels', 'channel')
Which produces the output:
[#channels]
channel = true
true
from docutils.parsers.rst.directives.images import Figure
class MyFigure(Figure):
def run(self):
# here I need to read the 'thumbnails_folder' setting
pass
def setup(app):
app.add_config_value('thumbnails_folder', '_thumbnails', 'env')
How can I access the config value in .run()? I read sources of Sphinx-contrib, but did not see things done in my way, so they accessed conf.py the way I can't. Or should I do it in a different manner?
All I want to do is translate this
.. figure:: image/image.jpg
into this:
.. image:: image/thumbnails/image.jpg
:target: image/image.jpg
Here's the extension code
(the thumbnail is generated with PIL). And also put the :target: into downloadable files (As I see, only builder instances can do this).
The build environment holds a reference to the Config object. Configuration variables can be retrieved from this object:
def run(self):
env = self.state.document.settings.env # sphinx.environment.BuildEnvironment
config = env.config # sphinx.config.Config
folder = config["thumbnails_folder"]
...
I have a Pyramid web application managed with zc.buildout. In it, I need to read a file on disk, which is located in a sub-directory of buildout directory.
The problem is with determining the path to the file - I do not want to hard-code the absolute path and just providing a relative path does not work when serving the app in production (supposedly because the working directory is different).
So the promising "hooks" I am thinking about are:
the "root" buildout directory, which I can address in buildout.cfg as ${buildout:directory} - however, I can't figure out how can I "export" it so it can be accessed by the Python code
the location of the Paster's .ini file which starts the app
Like #MartijnPieters suggests in a comment on your own answer, I'd use collective.recipe.template to generate an entry in the .ini. I wondered myself how I could then access that data in my project, so I worked it out :-)
Let's work our way backwards to what you need. First in your view code where you want the buildout directory:
def your_view(request):
buildout_dir = request.registry.settings['buildout_dir']
....
request.registry.settings (see documentation) is a "dictonary-like deployment settings object". See deployment settings, that's the **settings that gets passed into your main method like def main(global_config, **settings)
Those settings are what's in the [app:main] part of your deployment.ini or production.ini file. So add the buildout directory there:
[app:main]
use = egg:your_app
buildout_dir = /home/you/wherever/it/is
pyramid.reload_templates = true
pyramid.debug_authorization = false
...
But, and this is the last step, you don't want to have that hardcoded path in there. So generate the .ini with a template. The template development.ini.in uses a ${partname:variable} expansion language. in your case you need${buildout:directory}:
[app:main]
use = egg:your_app
buildout_dir = ${buildout:dir}
# ^^^^^^^^^^^^^^^
pyramid.reload_templates = true
pyramid.debug_authorization = false
...
Add a buildout part in buildout.cfg to generate development.ini from development.ini.in:
[buildout]
...
parts =
...
inifile
...
[inifile]
recipe = collective.recipe.template
input = ${buildout:directory}/development.ini.in
output = ${buildout:directory}/development.ini
Note that you can do all sorts of cool stuff with collective.recipe.template. ${serverconfig:portnumber} to generate a matching port number in your production.ini and in your your_site_name.nginx.conf, for instance. Have fun!
If the path to the file relative to the buildout root or location of paster.ini is always the same, which it seems it is from your question, you could set it in paster.ini:
[app:main]
...
config_file = %(here)s/path/to/file.txt
Then access it from the registry as in Reinout's answer:
def your_view(request):
config_file = request.registry.settings['config_file']
Here's a rather clumsy solution I've devised:
In buildout.cfg I used extra-paths option of zc.recipe.egg to add the buildout directory to sys.path:
....
[webserver]
recipe = zc.recipe.egg:scripts
eggs = ${buildout:eggs}
extra-paths = ${buildout:directory}
then I put a file called app_config.py into the buildout directory:
# This remembers the root of the installation (similar to {buildout:directory}
# so we can import it and use where we need access to the filesystem.
# Note: we could use os.getcwd() for that but it feels kinda wonky
# This is not directly related to Celery, we may want to move it somewhere
import os.path
INSTALLATION_ROOT = os.path.dirname(__file__)
Now we can import it in our Python code:
from app_config import INSTALLATION_ROOT
filename = os.path.join(INSTALLATION_ROOT, "somefile.ext")
do_stuff_with_file(filename)
If anyone knows a nicer solution you're welcome :)