I have a txt file containing some parameters. The txt file looks like this:
par1 = 10.81
par2 = 0.3
par3 = 0.5
I would like to read it in python and create automatically all the variables specified in the txt.
What is the best file to read a configuration file and create the parameters listed there?
in particular say that I have a class
how can I have the following?
class MyClass:
def __init__(self)
self.par1 = par1
self.par2 = par2
self.par3 = par3
DANGEROUS CODE AHEAD
You could use exec.
Consider the file foo.txt:
a = 1
b = 2
c = 3
and the code:
with open('foo.txt') as f:
for line in f:
exec(line)
print(a)
# 1
print(b)
# 2
print(c)
# 3
BUT, as the header says, this is a dangerous approach as foo.txt can be easily modified to contain
import os
os.remove('C:\system\a_very_important_system_file')
and exec will happily execute it.
THE SAFE APPROACH
Change foo.txt to JSON format:
{"a": 1,
"b": 2,
"c": 3}
And use the json module to load it in your code:
import json
with open('foo.txt') as f:
variables = json.load(f)
print(variables["a"])
# 1
EDIT
If you need to dynamically initiate instance attributes from an unknown JSON file you can use setattr:
import json
class MyClass:
def __init__(self):
with open('foo.txt') as f:
variables = json.load(f)
for key, value in variables.items():
setattr(self, key, value)
In addition to the answer provided by #DeepSpace, you could restrict the allowed values with a regular expression:
import re
string = """
par1 = 10.81
par2 = 0.3
par3 = 0.5
"""
rx = re.compile(r'^(?P<key>\w+)\s*=\s*(?P<value>[\d.]+)', re.MULTILINE)
for var in rx.finditer(string):
exec("{} = {}".format(var.group('key'), var.group('value')))
print(par1)
# 10.81
Here, only numerical values are allowed for the values (0-9.) and a-z_ for the variable names. Adjust as needed (ie to allow strings as well).
An alternative would be to use a container for the parameters, ie a dict:
params = {match.group('key'): match.group('value')
for match in rx.finditer(string)}
print(params)
# {'par1': '10.81', 'par2': '0.3', 'par3': '0.5'}
You would then call your variables via params['par1'].
After you read the file with readlines()
with open("sample.txt") as f:
fr = f.readlines()
fr = ['par1 = 10.81\n', 'par2 = 0.3\n', 'par3 = 0.5\n', 'par4 = 0.7\n', 'par5 = 0.9\n'] # Output
fr = [f[:-1] for f in fr]
frs = [f.split(" = ") for f in fr]
for i, j in frs:
exec("%s = %s" % (i,j))
You are looking for the eval function.
with open('foo.txt','r') as fil:
for line in fil:
eval(line)
Related
This is a heavily abstracted example where I build objects from variables stored in a .yaml file. I'm writing the reverse method to save them back as a new .yaml
I may create further objects via script, so the output yaml will in general be different.
I'm using .locals() to build a dictionary from the kwargs, and then .pop() to strip the ones I will not want to save.
This seems to work and do what I want, but I feel it is ugly. Am I missing a better, safer, or more pythonic way to do this?
I understand there is pickle and dill, but for the current question I'd like to restrict this to reading and writing yamls. (because)
note: if attributes are added later I don't want them saved. This is why I create ob.L right after instantiation.
Input .yaml:
bob:
args: {'x':1, 'y':2}
sue:
args: {'x':3, 'y':4}
Output .yaml:
bob:
args:
x: 1
y: 2
new:
args:
x: 5
y: 6
sue:
args:
x: 3
y: 4
Current script:
class A(object):
wow = 77
def __init__(self, name, x, y):
self.name = name
self.x = x
self.y = y
self.L = locals()
self.L.pop('name')
self.L.pop('self')
import yaml
with open('thing.yaml', 'r') as infile:
d = yaml.load(infile)
obs = []
for name, info in d.items():
ob = A(name, **info['args'])
obs.append(ob)
newob = A('new', 5, 6)
obs.append(newob)
newob.ignore_me = 777 # this should not be saved
# rebuild the yaml
d = dict()
for ob in obs:
info = dict()
info['args'] = ob.L
d[ob.name] = info
with open('newthing.yaml', 'w') as outfile:
yaml.dump(d, outfile, default_flow_style=False, allow_unicode=True)
I can't understand why you're doing any of this. All you need to do is to load the YAML, add your new items, and then dump it again.
with open('thing.yaml', 'r') as infile:
d = yaml.load(infile)
d['new'] = {'x': 5, 'y': 6}
with open('newthing.yaml', 'w') as outfile:
yaml.dump(d, outfile, default_flow_style=False, allow_unicode=True)
We have a sample .cfg file consist of key value pair. Based on user input, we need to update the value. I was thinking to update the value using configParser but file doesn't have any section (e.g. [My Section]). Based on the documentation it needs three values to set - section, key and value. Unfortunately, I will not be able to add any section marker, as this file is used by other tasks.
What would be the another way we can set the value based on key?
File example
some.status_file_mode = 1 # Some comment
some.example_time = 7200 # Some comment
As per the requirement, no change in the line. Spaces and comments needs to be same as is.
Use NamedTemporaryFile from the tempfile module it is not too hard to build a simple parser to update a file that looks like that:
Code:
def replace_key(filename, key, value):
with open(filename, 'rU') as f_in, tempfile.NamedTemporaryFile(
'w', dir=os.path.dirname(filename), delete=False) as f_out:
for line in f_in.readlines():
if line.startswith(key):
line = '='.join((line.split('=')[0], ' {}'.format(value)))
f_out.write(line)
# remove old version
os.unlink(filename)
# rename new version
os.rename(f_out.name, filename)
Test Code:
import os
import tempfile
replace_key('file1', 'some.example_time', 3)
Results:
some.status_file_mode = 1
some.example_time = 3
If you don't care about spacing, this works well for your case.
def replace_config(filename, key, value):
d = {}
with open(filename, "r+") as f:
for line in f:
k, v = line.split('=')
c = ""
try:
v, c = v.split('#')
except ValueError:
c = ""
d[k.strip()] = {'v': v.strip(), 'c': c.strip()}
f.seek(0)
f.truncate()
d[key]['v'] = value
for k, v in d.items():
if v["c"]:
text = "{} = {} # {}\n".format(k, v['v'], v['c'])
else:
text = "{} = {}\n".format(k, v['v'])
f.write(text)
replace_config('config.cfg', 'some.example_time', 3)
I'm trying to implement a simple helper class to interact with java-properties files. Fiddling with multiline properties I encountered a problem, that I can not get solved, maybe you can?
The unittest in the class first writes a multiline-property spanning over two lines to the property-file, then re-reads it and checks for equality. That just works. Now, if i use the class to add a third line to the property, it re-reads it with additional backslashes that I can't explain.
Here is my code:
#!/usr/bin/env python3
# -*- coding=UTF-8 -*-
import codecs
import os, re
import fileinput
import unittest
class ConfigParser:
reProp = re.compile(r'^(?P<key>[\.\w]+)=(?P<value>.*?)(?P<ext>[\\]?)$')
rePropExt = re.compile(r'(?P<value>.*?)(?P<ext>[\\]?)$')
files = []
def __init__(self, pathes=[]):
for path in pathes:
if os.path.isfile(path):
self.files.append(path)
def getOptions(self):
result = {}
key = ''
val = ''
with fileinput.input(self.files, inplace=False) as fi:
for line in fi:
m = self.reProp.match(line.strip())
if m:
key = m.group('key')
val = m.group('value')
result[key] = val
else:
m = self.rePropExt.match(line.rstrip())
if m:
val = '\n'.join((val, m.group('value')))
result[key] = val
fi.close()
return result
def setOptions(self, updates={}):
options = self.getOptions()
options.update(updates)
with fileinput.input(self.files, inplace=True) as fi:
for line in fi:
m = self.reProp.match(line.strip())
if m:
key = m.group('key')
nval = options[key]
nval = nval.replace('\n', '\\\n')
print('{}={}'.format(key,nval))
fi.close()
class test(unittest.TestCase):
files = ['test.properties']
props = {'test.m.a' : 'Johnson\nTanaka'}
def setUp(self):
for file in self.files:
f = codecs.open(file, encoding='utf-8', mode='w')
for key in self.props.keys():
val = self.props[key]
val = re.sub('\n', '\\\n', val)
f.write(key + '=' + val)
f.close()
def teardown(self):
pass
def test_read(self):
c = configparser(self.files)
for file in self.files:
for key in self.props.keys():
result = c.getOptions()
self.assertEqual(result[key],self.props[key])
def test_write(self):
c = ConfigParser(self.files)
changes = {}
for key in self.props.keys():
changes[key] = self.change_value(self.props[key])
c.setOptions(changes)
result = c.getOptions()
print('changes: ')
print(changes)
print('result: ')
print(result)
for key in changes.keys():
self.assertEqual(result[key],changes[key],msg=key)
def change_value(self, value):
return 'Smith\nJohnson\nTanaka'
if __name__ == '__main__':
unittest.main()
Output of the testrun:
C:\pyt>propertyfileparser.py
changes:
{'test.m.a': 'Smith\nJohnson\nTanaka'}
result:
{'test.m.a': 'Smith\nJohnson\\\nTanaka'}
Any hints welcome...
Since you are adding a backslash in front of new-lines when you are writing, you have to also remove them when you are reading. Uncommenting the line that substitutes '\n' with '\\n' solves the problem, but I expect this also means the file syntax is incorrect.
This happens only with the second line break, because you separate the value into an "oval" and an "nval" where the "oval" is the first line, and the "nval" the rest, and you only do the substitution on the nval.
It's also overkill to use regexp replacing to replace something that isn't a regexp. You can use val.replace('\n', '\\n') instead.
I'd do this parser very differently. Well, first of all, I wouldn't do it at all, I'd use an existing parser, but if I did, I'd read the file, line by line, while handling the line continuation issue, so that I had exactly one value per item in a list. Then I'd parse each item into a key and a value with a regexp, and stick that into a dictionary.
You instead parse each line separately and join continuation lines to the values after parsing, which IMO is completely backwards.
I have a file that looks like this
!--------------------------------------------------------------------------DISK
[DISK]
DIRECTION = 'OK'
TYPE = 'normal'
!------------------------------------------------------------------------CAPACITY
[CAPACITY]
code = 0
ID = 110
I want to read sections [DISK] and [CAPACITY].. there will be more sections like these. I want to read the parameters defined under those sections.
I wrote a following code:
file_open = open(myFile,"r")
all_lines = file_open.readlines()
count = len(all_lines)
file_open.close()
my_data = {}
section = None
data = ""
for line in all_lines:
line = line.strip() #remove whitespace
line = line.replace(" ", "")
if len(line) != 0: # remove white spaces between data
if line[0] == "[":
section = line.strip()[1:]
data = ""
if line[0] !="[":
data += line + ","
my_data[section] = [bit for bit in data.split(",") if bit != ""]
print my_data
key = my_data.keys()
print key
Unfortunately I am unable to get those sections and the data under that. Any ideas on this would be helpful.
As others already pointed out, you should be able to use the ConfigParser module.
Nonetheless, if you want to implement the reading/parsing yourself, you should split it up into two parts.
Part 1 would be the parsing at file level: splitting the file up into blocks (in your example you have two blocks: DISK and CAPACITY).
Part 2 would be parsing the blocks itself to get the values.
You know you can ignore the lines starting with !, so let's skip those:
with open('myfile.txt', 'r') as f:
content = [l for l in f.readlines() if not l.startswith('!')]
Next, read the lines into blocks:
def partition_by(l, f):
t = []
for e in l:
if f(e):
if t: yield t
t = []
t.append(e)
yield t
blocks = partition_by(content, lambda l: l.startswith('['))
and finally read in the values for each block:
def parse_block(block):
gen = iter(block)
block_name = next(gen).strip()[1:-1]
splitted = [e.split('=') for e in gen]
values = {t[0].strip(): t[1].strip() for t in splitted if len(t) == 2}
return block_name, values
result = [parse_block(b) for b in blocks]
That's it. Let's have a look at the result:
for section, values in result:
print section, ':'
for k, v in values.items():
print '\t', k, '=', v
output:
DISK :
DIRECTION = 'OK'
TYPE = 'normal'
CAPACITY :
code = 0
ID = 110
Are you able to make a small change to the text file? If you can make it look like this (only changed the comment character):
#--------------------------------------------------------------------------DISK
[DISK]
DIRECTION = 'OK'
TYPE = 'normal'
#------------------------------------------------------------------------CAPACITY
[CAPACITY]
code = 0
ID = 110
Then parsing it is trivial:
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('filename')
And getting data looks like this:
(Pdb) parser
<ConfigParser.SafeConfigParser instance at 0x100468dd0>
(Pdb) parser.get('DISK', 'DIRECTION')
"'OK'"
Edit based on comments:
If you're using <= 2.7, then you're a little SOL.. The only way really would be to subclass ConfigParser and implement a custom _read method. Really, you'd just have to copy/paste everything in Lib/ConfigParser.py and edit the values in line 477 (2.7.3):
if line.strip() == '' or line[0] in '#;': # add new comment characters in the string
However, if you're running 3'ish (not sure what version it was introduced in offhand, I'm running 3.4(dev)), you may be in luck: ConfigParser added the comment_prefixes __init__ param to allow you to customize your prefix:
parser = ConfigParser(comment_prefixes=('#', ';', '!'))
If the file is not big, you can load it and use Regexes to find parts that are of interest to you.
I am parsing a config file I would like to get only the values which are file paths in the file
for example the config file has
apache.access = /var/log/apache2/access.log
apache.errors = /var/log/apache2/errors.log
I would like to get only '/var/log/apache2/access.log' and '/var/log/apache2/errors.log' from the
COMMENT_CHAR = '#'
OPTION_CHAR = '='
def parse_config(filename):
options = {}
f = open(filename)
for line in f:
if COMMENT_CHAR in line:
line, comment = line.split(COMMENT_CHAR, 1)
if OPTION_CHAR in line:
option, value = line.split(OPTION_CHAR, 1)
option = option.strip()
value = value.strip()
options[option] = value
f.close()
return options
I tried this but it gives all the options and values in the file
apache.access : /var/log/apache2/access.log
apache.errors : /var/log/apache2/errors.log
First you have to define, what is a file path (1. does it have to exist? 2. does it have to be creatable in an existing directory? 3. anything else?). This tests whether the given string is a path pointing to an existing dir/file:
import os
# ...
[v for v in options.itervalues() if os.path.exists(v)]
Regular expressions is what you need. Here is little example how it could look
import re
f = open("apache.log")
for line in f:
m = re.match("apache\.(access|errors)\s+=\s+(.+)", line)
if None != m:
print (m.group(2))
f.close()