I am parsing a config file I would like to get only the values which are file paths in the file
for example the config file has
apache.access = /var/log/apache2/access.log
apache.errors = /var/log/apache2/errors.log
I would like to get only '/var/log/apache2/access.log' and '/var/log/apache2/errors.log' from the
COMMENT_CHAR = '#'
OPTION_CHAR = '='
def parse_config(filename):
options = {}
f = open(filename)
for line in f:
if COMMENT_CHAR in line:
line, comment = line.split(COMMENT_CHAR, 1)
if OPTION_CHAR in line:
option, value = line.split(OPTION_CHAR, 1)
option = option.strip()
value = value.strip()
options[option] = value
f.close()
return options
I tried this but it gives all the options and values in the file
apache.access : /var/log/apache2/access.log
apache.errors : /var/log/apache2/errors.log
First you have to define, what is a file path (1. does it have to exist? 2. does it have to be creatable in an existing directory? 3. anything else?). This tests whether the given string is a path pointing to an existing dir/file:
import os
# ...
[v for v in options.itervalues() if os.path.exists(v)]
Regular expressions is what you need. Here is little example how it could look
import re
f = open("apache.log")
for line in f:
m = re.match("apache\.(access|errors)\s+=\s+(.+)", line)
if None != m:
print (m.group(2))
f.close()
Related
I'm trying to write the text fields values to a file like this (where inputtxt is a Text widget):
input_file_name = inputtxt1.get('1.0', 'end-1c').replace(" ","")
num_of_compare_points = inputtxt2.get('1.0','end-1c')
VER = inputtxt3.get('1.0','end-1c')
mode = inputtxt4.get(1.0','end-1c')
executionMode = inputtxt5.get('1.0','end-1c')
numOfWorkers = inputtxt6.get ('1.0','end-1c')
sessionFile = inputtxt7.get('1.0','end-1c')
file = open("file_1.setup","w")
file.write (input_file_name + "\n" + num_of_compare_points + "\n" + Option+"\n"+VER+"\n"+mode+"\n"+executionMode+"\n"+numOfWorkers+"\n"+sessionFile)
This method was good but I want to write the variable name +the value to the file so that the user can fill the values from the file not only through GUI and the order in file will be irrelevant since we're storing in the variable itself , Example:
File_1.setup:
input_file_name = (some name the user can change it from here not from GUI)
num_of_compare_points = (some number the user can change it from here not from GUI)
.
.
.
etc
UPDATE: I'm using the file attribute:value in a restore finction which put the value of each attribute on its related text field ; the problem here that I have only the value but not the variable:value and the variables must be in order (I need the order to be irrelevent since I'll use the variable name)
def restore_info():
if os.stat('file_1.setup').st_size == 0:
print("Writing to setup file..")
else:
with open('file_1.setup','r') as f:
lines = list(line for line in (l.strip() for l in f) if line)
x = len(lines)
print (x)
for i in lines:
print (i)
if (x==7):
inputtxt1.insert('end',lines[0])
inputtxt2.insert('end',lines[1])
inputtxt3.insert['end',lines[2])
... etc
UPDATE2 : I've managed to split each line based on ":" but I need a way to tell the program where is variable and where is value
def restore_info():
if os.stat('file_1.setup').st_size == 0:
print("Writing to setup file..")
else:
with open('file_1.setup','r') as f:
lines = list(line for line in (l.strip() for l in f) if line)
x = len(lines)
print (x)
for i in lines:
splitted_i=i.split(":")
print (splitted_i)
UPDATE 3:Advantage of this step
User will be able to apply the value manually, and not only through GUI.
The order will be irrelevant.
Any amount of spaces should be allowed around and in between the variable name and its value.
It will be very easy to search file_1.setup and extract the value for each field.
What you need to do is to collect the data into a dict and dump it as key value pair
sessionFile = 'x'
executionMode = 'y'
data = {'sessionFile': sessionFile, 'executionMode': executionMode}
with open('out.txt', 'w') as f:
for k, v in data.items():
f.write(f'{k}:{v}\n')
out.txt
sessionFile:x
executionMode:y
have a Dict with multiple values in a tuple.
newhost = {'newhost.com': ('1.oldhost.com',
'2.oldhost.com',
'3.oldhost.com',
'4.oldhost.com')
}
I wanna open a existing file and search for lines in this file that contains a value of the oldhosts. A file can have multiple Account Lines. In example
Account: 1.oldhost.com username
Account: someotherhost username
When the line with 1.oldhost.com or 2.oldhost.com or 3.oldhost.com and so on is found i wanna replace it with the key form the dict newhost.com.
Can anyone help me? Searched alot, but didnt find the right thing.
Regards
Maybe something like this could get you started
infile_name = 'some_file.txt'
# Open and read the incoming file
with open(infile_name, 'r') as infile:
text = infile.read()
# Cycle through the dictionary
for newhost, oldhost_list in host_dict.items():
# Cycle through each possible old host
for oldhost in oldhost_list:
text.replace(oldhost, newhost)
outfile_name = 'some_other_file.txt'
# Write to file
with open(outfile_name, 'w') as outfile:
outfile.write(text)
Not claiming this to be the best solution, but it should be a good start for you.
To easily find the new host for a given old host, you should convert your data structure:
# your current structure
new_hosts = {
'newhost-E.com': (
'1.oldhost-E.com',
'2.oldhost-E.com',
),
'newhost-A.com': (
'1.oldhost-A.com',
'2.oldhost-A.com',
'3.oldhost-A.com',
),
}
# my proposal
new_hosts_2 = {
v: k
for k, v_list in new_hosts.items()
for v in v_list}
print(new_hosts_2)
# {
# '1.oldhost-E.com': 'newhost-E.com',
# '2.oldhost-E.com': 'newhost-E.com',
# '1.oldhost-A.com': 'newhost-A.com',
# '2.oldhost-A.com': 'newhost-A.com',
# '3.oldhost-A.com': 'newhost-A.com',
# }
This does repeat the new host names (the values in new_hosts_2), but it will allow you to quickly look up given an old host name:
some_old_host = 'x.oldhost.com'
the corresponding_new_host = new_hosts_2[some_old_host]
Now you just need to:
read the lines of the file
find the old hostname in that line
lookup the corresponding new host in new_hosts_2
replace that value in the line
write the line to a new file
Maybe like this:
with open(file_name_1, 'r') as fr:
with open(file_name_2, 'w') as fw:
for line in fr:
line = line.strip()
if len(line) > 0:
# logic to find the start and end position of the old host
start_i = ?
end_i = ?
# get and replace, but only if its found in 'new_hosts_2'
old_host = line[start_i:end_i]
if old_host in new_hosts_2:
line = line[:start_i] + new_hosts_2[old_host] + line[end_i:]
fw.write(line + '\n')
Thank you for your tips. I came up with this now and it is working fine.
import fileinput
textfile = 'somefile.txt'
curhost = 'newhost.com'
hostlist = {curhost: ('1.oldhost.com',
'2.oldhost.com',
'3.oldhost.com')
}
new_hosts_2 = {
v: k
for k, v_list in hostlist.items()
for v in v_list}
for line in fileinput.input(textfile, inplace=True):
line = line.rstrip()
if not line:
continue
for f_key, f_value in new_hosts_2.items():
if f_key in line:
line = line.replace(f_key, f_value)
print line
We have a sample .cfg file consist of key value pair. Based on user input, we need to update the value. I was thinking to update the value using configParser but file doesn't have any section (e.g. [My Section]). Based on the documentation it needs three values to set - section, key and value. Unfortunately, I will not be able to add any section marker, as this file is used by other tasks.
What would be the another way we can set the value based on key?
File example
some.status_file_mode = 1 # Some comment
some.example_time = 7200 # Some comment
As per the requirement, no change in the line. Spaces and comments needs to be same as is.
Use NamedTemporaryFile from the tempfile module it is not too hard to build a simple parser to update a file that looks like that:
Code:
def replace_key(filename, key, value):
with open(filename, 'rU') as f_in, tempfile.NamedTemporaryFile(
'w', dir=os.path.dirname(filename), delete=False) as f_out:
for line in f_in.readlines():
if line.startswith(key):
line = '='.join((line.split('=')[0], ' {}'.format(value)))
f_out.write(line)
# remove old version
os.unlink(filename)
# rename new version
os.rename(f_out.name, filename)
Test Code:
import os
import tempfile
replace_key('file1', 'some.example_time', 3)
Results:
some.status_file_mode = 1
some.example_time = 3
If you don't care about spacing, this works well for your case.
def replace_config(filename, key, value):
d = {}
with open(filename, "r+") as f:
for line in f:
k, v = line.split('=')
c = ""
try:
v, c = v.split('#')
except ValueError:
c = ""
d[k.strip()] = {'v': v.strip(), 'c': c.strip()}
f.seek(0)
f.truncate()
d[key]['v'] = value
for k, v in d.items():
if v["c"]:
text = "{} = {} # {}\n".format(k, v['v'], v['c'])
else:
text = "{} = {}\n".format(k, v['v'])
f.write(text)
replace_config('config.cfg', 'some.example_time', 3)
I have 2 numbers in two similar files. There is a new.txt and original.txt. They both have the same string in them except for a number. The new.txt has a string that says boothNumber="3". The original.txt has a string that says boothNumber="1".
I want to be able to read the new.txt, pick the number 3 out of it and replace the number 1 in original.txt.
Any suggestions? Here is what I am trying.
import re # used to replace string
import sys # some of these are use for other code in my program
def readconfig():
with open("new.text") as f:
with open("original.txt", "w") as f1:
for line in f:
match = re.search(r'(?<=boothNumber=")\d+', line)
for line in f1:
pattern = re.search(r'(?<=boothNumber=")\d+', line)
if re.search(pattern, line):
sys.stdout.write(re.sub(pattern, match, line))
When I run this, my original.txt gets completely cleared of any text.
I did a traceback and I get this:
in readconfig
for line in f1:
io.UnsupportedOperationo: not readable
UPDATE
I tried:
def readconfig(original_txt_path="original.txt",
new_txt_path="new.txt"):
with open(new_txt_path) as f:
for line in f:
if not ('boothNumber=') in line:
continue
booth_number = int(line.replace('boothNumber=', ''))
# do we need check if there are more than one 'boothNumber=...' line?
break
with open(original_txt_path) as f1:
modified_lines = [line.startswith('boothNumber=') if not line
else 'boothNumber={}'.format(booth_number)
for line in f1]
with open(original_txt_path, mode='w') as f1:
f1.writelines(modified_lines)
And I get error:
booth_number = int(line.replace('boothNumber=', ''))
ValueError: invalid literal for int() with base 10: '
(workstationID="1" "1" window=1= area="" extra parts of the line here)\n
the "1" after workstationID="1" is where the boothNumber=" " would normally go. When I open up original.txt, I see that it actually did not change anything.
UPDATE 3
Here is my code in full. Note, the file names are changed but I'm still trying to do the same thing. This is another idea or revision I had that is still not working:
import os
import shutil
import fileinput
import re # used to replace string
import sys # prevents extra lines being inputed in config
# example: sys.stdout.write
def convertconfig(pattern):
source = "template.config"
with fileinput.FileInput(source, inplace=True, backup='.bak') as file:
for line in file:
match = r'(?<=boothNumber=")\d+'
sys.stdout.write(re.sub(match, pattern, line))
def readconfig():
source = "bingo.config"
pattern = r'(?<=boothNumber=")\d+' # !!!!!!!!!! This probably needs fixed
with fileinput.FileInput(source, inplace=True, backup='.bak') as file:
for line in file:
if re.search(pattern, line):
fileinput.close()
convertconfig(pattern)
def copyfrom(servername):
source = r'//' + servername + '/c$/remotedirectory'
dest = r"C:/myprogram"
file = "bingo.config"
try:
shutil.copyfile(os.path.join(source, file), os.path.join(dest, file))
except:
print ("Error")
readconfig()
# begin here
os.system('cls' if os.name == 'nt' else 'clear')
array = []
with open("serverlist.txt", "r") as f:
for servername in f:
copyfrom(servername.strip())
bingo.config is my new file
template.config is my original
It's replacing the number in template.config with the literal string "r'(?<=boothNumber=")\d+'"
So template.config ends up looking like
boothNumber="r'(?<=boothNumber=")\d+'"
instead of
boothNumber="2"
To find boothNumber value we can use next regular expression (checked with regex101)
(?<=\sboothNumber=\")(\d+)(?=\")
Something like this should work
import re
import sys # some of these are use for other code in my program
BOOTH_NUMBER_RE = re.compile('(?<=\sboothNumber=\")(\d+)(?=\")')
search_booth_number = BOOTH_NUMBER_RE.search
replace_booth_number = BOOTH_NUMBER_RE.sub
def readconfig(original_txt_path="original.txt",
new_txt_path="new.txt"):
with open(new_txt_path) as f:
for line in f:
search_res = search_booth_number(line)
if search_res is None:
continue
booth_number = int(search_res.group(0))
# do we need check if there are more than one 'boothNumber=...' line?
break
else:
# no 'boothNumber=...' line was found, so next lines will fail,
# maybe we should raise exception like
# raise Exception('no line starting with "boothNumber" was found')
# or assign some default value
# booth_number = -1
# or just return?
return
with open(original_txt_path) as f:
modified_lines = []
for line in f:
search_res = search_booth_number(line)
if search_res is not None:
line = replace_booth_number(str(booth_number), line)
modified_lines.append(line)
with open(original_txt_path, mode='w') as f:
f.writelines(modified_lines)
Test
# Preparation
with open('new.txt', mode='w') as f:
f.write('some\n')
f.write('<jack Fill workstationID="1" boothNumber="56565" window="17" Code="" area="" section="" location="" touchScreen="False" secureWorkstation="false">')
with open('original.txt', mode='w') as f:
f.write('other\n')
f.write('<jack Fill workstationID="1" boothNumber="23" window="17" Code="" area="" section="" location="" touchScreen="False" secureWorkstation="false">')
# Invocation
readconfig()
# Checking output
with open('original.txt') as f:
for line in f:
# stripping newline character
print(line.rstrip('\n'))
gives
other
<jack Fill workstationID="1" boothNumber="56565" window="17" Code="" area="" section="" location="" touchScreen="False" secureWorkstation="false">
I'm trying to implement a simple helper class to interact with java-properties files. Fiddling with multiline properties I encountered a problem, that I can not get solved, maybe you can?
The unittest in the class first writes a multiline-property spanning over two lines to the property-file, then re-reads it and checks for equality. That just works. Now, if i use the class to add a third line to the property, it re-reads it with additional backslashes that I can't explain.
Here is my code:
#!/usr/bin/env python3
# -*- coding=UTF-8 -*-
import codecs
import os, re
import fileinput
import unittest
class ConfigParser:
reProp = re.compile(r'^(?P<key>[\.\w]+)=(?P<value>.*?)(?P<ext>[\\]?)$')
rePropExt = re.compile(r'(?P<value>.*?)(?P<ext>[\\]?)$')
files = []
def __init__(self, pathes=[]):
for path in pathes:
if os.path.isfile(path):
self.files.append(path)
def getOptions(self):
result = {}
key = ''
val = ''
with fileinput.input(self.files, inplace=False) as fi:
for line in fi:
m = self.reProp.match(line.strip())
if m:
key = m.group('key')
val = m.group('value')
result[key] = val
else:
m = self.rePropExt.match(line.rstrip())
if m:
val = '\n'.join((val, m.group('value')))
result[key] = val
fi.close()
return result
def setOptions(self, updates={}):
options = self.getOptions()
options.update(updates)
with fileinput.input(self.files, inplace=True) as fi:
for line in fi:
m = self.reProp.match(line.strip())
if m:
key = m.group('key')
nval = options[key]
nval = nval.replace('\n', '\\\n')
print('{}={}'.format(key,nval))
fi.close()
class test(unittest.TestCase):
files = ['test.properties']
props = {'test.m.a' : 'Johnson\nTanaka'}
def setUp(self):
for file in self.files:
f = codecs.open(file, encoding='utf-8', mode='w')
for key in self.props.keys():
val = self.props[key]
val = re.sub('\n', '\\\n', val)
f.write(key + '=' + val)
f.close()
def teardown(self):
pass
def test_read(self):
c = configparser(self.files)
for file in self.files:
for key in self.props.keys():
result = c.getOptions()
self.assertEqual(result[key],self.props[key])
def test_write(self):
c = ConfigParser(self.files)
changes = {}
for key in self.props.keys():
changes[key] = self.change_value(self.props[key])
c.setOptions(changes)
result = c.getOptions()
print('changes: ')
print(changes)
print('result: ')
print(result)
for key in changes.keys():
self.assertEqual(result[key],changes[key],msg=key)
def change_value(self, value):
return 'Smith\nJohnson\nTanaka'
if __name__ == '__main__':
unittest.main()
Output of the testrun:
C:\pyt>propertyfileparser.py
changes:
{'test.m.a': 'Smith\nJohnson\nTanaka'}
result:
{'test.m.a': 'Smith\nJohnson\\\nTanaka'}
Any hints welcome...
Since you are adding a backslash in front of new-lines when you are writing, you have to also remove them when you are reading. Uncommenting the line that substitutes '\n' with '\\n' solves the problem, but I expect this also means the file syntax is incorrect.
This happens only with the second line break, because you separate the value into an "oval" and an "nval" where the "oval" is the first line, and the "nval" the rest, and you only do the substitution on the nval.
It's also overkill to use regexp replacing to replace something that isn't a regexp. You can use val.replace('\n', '\\n') instead.
I'd do this parser very differently. Well, first of all, I wouldn't do it at all, I'd use an existing parser, but if I did, I'd read the file, line by line, while handling the line continuation issue, so that I had exactly one value per item in a list. Then I'd parse each item into a key and a value with a regexp, and stick that into a dictionary.
You instead parse each line separately and join continuation lines to the values after parsing, which IMO is completely backwards.