Develop Reference

How to Change dictionary values in python file from another file

I would like to change values in a Dict in another file. File1.py contains the code to edit the Dict, File2.py contains the Dict itself.
File1.py is generating a code to replace BTOK values only.
File1.py:
with open('file2.py', 'r') as file :
filedata = file.read()
print (filedata.str(BTK['btk1']))
for line in filedata:
line['btk1'] = BTok
with open('file2.py', 'w') as file:
file.write(line)
File2.py:
c = {
'id' : 'C80e3ce43c3ea3e8d1511ec',
'secret' : 'c10c371b4641010a750073925b0857'
}
rk = {
't1' : 'ZTkwMGE1MGEt',
}
BTK = {
'BTok' : '11eyJhbGc'
}

If you want to do this reliably, that is, so it works whether your strings are quoted with ', " or """, for whatever values they have and whatever newlines you want to put around values, then you may want to use ast to parse the source code and modify it. The only inconvenient with this is that module cannot, by itself, generate code, so you would need to install some additional dependency such as astor, for what is essentially a rather menial task. In any case, here is how you could do it that way:
import ast
import astor
# To read from file:
# with open('file2.py', 'r') as f: code = f.read()
code = """
c = {
'id' : 'C80e3ce43c3ea3e8d1511ec',
'secret' : 'c10c371b4641010a750073925b0857'
}
rk = {
't1' : 'ZTkwMGE1MGEt',
}
BTK = {
'BTok' : '11eyJhbGc'
}
"""
# Value to replace
KEY = 'BTok'
NEW_VALUE = 'new_btok'
# Parse code
m = ast.parse(code)
# Go through module statements
for stmt in m.body:
# Only look at assignments
if not isinstance(stmt, ast.Assign): continue
# Take right-hand side of the assignment
value = stmt.value
# Only look at dict values
if not isinstance(value, ast.Dict): continue
# Look for keys that match what we are looking for
replace_idx = [i for i, k in enumerate(value.keys)
if isinstance(k, ast.Str) and k.s == KEY]
# Replace corresponding values
for i in replace_idx:
value.values[i] = ast.Str(NEW_VALUE)
new_code = astor.to_source(m)
# To write to file:
# with open(`file2.py', 'w') as f: f.write(new_code)
print(new_code)
# c = {'id': 'C80e3ce43c3ea3e8d1511ec', 'secret':
# 'c10c371b4641010a750073925b0857'}
# rk = {'t1': 'ZTkwMGE1MGEt'}
# BTK = {'BTok': 'new_btok'}

Read text file and store into dictionary

I am trying to figure out to store content of the file into multiple values in the specific key.
Desired output:
{'city1':[Island-1,Island-3],'city2':[Island-2,Island-4]}
data.txt
city1-south:"London"
city1-south:"Paris"
city1-north:"Amsterdam"
city1-north:"Island-1"
city2-south:"Island-2"
city1-east:"Island-3"
city2-west:"Island-4"
def readFile(data_file):
data = open(data_file,"r")
d = {}
for line in data:
if 'Island' in line:
city,loc = line.rstrip("\n").split(":",1)
d[city] = loc
print (d)
data.close()
data_file = "data.txt"
readFile(data_file)
Current output:
{'city2-south': '"Island-2"', 'city2-west': '"Island-4"', 'city1-east': '"Island-3"', 'city1-north': '"Island-1"'}

I can not run your code now because config_file is not defined. I have made some modifications so that your code can run.
with open("data.txt") as data:
d = {'city1': [], 'city2': []}
for line in data:
if 'Island' in line:
city,loc = line.rstrip("\n").split(":",1)
for key in d.keys():
if key in city:
d[key].append(loc[1:-1])
print(d)
Results:
{'city1': ['Island-1', 'Island-3'], 'city2': ['Island-2', 'Island-4']}
For now Island-1 etc. can noly be output as string in the dictionary since otherwise python would treat them as variables.

How to replace one field of a line by another line in awk/sed/python?

input.txt
A(0,1,2)
...
B(A,3)
...
C(B,4,5)
If the first parameter of a function is not equal 0 but corresponding to a function name, I want to replace it with all of the corresponding function's parameters (e.g. to replace the first parameter 'A' in function B above with all parameters of function A). That is to expect
output.txt
A(0,1,2)
...
B(0,1,2,3)
...
C(0,1,2,3,4,5)
How can we do this with awk/sed or python?
EDIT:
One idea I have is to store the function name as variables and its parameters as values in bash. In python, we may use dict, and consider function names as keys, and its parameters as values. The implementation is not that easy.

Awk
awk -F'[()]' '
$2 !~ /^0,/ {
split($2, a, /,/)
sub(/^[^,]+/, val[a[1]], $2)
}
{
val[$1] = $2
print $1 "(" $2 ")"
}
' input.txt > output.txt
Where sub(/^[^,]+/, val[a[1]], $2) is used to match the first parameter in $2 and replace it with the value of val[a[1]] which is defined by the execution of val[$1] = $2 on previous lines.

Here's a solution in Python:
import re
with open('input.txt') as f:
data = f.read()
data = [line.strip() for line in data.split('\n') if line]
sets, output = {}, open('output.txt', 'w')
for line in data:
if line == '...':
output.write(line + '\n')
continue
sets[line[0]] = line[2:-1]
output.write(line[0] + '(')
for char in line[2:-1]:
if re.match(r'[\d,]', char):
output.write(char)
else:
output.write(sets[char])
output.write(')\n')
output.close()
Relevant documentation: open(), re.

Let lines be the lines of the input file. The following code will work if all parameters are integers or a functionname
funcs = {}
for line in lines:
match = re.search( '(.*)\((.*)\)', line)
if not match:
raise RuntimeError('Line does not match expectation')
function_name = match.group(1)
parameters = map(str.strip, match.group(2).split(','))
parameter_list = []
for parameter in parameters:
try:
parameter_list.append(int(parameter))
except ValueError:
parameter_list.extend( funcs.get(parameter, []) )
funcs[function_name] = parameter_list
for func_name, paras in sorted(funcs.items()):
print '{function}({parameters})'.format(
function=func_name,
parameters=', '.join(map(str, paras))
)

There are probably a ton of ways to do this but I think this is a simple way to do what you want.
import re
import sys
def convertLine(line):
if re.match("^\\w{1}\(.*\)$", line) is None:
return line
retVal = re.sub( "A", "0,1,2",line[1:])
retVal = re.sub( "B", "0,1,2,3",retVal)
retVal = re.sub( "C", "0,1,2,3,4,5",retVal)
return line[0:1]+retVal
def main():
for line in sys.stdin.read().splitlines():
print convertLine(line)
if __name__ == "__main__":
main()
usage:
python ReplaceProg.py < input.txt

if your file is like this
A(0,1,2)
B(A,3)
C(B,4,5)
using python:
f = open('inpu_file.txt').readlines()
f[0] = f[0].strip()
for i,x in enumerate(f):
if i > 0:
f[i]=re.sub(f[i-1][0],",".join(re.findall('\d+',f[i-1])),x).strip()
print f
output:
['A(0,1,2)', 'B(0,1,2,3)', 'C(0,1,2,3,4,5)']
i don't understand that ... in every alternate line, if its there tell me i can edit the code for that.

Kinda long but more modular:
import re
def build_dict(fobj):
d = dict()
for line in fobj:
match = re.match('^(\w)\((.*)\)', line)
fname = match.group(1)
fargs = match.group(2)
d[fname] = replace(fargs, d)
fobj.seek(0) # Reset cursor to start of file
return d
def replace(s, d):
for each in d:
if each in s:
s = s.replace(each, d[each])
return s
def split_paren(s):
index = s.index('(')
return s[:index], s[index:]
def write_replace(fobj, d):
outname = fobj.name[:-4] + '.out'
outfile = open(outname, 'w')
for line in fobj:
first, second = split_paren(line)
second = replace(second, d)
outfile.write(first + second)
outfile.close()
if __name__ == '__main__':
with open('test.txt', 'r') as f:
d = build_dict(f)
write_replace(f, d)

Parse the file section wise in python

I have a text (abc.txt) file having the following entry in text file:
[General]
Local=C:\Work\July\
path=C:\Work\July\abc
[Field1]
BB0B2BA8--EFE4-4567-B8AE-0204D4BF9F60=
[CustDetails]
BB0B2BA8-EFE4-4567-B8AE-0204D4BF9F60=NOthing
[DirName]
8e27822e-5f46-4f41=TEST
[URLNAME]
8e27822e-5f46=https://
[DestURL]
8e27822e-5f46=some_URL
I want to parse the abc.txt file and take into variable. like in variable
MYpath = C:\Work\July\abc
custdetails= Nothing
dir_name = TEST
URL_Name = https://
DestURL = some_URL
Thanks,

Using ConfigParser:
import ConfigParser
config = ConfigParser.ConfigParser()
config.read('abc.txt')
dic = {}
for section in config.sections():
for option in config.options(section):
res = config.get(section, option)
if res == '':
continue
dic.update({section: res})
print dic
Output:
{'DestURL': 'some_URL', 'URLNAME': 'https://', 'CustDetails': 'NOthing', 'DirName': 'TEST', 'General': 'C:\\Work\\July\\abc'}

You can use a dict here:
>>> dic = {}
with open('abc.txt') as f:
data = f.read()
lines = data.split('\n\n')
for line in lines:
line = line.split('\n')
field = line[0].strip('[]')
val = line[-1].split('=')[1]
if val:
dic[field] = val
...
>>> dic
{'DestURL': 'some_URL',
'URLNAME': 'https://',
'CustDetails': 'NOthing',
'DirName': 'TEST',
'General': 'C:\\Work\\July\\abc'}

file reading in python

So my whole problem is that I have two files one with following format(for Python 2.6):
#comments
config = {
#comments
'name': 'hello',
'see?': 'world':'ABC',CLASS=3
}
This file has number of sections like this. Second file has format:
[23]
[config]
'name'='abc'
'see?'=
[23]
Now the requirement is that I need to compare both files and generate file as:
#comments
config = {
#comments
'name': 'abc',
'see?': 'world':'ABC',CLASS=3
}
So the result file will contain the values from the first file, unless the value for same attribute is there in second file, which will overwrite the value. Now my problem is how to manipulate these files using Python.
Thanks in advance and for your previous answers in short time ,I need to use python 2.6

Was unable to find a beautiful solution due to the comments. This is tested and works for me, but requires Python 3.1 or higher:
from collections import OrderedDict
indenting = '\t'
def almost_py_read(f):
sections = OrderedDict()
contents = None
active = sections
for line in f:
line = line.strip()
if line.startswith('#'):
active[line] = None
elif line.endswith('{'):
k = line.split('=')[0].strip()
contents = OrderedDict()
active = contents
sections[k] = contents
elif line.endswith('}'):
active = sections
else:
try:
k, v = line.split(':')
k = k.strip()
v = v.strip()
active[k] = v
except:
pass
return sections
def almost_ini_read(f):
sections = OrderedDict()
contents = None
for line in f:
line = line.strip()
try:
k, v = line.split('=')
k = k.strip()
v = v.strip()
if v:
contents[k] = v
except:
if line.startswith('[') and line.endswith(']'):
contents = OrderedDict()
sections[line[1:-1]] = contents
print(sections)
return sections
def compilefiles(pyname, ininame):
sections = almost_py_read(open(pyname, 'rt'))
override_sections = almost_ini_read(open(ininame, "rt"))
for section_key, section_value in override_sections.items():
if not sections.get(section_key):
sections[section_key] = OrderedDict()
for k, v in section_value.items():
sections[section_key][k] = v
return sections
def output(d, indent=''):
for k, v in d.items():
if v == None:
print(indent+k)
elif v:
if type(v) == str:
print(indent+k+': '+v+',')
else:
print(indent+k+' = {')
output(v, indent+indenting)
print(indent+'}')
d = compilefiles('a.txt', 'b.ini')
output(d)
Output:
#comments
config = {
#comments
'name': 'abc',
'see?': 'world',
}

I had a really long and hard time to manage to write the following code.
I had difficulties to manage with commas. I wanted the updated file to have after the updating the same format as the file to update before the updating : lines end with a comma, except for the last one.
This code is crafted for the particular problem as exposed by the questioner and can't be used as-is for another type of problem. I know. It's the problem of using a code based on regex and not on a parser, I'm fully aware of that. But I think that it is a canvas that can be relatively easily adapted to other cases, by changing the regexes, which is a relatively readily process thanks to the malleability of regexes.
def file_updating(updating_filename,updating_data_extractor,filename_to_update):
# function whose name is hold by updating_data_extractor parameter
# is a function that
# extracts data from the file whose name is hold by updating_filename parameter
# and must return a tuple:
# ( updating dictionary , compiled regex )
updating_dico,pat = updating_data_extractor( updating_filename )
with open(filename_to_update,'r+') as f:
lines = f.readlines()
def jiji(line,dico = updating_dico ):
mat = pat.search(line.rstrip())
if mat and mat.group(3) in dico:
return '%s: %s,' % (mat.group(1),dico.pop(mat.group(3)))
else:
return line.rstrip(',') + ','
li = [jiji(line) for line in lines[0:-1] ] # [0:-1] because last line is '}'
front = (mit.group(2) for mit in ( pat.search(line) for line in lines ) if mit).next()
li.extend(front + '%s: %s,' % item for item in updating_dico.iteritems() )
li[-1] = li[-1].rstrip(',')
li.append('}')
f.seek(0,0)
f.writelines( '\n'.join(li) )
f.truncate()
Exemplifying code:
import re
bef1 = '''#comments
config =
{
#comments
'name': 'hello',
'arctic':01011101,
'summu': 456,
'see?': 'world',
'armorique': 'bretagne'
}'''
bef2 = '''#comments
config =
{
#comments
'name': 'abc',
'see?': { 'world':'india':'jagdev'},
}'''
def one_extractor(data_containing_filename):
with open(data_containing_filename) as g:
contg = re.search('\[(\d+)\].+\[config\](.*?)\[(\\1)\]',g.read(),re.DOTALL)
if contg:
updtgen = ( re.match("([^=]+)=[ \f\t\v]*([^ \f\t\v].*|)",line.strip())
for line in contg.group(2).splitlines() )
updating_data = dict( mi.groups() for mi in updtgen if mi and mi.group(2))
else:
from sys import exit
exit(updating_filename + " isn't a valid file for updating")
pat = re.compile("(([ \t]*)([^:]+)):\s*(.+),?")
return (updating_data,pat)
for bef in (bef1,bef2):
# file to update: rudu.txt
with open('rudu.txt','w') as jecr:
jecr.write(bef)
# updating data: renew_rudu.txt
with open('renew_rudu.txt','w') as jecr:
jecr.write('''[23]
[config]
'nuclear'= 'apocalypse'
'name'='abc'
'armorique'= 'BRETAGNE'
'arctic'=
'boloni'=7600
'see?'=
'summu'='tumulus'
[23]''')
print 'BEFORE ---------------------------------'
with open('rudu.txt') as lir:
print lir.read()
print '\nUPDATING DATA --------------------------'
with open('renew_rudu.txt') as lir:
print lir.read()
file_updating('renew_rudu.txt',one_extractor,'rudu.txt')
print '\nAFTER ================================='
with open('rudu.txt','r') as f:
print f.read()
print '\n\nX#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#\n'
Result:
>>>
BEFORE ---------------------------------
#comments
config =
{
#comments
'name': 'hello',
'arctic':01011101,
'summu': 456,
'see?': 'world',
'armorique': 'bretagne'
}
UPDATING DATA --------------------------
[23]
[config]
'nuclear'= 'apocalypse'
'name'='abc'
'armorique'= 'BRETAGNE'
'arctic'=
'boloni'=7600
'see?'=
'summu'='tumulus'
[23]
AFTER =================================
#comments,
config =,
{,
#comments,
'name': 'abc',
'arctic':01011101,
'summu': 'tumulus',
'see?': 'world',
'armorique': 'BRETAGNE',
'boloni': 7600,
'nuclear': 'apocalypse'
}
X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#
BEFORE ---------------------------------
#comments
config =
{
#comments
'name': 'abc',
'see?': { 'world':'india':'jagdev'},
}
UPDATING DATA --------------------------
[23]
[config]
'nuclear'= 'apocalypse'
'name'='abc'
'armorique'= 'BRETAGNE'
'arctic'=
'boloni'=7600
'see?'=
'summu'='tumulus'
[23]
AFTER =================================
#comments,
config =,
{,
#comments,
'name': 'abc',
'see?': { 'world':'india':'jagdev'},
'armorique': 'BRETAGNE',
'boloni': 7600,
'summu': 'tumulus',
'nuclear': 'apocalypse'
}
X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#X#
>>>
.
EDIT:
I have improved the code because I was still insatisfied. Now the "variable" front catches the blank characters ( ' ' or '\t' ) at the beginning of the data-containing lines in the file to be updated.
I had also forgot the instruction f.truncate() which is very important to not keep a tail of undesired characters.
I am satisfied to see that my code works well even with the following file in which a value is a dictionnary, as presented by Jagdev:
#comments
config =
{
#comments
'name': 'abc',
'see?': { 'world':'india':'jagdev'},
}
That confirms me in my choice to process line after line , and not trying to run through the entire file with a regex.
.
EDIT 2:
I again changed the code. The updating is performed by a function that takes as arguments :
the name of the updating file (the file containing the data used to udpdate another file)
and the function that is suited to extract the data from this particular updating file
Hence, it is possible to update a given file with data from various updating files. That makes the code more generic.

Very roughly (i.e. this hasn't been tested at all, and there are numerous imprvements that could be made such as the use of regex and/or pretty-printing):
dicts = []
with open('file1') as file1:
try:
file1content = file1.read()
eval(file1content )
file1content.strip(' ')
file1content.strip('\t')
for line in file1content.splitlines():
if '={' in line:
dicts.append(line.split('={').strip())
except:
print 'file1 not valid'
with open('file2') as file2:
filelines = file2.readlines()
while filelines:
while filelines and '[23]' not in filelines[0]:
filelines.pop(0)
if filelines:
filelines.pop(0)
dictname = filelines.pop(0).split('[')[1].split(']')[0]
if dictname not in dicts:
dicts.append(dictname)
exec(dictname + ' = {}')
while filelines and '[23]' not in filelines[0]:
line = filelines.pop(0)
[k,v] = line.split('=')
k.strip()
v.strip()
if v:
exec(dictname + '[k] = v')
with open('file3', 'w') as file3:
file3content = '\n'.join([`eval(dictname)` for dictname in dicts])
file3.write(file3content)