This question already has answers here:
How do I convert all strings in a list of lists to integers?
(15 answers)
Closed 1 year ago.
I'm currently learning Python and I'm having an issue with getting integer values from a text file (myfile.config). My goal is to be able to read a text file, find the integers then assign the said integers to some variables.
This is what my text file looks like (myFile.config):
someValue:100
anotherValue:1000
yetAnotherValue:-5
someOtherValueHere:5
This is what I've written so far:
import os.path
import numpy as np
# Check if config exists, otherwise generate a config file
def checkConfig():
if os.path.isfile('myFile.config'):
return True
else:
print("Config file not found - Generating default config...")
configFile = open("myFile.config", "w+")
configFile.write("someValue:100\rnotherValue:1000\ryetAnotherValue:-5\rsomeOtherValueHere:5")
configFile.close()
# Read the config file
def readConfig():
tempConfig = []
configFile = open('myFile.config', 'r')
for line in configFile:
cleanedField = line.strip() # remove \n from elements in list
fields = cleanedField.split(":")
tempConfig.append(fields[1])
configFile.close()
print(str(tempConfig))
return tempConfig
configOutput = np.asarray(readConfig())
someValue = configOutput[0]
anotherValue = configOutput[1]
yetAnotherValue = configOutput[2]
someOtherValueHere = configOutput[3]
One of the issues which I've noticed so far (if my current understanding of Python is correct) is that the elements in the list are being stored as strings. I've tried to correct this by converting the list to an array via the NumPy library, but it hasn't worked.
Thank you for taking the time to read this question.
You can use float() or int() to turn strings into either float or integer. So in this case you can just type
tempConfig.append(float(fields[1]))
or
tempConfig.append(int(fields[1]))
You have to call int for the conversion and I would use a dictionary for the result.
def read_config():
configuration = {}
with open('myFile.config', 'r') as config_file:
for line in config_file:
fields = line.split(':')
if len(fields) == 2:
configuration[fields[0].strip()] = int(fields[1])
print(configuration) # for debugging
return configuration
Now there's no need to create single variables like someValue or anotherValue. If you call the function with config = read_config() you have the values available as config['someValue'] and config['anotherValue'].
This is a much more flexible approach. Your current code will fail if you change the order of the lines in the configuration file. And if you add a fifth configuration entry you will have to change your code to create a new variable. The code in this answer can handle this by design.
With some eval magic, you get a dict out of the text file and if you insist, you can put them in the global namespace using globals()
def read_config():
config = '{' + open('myFile.config', 'r').read() + '}'
globals().update(eval(config.replace('{', '{"').replace(':', '":').replace('\n', ',"')))
Related
I am using yaml and pyyaml to configure my application.
Is it possible to configure something like this -
config.yml -
root:
repo_root: /home/raghhuveer/code/data_science/papers/cv/AlexNet_lght
data_root: $root.repo_root/data
service:
root: $root.data_root/csv/xyz.csv
yaml loading function -
def load_config(config_path):
config_path = os.path.abspath(config_path)
if not os.path.isfile(config_path):
raise FileNotFoundError("{} does not exist".format(config_path))
else:
with open(config_path) as f:
config = yaml.load(f, Loader=yaml.SafeLoader)
# logging.info(config)
logging.info("Config used for run - \n{}".format(yaml.dump(config, sort_keys=False)))
return DotDict(config)
Current Output-
root:
repo_root: /home/raghhuveer/code/data_science/papers/cv/AlexNet_lght
data_root: ${root.repo_root}/data
service:
root: ${root.data_root}/csv/xyz.csv
Desired Output -
root:
repo_root: /home/raghhuveer/code/data_science/papers/cv/AlexNet_lght
data_root: /home/raghhuveer/code/data_science/papers/cv/AlexNet_lght/data
service:
root: /home/raghhuveer/code/data_science/papers/cv/AlexNet_lght/data/csv/xyz.csv
Is this even possible with python? If so any help would be really nice.
Thanks in advance.
A general approach:
read the file as is
search for strings containing $:
determine the "path" of "variables"
replace the "variables" with actual values
An example, using recursive call for dictionaries and replaces strings:
import re, pprint, yaml
def convert(input,top=None):
"""Replaces $key1.key2 with actual values. Modifies input in-place"""
if top is None:
top = input # top should be the original input
if isinstance(input,dict):
ret = {k:convert(v,top) for k,v in input.items()} # recursively convert items
if input != ret: # in case order matters, do it one or several times more until no change happens
ret = convert(ret)
input.update(ret) # update original input
return input # return updated input (for the case of recursion)
if isinstance(input,str):
vars = re.findall(r"\$[\w_\.]+",input) # find $key_1.key_2.keyN sequences
for var in vars:
keys = var[1:].split(".") # remove dollar and split by dots to make "key chain"
val = top # starting from top ...
for k in keys: # ... for each key in the key chain ...
val = val[k] # ... go one level down
input = input.replace(var,val) # replace $key sequence eith actual value
return input # return modified input
# TODO int, float, list, ...
with open("in.yml") as f: config = yaml.load(f) # load as is
convert(config) # convert it (in-place)
pprint.pprint(config)
Output:
{'root': {'data_root': '/home/raghhuveer/code/data_science/papers/cv/AlexNet_lght/data',
'repo_root': '/home/raghhuveer/code/data_science/papers/cv/AlexNet_lght'},
'service': {'root': '/home/raghhuveer/code/data_science/papers/cv/AlexNet_lght/data/csv/xyz.csv'}}
Note: YAML is not that important here, would work also with JSON, XML or other formats.
Note2: If you use exclusively YAML and exclusively python, some answers from this post may be useful (using anchors and references and application specific local tags)
I am new to python and I know I can make the following code more concise using iteration (such as a for loop), I am just not sure how to
this is what I have so far
# Open file for reading
dataFileRead = open(filename, "r")
# Read file content into a list - to be completed - Part 1
SampleData = [line.rstrip('\n') for line in open(filename)]
print(SampleData)
variables = [mazeWidth, mazeHeight, aNumOfTreasures, aNumOfBombs, emptyCell, treasure, bomb, exitGate, boundary, boundarySide]
mazeWidth = SampleData[0]
mazeHeight = SampleData[1]
aNumOfTreasures = SampleData[2]
aNumOfBombs = SampleData[3]
emptyCell = SampleData[4]
treasure = SampleData[5]
bomb = SampleData[6]
mario = SampleData[7]
exitGate = SampleData[8]
boundary = SampleData[9]
boundarySide = SampleData[10]
any input helps! thank you
You can use a dictionary to hold the variables' names and values instead of having separate variables:
variable_names = ['mazeWidth', 'mazeHeight', 'aNumOfTreasures', 'aNumOfBombs', 'emptyCell', 'treasure', 'bomb', 'exitGate', 'boundary', 'boundarySide']
variables = {
name: SampleData[i] for i, name in enumerate(variable_names)
}
Later, if you want the value of the variable exitGate for example, you can use:
variables['exitGate']
And for assignment, use:
variables['exitGate'] = "some value"
If you want separate variables however, you can use this:
for i, name in enumerate(variable_names):
globals()[name] = SampleData[i]
And later you can access (get and set) the variables just as you normally would do (print(exitGate); exitGate = "some value").
If you really need these 11 variables to exist with their own names, abstain from any trick. 11 lines is no big deal.
Otherwise, keep working with the list SampleData.
If I were to use your method (which I wouldn't – I'll detail that later), I would edit your code as follows to combine SampleData with variables using zip() and dict().
# Open file for reading
dataFileRead = open(filename, "r")
# Read file content into a list - to be completed - Part 1
SampleData = [line.rstrip('\n') for line in open(filename)]
print(SampleData)
#previously known as variables
variable_names = ["mazeWidth", "mazeHeight", "aNumOfTreasures", "aNumOfBombs", "emptyCell", "treasure", "bomb", "exitGate", "boundary", "boundarySide"]
variables = dict(zip(variable_names, SampleData))
print(variables)
This will combine both lists into one dictionary. This way, if you want to access the number of bombs, or the width of the maze, all you have to do is write:
print(variables["aNumOfBombs"])
Dictionaries are useful like that. However, I would redo the system entirely. As your reading from a file anyway, I think you should make use of the json module and store the data that way. All of the code above would instead look like:
import json
with open(filename, "r") as var_file:
variables = json.load(var_file)
The only difference is how you structure the file you read from, which would instead look something like this:
{
"mazeWidth": 5,
"mazeHeight": 10,
"aNumOfTreasures": 4,
"aNumOfBombs": 16,
"emptyCell": "whatever",
"treasure": true,
"bomb": true,
"exitGate": false,
"boundary": "red",
"boundarySide": "NW"
}
Look into it!
I want to store values that I plan to later use for sorting pdfs on my computer using PyPDF2.
I thought that if I created a class and stored identifying info for each type of file (such as a descriptor, a string that is unique to the file and can be found by PyPDF2 later such as an account number, and the path where the file should be moved to) that would work. Something like this:
class File_Sort(object):
def __init__(self, identifier, file_text, file_path):
self.identifier = identifier
self.file_text = file_text
self.file_path = file_path
so an example input from me would be:
filetype0001 = File_Sort("Phone Bill", "123456", "/Users/Me/PhoneBills/")
I would like to be able to have users generate new file types via a series of raw_input questions, but I can't figure how to generate the variable to create a new instance, so that I can get:
filetype000[automatically incrementing number] = File_Sort(UserResponse1, UserResponse3, UserResponse3).
Creating the "filetype000[automatically incrementing number]" text itself seems easy enough with:
file_number += 1
file_name = "filetype" + str(file_number).zfill(4)
but how do you turn the generated file_name string into a variable and populate it?
It sounds like you're wanting to dynamically create variables. That's almost always a foolish thing to do. Instead, you should be using a data structure like a list or dictionary, indexed by the parts of the variable name you wanted to generate dynamically.
So instead of creating a list named filetype000, start with a list named filetypes, and append an inner list, so you can do filetypes[0] to get at it. Or if string names make more sense for your specific application, let filetypes be a dictionary, and access the inner lists with something like filetypes['pdf'].
I'm being a little vague here because I don't really understand all of your pseudocode. It's not at all obvious what the purpose of the [automatically incrementing number] parts of your example are, so I'm more or less ignoring those bits. You probably just want to start with an empty list and append values to it, rather than somehow initializing it to a specific size and magically indexing it.
so fyi this is what I ended up using:
file_descriptor = []
file_string = []
file_location = []
filetype_new = len(file_descriptor)
input_descriptor = raw_input("What is the description of the new file type? ")
file_descriptor.append(input_descriptor)
input_filestring = raw_input("What is unique string to search for in this file type? ")
file_string.append(input_filestring)
input_filelocation = raw_input("where should we put this file type? ")
file_location.append(input_filelocation)
print("file%s: %s, \t%s, \t%s" % (str(filetype_new+1).zfill(4), file_descriptor[filetype_new], file_string[filetype_new], file_location[filetype_new]))
review = raw_input("\nWould you like to review the current files? y/n ").lower()
while review not in "yn":
review = raw_input("Sorry, I don't understand. Would you like to review your file types? y/n ").lower()
print("There are currently sort instructions for %s filetypes: " % (len(file_descriptor)))
file_increment = 0
while file_increment in range(0, len(file_descriptor)):
print("file%s: %s, \t%s, \t%s" % (
str(file_increment + 1).zfill(4), file_descriptor[file_increment], file_string[file_increment],
file_location[file_increment]))
file_increment += 1
thanks for your advice.
I am using a new script (a) to extract information from an old script (b) to create a new file (c). I am looking for an equal sign in the old script (b) and want to modify the modification script (a) to make it automated.
The string is
lev1tolev2 'from=e119-b3331l1 mappars="simp:180" targ=enceladus.bi.def.3 km=0.6 lat=(-71.5,90) lon=(220,360)'
It is written in python 3.
The current output is fixed at
cam2map from=e119-b3331l1 to=rsmap-x map=enc.Ink.map pixres=mpp defaultrange=MAP res=300 minlat=-71.5 maxlat=90 minlon=220 maxlon=360
Currently, I have the code able to export a string of 0.6 for all of the iterations of lev1tolev2, but each one of these is going to be different.
cam2map = Call("cam2map")
cam2map.kwargs["from"] = old_lev1tolev2.kwargs["from"]
cam2map.kwargs["to"] = "rsmap-x"
cam2map.kwargs["map"] = "enc.Ink.map"
cam2map.kwargs["pixres"] = "mpp"
cam2map.kwargs["defaultrange"] = "MAP"
**cam2map.kwargs["res"] = float((old_lev1tolev2.kwargs["km"]))**
cam2map.kwargs["minlat"] = lat[0]
cam2map.kwargs["maxlat"] = lat[1]
cam2map.kwargs["minlon"] = lon[0]
cam2map.kwargs["maxlon"] = lon[1]
I have two questions, why is this not converting the string to a float? And, why is this not iterating over all of the lev1tolev2 commands as everything else in the code does?
The full code is available here.
https://codeshare.io/G6drmk
The problem occurred at a different location in the code.
def escape_kw_value(value):
if not isinstance(value, str):
return value
elif (value.startswith(('"', "'")) and value.endswith(('"', "'"))):
return value
# TODO escape the quote with \" or \'
#if value.startswith(('"', "'")) or value.endswith(('"', "'")):
# return value
if " " in value:
value = '"{}"'.format(value)
return value
it doesn't seem to clear to me, but from you syntax here :
**cam2map.kwargs["res"] = float((old_lev1tolev2.kwargs["km"]))**
I'd bet that cam2map.kwargs["res"] is a dict, and you thought that it would convert every values in the dict, using the ** syntax. The float built-in should then be called in a loop over the elements of the dict, or possible a list-comprehension as here :
cam2map.kwargs["res"] = dict()
for key, value in old_lev1tolev2.kwars["res"].items():
cam2map.kwargs["res"][key] = float(value)
Edit :
Ok so, it seems you took the string 'from=e119-b3331l1 mappars="simp:180" targ=enceladus.bi.def.3 km=0.6 lat=(-71.5,90) lon=(220,360)'
And then thought that calling youstring.kwargs would give you a dict, but it won't, you can probably parse it to a dict first, using some lib, or, you use mystring.split('=') and then work your way to a dict first, like that:
output = dict()
for one_bit in lev_1_lev2.split(' '):
key, value = one_bit.split('=')
output[key] = value
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I have 15 values that I want to get from a config file and store them in separate variables.
I am using
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read(configFile)
and it is a really good library.
Option #1
If I change the name of the variable and want it to match the config file entry I have to edit the corresponding line in the function
def fromConfig():
#open file
localOne = parser.get(section, 'one')
localTwo = parser.get(section, 'two')
return one, two
one = ''
two = ''
#etc
one, two = fromConfig()
Option #2
It is cleaner to see where the variables get their values from, but then I would be opening and closing the file for every variable
def getValueFromConfigFile(option):
#open file
value = parser.get(section, option)
return value
one = getValueFromConfigFile("one")
two = getValueFromConfigFile("two")
Option #3
This one doesn't make much sense since I have to have another list of all my variable names, but the function is cleaner.
def getValuesFromConfigFile(options):
#open file
values = []
for option in options:
values.append(parser.get(section, option))
return values
one = ''
two = ''
configList = ["one", "two"]
one, two = getValuesFromConfigFile(configList)
EDIT:
Here is my attempt at reading the file one and storing all values in a dict and then trying to use he values.
I have a multi-lined string and I am using
%(nl)s to be a new line character so then when I get the value
message = parser.get(section, 'message', vars={'nl':'\n'})
Here is my code:
from ConfigParser import SafeConfigParser
def getValuesFromConfigFile(configFile):
''' reads a single section of a config file as a dict '''
parser = SafeConfigParser()
parser.read(configFile)
section = parser.sections()[0]
options = dict(parser.items(section))
return options
options = getValuesFromConfigFile(configFile)
one = options["one"]
To get values from a single section as a dict:
options = dict(parser.items(section))
You could access individual values as usual: options["one"], options["two"]. In Python 3.2+ configparser provides dict-like access by itself.
For flexibility, to support updating config from a variety of source formats and/or centralize configuration management; you could define custom class that encapsulates parsing/access to config variables e.g.:
class Config(object):
# ..
def update_from_ini(self, inifile):
# read file..
self.__dict__.update(parser.items(section))
Individual values are available as instance attributes in this case: config.one, config.two.
A solution could be as well to use dictionaries & json which can make things verry easy & reusable
import json
def saveJson(fName, data):
f = open(fName, "w+")
f.write(json.dumps(data, indent=4))
f.close()
def loadJson(fName):
f = open(fName, "r")
data = json.loads(f.read())
f.close()
return data
mySettings = {
"one": "bla",
"two": "blabla"
}
saveJson("mySettings.json", mySettings)
myMoadedSettings = loadJson("mySettings.json")
print myMoadedSettings["two"]
As a possible solution:
module_variables = globals() # represents the current global symbol table
for name in ('one', 'two'):
module_variables[name] = parser.get(section, name)
print one, two