Good morning,
I have a configuration file with data like this:
[hostset 1]
ip = 192.168.122.136
user = test
password =
pkey = ~/.ssh/id_rsa
[hostset 2]
ip = 192.168.122.138
user = test
password =
pkey = ~/.ssh/id_rsa
I want to be able to join the ips of any given number of host sets in this configuration file if the other values are the same, so the ingested and formatted data would be stored in a dict, something like this:
{
ip: ['192.168.122.136', '192.168.122.138'],
user: 'test',
password: '',
pkey: '~/.ssh/id_rsa',
}
by doing something like:
from configparser import ConfigParser
def unpack(d):
return [value for key, value in d.items()]
def parse(configuration_file):
parser = ConfigParser()
parser.read(configuration_file)
hosts = [unpack(connection) for connection in [section for section in dict(parser).values()]][1:]
return [i for i in hosts]
if __name__ == '__main__':
parse('config.ini')
I can get a list of lists containing the elements of the configuration file, like this:
[['192.168.122.136', 'test', '', '~/.ssh/id_rsa'], ['192.168.122.138', 'test', '', '~/.ssh/id_rsa']]
Then I just need a way of comparing the two lists and if all elements are similar except for the ip, then join them into a list like:
[['192.168.122.136','192.168.122.138'], 'test', '', '~/.ssh/id_rsa']
So I would just need a smart way of doing this with a list of lists of no specific length and join all similar lists.
Got some help from a friend and solved the question. The key was making the values I wanted to compare into a tuple, making that tuple the key to a dictionary and the value the ips. From this. I can assert that if the tuple key already exists, then I will append the ip to the value.
from configparser import ConfigParser
from ast import literal_eval as literal
def unpack(d):
return [value for key, value in d.items()]
def parse(configuration_file):
parser = ConfigParser()
parser.read(configuration_file)
hosts = [unpack(connection) for connection in [section for section in dict(parser).values()]][1:]
d = dict()
for item in hosts:
try:
d[str((item[1:]))].append(item[0])
except KeyError:
d[str((item[1:]))] = [item[0]]
return d
if __name__ == '__main__':
for k, v in parse('config.ini').items():
print([v, *literal(k)])
In this solution, I presumed that the file format is exactly as described in the question:
First we split the host sets:
we suppose that your data is in rowdata variable
HostSets = rowdata.split("[hostset ") # first element is empty
Dict = {}
for i in range (1,len(HostSets)):
l = HostSets[i].split("ip = ")#two elements the first is trash
ip = l[1].split()[0]
conf =l[1].split("\n",1 )[1] #splits only the first element
try :
Dict[conf].append(ip)
except :
Dict[conf] = list()
Dict[conf].append(ip)
print('{')
for element in Dict:
print("ip: ",Dict[element],",",element)
print('}')
Related
I have written a short script to print some info based on information in a CSV file.
What I need to be able to do is to make the print function not print if there is no value for the key, or if there is a default value such as 'n/a'.
Or it may be that if there is a default or empty cell in the CSV, that it doesn't get added to the dictionary? Not sure what is the best option.
import csv
with open('lhcdes.csv', 'rb') as testcsv:
myfile = csv.DictReader(testcsv)
for row in myfile:
print 'Key1 %s' % row.get('Key1') + '\n' + 'and ' + 'Key2:%s ' % row.get('Key2') + 'Key3:%s ' % row.get('Key3:')
the CSV Format is as follow:
Key1,Key2,Key3,Key4,Key5,Key6
Gi0/3/0/1.1838,CustA,EU1,AN-12345,TAL12345,Host1_London
Gi0/3/0/1.2072,CustB,EU2,AN-12346,TAL12346,Host2_Manchester
Gi0/3/0/2.3761,CustB,EU3,AN-12347,TAL12347,Not Found
Gi0/3/0/3.3573,CustC,EU7,AN-12348,TAL12348,Host5_Swansea
Gi0/3/0/3.3702,CustD,EU5,AN-12349,N/A,Host4_Glasgow
Gi0/3/0/3.3917,CustB,EU6,AN-12350,TAL12350,Not Found
Gi0/3/0/3.3918,CustA,EU2,AN-12351,TAL12351,N/A
Gi0/3/0/3.3919,CustE,EU9,AN-12352,Not Found,Not Found
Gi0/3/0/3.3923,CustE,EU9,AN-12353,TAL12353,N/A
Gi0/3/0/4.512,CustC,EU8,AN-12354,TAL12354,Not Found
The output should look like
interface Gi0/3/0/1.1838
Client:CustA EU:EU1 IR:AN-12345 CR:TAL12345 R:Host1_London
interface Gi0/3/0/1.2072
Client:CustB EU:EU2 IR:AN-12346 CR:TAL12346 R:Host2_Manchester
Where info is absent or n/a
interface Gi0/3/0/3.3919
Client:CustE EU:EU9 IR:AN-12352
You need to test the contents of Key5 and Key6 for a missing value and then format your output accordingly:
import csv
missing = ['Not Found', None, 'N/A', '']
with open('lhcdes.csv', 'rb') as testcsv:
myfile = csv.DictReader(testcsv)
for row in myfile:
if row['Key5'] in missing or row['Key6'] in missing:
print 'interface {}\nClient: {} {} IR:{}'.format(
row['Key1'], row['Key2'], row['Key3'], row['Key4'])
else:
print 'interface {}\nClient: {} {} IR:{} CR:{} R:{}'.format(
row['Key1'], row['Key2'], row['Key3'], row['Key4'], row['Key5'], row['Key6'])
The script would display the following output:
interface Gi0/3/0/1.1838
Client: CustA EU1 IR:AN-12345 CR:TAL12345 R:Host1_London
interface Gi0/3/0/1.2072
Client: CustB EU2 IR:AN-12346 CR:TAL12346 R:Host2_Manchester
interface Gi0/3/0/2.3761
Client: CustB EU3 IR:AN-12347
interface Gi0/3/0/3.3573
Client: CustC EU7 IR:AN-12348 CR:TAL12348 R:Host5_Swansea
interface Gi0/3/0/3.3702
Client: CustD EU5 IR:AN-12349
interface Gi0/3/0/3.3917
Client: CustB EU6 IR:AN-12350
interface Gi0/3/0/3.3918
Client: CustA EU2 IR:AN-12351
interface Gi0/3/0/3.3919
Client: CustE EU9 IR:AN-12352
interface Gi0/3/0/3.3923
Client: CustE EU9 IR:AN-12353
interface Gi0/3/0/4.512
Client: CustC EU8 IR:AN-12354
One way would be to print the keys in a loop, given a condition:
import csv
with open('lhcdes.csv', 'rb') as testcsv:
myfile = csv.DictReader(testcsv)
keys = ['key1', 'key2', 'key3'] # a list of all keys you want to print
default_value = 'n/a' # the default value you want to skip
for row in myfile:
for key in keys:
value = row.get(key, default_value) # get the value at 'key', or the
# default value if 'key' was not found
if value == default_value:
continue # skip to the next item in the 'keys'
# list
print("{} {}".format(key, value)) # use .format over '%' syntax
This way, you skip any entries where there was either not value or whatever your default is.
I have a file of constant variables that I need to query and I am not sure how to go about it.
I have a database query which is returning user names and I need to find the matching user name in the file of constant variables.
The file looks like this:
SALES_MANAGER_01 = {"user_name": "BO01", "password": "password", "attend_password": "BO001",
"csm_password": "SM001", "employee_num": "BOSM001"}
There is just a bunch of users just like the one above.
My function looks like this:
#attr("user_test")
def test_get_user_for_login(self):
application_code = 'BO'
user_from_view = self.select_user_for_login(application_code=application_code)
users = [d['USER'] for d in user_from_view]
user_with_ent = choice(users)
user_wo_ent = user_with_ent[-4:]
password = ""
global_users = dir(gum)
for item in global_users:
if user_wo_ent not in item.__getattr__("user_name"):
user_with_ent = choice(users)
user_wo_ent = user_with_ent[-4:]
else:
password = item.__getattr__("password")
print(user_wo_ent, password)
global_users = dir(gum) is my file of constants. So I know I am doing something wrong since I am getting an attribute error AttributeError: 'str' object has no attribute '__getattr__', I am just not sure how to go about resolving it.
You should reverse your looping as you want to compare each item to your match condition. Also, you have a dictionary, so use it to do some heavy lifting.
You need to add some imports
import re
from ast import literal_eval
I've changed the dir(gum) bit to be this function.
def get_global_users(filename):
gusers = {} # create a global users dict
p_key = re.compile(ur'\b\w*\b') # regex to get first part, e.g.. SALES_MANAGER_01
p_value = re.compile(ur'\{.*\}') # regex to grab everything in {}
with (open(filename)) as f: # open the file and work through it
for line in f: # for each line
gum_key = p_key.match(line) # pull out the key
gum_value = p_value.search(line) # pull out the value
''' Here is the real action. update a dictionary
with the match of gum_key and with match of gum_value'''
gusers[gum_key.group()] = literal_eval(gum_value.group())
return(gusers) # return the dictionary
The bottom of your existing code is replaced with this.
global_users = get_global_users(gum) # assign return to global_users
for key, value in global_users.iteritems(): # walk through all key, value pairs
if value['user_name'] != user_wo_ent:
user_with_ent = choice(users)
user_wo_ent = user_with_ent[-4:]
else:
password = value['password']
So a very simple answer was get the dir of the constants file then parsing over it like so:
global_users = dir(gum)
for item in global_users:
o = gum.__dict__[item]
if type(o) is not dict:
continue
if gum.__dict__[item].get("user_name") == user_wo_ent:
print(user_wo_ent, o.get("password"))
else:
print("User was not in global_user_mappings")
I was able to find the answer by doing the following:
def get_user_for_login(application_code='BO'):
user_from_view = BaseServiceTest().select_user_for_login(application_code=application_code)
users = [d['USER'] for d in user_from_view]
user_with_ent = choice(users)
user_wo_ent = user_with_ent[4:]
global_users = dir(gum)
user_dict = {'user_name': '', 'password': ''}
for item in global_users:
o = gum.__dict__[item]
if type(o) is not dict:
continue
if user_wo_ent == o.get("user_name"):
user_dict['user_name'] = user_wo_ent
user_dict['password'] = o.get("password")
return user_dict
So i have a list of elements:
elements = [room1, room2, room3]
I also have a list of key/value attributes that each room has:
keys = ["level", "finish1", "finish2"]
values = [["ground", "paint1", "carpet1"],["ground", "paint1", "paint2"], ["second level", "paint1", "paint2"]]
is there a way to serialize this two lists into a json file structured like this:
{'room1': [{'level': 'ground', 'finish1': 'paint1', 'finish2': 'carpet1'}],'room2': [{'level': 'ground', 'finish1': 'paint1', 'finish2': 'paint2'}],'room3': [{'level': 'second level', 'finish1': 'paint1', 'finish2': 'paint2'}]}
I am on this weird platform that doesnt support dictionaries so I created a class for them:
class collection():
def __init__(self,name,key,value):
self.name = name
self.dict = {}
self.dict[key] = value
def __str__(self):
x = str(self.name) + " collection"
for key,value in self.dict.iteritems():
x = x + '\n'+ ' %s= %s ' % (key, value)
return x
then i found a peiece of code that would allow me to create a basic json code from two parallel lists:
def json_list(keys,values):
lst = []
for pn, dn in zip(values, keys):
d = {}
d[dn]=pn
lst.append(d)
return json.dumps(lst)
but this code desnt give me the {room1: [{ ... structure
Any ideas would be great. This software I am working with is based on IronPython2.7
Ok, so the above worked great. I got a great feedback from Comments. I have one more variation that I didnt account for. Sometimes when I try to mix more than singe element type (rooms, columns etc) they might not have the same amount of attributes. For example a room can have (level, finish and finish) while column might have only thickness and material. If i kept it all organized in parallel lists key/value is it possible to modify the definition below:
keys = [[thickness, material],[level,finish,finish]]
values = [[100,paint],[ground,paint,paint]]
elements = [column,room]
How would i need to modify the definition below to make it work? Again I want to export a json file.
I don't know how Python can even work without dictionaries, so please just test this and tell me the error it shows you:
import json
elements = ['r1','r2','r3']
keys = ["level", "finish1", "finish2"]
values = [["ground", "paint1", "carpet1"],["ground", "paint1", "paint2"], ["second level", "paint1", "paint2"]]
d = dict()
for (index, room) in enumerate(elements):
d[room] = dict()
for (index2, key) in enumerate(keys):
d[room][key] = values[index][index2]
print json.dumps(d)
This may work.
#-*- encoding: utf-8 -*-
import json
elements = ["room1", "room2", "room3"]
keys = ["level", "finish1", "finish2"]
values = [["ground", "paint1", "carpet1"],["ground", "paint1", "paint2"], ["second level", "paint1", "paint2"]]
what_i_want = dict((room, [dict(zip(keys, value))])
for room, value in zip(elements, values))
print(json.dumps(what_i_want))
Alright, so basically I have a Google script that searches for a keyword. The results look like:
http://www.example.com/user/1234
http://www.youtube.com/user/125
http://www.forum.com/user/12
What could I do to organize these results like this?:
Forums:
http://www.forum.com/user/12
YouTubes:
http://www.youtube.com/user/125
Unidentified:
http://www.example.com/user/1234
By the way I'm organizing them with keywords. If the url has "forum" in it then it goes to the forum list, if it has YouTube it goes to the YouTube list, but if no keywords match up then it goes to unidentified.
1/. Create a dict, and assign an empty list to each keyword you have.
eg
my_dict = {'forums':[],'youtube':[],'unidentified':[]}
2/.Iterate over your urls.
3/. Generate a key for your url,domain name in your case, you can extract the key using re regex module.
4/ Check the dictionary ( of step#1) for this key, if it does not exist, assign it to 'unidentified key, if it exists, append this url to the list in the dictionary with that key.
Something like this? I guess you will be able to adapt this example to your needs
import pprint
import re
urls = ['http://www.example.com/user/1234',
'http://www.youtube.com/user/126',
'http://www.youtube.com/user/125',
'http://www.forum.com/useryoutube/12']
pattern = re.compile('//www\.(\w+)\.')
keys = ['forum', 'youtube']
results = dict()
for u in urls:
ms = pattern.search(u)
key = ms.group(1)
if key in keys:
results.setdefault(key, []).append(u)
pprint.pprint(results)
import urlparse
urls = """
http://www.example.com/user/1234
http://www.youtube.com/user/125
http://www.forum.com/user/12
""".split()
categories = {
"youtube.com": [],
"forum.com": [],
"unknown": [],
}
for url in urls:
netloc = urlparse.urlparse(url).netloc
if netloc.count(".") == 2:
# chop sub-domain
netloc = netloc.split(".", 1)[1]
if netloc in categories:
categories[netloc].append(url)
else:
categories["unknown"].append(url)
print categories
Parse the urls. Find the category. Append the full url
You should probably keep your sorted results in a dictionary and the unsorted ones in a list. You could then sort it like so:
categorized_results = {"forum": [], "youtube": []}
uncategorized_results = []
for i in results:
i = i.split(".")
for k in categorized_results:
j = True
if k in i:
categorized_results[k].append(i)
j = False
if j:
uncategorized_results.append(i)
If you'd like to output it neatly:
category_aliases: {"forum": "Forums:", "youtube": "Youtubes:"}
for i in categorized_results:
print(category_aliases[i])
for j in categorized_results[i]:
print(j)
print("\n")
print("Unidentified:")
print("\n".join(uncategorized_results)) # Let's not put in another for loop.
How about this:
from urlparse import urlparse
class Organizing_Results(object):
CATEGORY = {'example': [], 'youtube': [], 'forum': []}
def __init__(self):
self.url_list = []
def add_single_url(self, url):
self.url_list.append(urlparse(url))
def _reduce_result_list(self, acc, element):
for c in self.CATEGORY:
if c in element[1]:
return self.CATEGORY[c].append(element)
return self.CATEGORY['example'].append(element)
def get_result(self):
reduce(lambda x, y: c._reduce_result_list(x, y), c.url_list, [])
return self.CATEGORY
c = Organizing_Results()
c.add_single_url('http://www.example.com/user/1234')
c.add_single_url('http://www.youtube.com/user/1234')
c.add_single_url('http://www.unidentified.com/user/1234')
c.get_result()
You can easy broaden the class with more functions as you need.
How can I convert the result of a ConfigParser.items('section') to a dictionary to format a string like here:
import ConfigParser
config = ConfigParser.ConfigParser()
config.read('conf.ini')
connection_string = ("dbname='%(dbname)s' user='%(dbuser)s' host='%(host)s' "
"password='%(password)s' port='%(port)s'")
print connection_string % config.items('db')
Have you tried
print connection_string % dict(config.items('db'))
?
How I did it in just one line.
my_config_parser_dict = {s:dict(config.items(s)) for s in config.sections()}
No more than other answers but when it is not the real businesses of your method and you need it just in one place use less lines and take the power of dict comprehension could be useful.
This is actually already done for you in config._sections. Example:
$ cat test.ini
[First Section]
var = value
key = item
[Second Section]
othervar = othervalue
otherkey = otheritem
And then:
>>> from ConfigParser import ConfigParser
>>> config = ConfigParser()
>>> config.read('test.ini')
>>> config._sections
{'First Section': {'var': 'value', '__name__': 'First Section', 'key': 'item'}, 'Second Section': {'__name__': 'Second Section', 'otherkey': 'otheritem', 'othervar': 'othervalue'}}
>>> config._sections['First Section']
{'var': 'value', '__name__': 'First Section', 'key': 'item'}
Edit: My solution to the same problem was downvoted so I'll further illustrate how my answer does the same thing without having to pass the section thru dict(), because config._sections is provided by the module for you already.
Example test.ini:
[db]
dbname = testdb
dbuser = test_user
host = localhost
password = abc123
port = 3306
Magic happening:
>>> config.read('test.ini')
['test.ini']
>>> config._sections
{'db': {'dbname': 'testdb', 'host': 'localhost', 'dbuser': 'test_user', '__name__': 'db', 'password': 'abc123', 'port': '3306'}}
>>> connection_string = "dbname='%(dbname)s' user='%(dbuser)s' host='%(host)s' password='%(password)s' port='%(port)s'"
>>> connection_string % config._sections['db']
"dbname='testdb' user='test_user' host='localhost' password='abc123' port='3306'"
So this solution is not wrong, and it actually requires one less step. Thanks for stopping by!
I know this was asked a long time ago and a solution chosen, but the solution selected does not take into account defaults and variable substitution. Since it's the first hit when searching for creating dicts from parsers, thought I'd post my solution which does include default and variable substitutions by using ConfigParser.items().
from ConfigParser import SafeConfigParser
defaults = {'kone': 'oneval', 'ktwo': 'twoval'}
parser = SafeConfigParser(defaults=defaults)
parser.set('section1', 'kone', 'new-val-one')
parser.add_section('section1')
parser.set('section1', 'kone', 'new-val-one')
parser.get('section1', 'ktwo')
parser.add_section('section2')
parser.get('section2', 'kone')
parser.set('section2', 'kthree', 'threeval')
parser.items('section2')
thedict = {}
for section in parser.sections():
thedict[section] = {}
for key, val in parser.items(section):
thedict[section][key] = val
thedict
{'section2': {'ktwo': 'twoval', 'kthree': 'threeval', 'kone': 'oneval'}, 'section1': {'ktwo': 'twoval', 'kone': 'new-val-one'}}
A convenience function to do this might look something like:
def as_dict(config):
"""
Converts a ConfigParser object into a dictionary.
The resulting dictionary has sections as keys which point to a dict of the
sections options as key => value pairs.
"""
the_dict = {}
for section in config.sections():
the_dict[section] = {}
for key, val in config.items(section):
the_dict[section][key] = val
return the_dict
For an individual section, e.g. "general", you can do:
dict(parser['general'])
Another alternative would be:
config.ini
[DEFAULT]
potato=3
[foo]
foor_property=y
potato=4
[bar]
bar_property=y
parser.py
import configparser
from typing import Dict
def to_dict(config: configparser.ConfigParser) -> Dict[str, Dict[str, str]]:
"""
function converts a ConfigParser structure into a nested dict
Each section name is a first level key in the the dict, and the key values of the section
becomes the dict in the second level
{
'section_name': {
'key': 'value'
}
}
:param config: the ConfigParser with the file already loaded
:return: a nested dict
"""
return {section_name: dict(config[section_name]) for section_name in config.sections()}
main.py
import configparser
from parser import to_dict
def main():
config = configparser.ConfigParser()
# By default section names are parsed to lower case, optionxform = str sets to no conversion.
# For more information: https://docs.python.org/3/library/configparser.html#configparser-objects
# config.optionxform = str
config.read('config.ini')
print(f'Config read: {to_dict(config)}')
print(f'Defaults read: {config.defaults()}')
if __name__ == '__main__':
main()
In Python +3.6 you could do this
file.ini
[SECTION1]
one = 1
two = 2
[SECTION2]
foo = Hello
bar = World
[SECTION3]
param1 = parameter one
param2 = parameter two
file.py
import configparser
cfg = configparser.ConfigParser()
cfg.read('file.ini')
# Get one section in a dict
numbers = {k:v for k, v in cfg['SECTION1'].items()}
If you need all sections listed you should use cfg.sections()
Combining Michele d'Amico and Kyle's answer (no dict),
produces a less readable but somehow compelling:
{i: {i[0]: i[1] for i in config.items(i)} for i in config.sections()}
Here is another approach using Python 3.7 with configparser and ast.literal_eval:
game.ini
[assets]
tileset = {0:(32, 446, 48, 48),
1:(96, 446, 16, 48)}
game.py
import configparser
from ast import literal_eval
config = configparser.ConfigParser()
config.read('game.ini')
# convert a string to dict
tileset = literal_eval(config['assets']['tileset'])
print('tileset:', tileset)
print('type(tileset):', type(tileset))
output
tileset: {0: (32, 446, 48, 48), 1: (96, 446, 16, 48)}
type(tileset): <class 'dict'>