Develop Reference

How to align print output by field when printing key,value pairs in Python [duplicate]

This question already has answers here:
String formatting: Columns in line
(4 answers)
Closed 3 years ago.
I want to print the content of a Python config parser object.
The following code
for section in system_config.config.sections():
print ("\n[" + section + "]")
for (key, value) in system_config.config.items(section):
print(key + "=" + value)
prints
[GENERAL]
data_roots=[["c:\\data", "/data"] , ["d:\\data2", "/data2"]]
test_data_folder=c:\data\atp-test-data
mount_render_python_apps=false
mount_render_modules=false
host_memory=24
host_number_of_cores=4
at_core_threads=15
For readability, the following is preferable:
[GENERAL]
data_roots = [["c:\\data", "/data"] , ["d:\\data2", "/data2"]]
test_data_folder = c:\data\atp-test-data
mount_render_python_apps = false
mount_render_modules = false
host_memory = 24
host_number_of_cores = 4
at_core_threads = 15
In C++ this can be achieved by setting the 'width' of the first field when using the stream operator '<<'.
Question is, how to do this with Python?

You can use {:<30} format to align strings up to 30 of length to left
here is a full example:
import random
import string
def randomstr():
letters = string.ascii_lowercase
return ''.join(random.choice(letters) for i in range(random.randint(1,30)))
for _ in range(10):
print('{:<30} = {}'.format(randomstr(), randomstr()))
Sample output
ohpy = bxqoknodteueocokveygkdxmzzxubi
rsulmvnqeyeihchanxrggorlm = vtfeu
cvuhpavispkfbttbadt = d
dgfcqtswqjvywosiikkjdmpyvjhoo = ijx
ainrzifrjrkqfanrxyczs = aluoaoizxtmcrvqv
zpujlyopvrucjqugtaamu = pezh
eot = uizfrxpkjywtlxbgzhrcuuj
hfavmswauekyrtgzrhyxwmbgcyzfq = znwfpuosysirtbkiiimzjkifbueq
qxsqzwkyafcwjrjwnwlradrudush = barehtexzpku
hntgerexophiqbafmwfwdomas = frtsmtakcfztlwfesiijacbmocksqq
You may not know the maximum length of your key strings. But you can do it with something like this in your case maxlen = max(len(k) for k in system_config.config.keys()) and using maxlen in format like this '{:<{width}} = {}'.format(randomstr(), randomstr(), width=maxlen)

.format method of a string, or an f-string.
print( '{:<20}={}'.format(key,value) ) # python 2 friendly
#or
print( f'{key:<20}={value}' )
20 is a guess at the width, I haven't counted it.

Python's built-in str class has an ljust method which performs left text justification given a width and an optional character to fill with.
for section in system_config.config.sections():
print ("\n[" + section + "]")
max_len = max(len(key) for key, _ in system_config.config.items(section))
for key, value in system_config.config.items(section):
print(f'{key.ljust(max_len)} = {value}')

Convert string with alternating key value pairs to a dictionary

I have that weird string (single line) where first field is a key, second is a value. It looks like this:
key1\val1\key2\val2\key3\val3\...\keyn\valn
What would be the best way to convert such notation to python dictionary?

Just use a temporary list to split your string to:
s = 'key1\\val1\\key2\\val2\\key3\\val3'
temp = s.split('\\')
d = {k: v for k, v in zip(temp[0::2], temp[1::2])}

Simple answer.
a = "key1\\val1\\key2\\val2\\key3\\val3"
b = a.split('\\')
dc = {}
for i in range(0,len(b), 2):
dc[b[i]]=b[i+1]

Here is what I came up with:
import re
string = 'key1\\val1\\key2\\val2\\key3\\val3'
dictionary = {match.group(1): match.group(2) for match in re.finditer(r'(\w+)\\(\w+)', string)}
print dictionary
However, note that this would work only if the key values are only characters (no space or underscores and stuff like that). In order to accomodate to such different cases, you would have to modify the simple regex I am using in the above code.

This does it without any imports:
s = """key1\\val1\\key2\\val2\\key3\\val3\\...\\keyn\\valn"""
spl = s.split("\\")
m = {}
for i in range(0, len(spl)-1, 2):
m[spl[i]] = spl[i+1]
print(m)

use split and itertools.islice
import itertools
def parse(ss):
inp = ss.split('\\')
keys, vals = itertools.tee(inp)
keys = itertools.islice(keys,0,None,2)
vals = itertools.islice(vals,1,None,2)
nd = {}
for key,val in zip(keys,vals):
nd[key] = val
return nd

In Python 2.7.12:
line = "key1\\val1\\key2\\val2\\key3\\val3"
line_data = line.split("\\")
line_dict = {}
print line_data
for i in range(0, len(line_data), 2):
key = line_data[i]
value = line_data[i+1]
line_dict[key] = value
print line_dict

Split list based on first character - Python

I am new to Python and can't quite figure out a solution to my Problem. I would like to split a list into two lists, based on what the list item starts with. My list looks like this, each line represents an item (yes this is not the correct list notation, but for a better overview i'll leave it like this) :
***
**
.param
+foo = bar
+foofoo = barbar
+foofoofoo = barbarbar
.model
+spam = eggs
+spamspam = eggseggs
+spamspamspam = eggseggseggs
So I want a list that contains all lines starting with a '+' between .param and .model and another list that contains all lines starting with a '+' after model until the end.
I have looked at enumerate() and split(), but since I have a list and not a string and am not trying to match whole items in the list, I'm not sure how to implement them.
What I have is this:
paramList = []
for line in newContent:
while line.startswith('+'):
paramList.append(line)
if line.startswith('.'):
break
This is just my try to create the first list. The Problem is, the code reads the second block of '+'s as well because break just Exits the while Loop, not the for Loop.
I hope you can understand my question and thanks in advance for any pointers!

What you want is really a simple task that can be accomplish using list slices and list comprehension:
data = ['**','***','.param','+foo = bar','+foofoo = barbar','+foofoofoo = barbarbar',
'.model','+spam = eggs','+spamspam = eggseggs','+spamspamspam = eggseggseggs']
# First get the interesting positions.
param_tag_pos = data.index('.param')
model_tag_pos = data.index('.model')
# Get all elements between tags.
params = [param for param in data[param_tag_pos + 1: model_tag_pos] if param.startswith('+')]
models = [model for model in data[model_tag_pos + 1: -1] if model.startswith('+')]
print(params)
print(models)
Output
>>> ['+foo = bar', '+foofoo = barbar', '+foofoofoo = barbarbar']
>>> ['+spam = eggs', '+spamspam = eggseggs']
Answer to comment:
Suppose you have a list containing numbers from 0 up to 5.
l = [0, 1, 2, 3, 4, 5]
Then using list slices you can select a subset of l:
another = l[2:5] # another is [2, 3, 4]
That what we are doing here:
data[param_tag_pos + 1: model_tag_pos]
And for your last question: ...how does python know param are the lines in data it should iterate over and what exactly does the first paramin param for paramdo?
Python doesn't know, You have to tell him.
First param is a variable name I'm using here, it cuold be x, list_items, whatever you want.
and I will translate the line of code to plain english for you:
# Pythonian
params = [param for param in data[param_tag_pos + 1: model_tag_pos] if param.startswith('+')]
# English
params is a list of "things", for each "thing" we can see in the list `data`
from position `param_tag_pos + 1` to position `model_tag_pos`, just if that "thing" starts with the character '+'.

data = {}
for line in newContent:
if line.startswith('.'):
cur_dict = {}
data[line[1:]] = cur_dict
elif line.startswith('+'):
key, value = line[1:].split(' = ', 1)
cur_dict[key] = value
This creates a dict of dicts:
{'model': {'spam': 'eggs',
'spamspam': 'eggseggs',
'spamspamspam': 'eggseggseggs'},
'param': {'foo': 'bar',
'foofoo': 'barbar',
'foofoofoo': 'barbarbar'}}

I am new to Python
Whoops. Don't bother with my answer then.
I want a list that contains all lines starting with a '+' between
.param and .model and another list that contains all lines starting
with a '+' after model until the end.
import itertools as it
import pprint
data = [
'***',
'**',
'.param',
'+foo = bar',
'+foofoo = barbar',
'+foofoofoo = barbarbar',
'.model',
'+spam = eggs',
'+spamspam = eggseggs',
'+spamspamspam = eggseggseggs',
]
results = [
list(group) for key, group in it.groupby(data, lambda s: s.startswith('+'))
if key
]
pprint.pprint(results)
print '-' * 20
print results[0]
print '-' * 20
pprint.pprint(results[1])
--output:--
[['+foo = bar', '+foofoo = barbar', '+foofoofoo = barbarbar'],
['+spam = eggs', '+spamspam = eggseggs', '+spamspamspam = eggseggseggs']]
--------------------
['+foo = bar', '+foofoo = barbar', '+foofoofoo = barbarbar']
--------------------
['+spam = eggs', '+spamspam = eggseggs', '+spamspamspam = eggseggseggs']
This thing here:
it.groupby(data, lambda x: x.startswith('+')
...tells python to create groups from the strings according to their first character. If the first character is a '+', then the string gets put into a True group. If the first character is not a '+', then the string gets put into a False group. However, there are more than two groups because consecutive False strings will form a group, and consecutive True strings will form a group.
Based on your data, the first three strings:
***
**
.param
will create one False group. Then, the next strings:
+foo = bar
+foofoo = barbar
+foofoofoo = barbarbar
will create one True group. Then the next string:
'.model'
will create another False group. Then the next strings:
'+spam = eggs'
'+spamspam = eggseggs'
'+spamspamspam = eggseggseggs'
will create another True group. The result will be something like:
{
False: [strs here],
True: [strs here],
False: [strs here],
True: [strs here]
}
Then it's just a matter of picking out each True group: if key, and then converting the corresponding group to a list: list(group).
Response to comment:
where exactly does python go through data, like how does it know s is
the data it's iterating over?
groupby() works like do_stuff() below:
def do_stuff(items, func):
for item in items:
print func(item)
#Create the arguments for do_stuff():
data = [1, 2, 3]
def my_func(x):
return x + 100
#Call do_stuff() with the proper argument types:
do_stuff(data, my_func) #Just like when calling groupby(), you provide some data
#and a function that you want applied to each item in data
--output:--
101
102
103
Which can also be written like this:
do_stuff(data, lambda x: x + 100)
lambda creates an anonymous function, which is convenient for simple functions which you don't need to refer to by name.
This list comprehension:
[
list(group)
for key, group in it.groupby(data, lambda s: s.startswith('+'))
if key
]
is equivalent to this:
results = []
for key, group in it.groupby(data, lambda s: s.startswith('+') ):
if key:
results.append(list(group))
It's clearer to explicitly write a for loop, however list comprehensions execute much faster. Here is some detail:
[
list(group) #The item you want to be in the results list for the current iteration of the loop here:
for key, group in it.groupby(data, lambda s: s.startswith('+')) #A for loop
if key #Only include the item for the current loop iteration in the results list if key is True
]

I would suggest doing things step by step.
1) Grab every word from the array separately.
2) Grab the first letter of the word.
3) Look if that is a '+' or '.'
Example code:
import re
class Dark():
def __init__(self):
# Array
x = ['+Hello', '.World', '+Hobbits', '+Dwarves', '.Orcs']
xPlus = []
xDot = []
# Values
i = 0
# Look through every word in the array one by one.
while (i != len(x)):
# Grab every word (s), and convert to string (y).
s = x[i:i+1]
y = '\n'.join(s)
# Print word
print(y)
# Grab the first letter.
letter = y[:1]
if (letter == '+'):
xPlus.append(y)
elif (letter == '.'):
xDot.append(y)
else:
pass
# Add +1
i = i + 1
# Print lists
print(xPlus)
print(xDot)
#Run class
Dark()

Python: Concatenate similiar objects in List

I have a list containing strings as ['Country-Points'].
For example:
lst = ['Albania-10', 'Albania-5', 'Andorra-0', 'Andorra-4', 'Andorra-8', ...other countries...]
I want to calculate the average for each country without creating a new list. So the output would be (in the case above):
lst = ['Albania-7.5', 'Andorra-4.25', ...other countries...]
Would realy appreciate if anyone can help me with this.
EDIT:
this is what I've got so far. So, "data" is actually a dictionary, where the keys are countries and the values are list of other countries points' to this country (the one as Key). Again, I'm new at Python so I don't realy know all the built-in functions.
for key in self.data:
lst = []
index = 0
score = 0
cnt = 0
s = str(self.data[key][0]).split("-")[0]
for i in range(len(self.data[key])):
if s in self.data[key][i]:
a = str(self.data[key][i]).split("-")
score += int(float(a[1]))
cnt+=1
index+=1
if i+1 != len(self.data[key]) and not s in self.data[key][i+1]:
lst.append(s + "-" + str(float(score/cnt)))
s = str(self.data[key][index]).split("-")[0]
score = 0
self.data[key] = lst

itertools.groupby with a suitable key function can help:
import itertools
def get_country_name(item):
return item.split('-', 1)[0]
def get_country_value(item):
return float(item.split('-', 1)[1])
def country_avg_grouper(lst) :
for ctry, group in itertools.groupby(lst, key=get_country_name):
values = list(get_country_value(c) for c in group)
avg = sum(values)/len(values)
yield '{country}-{avg}'.format(country=ctry, avg=avg)
lst[:] = country_avg_grouper(lst)
The key here is that I wrote a function to do the change out of place and then I can easily make the substitution happen in place by using slice assignment.

I would probabkly do this with an intermediate dictionary.
def country(s):
return s.split('-')[0]
def value(s):
return float(s.split('-')[1])
def country_average(lst):
country_map = {}|
for point in lst:
c = country(pair)
v = value(pair)
old = country_map.get(c, (0, 0))
country_map[c] = (old[0]+v, old[1]+1)
return ['%s-%f' % (country, sum/count)
for (country, (sum, count)) in country_map.items()]
It tries hard to only traverse the original list only once, at the expense of quite a few tuple allocations.

I want my parser to return a list of strings, but it returns a blank list

I have a parser that reads in a long octet string, and I want it to print out smaller strings based on the parsing details. It reads in a hexstring which is as follows
The string will be in a format like so:
01046574683001000004677265300000000266010000
The format of the interface contained in the hex is like so:
version:length_of_name:name:op_status:priority:reserved_byte
==
01:04:65746830:01:00:00
== (when converted from hex)
01:04:eth0:01:00:00
^ this is 1 segment of the string , represents eth0 (I inserted the : to make it easier to read). At the minute, however, my code returns a blank list, and I don't know why. Can somebody help me please!
def octetChop(long_hexstring, from_ssh_):
startpoint_of_interface_def=0
# As of 14/8/13 , the network operator has not been implemented
network_operator_implemented=False
version_has_been_read = False
position_of_interface=0
chopped_octet_list = []
#This while loop moves through the string of the interface, based on the full length of the container
try:
while startpoint_of_interface_def < len(long_hexstring):
if version_has_been_read == True:
pass
else:
if startpoint_of_interface_def == 0:
startpoint_of_interface_def = startpoint_of_interface_def + 2
version_has_been_read = True
endpoint_of_interface_def = startpoint_of_interface_def+2
length_of_interface_name = long_hexstring[startpoint_of_interface_def:endpoint_of_interface_def]
length_of_interface_name_in_bytes = int(length_of_interface_name) * 2 #multiply by 2 because its calculating bytes
end_of_interface_name_point = endpoint_of_interface_def + length_of_interface_name_in_bytes
hex_name = long_hexstring[endpoint_of_interface_def:end_of_interface_name_point]
text_name = hex_name.decode("hex")
print "the text_name is " + text_name
operational_status_hex = long_hexstring[end_of_interface_name_point:end_of_interface_name_point+2]
startpoint_of_priority = end_of_interface_name_point+2
priority_hex = long_hexstring[startpoint_of_priority:startpoint_of_priority+2]
#Skip the reserved byte
network_operator_length_startpoint = startpoint_of_priority+4
single_interface_string = long_hexstring[startpoint_of_interface_def:startpoint_of_priority+4]
print single_interface_string + " is chopped from the octet string"# - keep for possible debugging
startpoint_of_interface_def = startpoint_of_priority+4
if network_operator_implemented == True:
network_operator_length = long_hexstring[network_operator_length_startpoint:network_operator_length_startpoint+2]
network_operator_length = int(network_operator_length) * 2
network_operator_start_point = network_operator_length_startpoint+2
network_operator_end_point = network_operator_start_point + network_operator_length
network_operator = long_hexstring[network_operator_start_point:network_operator_end_point]
#
single_interface_string = long_hexstring[startpoint_of_interface_def:network_operator_end_point]
#set the next startpoint if there is one
startpoint_of_interface_def = network_operator_end_point+1
else:
self.network_operator = None
print single_interface_string + " is chopped from the octet string"# - keep for possible debugging
#This is where each individual interface is stored, in a list for comparison.
chopped_octet_list.append(single_interface_string)
finally:
return chopped_octet_list

The reason your code is returning a blank list is the following: In this line:
else:
self.network_operator = None
self is not defined so you get a NameError exception. This means that the try jumps directly to the the finally clause without ever executing the part where you:
chopped_octet_list.append(single_interface_string)
As a consequence the list remains empty. In any case the code is overly complicated for such a task, I would follow one of the other answers.

I hope I got you right. You got a hex-string which contains various interface definition. Inside each interface definition the second octet describes the length of the name of the interface.
Lets say the string contains the interfaces eth0 and eth01 and looks like this (length 4 for eth0 and length 5 for eth01):
01046574683001000001056574683031010000
Then you can split it like this:
def splitIt (s):
tokens = []
while s:
length = int (s [2:4], 16) * 2 + 10 #name length * 2 + 10 digits for rest
tokens.append (s [:length] )
s = s [length:]
return tokens
This yields:
['010465746830010000', '01056574683031010000']

To add onto Hyperboreus's answer, here's a simple way to parse the interface strings once you split them:
def parse(s):
version = int(s[:2], 16)
name_len = int(s[2:4], 16)
name_end = 4 + name_len * 2
name = s[4:name_end].decode('hex')
op_status = int(s[name_end:name_end+2], 16)
priority = int(s[name_end+2:name_end+4], 16)
reserved = s[name_end+4:name_end+6]
return version, name_len, name, op_status, priority, reserved
Here's the output:
>>> parse('010465746830010000')
(1, 4, 'eth0', 1, 0, '00')

Check if the following helps. Call parse method below and pass a string stream into it, then iterate to get card infos (hope I got you right :)) parse will return you tuple(s) of the desired info.
>>> def getbytes(hs):
"""Returns a generator of bytes from a hex string"""
return (int(hs[i:i+2],16) for i in range(0,len(hs)-1,2))
>>> def get_single_card_info(g):
"""Fetches a single card info from a byte generator"""
v = g.next()
l = g.next()
name = "".join(chr(x) for x in map(lambda y: y.next(),[g]*l))
return (str(v),name,g.next(),g.next(),g.next())
>>> def parse(hs):
"""Parses a hex string stream and returns a generator of card infos"""
bs = getbytes(hs)
while True:
yield get_single_card_info(bs)
>>> c = 1
>>> for card in parse("01046574683001000001056574683031010000"):
print "Card:{0} -> Version:{1}, Id:{2}, Op_stat:{3}, priority:{4}, reserved:{5} bytes".format(c,*card)
c = c + 1
Card:1 -> Version:1, Id:eth0, Op_stat:1, priority:0, reserved:0 bytes
Card:2 -> Version:1, Id:eth01, Op_stat:1, priority:0, reserved:0 bytes

Pyparsing includes a built-in expression for parsing a counted array of elements, so this would take care of your 'name' field nicely. Here's the whole parser:
from pyparsing import Word,hexnums,countedArray
# read in 2 hex digits, convert to integer at parse time
octet = Word(hexnums,exact=2).setParseAction(lambda t:int(t[0],16))
# read in a counted array of octets, convert to string
nameExpr = countedArray(octet, intExpr=octet)
nameExpr.setParseAction(lambda t: ''.join(map(chr,t[0])))
# define record expression, with named results
recordExpr = (octet('version') + nameExpr('name') + octet('op_status') +
octet('priority') #+ octet('reserved'))
Parsing your sample:
sample = "01046574683001000004677265300000000266010000"
for rec in recordExpr.searchString(sample):
print rec.dump()
Gives:
[1, 'eth0', 1, 0]
- name: eth0
- op_status: 1
- priority: 0
- version: 1
[0, 'gre0', 0, 0]
- name: gre0
- op_status: 0
- priority: 0
- version: 0
[0, 'f\x01', 0, 0]
- name: f
- op_status: 0
- priority: 0
- version: 0
The dump() method shows results names that you can use to access the individually parsed bits, like rec.name or rec.version.
(I commented out the reserved byte, else the second entry wouldn't parse correctly. Also, the third entry contains a name with a \x01 byte.)