Develop Reference

How is this Python 2.7 setdefault/ defaultdict code giving a nested result?

This seems to be a pretty straightforward use of setdefault and defaultdict which I am not able to understand, it would be great if someone could explain "why" the code below works.
d = {}
for name in ['foo', 'bar', 'bars']:
t = d
for char in name:
t = t.setdefault(char,{}) # Should be a empty {}
print d
# Prints {'b': {'a': {'r': {'s': {}}}}, 'f': {'o': {'o': {}}}}
I am not able to understand how this piece of code works. When the line t = t.setdefault(char,{}) executes it should assign a empty dictionary to t, but how is it affecting d such that d ends up becoming a nested dictionary?
Also, what would be the equivalent of the above if I were to use defaultdict. I came up with this which is wrong:
d1 = defaultdict(dict)
for name in ['foo', 'bar', 'bars']:
t1 = d1
for char in name:
t1 = t1[char]
print d1
It would be great if someone could point out as to how one should understand defaultdicts

I'll walk through the loop one step at a time, and explain how it proceeds to assign the nested dicts:
name = 'foo'
t = d # both t and d point to the same empty dict object
char = 'f'
t = t.setdefault(char,{})
# the first thing evaluated is the right hand side:
# now d['f'] = {}, since that key wasn't in the dict
# t points to the same object here
# now the result of the left side (a new empty dict) is assigned to `t`.
# this empty dict is also the exact *same* object referenced by d['f'] as well though!
# so at this point d['f'] = {}, and t = {}, and both those dicts are the same!
char = 'o'
t = t.setdefault(char,{})
# eval the right side again, so now t['o'] = {}, but remember d['f'] == t
# so really d['f'] = {'o':{}}
# and again we assign the result of the right side to a brand new `t`
# so now d['f']['o'] = {}, and t['o'] = {}, and these empty dicts are
# again referencing the same object
char = 'o'
t = t.setdefault(char,{})
# our `t` from last time is empty, so it gets assigned the same as before
# and now d['f']['o']['o'] = {}
name = 'bar'
t = d # re-start this, but with d['f']['o']['o'] = {}
char = 'b'
#...everything proceeds as before - since 'b' is not in `d`,
# we start generating nested dicts again
# ...
...
name = 'bars'
# main difference here is that d['b']['a']['r'] exists,
# so we end up just adding the 's':{} to the end
As for the defaultdict equivalent, that is a little trickier.
The issue is you need defaultdict's all-the-way-down
I found a way to do that with a little function here
from collections import defaultdict
def fix(f):
return lambda *args, **kwargs: f(fix(f), *args, **kwargs)
d1 = fix(defaultdict)()
for name in ['foo', 'bar', 'bars']:
t1 = d1
for char in name:
t1 = t1[char]
print d1

For the first part, the line t = d does not make a copy of d. It only creates a new reference to d and stores it in t. t and d now refer to the same object; in other words, you have only one object, but two names for that object. Since that object is a mutable object (a dict, in this case), changing t also changes d since there is only one object. While that's necessary here, if for some reason in other code you want to make a copy of a mutable object and operate on the copy without modifying the original, you need to import copy and use copy.deepcopy().
On the second, the defaultdict() constructor expects, as its first argument, a callable that accepts no arguments and returns the default value. However, for this case, that return value would need to be another defaultdict with a callable returning another defaultdict with a callable returning yet another... etc. It's infinite recursion.
So there is no defaultdict equivalent to this code. Instead, the original version with setdefault and plain dicts is probably the best and most Pythonic way to do it.

how setdefault works in dictionary
# case 1
d = {}
temp = d.setdefault("A")
print "d = ", d
print "temp = ", temp
print "id of d = ", id(d), "id of temp = ", id(temp)
# output
d = {'A': None}
temp = None
id of d = 140584110017624, id of temp = 9545840 # memory locations of d, temp
# case 2
d = {}
temp = d.setdefault("A", "default Value")
print "d = ", d
print "temp = ", temp
print "id of d = ", id(d), "id of temp = ", id(temp)
# output
d = {'A': "default Value"}
temp = "default Value"
id of d = 140584110017624, id of temp = 9545840 # memory locations of d, temp
I your code t=d means memory location of t and d both are same.
so, when the code t = t.setdefault(char,{}) excecutes first t.setdefault(char,{}) executes and changes the content in memory location of t, then it returns the content then it assigns the new memory location to name t and assigns the returned value to it.
memory location of t and d are same that's the reason d is getting affected.

Convert strings of assignments to dicts in Python

I have the following strings - they are assignment commands:
lnodenum = 134241
d1 = 0.200000
jobname = 'hcalfzp'
Is there a way to convert this string, containing variables and values, to keys and values of a dictionary? It'd be equivalent to executing the below command:
my_dict = dict(lnodenum = 134241, d1 = 0.200000, jobname = 'hcalfzp')
I guess this is one way to do it:
my_str = """lnodenum = 134241
d1 = 0.200000
jobname = 'hcalfzp' """
exec('my_dict = dict(%s)' % ','.join(my_str.split('\n')))
Not sure whether anyone would think of a more concise way?
Note: I am using my own code for scientific computing, so I don't mind having code with safety concerns like malicious input data. But, I do prefer to have code that is shorter and easier to read :)

There's no need for exec. Just use a regular assignment with the dict function.
my_str = """lnodenum = 134241
d1 = 0.200000
jobname = 'hcalfzp' """
my_dict = dict(pair for pair in (line.strip().split(' = ') for line in my_str.splitlines()))
Result:
>>> my_dict
{'d1': '0.200000', 'lnodenum': '134241', 'jobname': "'hcalfzp'"}
Or, if you'd like to parse each object, use ast.literal_eval with a comprehension:
import ast
my_dict = {k:ast.literal_eval(v) for k,v in (line.strip().split(' = ') for line in my_str.splitlines())}
Result:
>>> my_dict
{'d1': 0.2, 'lnodenum': 134241, 'jobname': 'hcalfzp'}

Dividing dictionary into nested dictionaries, based on the key's name on Python 3.4

I have the following dictionary (short version, real data is much larger):
dict = {'C-STD-B&M-SUM:-1': 0, 'C-STD-B&M-SUM:-10': 4.520475, 'H-NSW-BAC-ART:-9': 0.33784000000000003, 'H-NSW-BAC-ART:0': 0, 'H-NSW-BAC-ENG:-59': 0.020309999999999998, 'H-NSW-BAC-ENG:-6': 0,}
I want to divide it into smaller nested dictionaries, depending on a part of the key name.
Expected output would be:
# fixed closing brackets
dict1 = {'C-STD-B&M-SUM: {'-1': 0, '-10': 4.520475}}
dict2 = {'H-NSW-BAC-ART: {'-9': 0.33784000000000003, '0': 0}}
dict3 = {'H-NSW-BAC-ENG: {'-59': 0.020309999999999998, '-6': 0}}
Logic behind is:
dict1: if the part of the key name is 'C-STD-B&M-SUM', add to dict1.
dict2: if the part of the key name is 'H-NSW-BAC-ART', add to dict2.
dict3: if the part of the key name is 'H-NSW-BAC-ENG', add to dict3.
Partial code so far:
def divide_dictionaries(dict):
c_std_bem_sum = {}
for k, v in dict.items():
if k[0:13] == 'C-STD-B&M-SUM':
c_std_bem_sum = k[14:17], v
What I'm trying to do is to create the nested dictionaries that I need and then I'll create the dictionary and add the nested one to it, but I'm not sure if it's a good way to do it.
When I run the code above, the variable c_std_bem_sum becomes a tuple, with only two values that are changed at each iteration. How can I make it be a dictionary, so I can later create another dictionary, and use this one as the value for one of the keys?

One way to approach it would be to do something like
d = {'C-STD-B&M-SUM:-1': 0, 'C-STD-B&M-SUM:-10': 4.520475, 'H-NSW-BAC-ART:-9': 0.33784000000000003, 'H-NSW-BAC-ART:0': 0, 'H-NSW-BAC-ENG:-59': 0.020309999999999998, 'H-NSW-BAC-ENG:-6': 0,}
def divide_dictionaries(somedict):
out = {}
for k,v in somedict.items():
head, tail = k.split(":")
subdict = out.setdefault(head, {})
subdict[tail] = v
return out
which gives
>>> dnew = divide_dictionaries(d)
>>> import pprint
>>> pprint.pprint(dnew)
{'C-STD-B&M-SUM': {'-1': 0, '-10': 4.520475},
'H-NSW-BAC-ART': {'-9': 0.33784000000000003, '0': 0},
'H-NSW-BAC-ENG': {'-59': 0.020309999999999998, '-6': 0}}
A few notes:
(1) We're using nested dictionaries instead of creating separate named dictionaries, which aren't convenient.
(2) We used setdefault, which is a handy way to say "give me the value in the dictionary, but if there isn't one, add this to the dictionary and return it instead.". Saves an if.
(3) We can use .split(":") instead of hardcoding the width, which isn't very robust -- at least assuming that's the delimiter, anyway!
(4) It's a bad idea to use dict, the name of a builtin type, as a variable name.

That's because you're setting your dictionary and overriding it with a tuple:
>>> a = 1, 2
>>> print a
>>> (1,2)
Now for your example:
>>> def divide_dictionaries(dict):
>>> c_std_bem_sum = {}
>>> for k, v in dict.items():
>>> if k[0:13] == 'C-STD-B&M-SUM':
>>> new_key = k[14:17] # sure you don't want [14:], open ended?
>>> c_std_bem_sum[new_key] = v
Basically, this grabs the rest of the key (or 3 characters, as you have it, the [14:None] or [14:] would get the rest of the string) and then uses that as the new key for the dict.

Change nesting order of triple-nested Python dictionary

I have dictionary with three-level nesting, eg:
d = {
'sp1':{
'a1':{'c1':2,'c2':3},
'a2':{'c3':1,'c4':4}
},
'sp2':{
'a1':{'c1':3,'c2':3},
'a2':{'c3':2,'c4':0}
}
}
All 2nd-level dictionaries contain the same elements, so I want to change it to
d2 = {'a1':{'c1':{'sp1':2,'sp2':3}, 'c2':{'sp1':3,'sp2':3}}}
i.e. essentially switch nesting order. But when I write code like
d2 = {}
d2['a1']['c1']['sp1'] = 2
It just throws KeyError with whatever values happens to be 'a1'. How do I perform such operation?

If you are doing it manually like the snippet you tried, this is how you should be doing it:
>>> d = {
... 'sp1':{
... 'a1':{'c1':2,'c2':3},
... 'a2':{'c3':1,'c4':4}
... },
... 'sp2':{
... 'a1':{'c1':3,'c2':3},
... 'a2':{'c3':2,'c4':0}
... }
... }
>>>
>>> e = {}
>>> e['a1'] = {}
>>> e['a1']['c1'] = {}
>>> e['a1']['c1']['sp1'] = d['sp1']['a1']['c1']
>>> e['a1']['c2'] = {}
>>> e['a1']['c2']['sp1'] = d['sp1']['a1']['c2']
>>> e['a2'] = {}
>>> e['a2']['c1'] = {}
>>> e['a2']['c2'] = {}
>>> e['a1']['c1']['sp2'] = d['sp2']['a1']['c1']
>>> e['a1']['c2']['sp2'] = d['sp2']['a1']['c2']
>>> e
{'a1': {'c2': {'sp1': 3, 'sp2': 3}, 'c1': {'sp1': 2, 'sp2': 3}}}
>>>
But it is unclear as to why you are doing it. As OmnipotentEntity suggested in the comments, may be you need to use a different data structure to store the data.

To do this, you can use defaultdict, which allows you to define a default initialization action on a dict.
In your case, you want a reverse-order recursive defaultdict, with a classmethod
reverse_recursive_make() which unrolls and reverses the key order:
when passed in a key-value pair or None, returns a (toplevel) dict
when passed in a dict, recurses into each of the {k:v} pairs
I'm not going to write the code for that because what you want can be much more easily achieved with SQL, like I commented.
FOOTNOTE: your version with lambdas (comment below) is perfect.
(If you insist on using dicts, and not some other data structure)

something like this should work
d_final = {}
for k in d.keys():
d2 = d[k]
for k2 in d2.keys():
d3 = d2[k2]
for k3 in d3.keys():
d4 = d_final.get(k2,{})
d4[k] = d3[k3]
d_final[k2] = d4
I may have my indexing off a little, but that should be about right.

Dictionary with named values?

I have a file like this :
A X V1
A Y V2
B X V3
B Y V4
Let's say the first column is a model type, second column is a version number and third is the value of something related.
I would like to answer the question : "What is the value of model A, version X ?"
For all values and all versions.
I wanted to use a dict but i only know dicts with one value for each keys. This one here needs two keys, ie something like :
d[model][version] = value
How would you do this ?

You can nest dictionaries:
d['A'] = {}
d['A']['X'] = 'V1'
or you can use tuple keys instead:
d[('A', 'X')] = 'V1'
Nesting would make it easier to list all known versions for a given model:
versions_for_model = d['A'].keys()
Creating a nested dictionary setup can be simplified a little by using collections.defaultdict():
d = defaultdict(dict)
d['A']['X'] = 'V1'
Here trying to access d['A'] automatically creates a new dictionary value.

with open("Input.txt") as inputFile:
lines = [line.strip().split() for line in inputFile]
result = {}
for k1, k2, v in lines:
result.setdefault(k1, {})[k2] = v
print result
Output
{'A': {'Y': 'V2', 'X': 'V1'}, 'B': {'Y': 'V4', 'X': 'V3'}}
You can access the individual elements like this
print result["A"]["Y"]
Output
V2