Develop Reference

Reshaping a large dictionary

I am working on xbrl document parsing. I got to a point where I have a large dic structured like this....
sample of a dictionary I'm working on
Since it's bit challenging to describe the pattern of what I'm trying to achieve I just put an example of what I'd like it to be...
sample of what I'm trying to achieve
Since I'm fairly new to programing, I'm hustling for days with this. Trying different approaches with loops, list and dic comprehension starting from here...
for k in storage_gaap:
if 'context_ref' in storage_gaap[k]:
for _k in storage_gaap[k]['context_ref']:
storage_gaap[k]['context_ref']={_k}```
storage_gaap being the master dictionary. Sorry for attaching pictures, but it's just much clearer to see the dictionary
I'd really appreciate any and ever help

Here's a solution using zip and dictionary comprehension to do what you're trying to do using toy data in a similar structure.
import itertools
import pprint
# Sample data similar to provided screenshots
data = {
'a': {
'id': 'a',
'vals': ['a1', 'a2', 'a3'],
'val_num': [1, 2, 3]
},
'b': {
'id': 'b',
'vals': ['b1', 'b2', 'b3'],
'val_num': [4, 5, 6]
}
}
# Takes a tuple of keys, and a list of tuples of values, and transforms them into a list of dicts
# i.e ('id', 'val'), [('a', 1), ('b', 2) => [{'id': 'a', 'val': 1}, {'id': 'b', 'val': 2}]
def get_list_of_dict(keys, list_of_tuples):
list_of_dict = [dict(zip(keys, values)) for values in list_of_tuples]
return list_of_dict
def process_dict(key, values):
# Transform the dict with lists of values into a list of dicts
list_of_dicts = get_list_of_dict(('id', 'val', 'val_num'), zip(itertools.repeat(key, len(values['vals'])), values['vals'], values['val_num']))
# Dictionary comprehension to group them based on the 'val' property of each dict
return {d['val']: {k:v for k,v in d.items() if k != 'val'} for d in list_of_dicts}
# Reorganize to put dict under a 'context_values' key
processed = {k: {'context_values': process_dict(k, v)} for k,v in data.items()}
# {'a': {'context_values': {'a1': {'id': 'a', 'val_num': 1},
# 'a2': {'id': 'a', 'val_num': 2},
# 'a3': {'id': 'a', 'val_num': 3}}},
# 'b': {'context_values': {'b1': {'id': 'b', 'val_num': 4},
# 'b2': {'id': 'b', 'val_num': 5},
# 'b3': {'id': 'b', 'val_num': 6}}}}
pprint.pprint(processed)

Ok, Here is the updated solution from my case. Catch for me was the was the zip function since it only iterates over the smallest list passed. Solution was the itertools.cycle method Here is the code:
data = {'us-gaap_WeightedAverageNumberOfDilutedSharesOutstanding': {'context_ref': ['D20210801-20220731',
'D20200801-20210731',
'D20190801-20200731',
'D20210801-20220731',
'D20200801-20210731',
'D20190801-20200731'],
'decimals': ['-5',
'-5',
'-5',
'-5',
'-5',
'-5'],
'id': ['us-gaap:WeightedAverageNumberOfDilutedSharesOutstanding'],
'master_id': ['us-gaap_WeightedAverageNumberOfDilutedSharesOutstanding'],
'unit_ref': ['shares',
'shares',
'shares',
'shares',
'shares',
'shares'],
'value': ['98500000',
'96400000',
'96900000',
'98500000',
'96400000',
'96900000']},
def get_list_of_dict(keys, list_of_tuples):
list_of_dict = [dict(zip(keys, values)) for values in list_of_tuples]
return list_of_dict
def process_dict(k, values):
list_of_dicts = get_list_of_dict(('context_ref', 'decimals', 'id','master_id','unit_ref','value'),
zip((values['context_ref']),values['decimals'],itertools.cycle(values['id']),
itertools.cycle(values['master_id']),values['unit_ref'], values['value']))
return {d['context_ref']: {k:v for k,v in d.items()if k != 'context_ref'} for d in list_of_dicts}
processed = {k: {'context_values': process_dict(k, v)} for k,v in data.items()}
pprint.pprint(processed)

Dictionary values are in "set", how can I manage to remove the sets?

dict = {'a': {'Islamabad'}, 'b' : {'Islamabad'}, 'c': {'Paris'},
'd': {'Bern'}, 'e': {'Moscow'}}
result wanted:
dict = {'a': 'Islamabad', 'b' : 'Islamabad', 'c': 'Paris',
'd': 'Bern', 'e': 'Moscow'}

A simple way to get one element from a set is set.pop().
dct = {'a': {'Islamabad'}, 'b' : {'Islamabad'}, 'c': {'Paris'},
'd': {'Bern'}, 'e': {'Moscow'}}
dct = {k: v.pop() for k, v in dct.items()}
print(dct)
v.pop() modifies the original set v. If this is not desired, use next(iter(v)).
Update: Or use an assignment with a [target_list] or (target_list) (--> comment from #Kelly Bundy)
P.S. don't use dict as a variable name

You can convert set to string using * with str() function ( str(*set) ). This way it will remove brackets from the string
You have to loop over the dict and change each value to string like this
dict1 = {'a': {'Islamabad'}, 'b' : {'Islamabad'}, 'c': {'Paris'}, 'd': {'Bern'}, 'e': {'Moscow'}}
dict1 = {k: str(*v) for k, v in dict1.items()}
output: {'a': 'Islamabad', 'b': 'Islamabad', 'c': 'Paris', 'd': 'Bern', 'e': 'Moscow'}
and also, don't use "dict" to name your dictionaries. "dict" is python built-in word which may lead to error in some scenarios

Extremely poor code, please don't use this in any other cases than your own, but this works with your current example.
import ast
dictt = {'a': {'Islamabad'}, 'b' : {'Islamabad'}, 'c': {'Paris'},
'd': {'Bern'}, 'e': {'Moscow'}}
dictt = str(dictt).replace("{","") #Convert to string and remove {
dictt = "{" + dictt.replace("}","") +"}" #remove } and re-add outer { }
dictt = ast.literal_eval(dictt) #Back to dict
print(dictt)
output:
{'a': 'Islamabad', 'b': 'Islamabad', 'c': 'Paris', 'd': 'Bern', 'e': 'Moscow'}

Convert a list of objects to a dictionary of lists

I have a list of JSON objects, already sorted (by time let's say). Each JSON object has type and status. For example:
[
{'type': 'A', 'status': 'ok'},
{'type': 'B', 'status': 'ok'},
{'type': 'A', 'status': 'fail'}
]
I'd like to convert it to:
{
'A': ['ok', 'fail'],
'B': ['ok']
}
So of course it's an easy task but I'm looking for the Pythonic way doing that, so it will spare me a few lines of code

I don't know if there is a one-liner but you can make use of setdefault or defaultdict to achieve the desired result as:
data = [
{'type': 'A', 'status': 'ok'},
{'type': 'B', 'status': 'ok'},
{'type': 'A', 'status': 'fail'}
]
Using setdefault():
res = {}
for elt in data:
res.setdefault(elt['type'], []).append(elt['status'])
Output:
{'A': ['ok', 'fail'], 'B': ['ok']}
Using defaultdict:
from collections import defaultdict
res = defaultdict(list)
for elt in data:
res[elt['type']].append(elt['status'])
Output:
defaultdict(<class 'list'>, {'A': ['ok', 'fail'], 'B': ['ok']})

Since your new structure depends on older (already processed) keys also, I doubt you can use dictionary comprehension (which operates on per-key basis) to achieve the required final state.
I would prefer to use defaultdict where each default key is a list.
from collections import defaultdict
new_data = defaultdict(list)
for item in data:
new_data[item['type']].append(item['status'])
print(new_data)
which gives output:
defaultdict(<class 'list'>, {'A': ['ok', 'fail'], 'B': ['ok']})

A one-liner with multiple list comprehensions and sorted(set(…)) to remove the duplicate keys:
original = [
{"type": "A", "status": "ok"},
{"type": "B", "status": "ok"},
{"type": "A", "status": "fail"},
]
result = {
key: [x["status"] for x in original if x["type"] == key]
for key in sorted(set([x["type"] for x in original]))
}
Output:
{'A': ['ok', 'fail'], 'B': ['ok']}

How to merge data from multiple dictionaries with repeating keys?

I have two dictionaries:
dict1 = {'a': '2', 'b': '10'}
dict2 = {'a': '25', 'b': '7'}
I need to save all the values for same key in a new dictionary.
The best i can do so far is: defaultdict(<class 'list'>, {'a': ['2', '25'], 'b': ['10', '7']})
dd = defaultdict(list)
for d in (dict1, dict2):
for key, value in d.items():
dd[key].append(value)
print(dd)
that does not fully resolve the problem since a desirable result is:
a = {'dict1':'2', 'dict2':'25'}
b = {'dict2':'10', 'dict2':'7'}
Also i possibly would like to use new dictionary key same as initial dictionary name

Your main problem is that you're trying to cross the implementation boundary between a string value and a variable name. This is almost always bad design. Instead, start with all of your labels as string data:
table = {
"dict1": {'a': '2', 'b': '10'},
"dict2": {'a': '25', 'b': '7'}
}
... or, in terms of your original post:
table = {
"dict1": dict1,
"dict2": dict2
}
From here, you should be able to invert the levels to obtain
invert = {
"a": {'dict1': '2', 'dict2': '25'},
"b": {'dict2': '10', 'dict2': '7'}
}
Is that enough to get your processing where it needs to be? Keeping the data in comprehensive dicts like this, will make it easier to iterate through the sub-dicts as needed.

As #Prune suggested, structuring your result as a nested dictionary will be easier:
{'a': {'dict1': '2', 'dict2': '25'}, 'b': {'dict1': '10', 'dict2': '7'}}
Which could be achieved with a dict comprehension:
{k: {"dict%d" % i: v2 for i, v2 in enumerate(v1, start=1)} for k, v1 in dd.items()}
If you prefer doing it without a comprehension, you could do this instead:
result = {}
for k, v1 in dd.items():
inner_dict = {}
for i, v2 in enumerate(v1, start=1):
inner_dict["dict%d" % i] = v2
result[k] = inner_dict
Note: This assumes you want to always want to keep the "dict1", "dict2",... key structure.

Creating a nested dict out of a list with unknown length

I'm trying to get a method that gets 3 lists and turns them into a nested dict.
The first and the second list can have any amount of entries > 0.
The values list always has len(firstlist) * len(secondlist) entries.
For example:
givenlist1 = ["First", "Second"]
givenlist2 = ["A.B.D", "A.Y.Z", "A.B.E"]
Values = ["10", "2", "3", "4", "1", "3"]
Should return a dict like this:
{'First': {'A': {'B': {'D': '10', 'E': '3'}, 'Y': {'Z': '2'}}},
'Second': {'A': {'B': {'D': '4', 'E': '3'}, 'Y': {'Z': '1'}}}}
I tried a lot with .update but I just can't get an idea how to do it with a variable amount of entries in the second list.

You can use itertools.product to get the required combinations of the entries in givenlist1 and givenlist2, and use zip to associate them with the corresponding items from values. Then you need to .split the individual letter keys from the letters in the givenlist2 items to get the nested keys, creating new dicts as necessary.
from itertools import product
from pprint import pprint
givenlist1 = ["First", "Second"]
givenlist2 = ["A.B.D", "A.Y.Z", "A.B.E"]
values = ["10", "2", "3", "4", "1", "3"]
result = {k1: {} for k1 in givenlist1}
for (k1, k2), v in zip(product(givenlist1, givenlist2), values):
d = result[k1]
keys = k2.split('.')
for k in keys[:-1]:
d = d.setdefault(k, {})
d[keys[-1]] = v
pprint(result)
output
{'First': {'A': {'B': {'D': '10', 'E': '3'}, 'Y': {'Z': '2'}}},
'Second': {'A': {'B': {'D': '4', 'E': '3'}, 'Y': {'Z': '1'}}}}
Here's a less compact but possibly more readable way to write the inner for loop:
for k in keys[:-1]:
if k not in d:
d[k] = {}
d = d[k]