Related
I am trying to access a specific key in a nest dictionary, then match its value to a string in a list. If the string in the list contains the string in the dictionary value, I want to override the dictionary value with the list value. below is an example.
my_list = ['string1~', 'string2~', 'string3~', 'string4~', 'string5~', 'string6~']
my_iterable = {'A':'xyz',
'B':'string6',
'C':[{'B':'string4', 'D':'123'}],
'E':[{'F':'321', 'B':'string1'}],
'G':'jkl'
}
The key I'm looking for is B, the objective is to override string6 with string6~, string4 with string4~, and so on for all B keys found in the my_iterable.
I have written a function to compute the Levenshtein distance between two strings, but I am struggling to write an efficient ways to override the values of the keys.
def find_and_replace(key, dictionary, original_list):
for k, v in dictionary.items():
if k == key:
#function to check if original_list item contains v
yield v
elif isinstance(v, dict):
for result in find_and_replace(key, v, name_list):
yield result
elif isinstance(v, list):
for d in v:
if isinstance(d, dict):
for result in find_and_replace(key, d, name_list):
yield result
if I call
updated_dict = find_and_replace('B', my_iterable, my_list)
I want updated_dict to return the below:
{'A':'xyz',
'B':'string6~',
'C':[{'B':'string4~', 'D':'123'}],
'E':[{'F':'321', 'B':'string1~'}],
'G':'jkl'
}
Is this the right approach to the most efficient solution, and how can I modify it to return a dictionary with the updated values for B?
You can use below code. I have assumed the structure of input dict to be same throughout the execution.
# Input List
my_list = ['string1~', 'string2~', 'string3~', 'string4~', 'string5~', 'string6~']
# Input Dict
# Removed duplicate key "B" from the dict
my_iterable = {'A':'xyz',
'B':'string6',
'C':[{'B':'string4', 'D':'123'}],
'E':[{'F':'321', 'B':'string1'}],
'G':'jkl',
}
# setting search key
search_key = "B"
# Main code
for i, v in my_iterable.items():
if i == search_key:
if not isinstance(v,list):
search_in_list = [i for i in my_list if v in i]
if search_in_list:
my_iterable[i] = search_in_list[0]
else:
try:
for j, k in v[0].items():
if j == search_key:
search_in_list = [l for l in my_list if k in l]
if search_in_list:
v[0][j] = search_in_list[0]
except:
continue
# print output
print (my_iterable)
# Result -> {'A': 'xyz', 'B': 'string6~', 'C': [{'B': 'string4~', 'D': '123'}], 'E': [{'F': '321', 'B': 'string1~'}], 'G': 'jkl'}
Above can has scope of optimization using list comprehension or using
a function
I hope this helps and counts!
In some cases, if your nesting is kind of complex you can treat the dictionary like a json string and do all sorts of replacements. Its probably not what people would call very pythonic, but gives you a little more flexibility.
import re, json
my_list = ['string1~', 'string2~', 'string3~', 'string4~', 'string5~', 'string6~']
my_iterable = {'A':'xyz',
'B':'string6',
'C':[{'B':'string4', 'D':'123'}],
'E':[{'F':'321', 'B':'string1'}],
'G':'jkl'}
json_str = json.dumps(my_iterable, ensure_ascii=False)
for val in my_list:
json_str = re.sub(re.compile(f"""("[B]":\\W?")({val[:-1]})(")"""), r"\1" + val + r"\3", json_str)
my_iterable = json.loads(json_str)
print(my_iterable)
I created two dictionaries. Each is based on a different query of the same database. There is a key and four fields from the database in each dictionary. I want to find all rows of dict_x that are in are in dict_y.
for row in dict_x:
if dict_y.values() not in dict_x.values():
del dict_x[row]
print 'Length dict_x', len(dict_x)
This returns the error
TypeError: 'type' object does not support item deletion
This will work as long as the elements in the array will always be in the same order.
dict_x = {'hi': ['hello', 'hi'], 'bye': ['good bye', 'bye']}
dict_y = {'hi': ['hello', 'hi']}
dict_z = dict()
for key, row in dict_x.items():
if row in dict_y.values():
dict_z[key] = row
print(dict_z)
If the elements won't be in the same order then you'll have to do this:
dict_x = {'hi': ['hi', 'hello'], 'bye': ['good bye', 'bye']}
dict_y = {'hi': ['hello', 'hi']}
dict_z = dict()
for x_key, x_row in dict_x.items():
for y_key, y_row in dict_y.items():
if set(x_row).intersection(y_row):
dict_z[x_key] = y_row
print(dict_z)
>>> dict_a = {'a':[1,1,2,3]}
>>> dict_b = {'b':[1,2]}
>>> for a_key, b_key in zip(dict_a.keys(), dict_b.keys()):
... print [i for i in dict_a[a_key] if i in set(dict_b[b_key])]
...
[1, 1, 2]
The steps to solve the problem would be to
Invert the key value pairs of the dictionaries
Identify the common intersecting keys
Loop through the keys and check if their values match
The code could look something like below
dict_x = {v: k for k, v in dict_x.items()}
dict_y = {v: k for k, v in dict_y.items()}
for key in dict_x.keys() & dict_y.keys():
print(key, dict_x[key])
print(key, dict_y[key])
Here is the dict comprehension equivalent in python 3
result_set = {key: dict_x[key] for key in dict_x.keys() & dict_y.keys() if dict_x[key] == dict_y[key]}
This might help, it will return False if the dict is equal and true if not. I know its other way around
def compare_dict(
dict_1: Dict[Any, Any], dict_2: Dict[Any, Any]
):
new_key = any([False if key in dict_1 else True for key in dict_2])
delete_key = any(
[False if key in dict_2 else True for key in dict_1]
)
if new_key or delete_key:
return True
else:
values_mismatch_flag = any(
[
True if v != dict_1[k] else False
for k, v in dict_2.items()
]
)
if values_mismatch_flag:
return True
return False
I am receiving data in batches from an API in JSON format. I wish to store only the values, in a list.
The raw data looks like this and will always look like this, i.e: all {...} will look like the first example:
data = content.get('data')
>>> [{'a':1, 'b':{'c':2, 'd':3}, 'e':4}, {...}, {...}, ...]
The nested dictionary is making this harder; I need this unpacked as well.
Here is what I have, which works but it feels so bad:
unpacked = []
data = content.get('data')
for d in data:
item = []
for k, v in d.items():
if k == 'b':
for val in v.values():
item.append(val)
else:
item.append(v)
unpacked.append(item)
Output:
>>> [[1,2,3,4], [...], [...], ...]
How can I improve this?
You could use a recursive function and some type tests:
data = [{'a':1, 'b':{'c':2, 'd':3}, 'e':4}, {'f':5,'g':6}]
def extract_nested_values(it):
if isinstance(it, list):
for sub_it in it:
yield from extract_nested_values(sub_it)
elif isinstance(it, dict):
for value in it.values():
yield from extract_nested_values(value)
else:
yield it
print(list(extract_nested_values(data)))
# [1, 2, 3, 4, 5, 6]
Note that it outputs a flat generator, not a list of lists.
Assuming your dictionaries do not contain inner lists, you could define a simple routine to unpack a nested dictionary, and iterate through each item in data using a loop.
def unpack(data):
for k, v in data.items():
if isinstance(v, dict):
yield from unpack(v)
else:
yield v
Note that this function is as simple as it is thanks to the magic of yield from. Now, let's call it with some data.
data = [{'a':1, 'b':{'c':2, 'd':3}, 'e':4}, {'f':5,'g':6}] # Data "borrowed" from Kaushik NP
result = [list(unpack(x)) for x in data]
print(result)
[[2, 3, 1, 4], [5, 6]]
Note the lack of order in your result, because of the arbitrary order of dictionaries.
For completeness, based on the excellent answer of Eric Duminil, here is a function that returns the maximum depth of a nested dict or list:
def depth(it, count=0):
"""Depth of a nested dict.
# Arguments
it: a nested dict or list.
count: a constant value used in internal calculations.
# Returns
Numeric value.
"""
if isinstance(it, list):
if any(isinstance(v, list) or isinstance(v, dict) for v in it):
for v in it:
if isinstance(v, list) or isinstance(v, dict):
return depth(v, count + 1)
else:
return count
elif isinstance(it, dict):
if any(isinstance(v, list) or isinstance(v, dict) for v in it.values()):
for v in it.values():
if isinstance(v, list) or isinstance(v, dict):
return depth(v, count + 1)
else:
return count
else:
return count
In the Python tradition, it is zero-based.
Other answers (especially #COLDSPEED's) have already covered the situation, but here is a slightly different code based on the old adage it's better to ask forgiveness than permission , which I tend to prefer to type checking:
def unpack(data):
try:
for value in data.values():
yield from unpack(value)
except AttributeError:
yield data
data = [{'a':1, 'b':{'c':2, 'd':3}, 'e':4}]
unpacked = [list(unpack(item)) for item in data]
Doing recursively :
def traverse(d):
for key,val in d.items():
if isinstance(val, dict):
traverse(val)
else:
l.append(val)
out=[]
for d in data:
l=[]
traverse(d)
out.append(l)
print(out)
#driver values :
IN : data = [{'a':1, 'b':{'c':2, 'd':3}, 'e':4}, {'f':5,'g':6}]
OUT : out = [[1, 2, 3, 4], [5, 6]]
EDIT : A better way to do this is using yield so as not to have to rely on global variables as in the first method.
def traverse(d):
for key,val in d.items():
if isinstance(val, dict):
yield from traverse(val)
else:
yield val
out = [list(traverse(d)) for d in data]
I need to write a function which is doing following work
Find a string as value in a dictionary of dictionaries and return its key
(1st key if found in main dictionary, 2nd key if found in sub dictionary).
Source Code
Here is the function which I try to implement, but it works incorrect as I can't find any answer of how to convert list into dictionary as in this case the following error occurs
for v, k in l:
ValueError: need more than 1 value to unpack
def GetKeyFromDictByValue(self, dictionary, value_to_find):
""""""
key_list = [k for (k, v) in dictionary.items() if v == value_to_find]
if key_list.__len__() is not 0:
return key_list[0]
else:
l = [s for s in dictionary.values() if ":" in str(s)]
d = defaultdict(list)
for v, k in l:
d[k].append(v)
print d
dict = {'a': {'a1': 'a2'}, "aa": "aa1", 'aaa': {'aaa1': 'aaa2'}}
print GetKeyFromDictByValue(dict, "a2")
I must do this on Python 2.5
You created a list of only the dictionary values, but then try to loop over it as if it already contains both keys and values of those dictionaries. Perhaps you wanted to loop over each matched dictionary?
l = [v for v in dictionary.values() if ":" in str(v)]
d = defaultdict(list)
for subdict in l:
for k, v in subdict.items():
I'd instead flatten the structure:
def flatten(dictionary):
for key, value in dictionary.iteritems():
if isinstance(value, dict):
# recurse
for res in flatten(value):
yield res
else:
yield key, value
then just search:
def GetKeyFromDictByValue(self, dictionary, value_to_find):
for key, value in flatten(dictionary):
if value == value_to_find:
return key
Demo:
>>> sample = {'a': {'a1': 'a2'}, "aa": "aa1", 'aaa': {'aaa1': 'aaa2'}}
>>> GetKeyFromDictByValue(None, sample, "a2")
'a1'
Let's say I have a pretty complex dictionary.
{'fruit':'orange','colors':{'dark':4,'light':5}}
Anyway, my objective is to scan every key in this complex multi-level dictionary. Then, append "abc" to the end of each key.
So that it will be:
{'fruitabc':'orange','colorsabc':{'darkabc':4,'lightabc':5}}
How would you do that?
Keys cannot be changed. You will need to add a new key with the modified value then remove the old one, or create a new dict with a dict comprehension or the like.
For example like this:
def appendabc(somedict):
return dict(map(lambda (key, value): (str(key)+"abc", value), somedict.items()))
def transform(multilevelDict):
new = appendabc(multilevelDict)
for key, value in new.items():
if isinstance(value, dict):
new[key] = transform(value)
return new
print transform({1:2, "bam":4, 33:{3:4, 5:7}})
This will append "abc" to each key in the dictionary and any value that is a dictionary.
EDIT: There's also a really cool Python 3 version, check it out:
def transform(multilevelDict):
return {str(key)+"abc" : (transform(value) if isinstance(value, dict) else value) for key, value in multilevelDict.items()}
print(transform({1:2, "bam":4, 33:{3:4, 5:7}}))
I use the following utility function that I wrote that takes a target dict and another dict containing the translation and switches all the keys according to it:
def rename_keys(d, keys):
return dict([(keys.get(k), v) for k, v in d.items()])
So with the initial data:
data = { 'a' : 1, 'b' : 2, 'c' : 3 }
translation = { 'a' : 'aaa', 'b' : 'bbb', 'c' : 'ccc' }
We get the following:
>>> data
{'a': 1, 'c': 3, 'b': 2}
>>> rename_keys(data, translation)
{'aaa': 1, 'bbb': 2, 'ccc': 3}
>>> mydict={'fruit':'orange','colors':{'dark':4,'light':5}}
>>> def f(mydict):
... return dict((k+"abc",f(v) if hasattr(v,'keys') else v) for k,v in mydict.items())
...
>>> f(mydict)
{'fruitabc': 'orange', 'colorsabc': {'darkabc': 4, 'lightabc': 5}}
My understanding is that you can't change the keys, and that you would need to make a new set of keys and assign their values to the ones the original keys were pointing to.
I'd do something like:
def change_keys(d):
if type(d) is dict:
return dict([(k+'abc', change_keys(v)) for k, v in d.items()])
else:
return d
new_dict = change_keys(old_dict)
here's a tight little function:
def keys_swap(orig_key, new_key, d):
d[new_key] = d.pop(orig_key)
for your particular problem:
def append_to_dict_keys(appendage, d):
#note that you need to iterate through the fixed list of keys, because
#otherwise we will be iterating through a never ending key list!
for each in d.keys():
if type(d[each]) is dict:
append_to_dict_keys(appendage, d[each])
keys_swap(each, str(each) + appendage, d)
append_to_dict_keys('abc', d)
#! /usr/bin/env python
d = {'fruit':'orange', 'colors':{'dark':4,'light':5}}
def add_abc(d):
newd = dict()
for k,v in d.iteritems():
if isinstance(v, dict):
v = add_abc(v)
newd[k + "abc"] = v
return newd
d = add_abc(d)
print d
Something like that
def applytoallkeys( dic, func ):
def yielder():
for k,v in dic.iteritems():
if isinstance( v, dict):
yield func(k), applytoallkeys( v, func )
else:
yield func(k), v
return dict(yielder())
def appendword( s ):
def appender( x ):
return x+s
return appender
d = {'fruit':'orange','colors':{'dark':4,'light':5}}
print applytoallkeys( d, appendword('asd') )
I kinda like functional style, you can read just the last line and see what it does ;-)
You could do this with recursion:
import collections
in_dict={'fruit':'orange','colors':{'dark':4,'light':5}}
def transform_dict(d):
out_dict={}
for k,v in d.iteritems():
k=k+'abc'
if isinstance(v,collections.MutableMapping):
v=transform_dict(v)
out_dict[k]=v
return out_dict
out_dict=transform_dict(in_dict)
print(out_dict)
# {'fruitabc': 'orange', 'colorsabc': {'darkabc': 4, 'lightabc': 5}}
you should also consider that there is the possibility of nested dicts in nested lists, which will not be covered by the above solutions. This function ads a prefix and/or a postfix to every key within the dict.
def transformDict(multilevelDict, prefix="", postfix=""):
"""adds a prefix and/or postfix to every key name in a dict"""
new_dict = multilevelDict
if prefix != "" or postfix != "":
new_key = "%s#key#%s" % (prefix, postfix)
new_dict = dict(map(lambda (key, value): (new_key.replace('#key#', str(key)), value), new_dict.items()))
for key, value in new_dict.items():
if isinstance(value, dict):
new_dict[key] = transformDict(value, prefix, postfix)
elif isinstance(value, list):
for index, item in enumerate(value):
if isinstance(item, dict):
new_dict[key][index] = transformDict(item, prefix, postfix)
return new_dict
for k in theDict: theDict[k+'abc']=theDict.pop(k)
I use this for converting docopt POSIX-compliant command-line keys to PEP8 keys
(e.g. "--option" --> "option", "" --> "option2", "FILENAME" --> "filename")
arguments = docopt.docopt(__doc__) # dictionary
for key in arguments.keys():
if re.match('.*[-<>].*', key) or key != key.lower():
value = arguments.pop(key)
newkey = key.lower().translate(None, '-<>')
arguments[newkey] = value
Hi I'm a new user but finding an answer for same question, I can't get anything fully functional to my problem, I make this little piece of cake with a full nested replace of keys, you can send list with dict or dict.
Finally your dicts can have list with dict or more dict nested and it is all replaced with your new key needs.
To indicate who key want replace with a new key use "to" parameter sending a dict.
See at end my little example.
P/D: Sorry my bad english. =)
def re_map(value, to):
"""
Transform dictionary keys to map retrieved on to parameters.
to parameter should have as key a key name to replace an as value key name
to new dictionary.
this method is full recursive to process all levels of
#param value: list with dictionary or dictionary
#param to: dictionary with re-map keys
#type to: dict
#return: list or dict transformed
"""
if not isinstance(value, dict):
if not isinstance(value, list):
raise ValueError(
"Only dict or list with dict inside accepted for value argument.") # #IgnorePep8
if not isinstance(to, dict):
raise ValueError("Only dict accepted for to argument.")
def _re_map(value, to):
if isinstance(value, dict):
# Re map dictionary key.
# If key of original dictionary is not in "to" dictionary use same
# key otherwise use re mapped key on new dictionary with already
# value.
return {
to.get(key) or key: _re_map(dict_value, to)
for key, dict_value in value.items()
}
elif isinstance(value, list):
# if value is a list iterate it a call _re_map again to parse
# values on it.
return [_re_map(item, to) for item in value]
else:
# if not dict or list only return value.
# it can be string, integer or others.
return value
result = _re_map(value, to)
return result
if __name__ == "__main__":
# Sample test of re_map method.
# -----------------------------------------
to = {"$id": "id"}
x = []
for i in range(100):
x.append({
"$id": "first-dict",
"list_nested": [{
"$id": "list-dict-nested",
"list_dic_nested": [{
"$id": "list-dict-list-dict-nested"
}]
}],
"dict_nested": {
"$id": "non-nested"
}
})
result = re_map(x, to)
print(str(result))
A functional (and flexible) solution: this allows an arbitrary transform to be applied to keys (recursively for embedded dicts):
def remap_keys(d, keymap_f):
"""returns a new dict by recursively remapping all of d's keys using keymap_f"""
return dict([(keymap_f(k), remap_keys(v, keymap_f) if isinstance(v, dict) else v)
for k,v in d.items()])
Let's try it out; first we define our key transformation function, then apply it to the example:
def transform_key(key):
"""whatever transformation you'd like to apply to keys"""
return key + "abc"
remap_keys({'fruit':'orange','colors':{'dark':4,'light':5}}, transform_key)
{'fruitabc': 'orange', 'colorsabc': {'darkabc': 4, 'lightabc': 5}}
(note: if you're still on Python 2.x, you'll need to replace d.items() on the last line with d.iteritems() -- thanks to #Rudy for reminding me to update this post for Python 3).
Based on #AndiDog's python 3 version and similar to #sxc731's version but with a flag for whether to apply it recursively:
def transform_keys(dictionary, key_fn, recursive=True):
"""
Applies function to keys and returns as a new dictionary.
Example of key_fn:
lambda k: k + "abc"
"""
return {key_fn(key): (transform_keys(value, key_fn=key_fn, recursive=recursive)
if recursive and isinstance(value, dict) else value)
for key, value in dictionary.items()}