Related
I am parsing unknown nested json object, I do not know the structure nor depth ahead of time. I am trying to search through it to find a value. This is what I came up with, but I find it fugly. Could anybody let me know how to make this look more pythonic and cleaner?
def find(d, key):
if isinstance(d, dict):
for k, v in d.iteritems():
try:
if key in str(v):
return 'found'
except:
continue
if isinstance(v, dict):
for key,value in v.iteritems():
try:
if key in str(value):
return "found"
except:
continue
if isinstance(v, dict):
find(v)
elif isinstance(v, list):
for x in v:
find(x)
if isinstance(d, list):
for x in d:
try:
if key in x:
return "found"
except:
continue
if isinstance(v, dict):
find(v)
elif isinstance(v, list):
for x in v:
find(x)
else:
if key in str(d):
return "found"
else:
return "Not Found"
It is generally more "Pythonic" to use duck typing; i.e., just try to search for your target rather than using isinstance. See What are the differences between type() and isinstance()?
However, your need for recursion makes it necessary to recurse the values of the dictionaries and the elements of the list. (Do you also want to search the keys of the dictionaries?)
The in operator can be used for both strings, lists, and dictionaries, so no need to separate the dictionaries from the lists when testing for membership. Assuming you don't want to test for the target as a substring, do use isinstance(basestring) per the previous link. To test whether your target is among the values of a dictionary, test for membership in your_dictionary.values(). See Get key by value in dictionary
Because the dictionary values might be lists or dictionaries, I still might test for dictionary and list types the way you did, but I mention that you can cover both list elements and dictionary keys with a single statement because you ask about being Pythonic, and using an overloaded oeprator like in across two types is typical of Python.
Your idea to use recursion is necessary, but I wouldn't define the function with the name find because that is a Python built-in which you will (sort of) shadow and make the recursive call less readable because another programmer might mistakenly think you're calling the built-in (and as good practice, you might want to leave the usual access to the built in in case you want to call it.)
To test for numeric types, use `numbers.Number' as described at How can I check if my python object is a number?
Also, there is a solution to a variation of your problem at https://gist.github.com/douglasmiranda/5127251 . I found that before posting because ColdSpeed's regex suggestion in the comment made me wonder if I were leading you down the wrong path.
So something like
import numbers
def recursively_search(object_from_json, target):
if isinstance(object_from_json, (basestring, numbers.Number)):
return object_from_json==target # the recursion base cases
elif isinstance(object_from_json, list):
for element in list:
if recursively_search(element, target):
return True # quit at first match
elif isinstance(object_from_json, dict):
if target in object_from_json:
return True # among the keys
else:
for value in object_from_json.values():
if recursively_search(value, target):
return True # quit upon finding first match
else:
print ("recursively_search() did not anticipate type ",type(object_from_json))
return False
return False # no match found among the list elements, dict keys, nor dict values
I have several nested dictionaries within lists, and I need to verify if a specific path exist e.g.
dict1['layer1']['layer2'][0]['layer3']
How can I check with an IF statement if the path is valid?
I was thinking to
if dict1['layer1']['layer2'][0]['layer3'] :
but it doesn't work
Here's the explicit short code with try/except:
try:
dict1['layer1']['layer2'][0]['layer3']
except KeyError:
present = False
else:
present = True
if present:
...
To get the element:
try:
obj = dict1['layer1']['layer2'][0]['layer3']
except KeyError:
obj = None # or whatever
I wanted to propose another solution, because I've been thinking about it too.
if not dict1.get("layer1", {}).get("layer2", {})[0].get("layer3", {}):
...
dict.get() attempts to get the key at each stage.
If the key is not present, an empty dict will be returned, instead of the nested dict (this is needed, because trying to call .get() on the default return of None will yield an AttributeError).
If the return is empty, it will evaluate to false.
So, this wouldn't work if the final result was an empty dict anyway, but if you can guarantee the results will be filled, this is an alternative that is fairly simple.
As far as I know, you've to go step by step, i.e.:
if 'layer1' in dict1:
if 'layer2' in dict1['layer1']
ans so on...
If you don't want to go the try/except route, you could whip up a quick method to do this:
def check_dict_path(d, *indices):
sentinel = object()
for index in indices:
d = d.get(index, sentinel)
if d is sentinel:
return False
return True
test_dict = {1: {'blah': {'blob': 4}}}
print check_dict_path(test_dict, 1, 'blah', 'blob') # True
print check_dict_path(test_dict, 1, 'blah', 'rob') # False
This might be redundant if you're also trying to retrieve the object at that location (rather than just verify whether the location exists). If that's the case, the above method can easily be updated accordingly.
Here is a similar question with the answer I would recommend:
Elegant way to check if a nested key exists in a python dict
Using recursive function:
def path_exists(path, dict_obj, index = 0):
if (type(dict_obj) is dict and path[index] in dict_obj.keys()):
if (len(path) > (index+1)):
return path_exists(path, dict_obj[path[index]], index + 1)
else:
return True
else:
return False
Where path is a list of strings representing the nested keys.
Is there are more readable way to check if a key buried in a dict exists without checking each level independently?
Lets say I need to get this value in a object buried (example taken from Wikidata):
x = s['mainsnak']['datavalue']['value']['numeric-id']
To make sure that this does not end with a runtime error it is necessary to either check every level like so:
if 'mainsnak' in s and 'datavalue' in s['mainsnak'] and 'value' in s['mainsnak']['datavalue'] and 'nurmeric-id' in s['mainsnak']['datavalue']['value']:
x = s['mainsnak']['datavalue']['value']['numeric-id']
The other way I can think of to solve this is wrap this into a try catch construct which I feel is also rather awkward for such a simple task.
I am looking for something like:
x = exists(s['mainsnak']['datavalue']['value']['numeric-id'])
which returns True if all levels exists.
To be brief, with Python you must trust it is easier to ask for forgiveness than permission
try:
x = s['mainsnak']['datavalue']['value']['numeric-id']
except KeyError:
pass
The answer
Here is how I deal with nested dict keys:
def keys_exists(element, *keys):
'''
Check if *keys (nested) exists in `element` (dict).
'''
if not isinstance(element, dict):
raise AttributeError('keys_exists() expects dict as first argument.')
if len(keys) == 0:
raise AttributeError('keys_exists() expects at least two arguments, one given.')
_element = element
for key in keys:
try:
_element = _element[key]
except KeyError:
return False
return True
Example:
data = {
"spam": {
"egg": {
"bacon": "Well..",
"sausages": "Spam egg sausages and spam",
"spam": "does not have much spam in it"
}
}
}
print 'spam (exists): {}'.format(keys_exists(data, "spam"))
print 'spam > bacon (do not exists): {}'.format(keys_exists(data, "spam", "bacon"))
print 'spam > egg (exists): {}'.format(keys_exists(data, "spam", "egg"))
print 'spam > egg > bacon (exists): {}'.format(keys_exists(data, "spam", "egg", "bacon"))
Output:
spam (exists): True
spam > bacon (do not exists): False
spam > egg (exists): True
spam > egg > bacon (exists): True
It loop in given element testing each key in given order.
I prefere this to all variable.get('key', {}) methods I found because it follows EAFP.
Function except to be called like: keys_exists(dict_element_to_test, 'key_level_0', 'key_level_1', 'key_level_n', ..). At least two arguments are required, the element and one key, but you can add how many keys you want.
If you need to use kind of map, you can do something like:
expected_keys = ['spam', 'egg', 'bacon']
keys_exists(data, *expected_keys)
You could use .get with defaults:
s.get('mainsnak', {}).get('datavalue', {}).get('value', {}).get('numeric-id')
but this is almost certainly less clear than using try/except.
Python 3.8 +
dictionary = {
"main_key": {
"sub_key": "value",
},
}
if sub_key_value := dictionary.get("main_key", {}).get("sub_key"):
print(f"The key 'sub_key' exists in dictionary[main_key] and it's value is {sub_key_value}")
else:
print("Key 'sub_key' doesn't exists or their value is Falsy")
Extra
A little but important clarification.
In the previous code block, we verify that a key exists in a dictionary but that its value is also Truthy.
Most of the time, this is what people are really looking for, and I think this is what the OP really wants. However, it is not really the most "correct" answer, since if the key exists but its value is False, the above code block will tell us that the key does not exist, which is not true.
So, I leet here a more correct answer:
dictionary = {
"main_key": {
"sub_key": False,
},
}
if "sub_key" in dictionary.get("main_key", {}):
print(f"The key 'sub_key' exists in dictionary[main_key] and it's value is {dictionary['main_key']['sub_key']}")
else:
print("Key 'sub_key' doesn't exists")
Try/except seems to be most pythonic way to do that.
The following recursive function should work (returns None if one of the keys was not found in the dict):
def exists(obj, chain):
_key = chain.pop(0)
if _key in obj:
return exists(obj[_key], chain) if chain else obj[_key]
myDict ={
'mainsnak': {
'datavalue': {
'value': {
'numeric-id': 1
}
}
}
}
result = exists(myDict, ['mainsnak', 'datavalue', 'value', 'numeric-id'])
print(result)
>>> 1
I suggest you to use python-benedict, a solid python dict subclass with full keypath support and many utility methods.
You just need to cast your existing dict:
s = benedict(s)
Now your dict has full keypath support and you can check if the key exists in the pythonic way, using the in operator:
if 'mainsnak.datavalue.value.numeric-id' in s:
# do stuff
Here the library repository and the documentation:
https://github.com/fabiocaccamo/python-benedict
Note: I am the author of this project
You can use pydash to check if exists: http://pydash.readthedocs.io/en/latest/api.html#pydash.objects.has
Or get the value (you can even set default - to return if doesn't exist): http://pydash.readthedocs.io/en/latest/api.html#pydash.objects.has
Here is an example:
>>> get({'a': {'b': {'c': [1, 2, 3, 4]}}}, 'a.b.c[1]')
2
The try/except way is the most clean, no contest. However, it also counts as an exception in my IDE, which halts execution while debugging.
Furthermore, I do not like using exceptions as in-method control statements, which is essentially what is happening with the try/catch.
Here is a short solution which does not use recursion, and supports a default value:
def chained_dict_lookup(lookup_dict, keys, default=None):
_current_level = lookup_dict
for key in keys:
if key in _current_level:
_current_level = _current_level[key]
else:
return default
return _current_level
The accepted answer is a good one, but here is another approach. It's a little less typing and a little easier on the eyes (in my opinion) if you end up having to do this a lot. It also doesn't require any additional package dependencies like some of the other answers. Have not compared performance.
import functools
def haskey(d, path):
try:
functools.reduce(lambda x, y: x[y], path.split("."), d)
return True
except KeyError:
return False
# Throwing in this approach for nested get for the heck of it...
def getkey(d, path, *default):
try:
return functools.reduce(lambda x, y: x[y], path.split("."), d)
except KeyError:
if default:
return default[0]
raise
Usage:
data = {
"spam": {
"egg": {
"bacon": "Well..",
"sausages": "Spam egg sausages and spam",
"spam": "does not have much spam in it",
}
}
}
(Pdb) haskey(data, "spam")
True
(Pdb) haskey(data, "spamw")
False
(Pdb) haskey(data, "spam.egg")
True
(Pdb) haskey(data, "spam.egg3")
False
(Pdb) haskey(data, "spam.egg.bacon")
True
Original inspiration from the answers to this question.
EDIT: a comment pointed out that this only works with string keys. A more generic approach would be to accept an iterable path param:
def haskey(d, path):
try:
functools.reduce(lambda x, y: x[y], path, d)
return True
except KeyError:
return False
(Pdb) haskey(data, ["spam", "egg"])
True
I had the same problem and recent python lib popped up:
https://pypi.org/project/dictor/
https://github.com/perfecto25/dictor
So in your case:
from dictor import dictor
x = dictor(s, 'mainsnak.datavalue.value.numeric-id')
Personal note:
I don't like 'dictor' name, since it doesn't hint what it actually does. So I'm using it like:
from dictor import dictor as extract
x = extract(s, 'mainsnak.datavalue.value.numeric-id')
Couldn't come up with better naming than extract. Feel free to comment, if you come up with more viable naming. safe_get, robust_get didn't felt right for my case.
Another way:
def does_nested_key_exists(dictionary, nested_key):
exists = nested_key in dictionary
if not exists:
for key, value in dictionary.items():
if isinstance(value, dict):
exists = exists or does_nested_key_exists(value, nested_key)
return exists
The selected answer works well on the happy path, but there are a couple obvious issues to me. If you were to search for ["spam", "egg", "bacon", "pizza"], it would throw a type error due to trying to index "well..." using the string "pizza". Like wise, if you replaced pizza with 2, it would use that to get the index 2 from "Well..."
Selected Answer Output Issues:
data = {
"spam": {
"egg": {
"bacon": "Well..",
"sausages": "Spam egg sausages and spam",
"spam": "does not have much spam in it"
}
}
}
print(keys_exists(data, "spam", "egg", "bacon", "pizza"))
>> TypeError: string indices must be integers
print(keys_exists(data, "spam", "egg", "bacon", 2)))
>> l
I also feel that using try except can be a crutch that we might too quickly rely on. Since I believe we already need to check for the type, might as well remove the try except.
Solution:
def dict_value_or_default(element, keys=[], default=Undefined):
'''
Check if keys (nested) exists in `element` (dict).
Returns value if last key exists, else returns default value
'''
if not isinstance(element, dict):
return default
_element = element
for key in keys:
# Necessary to ensure _element is not a different indexable type (list, string, etc).
# get() would have the same issue if that method name was implemented by a different object
if not isinstance(_element, dict) or key not in _element:
return default
_element = _element[key]
return _element
Output:
print(dict_value_or_default(data, ["spam", "egg", "bacon", "pizza"]))
>> INVALID
print(dict_value_or_default(data, ["spam", "egg", "bacon", 2]))
>> INVALID
print(dict_value_or_default(data, ["spam", "egg", "bacon"]))
>> "Well..."
Here's my small snippet based on #Aroust's answer:
def exist(obj, *keys: str) -> bool:
_obj = obj
try:
for key in keys:
_obj = _obj[key]
except (KeyError, TypeError):
return False
return True
if __name__ == '__main__':
obj = {"mainsnak": {"datavalue": {"value": "A"}}}
answer = exist(obj, "mainsnak", "datavalue", "value", "B")
print(answer)
I added TypeError because when _obj is str, int, None, or etc, it would raise that error.
I wrote a data parsing library called dataknead for cases like this, basically because i got frustrated by the JSON the Wikidata API returns as well.
With that library you could do something like this
from dataknead import Knead
numid = Knead(s).query("mainsnak/datavalue/value/numeric-id").data()
if numid:
# Do something with `numeric-id`
Using dict with defaults is concise and appears to execute faster than using consecutive if statements.
Try it yourself:
import timeit
timeit.timeit("'x' in {'a': {'x': {'y'}}}.get('a', {})")
# 0.2874350370002503
timeit.timeit("'a' in {'a': {'x': {'y'}}} and 'x' in {'a': {'x': {'y'}}}['a']")
# 0.3466246419993695
I have written a handy library for this purpose.
I am iterating over ast of the dict and trying to check if a particular key is present or not.
Do check this out.
https://github.com/Agent-Hellboy/trace-dkey
If you can suffer testing a string representation of the object path then this approach might work for you:
def exists(str):
try:
eval(str)
return True
except:
return False
exists("lst['sublist']['item']")
one can try to use this for checking whether key/nestedkey/value is in nested dict
import yaml
#d - nested dictionary
if something in yaml.dump(d, default_flow_style=False):
print(something, "is in", d)
else:
print(something, "is not in", d)
There are many great answers. here is my humble take on it. Added check for array of dictionaries as well. Please note that I am not checking for arguments validity. I used part Arnot's code above. I added this answer because a I got a use case that requires checking array or dictionaries in my data.
Here is the code:
def keys_exists(element, *keys):
'''
Check if *keys (nested) exists in `element` (dict).
'''
retval=False
if isinstance(element,dict):
for key,value in element.items():
for akey in keys:
if element.get(akey) is not None:
return True
if isinstance(value,dict) or isinstance(value,list):
retval= keys_exists(value, *keys)
elif isinstance(element, list):
for val in element:
if isinstance(val,dict) or isinstance(val,list):
retval=keys_exists(val, *keys)
return retval
This question already has answers here:
Why does my recursive function return None?
(4 answers)
Closed 6 months ago.
I have a function that takes a key and traverses nested dicts to return the value regardless of its depth. However, I can only get the value to print, not return. I've read the other questions on this issue and and have tried 1. implementing yield 2. appending the value to a list and then returning the list.
def get_item(data,item_key):
# data=dict, item_key=str
if isinstance(data,dict):
if item_key in data.keys():
print data[item_key]
return data[item_key]
else:
for key in data.keys():
# recursion
get_item(data[key],item_key)
item = get_item(data,'aws:RequestId')
print item
Sample data:
data = OrderedDict([(u'aws:UrlInfoResponse', OrderedDict([(u'#xmlns:aws', u'http://alexa.amazonaws.com/doc/2005-10-05/'), (u'aws:Response', OrderedDict([(u'#xmlns:aws', u'http://awis.amazonaws.com/doc/2005-07-11'), (u'aws:OperationRequest', OrderedDict([(u'aws:RequestId', u'4dbbf7ef-ae87-483b-5ff1-852c777be012')])), (u'aws:UrlInfoResult', OrderedDict([(u'aws:Alexa', OrderedDict([(u'aws:TrafficData', OrderedDict([(u'aws:DataUrl', OrderedDict([(u'#type', u'canonical'), ('#text', u'infowars.com/')])), (u'aws:Rank', u'1252')]))]))])), (u'aws:ResponseStatus', OrderedDict([(u'#xmlns:aws', u'http://alexa.amazonaws.com/doc/2005-10-05/'), (u'aws:StatusCode', u'Success')]))]))]))])
When I execute, the desired value prints, but does not return:
>>>52c7e94b-dc76-2dd6-1216-f147d991d6c7
>>>None
What is happening? Why isn't the function breaking and returning the value when it finds it?
A simple fix, you have to find a nested dict that returns a value. You don't need to explicitly use an else clause because the if returns. You also don't need to call .keys():
def get_item(data, item_key):
if isinstance(data, dict):
if item_key in data:
return data[item_key]
for key in data:
found = get_item(data[key], item_key)
if found:
return found
return None # Explicit vs Implicit
>>> get_item(data, 'aws:RequestId')
'4dbbf7ef-ae87-483b-5ff1-852c777be012'
One of the design principles of python is EAFP (Easier to Ask for Forgiveness than Permission), which means that exceptions are more commonly used than in other languages. The above rewritten with EAFP design:
def get_item(data, item_key):
try:
return data[item_key]
except KeyError:
for key in data:
found = get_item(data[key], item_key)
if found:
return found
except (TypeError, IndexError):
pass
return None
As other people commented, you need return statement in else blocks, too. You have two if blocks so you would need two more return statement. Here is code that does what you may want
from collections import OrderedDict
def get_item(data,item_key):
result = []
if isinstance(data, dict):
for key in data:
if key == item_key:
print data[item_key]
result.append(data[item_key])
# recursion
result += get_item(data[key],item_key)
return result
return result
Your else block needs to return the value if it finds it.
I've made a few other minor changes to your code. You don't need to do
if item_key in data.keys():
Instead, you can simply do
if item_key in data:
Similarly, you don't need
for key in data.keys():
You can iterate directly over a dict (or any class derived from a dict) to iterate over its keys:
for key in data:
Here's my version of your code, which should run on Python 2.7 as well as Python 3.
from __future__ import print_function
from collections import OrderedDict
def get_item(data, item_key):
if isinstance(data, dict):
if item_key in data:
return data[item_key]
for val in data.values():
v = get_item(val, item_key)
if v is not None:
return v
data = OrderedDict([(u'aws:UrlInfoResponse',
OrderedDict([(u'#xmlns:aws', u'http://alexa.amazonaws.com/doc/2005-10-05/'), (u'aws:Response',
OrderedDict([(u'#xmlns:aws', u'http://awis.amazonaws.com/doc/2005-07-11'), (u'aws:OperationRequest',
OrderedDict([(u'aws:RequestId', u'4dbbf7ef-ae87-483b-5ff1-852c777be012')])), (u'aws:UrlInfoResult',
OrderedDict([(u'aws:Alexa',
OrderedDict([(u'aws:TrafficData',
OrderedDict([(u'aws:DataUrl',
OrderedDict([(u'#type', u'canonical'), ('#text', u'infowars.com/')])),
(u'aws:Rank', u'1252')]))]))])), (u'aws:ResponseStatus',
OrderedDict([(u'#xmlns:aws', u'http://alexa.amazonaws.com/doc/2005-10-05/'),
(u'aws:StatusCode', u'Success')]))]))]))])
item = get_item(data, 'aws:RequestId')
print(item)
output
4dbbf7ef-ae87-483b-5ff1-852c777be012
Note that this function returns None if the isinstance(data, dict) test fails, or if the for loop fails to return. It's generally a good idea to ensure that every possible return path in a recursive function has an explicit return statement, as that makes it clearer what's happening, but IMHO it's ok to leave those returns implicit in this fairly simple function.
This is probably a very basic question, but I looked through the python documentation on exceptions and couldn't find it.
I'm trying to read a bunch of specific values from a dictionary and insert slices of these into another dictionary.
for item in old_dicts:
try:
new_dict['key1'] = item['dog1'][0:5]
new_dict['key2'] = item['dog2'][0:10]
new_dict['key3'] = item['dog3'][0:3]
new_dict['key4'] = item['dog4'][3:11]
except KeyError:
pass
Now, if Python encounters a key error at ['dog1'], it seems to abort the current iteration and go to the next item in old_dicts. I'd like it to go to the next line in the loop instead. Do I have to insert an exception instruction for each row?
Make it a function:
def newdog(self, key, dog, a, b)
try:
new_dict[key] = item[dog][a:b]
except KeyError:
pass
I didn't run the above code but something like that should work, a modularization. Or what you could do is prepare so that it checks all values and removes all values that are not in the dictionary, but that will probably be more code than an exception for each row.
for item in old_dicts:
for i, (start, stop) in enumerate([(0,5), (0,10), (0,3), (3,11)], 1):
try:
new_dict['key' + str(i)] = item['dog' + str(i)][start:stop]
except KeyError:
pass
Assuming that you know the values in the keys will be valid, why not just forgo exceptions all together and check for the keys?
for item in old_dicts:
if 'dog1' in item:
new_dict['key1'] = item['dog1'][0:5]
if 'dog2' in item:
new_dict['key2'] = item['dog2'][0:10]
if 'dog3' in item:
new_dict['key3'] = item['dog3'][0:3]
if 'dog4' in item:
new_dict['key4'] = item['dog4'][3:11]
Yes, you do. It will be messy and unpleasant to look at, but you do need an exception handler for every call.
That said, you can write your code a little differently to make life easier on yourself. Consider this:
def set_new_dict_key(new_dict, key, item, path):
try:
for path_item in path:
item = item[path_item]
except KeyError:
pass
else:
new_dict[key] = item
for item in old_dicts:
set_new_dict_key(new_dict, 'key1', item, ['dog1', slice(0, 5)])
set_new_dict_key(new_dict, 'key2', item, ['dog2', slice(0, 10)])
set_new_dict_key(new_dict, 'key3', item, ['dog3', slice(0, 3)])
set_new_dict_key(new_dict, 'key4', item, ['dog4', slice(3, 11)])
Yes, you will. Best practice is to keep your try blocks as small as possible. Only write the code you possibly expect an exception from in the try block. Note that you also have an else for the try and except statements, that only gets executed when the try ran without an exception:
try:
// code that possibly throws exception
except Exception:
// handle exception
else:
// do stuff that should be done if there was no exception