I need a bit of Python refactoring advice.
I have a list of dict objects (new_monitors), which can be empty. When there are new monitors, however, I want to add a bunch of fields to those monitors.
For each monitor, I would like to append all not None fields from the DogDump.HIDE_FIELDS dict:
if new_monitors:
for monitor in new_monitors:
for key, value in DogDump.HIDE_FIELDS.items():
if value:
monitor[key] = value
Note: This snippet below worked very well, but it included all of the None fields. I do not want the None fields!
if new_monitors:
for monitor in new_monitors:
monitor.update(DogDump.HIDE_FIELDS)
How can I refactor this snippet that looks more pythonic, but still maintain good readability?
Not sure which is really the most "pythonic" way of handling your need to filter DogDump.HIDE_FIELDS dict before adding the relevant key / value pairs to your monitor dict. One way would be to perform the "filtering" with dict comprehension.
Also, I would think that you could "filter" your DogDump.HIDE_FIELDS dict before your loop rather than repeating this operation for each loop iteration (unless there are other operations taking place that mutate DogDump.HIDE_FIELDS while you are iterating).
Example of "filtering" with dict comprehension (dump refers to your DogDump.HIDE_FIELDS dict):
monitor = {'key': 'value'}
dump = {'a': 1, 'b': None}
dump_filtered = {k:v for (k,v) in dump.items() if v}
monitor.update(dump_filtered)
print(monitor)
# OUTPUT
# {'key': 'value', 'a': 1}
Related
I was trying to iterate over a list of values, craft a dictionary to save each value in a structured way, and then append the dictionary to a new list of results, but I found an unexpected behavior.
Below is an example:
values_list = [1,2,3]
# Basic dict
result_dict = {
'my_value': ''
}
# Iterate, craft a dictionary, and append result
dicts_list = []
for value in values_list:
result_dict.update({'my_value': value})
dicts_list.append(result_dict)
print(dicts_list)
As you can see, first I create a basic dictionary, then I'm iterating over the list of values and updating the dictionary, at the end I'm appending the crafted dictionary to a separate list of results (dicts_list).
As a result I was expecting:
[{'my_value': 1}, {'my_value': 2}, {'my_value': 3}]
but instead I was getting:
[{'my_value': 3}, {'my_value': 3}, {'my_value': 3}]
It looks like every iteration is not only updating the basic dictionary – which is expected – but also the dictionaries already appended to the list of results on the previous iteration.
To fix the issue, I nested the basic dictionary under the for loop:
values_list = [1,2,3]
# Iterate, craft a dictionary, and append result
dicts_list = []
for value in values_list:
result_dict = {'my_value': ''}
result_dict.update({'my_value': value})
dicts_list.append(result_dict)
print(dicts_list)
Can anyone explain what is wrong with the first approach? How is the loop causing the list of appended dictionaries to be updated?
Thanks for any advice! :)
Franz
As explained in the comment, you're appending the same dictionary in each iteration because update() modifies the result_dict rather than returning a copy. So, the only change you need to do is to append a copy of the crafted dictionary. For example:
values_list = [1,2,3]
# Basic dict
result_dict = {
'my_value': ''
}
# Iterate, craft a dictionary, and append result
dicts_list = []
for value in values_list:
result_dict.update({'my_value': value})
dicts_list.append(dict(result_dict)) # <--- this is the only change
print(dicts_list)
To gain understanding of How is the loop causing the list of appended dictionaries to be updated? you can use Python Tutor: Visualize code in Python with the code you provided in your question to see the effect of executing the code line by line with the final result being following visualization:
I suggest you read also Facts and myths about Python names and values.
I have a django queryset containing information about network things. Primary key is IP.
Then, I have a dictionary with IPs as keys containing some data.
I would like to display all IPs from either of the two sources in order, together with some info.
If an IP exists in the queryset, I don't care about the dictionary value.
For example, given:
<Queryset[{'ip':'10.2.2.2', 'type':'laptop', ...},
{'ip':'10.4.4.4', 'type':'camera', ...},
{'ip':'10.5.5.5', 'type':'iot', ...},
{'ip':'10.8.8.8', 'type':'switch', ...}]>
and:
{'10.1.1.1': 'foo', '10.4.4.4':'xxx', '10.6.6.6': 'bar', '10.9.9.9': 'baz'}
I want the following output:
10.1.1.1 foo
10.2.2.2 type: laptop ...
10.4.4.4 type: camera ...
10.5.5.5 type: iot ...
10.6.6.6 bar
10.8.8.8 type: switch ...
10.9.9.9 baz
I could do nested loops, but is there a neat pythonic way?
You really don't want to nest the loops. Since you don't care about the dict entry if it's in the query set, you can update the dict with the items from the query set, and those will be overwritten. Then you can sort the dict items all at once.
d.update(qs.values_list('ip', 'type'))
for k, v in sorted(d.items()):
print(k, v)
Where d is the dict and qs is the query set.
If you still need the type: part printed off for items originally in the query set only, you can do the update a little differently.
d.update((k, 'type: '+v) for k, v in qs.values_list('ip', 'type'))
This prepends the string type: to every value from the query set before putting it in the dict.
It's not just type. For each IP I want either the whole object, or the string from the dictionary
Something like this, then?
d.update((o.ip, o) for o in qs)
You can set the value side of the pair to be whatever you want in the generator expression.
Say I have a list of dictionaries.
each dict in the list has 3 elements.
Name, id and status.
list_of_dicts = [{'id':1, 'name':'Alice', 'status':0},{'id':2, 'name':'Bob', 'status':0},{'id':3, 'name':'Robert', 'status':1}]
so I get:
In[20]: print list_of_dicts
Out[20]:
[{'id': 1, 'name': 'Alice', 'status': 0},
{'id': 2, 'name': 'Bob', 'status': 0},
{'id': 3, 'name': 'Robert', 'status': 1}]
If i recieve a name, how can I get its status without iterating on the list?
e.g. I get 'Robert' and I want to output 1. thank you.
for example you can use pandas
import pandas as pd
list_of_dicts = [{'id':1, 'name':'Alice', 'status':0},{'id':2, 'name':'Bob', 'status':0},{'id':3, 'name':'Robert', 'status':1}]
a = pd.DataFrame(list_of_dicts)
a.loc[a['name'] == 'Robert']
and play with dataframes its very fast because write on c++ and easy (like sql queries)
As you found you have to iterate (unless you are able to change your data structure to an enclosing dict) why don't you just do it?
>>> [d['status'] for d in list_of_dicts if d['name']=='Robert']
[1]
Despite this, I recommend considering a map type (like dict) every time you see some 'id' field in a proposed data structure. If it's there you probably want to use it for general identification, instead of carrying dicts around. They can be used for relations also, and transfer easily into a relational database if you need it later.
I don't think you can do what you ask without iterating through the dictionary:
Best case, you'll find someone that suggests you a method that hides the iteration.
If what really concerns you is the speed, you may break your iteration as soon as you find the first valid result:
for iteration, elements in enumerate(list_of_dicts):
if elements['name'] == "Robert":
print "Elements id: ", elements['id']
break
print "Iterations: ", iteration
# OUTPUT: Elements id: 3, Iterations: 1
Note that numbers of iteration may vary, since dictionaries are not indexed, and if you have more "Roberts", only for one the "id" will be printed
It's not possible to do this without iteration.
However, but you can transform you dictionary into a different data structure, such as a dictionary where names are the keys:
new_dict = {person["name"]: {k: v for k, v in person.items() if k != "name"} for person in list_of_dicts}
Then you can get the status like so:
new_dict["Robert"]["status"]
# 1
Additionally, as #tobias_k mentions in the comments, you can keep the internal dictionary the same:
{person["name"]: person for person in list_of_dicts}
The only issue with the above approaches is that it can't handle multiple names. You can instead add the unique id into the key to differentiate between names:
new_dict = {(person["name"], person["id"]): person["status"] for person in list_of_dicts}
Which can be called like this:
new_dict["Robert", 3]
# 1
Even though it takes extra computation(only once) to create these data structures, the lookups afterwards will be O(1), instead of iterating the list every time when you want to search a name.
Your list_of_dicts cannot be reached without a loop so for your desire your list should be modified a little like 1 dict and many lists in it:
list_of_dicts_modified = {'name':['Alice', 'Bob', 'Robert'],'id':[1, 2, 3], 'status': [0, 0, 1]}
index = list_of_dicts_modified['name'].index(input().strip())
print('Name: {0} ID: {1} Status: {2}'.format(list_of_dicts_modified['name'][index], list_of_dicts_modified['id'][index], list_of_dicts_modified['status'][index]))
Output:
C:\Users\Documents>py test.py
Alice
Name: Alice ID: 1 Status: 0
To elaborate
sample_dict = {"key1" : {"key2" : "value_of_interest}}
Now I want to be able to make the following call
val = sample_dict.recursive_get("key2", None)
I can think of a number of cases where this would be useful. For example, getting a key-value from an api response that came as JSON.
And I know it is not difficult to write to such a function, but my question is - why is it not already included?
If it leads to bad design, please provide some examples.
Suppose, as several commenters pointed out, the key pointing to your "value of interest" appears multiple times in the nested hierarchy:
sample_dict = {'key1': {'key1': 'value_of_interest'}}
val = sample_dict.recursive_get('key1', None)
What is the value of val? The inner dict? 'value_of_interest'? What should it be? Which answer violates the principle of least astonishment?
What about a dict like this?
sample_dict = {
'key1': [
{'key1': 'value_of_interest'},
{'key2': 'less_interesting_value'},
],
}
Should the recursion check at every level for a possible list or array? It's not uncommon to see dicts like this, especially when formed from JSON.
The point is there's a lot of different dicts out there, which can be nested in a lot of different ways. Python is pretty well-equipped to handle them all fairly elegantly. No definition of recursive_get that I can think of, though, is.
def recursive(dic, key, out=[]):
if out: return
if key in dic: out += [dic[key]]
for k, v in dic.items():
if isinstance(v, dict): recursive(v, key)
return out
think i wrote you a depth first search?
This is more of a theoretical question than anything. I have 3 dictionaries that have different structure/keys but the values may be the same. E.g.
dict1 = {u'd':{u'results':[{u'number':'1', u'dispURL':u'www.site.com'},
{u'number':u'2', u'dispURL':u'www.othersite.com'}]
}}
dict2 = {u'result_page':{u'url':u'www.greatsite.com', u'pos':u'1'},
{u'url':u'www.site.com', u'pos':u'2'}}
dict3 = {u'hits':[{u'displayurl':u'www.othersite.com', u'index':u'1'},
{u'displayurl':u'www.site.com', u'index':u'2'}]
}
Note how dict1 has {u'd': before the {u'results':, also dict1 and dict3 have the different key/values enclosed in square parentheses. Also note how the keys/values layout is different in dict1 with the number key coming before the url instead of after like in dict2 and dict3, as well as the key names being different in each dictionary.
I have 3 large dictionaries like this and I need to compare them by the position of each url for scoring purposes. i.e.
if dict1[www.site.com index] > dict2[www.site.com index]:
dict1[www.site.com] score +1
I know the code snippet isn't correct it's just for illustration. What I want to know is what do I need to do with the dictionaries to be able to perform a comparison like this? I was thinking of taking the required data from each dictionary and putting them into 3 new dictionaries with uniform keys and structure, or even 1 new dictionary. But my program has to be computationally quite fast so I don't know how this would affect it. Any of you more experienced python programmers want to have a say on this?
The most effective way to approach this is to convert your data into canonical {url:value} format.
For example:
dict1 = {data[u'dispURL']:int(data[u'number']) for data in dict1[u'd'][u'results']}
dict2 = {data[u'url']:int(data[u'pos']) for data in dict2[u'result_page']}
dict3 = {data[u'displayurl']:int(data[u'index']) for data in dict3[u'hits']}
Now they look like
dict1 = {u'www.othersite.com': 2, u'www.site.com': 1}
dict2 = {u'www.greatsite.com': 1, u'www.site.com': 2}
dict3 = {u'www.othersite.com': 1, u'www.site.com': 2}
and your comparison looks like
for url in dict1:
if dict1[url] > dict2[url]:
# do something