Develop Reference

Why? - Constructing a list of dictionaries from a loop doesn't work unless the dictionary is recreated on every iteration [duplicate]

This question already has answers here:
Appending a dictionary to a list - I see a pointer like behavior
(3 answers)
Closed 2 years ago.
I was really stumped to find that creating a list of dictionaries from a loop does not yield the expected results unless the dictionary is recreated on each iteration.
The following examples are contrived and just serve as a minimal repex.
Two things that do work as expected:
l = list()
for i in range(1, 4):
d = dict() # dict recreated on every iteration
d['index'] = i
l.append(d)
print(l)
print([{'index': i} for i in range(1, 4)])
They both print:
[{'index': 1}, {'index': 2}, {'index': 3}]
The thing that does not work as expected:
d = dict() # dict created once only
l = list()
for i in range(1, 4):
d['index'] = i
l.append(d)
print(l)
Produces:
[{'index': 3}, {'index': 3}, {'index': 3}]
I'd have expected that the existing dictionary's value referred to by index would simply be overwritten on every pass and then added to the list and that I'd get a little performance improvement (in reality the dict is much larger).
It almost appears as if l.append just added references instead of passing values.
Am I missing something embarrassingly obvious?

"It almost appears as if l.append just added references instead of passing values.": That's it; you didn't miss anything.

Like other people said,Python will pass the reference.But you could do like:
for i in range(1, 4):
d['index'] = i
l.append(d.copy())
To get the result you want.

Function which returns dictionary overwriting all dictionaries

Brief description of code:
The main code first makes a blank dictionary, which is passed on to my function. The function tallies how many of each number and updates the dictionary which is then returned. However when the function executes, it overwrites the input 'blank_dictionary' to be the same as the dictionary it returns ('new_dictionary'). Why does this happen? I want the 'dictionary' in the main code to remain blank throughout so that it can be reused.
def index_list(lst, blank_dictionary):
new_dictionary = blank_dictionary
for i in lst:
new_dictionary[i] += 1
return new_dictionary
number = 1
maximum = 3
numbers = range(1,maximum+1)
dictionary = {}
for i in numbers:
dictionary[i] = 0
print ('original blank dictionary', dictionary)
new_dictionary = index_list([3,3,3],dictionary)
print ('new dictionary which indexed the list', new_dictionary)
print ('should still be blank, but isnt', dictionary)
Outputs:
original blank dictionary {1: 0, 2: 0, 3: 0}
new dictionary which indexed the list {1: 0, 2: 0, 3: 3}
should still be blank, but isnt {1: 0, 2: 0, 3: 3}
Thanks very much

You are setting new_dictionary to the reference to blank_dictionary. Change the line to new_dictionary = dict(blank_dictionary) and you will be fine. Using the dict() constructor function will make a new new_dictionary and so blank_dictionary will not be modified.
You might want to investigate the defaultdict in the collections module. If you only need to count the number of times each element appears, consider collections.counter.

This behavior is not limited to dicts. In Python, any time you pass a mutable object to a function, the function operates on the original object, not a copy. This is not true for immutable objects like tuples and strings.
However in this case there is no reason to pass a blank dictionary to the function in the first place. The function can create a new dictionary and return it.

Python best way to assume a value from dictionary?

I am reading a dictionary from external source, let's say
data = {'name': 'James', 'gender': 'male'}
And sometimes
data = {'name': 'James', 'gender': 'male', 'article': {'title':'abc'}}
And sometimes
data = {'name': 'James', 'gender': 'male', 'article': None}
I know that I can use .get(key, default) when I am not sure if articles exists in data:
articles = data.get('article', {}).get('title')
But sometimes, they provide the element with None value, so the above doesn't work and cause error, and need to become:
articles = data.get('article') or {}
But this requires me to break it into 2 statements instead of chaining up to get values from article as mentioned earlier.
Is there a more elegant way to do that, something like:
data.get('article', {}, ignore=[None])
or
data.get_ignore_none('article', {})

By default .get() will return None if the key doesn't exist. In your case you are returning an empty dictionary.
Now, I don't know what error is being raised, but I am sure its from get_stuff(article) rather than your list comprehension.
You have a few ways to solve this:
Modify get_stuff so that it takes the value directly, rather than each element. This way, you are just passing it [get_stuff(value) for value in data.get('articles')]. Now, in get_stuff, you simply do this:
def get_stuff(foo):
if not foo:
return None
for item in foo:
do stuff
return normal_results
Add a filter in your list comprehension:
[get_stuff(foo) for foo in data.get('articles') if data.get('articles')]

There's nothing wrong with using exceptions to break out early in this case. I'm assuming you want the title value, or None, no matter what the data is. The following function will work (for Python 3).
def get_title(d):
try:
return data.get("article").get("title")
except AttributeError:
return None
If the outer dictionary gets a None as value or by default it will raise the AttributeError on the None object which you just catch.

First off you seem to think that using an or expression to discard false-y results from data.get('article') can only be done in two statements like the following:
temp = data.get('article') or {}
articles = temp.get("title")
However you can just put brackets around the first expression and call .get("title") on it's return value directly:
articles = (data.get('article') or {}).get("title")
But I feel this is not particularly readable or efficient, when 'article' missing or None then you are creating a new mapping and checking it for "title" unnecessarily.
One possible solution is to use a function like the following:
def nested_get(mapping, *keys):
"""gets a value from a nested dictionary,
if any key along the way is missing or None then None is returned
will raise an AttributeError if a value in the chain is not a dictionary (support the .get method)"""
current = mapping
for item in keys:
current = current.get(item)
if current is None:
return None
return current
Then you would do nested_get(data, "article", "title") to try to get data["article"]["title"] without throwing errors if data["article"] is None or missing.
I tested this with the following code:
test_cases = [{'name': 'James', 'gender': 'male'},
{'name': 'James', 'gender': 'male', 'article': {'title':'abc'}},
{'name': 'James', 'gender': 'male', 'article': None}]
for case in test_cases:
print(case)
print(nested_get(case,"article","title"))
print()
#the follwing will raise an error since mapping["a"] would need to be a dict
nested_get({"a":5}, "a","b")

how about this
>>> data = {1:(42,23)}
>>> [x for x in data.get(1) or []]
[42, 23]
>>> [x for x in data.get(32) or []]
[]
use or to change to your default value in case you get None or something that eval to false
Edit:
In the same way you can or and brackets to get desired output in one line
articles = (data.get('article') or {}).get('title')
and with just that you handle all 3 cases.
you can also define get_ignore_none like for example
def get_ignore_none(data_dict, key, default=None):
if key in data_dict:
value = data_dict[key]
return value if value is not None else default
return default

Since you're loading this data from an external source, one option is a preprocessing step as soon as you load it:
from collections import Mapping
def remove_none(d):
for k, v in d.items():
if v is None:
del d[k]
if isinstance(v, Mapping):
remove_none(v)
data = load_data_from_somewhere()
remove_none(data)
Now you can just use get everywhere you need to:
articles = data.get('article', {}).get('title')

How to remove extra list from the result in python

Result has extra brackets, how can I remove them in python?
I am calling two methods where one returns a tuple of dictionaries and the second returns a list of tuples of dictionaries.
print method_A() // ({'id': 6}, {'id': 9})
print method_B() // [({'id': 6}, {'id': 9})]
How can I remove the list from the result of second method?
I tried it with type checking and has worked but I want to know if there is any easy to way to do it.
I tried following code:
resultA = method_A() // ({'id': 6}, {'id': 9})
resultB method_B() // [({'id': 6}, {'id': 9})]
if type(resultA) == list:
resultA = resultA[0]
if type(resultB) == list:
resultB = resultB[0]
or directly I can use resultB[0] if I know it

If one of these methods always returns a list with the tuple element, just use indexing:
resultA = method_A()
resultB = method_B()[0]
If either method sometimes returns a list object and sometimes just the tuple, use a function:
def unwrap(v):
v[0] if isinstance(v, list) else v
and use that:
resultA = unwrap(method_A())
resultB = unwrap(method_B())
Then contact whomever created those methods and talk to them sternly about consistency in API design.

Simple 1 line For Loops

Typically I have used list comprehensions to iterate and filter by data (i.e dicts etc) within the need to write multiple line for loops.
[x['a']["b"] for x in s["items"] if x["car"] == "ford"]
However this returns a list such as :
[False]
Not a massive problem as I can write
[x['a']["b"] for x in s["items"] if x["car"] == "ford"][0]
However is there a way either with list comprehensions or another way to write a for loop which an if condition so that I only get a string returned ?
Edit : In other words how can I place the following onto a single line and return a string,
for x in s["items"]:
if x["car"] == "ford":
print x['a']['b']
Thanks,

If I understand correctly, you want to short-circuit at the first match. Use next along with a generator expression:
>>> s = {'items': [{'a': {'b': 'potato'}, 'car': 'ford'}, {'a': {'b': 'spam'}, 'car': 'honda'}]}
>>> next(x['a']['b'] for x in s['items'] if x['car'] == "ford")
'potato'

As you have not shown that dict s, i have tested it with the possible data and it works fine:
>>> s = {'items': [{'a': {'b': 1}, 'car': 'ford'}, {'a': {'b': 1}, 'car': 'honda'}]}
>>> print [x['a']['b'] for x in s['items'] if x['car'] == "ford"]
[1]

There is nothing in the syntax of your problem that guarantees that there is only value in s that satisfies the criterion. (I.E., for an arbitrary dict s, there could be more than one.)
You may be able to guarantee that to be the case, but that is external to (this part of) the code.
Hence python isn't going to be able to automatically enforce that.