Develop Reference

making old keys the values for a new dictionary with list comprehension

I am trying to make a new list of dictionaries using list comprehension. I have an old list that has 'age' and 'email' keys, with their associated values. I am wanting to create a new list of dictionaries, where 'age' and 'email' are the VALUES of new keys called 'new_age', and 'new_email'.
How would I accomplish this?
entries = [{'age': 65, 'name': 'Tim', 'email': 'tim#bob.com'},{'age': 72, 'name': 'Andy', 'email': 'andy#bob.com'},{'age': 50, 'name': 'Bob', 'email': 'bob#bob.com'}, {'age': 30, 'name': 'Shelly', 'email': 'shelly#shelly.com'}]
x =[{dictionary['new_age'],dictionary['new_email']} for dictionary in entries if dictionary['age'] >= 50]
so my new list of dictionaries 'x' is supposed to make a new list of dictionaries if 'age' >= 50, and then I want just the 'age' and 'email' of that entry in a new dictionary.
so the form will look like this
[{'new_age': 65, 'new_email': bob#bob.com}, {}, etc]
my x list is just an example and it prints out the email, and age if age is above 50 but I need the key pairs in there as well, and this is where I am stuck.

{dictionary['new_age'],dictionary['new_email']} creates a set, not a dictionary (and it wouldn't work anyway because dictionary, which is an element of entries, doesn't contain the keys new_age and new_email)
To create a dictionary, you need key-value pairs like so:
[
{'new_email': dictionary['email'], 'new_age': dictionary['age']}
for dictionary in entries if dictionary['age'] >= 50
]
which gives what you're looking for:
[{'new_email': 'tim#bob.com', 'new_age': 65},
{'new_email': 'andy#bob.com', 'new_age': 72},
{'new_email': 'bob#bob.com', 'new_age': 50}]

You just need to fix how you're doing the dictionary comprehension
x = [
{'new_age': dictionary['age'], 'new_email': dictionary['email']}
for dictionary in entries if dictionary['age'] >= 50
]

Filtering through a list with embedded dictionaries

I've got a json format list with some dictionaries within each list, it looks like the following:
[{"id":13, "name":"Albert", "venue":{"id":123, "town":"Birmingham"}, "month":"February"},
{"id":17, "name":"Alfred", "venue":{"id":456, "town":"London"}, "month":"February"},
{"id":20, "name":"David", "venue":{"id":14, "town":"Southampton"}, "month":"June"},
{"id":17, "name":"Mary", "venue":{"id":56, "town":"London"}, "month":"December"}]
The amount of entries within the list can be up to 100. I plan to present the 'name' for each entry, one result at a time, for those that have London as a town. The rest are of no use to me. I'm a beginner at python so I would appreciate a suggestion in how to go about this efficiently. I initially thought it would be best to remove all entries that don't have London and then I can go through them one by one.
I also wondered if it might be quicker to not filter but to cycle through the entire json and select the names of entries that have the town as London.

You can use filter:
data = [{"id":13, "name":"Albert", "venue":{"id":123, "town":"Birmingham"}, "month":"February"},
{"id":17, "name":"Alfred", "venue":{"id":456, "town":"London"}, "month":"February"},
{"id":20, "name":"David", "venue":{"id":14, "town":"Southampton"}, "month":"June"},
{"id":17, "name":"Mary", "venue":{"id":56, "town":"London"}, "month":"December"}]
london_dicts = filter(lambda d: d['venue']['town'] == 'London', data)
for d in london_dicts:
print(d)
This is as efficient as it can get because:
The loop is written in C (in case of CPython)
filter returns an iterator (in Python 3), which means that the results are loaded to memory one by one as required

One way is to use list comprehension:
>>> data = [{"id":13, "name":"Albert", "venue":{"id":123, "town":"Birmingham"}, "month":"February"},
{"id":17, "name":"Alfred", "venue":{"id":456, "town":"London"}, "month":"February"},
{"id":20, "name":"David", "venue":{"id":14, "town":"Southampton"}, "month":"June"},
{"id":17, "name":"Mary", "venue":{"id":56, "town":"London"}, "month":"December"}]
>>> [d for d in data if d['venue']['town'] == 'London']
[{'id': 17,
'name': 'Alfred',
'venue': {'id': 456, 'town': 'London'},
'month': 'February'},
{'id': 17,
'name': 'Mary',
'venue': {'id': 56, 'town': 'London'},
'month': 'December'}]

Prevent Overwrite of Nested Dictionary Values while Creating Dictionary in a For Loop

I am very new to Python and am trying to write a script that loops through an existing dictionary and reorganizes the data so it is in a nested dictionary. The existing dictionary is created from some code I found online that turns a row from an SQL query into a dictionary. I know this is a bit redundant, I just don't know how to edit this code to make it display what I want.
Here's the script: https://geert.vanderkelen.org/2010/fetching-rows-as-dictionaries-with-mysql-connectorpython/
Anyway, when I do this, the nested dictionary automatically overwrites previous nested dictionaries even though the dictionary key for the nested dictionary is changing. I've searched and found some other StackOverflow questions with the same problem but have not been able to find a solution for my code.
Here's my relevant code:
row=curs.fetchone
d={}
D={}
while row is not None:
station=row[u'STATION']
date_obs=row[u'DATE_OBSERVATION']
temperature=row[u'TMPC']
altimeter=row[u'ALTM']
dewpoint=row[u'DWPC']
windspeed=row[u'SPED']
winddirection=row[u'DRCT']
for i in ('date_obs', 'temperature', 'altimeter', 'dewpoint', 'windspeed', 'winddirection'):
d[i]=locals()[i]
D[station]=d
row = curs.fetchone()
print D
I will get something like this (although with many more dictionary entries):
{u'KBFI': {date_obs': datetime.datetime(2017, 7, 19, 16, 56),
'temperature': Decimal('21.00' 'dewpoint: 'Decimal('4.00'),
'altimeter': Decimal('30.10'), 'windspeed': Decimal('3.00'),
'winddirection': Decimal('310.00')}, u'KKLS': {date_obs':
datetime.datetime(2017, 7, 19, 16, 56), 'temperature': Decimal('21.00'
'dewpoint: 'Decimal('4.00'), 'altimeter': Decimal('30.10'),
'windspeed': Decimal('3.00'), 'winddirection': Decimal('310.00')}}
And want something like:
{u'KBFI': {date_obs': datetime.datetime(2017, 7, 19, 16, 53),
'temperature': Decimal('19.00' 'dewpoint: 'Decimal('5.00'),
'altimeter': Decimal('30.06'), 'windspeed': Decimal('4.00'),
'winddirection': Decimal('270.00')}, u'KKLS': {date_obs':
datetime.datetime(2017, 7, 19, 16, 56), 'temperature': Decimal('21.00'
'dewpoint: 'Decimal('4.00'), 'altimeter': Decimal('30.10'),
'windspeed': Decimal('3.00'), 'winddirection': Decimal('310.00')}}

You create a d dictionary with:
d = {}
Then, in each loop, you create a new entry in your main D dictionary with:
D[station] = d
But each time, d refers to the very same dictionary. You have updated its values in each loop with d[i] = ...., but it still remains the same object.
So, when you print D, D['KBFI'] and D['KKLS'] refer to the same dictionary object, whose values are the ones you gave it in the last loop.
In Python parlance, we say that the dict objects are mutable: you can change what they contain without creating a new object.
In order to obtain the behaviour you want, you could create a new d in each loop:
while row is not None:
station = row[u'STATION']
# ....
d = {} # Here, we create a new dict
for i in ('date_obs', 'temperature', 'altimeter', 'dewpoint', 'windspeed', 'winddirection'):
d[i] = locals()[i]
D[station] = d # D[station] now refers to our new dict
But this hack with locals() is really ugly... You'd rather use:
D={}
row = curs.fetchone
while row is not None:
station = row[u'STATION']
D[station] = {'date_obs': row[u'DATE_OBSERVATION'],
'temperature': row[u'TMPC'],
'altimeter': row[u'ALTM'],
'dewpoint': row[u'DWPC'],
'windspeed': row[u'SPED'],
'winddirection': row[u'DRCT']
}
row = curs.fetchone()
print D
Or you could keep a table of correspondance between your key names and the field names, and rewrite your code as:
fields = {'date_obs': u'DATE_OBSERVATION', 'temperature': u'TMPC'} # and so on...
D={}
row = curs.fetchone
while row is not None:
station = row[u'STATION']
D[station] = { key: row[field] for key, field in fields.items() }
row = curs.fetchone()
print D
using a dictionary comprehension.

Use list of indices to manipulate a nested dictionary

I'm trying to perform operations on a nested dictionary (data retrieved from a yaml file):
data = {'services': {'web': {'name': 'x'}}, 'networks': {'prod': 'value'}}
I'm trying to modify the above using the inputs like:
{'services.web.name': 'new'}
I converted the above to a list of indices ['services', 'web', 'name']. But I'm not able to/not sure how to perform the below operation in a loop:
data['services']['web']['name'] = new
That way I can modify dict the data. There are other values I plan to change in the above dictionary (it is extensive one) so I need a solution that works in cases where I have to change, EG:
data['services2']['web2']['networks']['local'].
Is there a easy way to do this? Any help is appreciated.

You may iterate over the keys while moving a reference:
data = {'networks': {'prod': 'value'}, 'services': {'web': {'name': 'x'}}}
modification = {'services.web.name': 'new'}
for key, value in modification.items():
keyparts = key.split('.')
to_modify = data
for keypart in keyparts[:-1]:
to_modify = to_modify[keypart]
to_modify[keyparts[-1]] = value
print(data)
Giving:
{'networks': {'prod': 'value'}, 'services': {'web': {'name': 'new'}}}

dict.update overwrites existing keys, how to avoid?

When using the update function for a dictionary in python where you are merging two dictionaries and the two dictionaries have the same keys they are apparently being overwritten.
A simple example:
simple_dict_one = {'name': "tom", 'age': 20}
simple_dict_two = {'name': "lisa", 'age': 17}
simple_dict_one.update(simple_dict_two)
After the dicts are merged the following dict remains:
{'age': 17, 'name': 'lisa'}
So if you have the same key in both dict only one remains (the last one apparently).
If i have a lot of names for several sources i would probably want a temp dict from each of those and then want to add it to a whole bigger dict.
Is there a way to merge two dicts and still keep all the keys ? I guess you are only suppose to have one unique key but then how would i merge two dicts without loosing data

Well i have several sources i gather information from, for example an
ldap database and other sources where i have python functions that
create a temp dict each but i want a complete dict at the end that
sort of concatenates or displays all information gathered from all the
sources.. so i would have one dict holding all the info
What you are trying to do with the 'merging' is not quite ideal. As you said yourself
I guess you are only suppose to have one unique key
Which makes it relatively and unnecessarily hard to gather all your information in one dict.
What you could do, instead of calling .update() on the existing dict, is add a sub-dict. Its key could be the name of the source from which you gathered the information. The value could be the dict you receive from the source, and if you need to store more than 1 dict of the same source you can store them in a list.
Example
>>> data = {}
>>> person_1 = {'name': 'lisa', 'age': 17}
>>> person_2 = {'name': 'tom', 'age': 20}
>>> data['people'] = [person_1, person_2]
>>> data
{'people': [{'age': 17, 'name': 'lisa'}, {'age': 20, 'name': 'tom'}]}
Then whenever you need to add newly gathered information, you just add a new entry to the data dict
>>> ldap_data = {'foo': 1, 'bar': 'baz'} # just some dummy data
>>> data['ldap_data'] = ldap_data
>>> data
{'people': [{'age': 17, 'name': 'lisa'}, {'age': 20, 'name': 'tom'}],
'ldap_data': {'foo': 1, 'bar': 'baz'}}
The source-specific data is easily extractable from the data dict
>>> data['people']
[{'age': 17, 'name': 'lisa'}, {'age': 20, 'name': 'tom'}]
>>> data['ldap_data']
{'foo': 1, 'bar': 'baz'}