Develop Reference

Parse a json/dictionary with same key values

I currently have a list variable that looks like this:
list_of_dicts = [{"Away_Team":"KC", "Home_Team":"NYY"},
{"Away_Team":"TB", "Home_Team":"MIA"},
{"Away_Team":"TOR", "Home_Team":"BOS"},
]
As you can see, there are multiple keys with the same names, pertaining to the game matchups.
When I try to use:
print(json.dumps(list_of_dicts[0], indent=4, sort_keys=True))
...it only prints out the first matchup due to the same keys:
{
"Away_Team": "KC",
"Home_Team": "NYY"
}
How can I convert this list_of_dicts variable into something like the following output so I can use it like a valid dictionary or json object?
{
"Away_Team_1":"KC", "Home_Team_1":"NYY",
"Away_Team_2":"TB", "Home_Team_2":"MIA",
"Away_Team_3":"TOR", "Home_Team_3":"BOS",
}
This output doesn't need to be exactly that if a better solution is available, this is just to give you an idea of how I'd like to be able to parse the data.
The list_of_dicts variable can be of varying sizes, I've shown 3 here, but it could contain 1 or 10 matchups, it varies, so the solution needs to be dynamic to that.

You can add suffixes to the keys with enumerate:
list_of_dicts2 = [{f"{k}_{i}":v for k,v in d.items()} for i,d in enumerate(list_of_dicts, start=1)]

One option is to use pandas:
pd.DataFrame(list_of_dicts).to_csv('filename.csv', index=False)
gives
Away_Team,Home_Team
KC,NYY
TB,MIA
TOR,BOS
Now the index is implied by the row, and if you load it back in you'll have those indices. Pandas also supports to_json if you are hard set on using json though. You can even recover your original list from a dataframe using .to_dict(orient='records')

Data structure is important. You really don't need a dictionary for this. Simply reducing to a list of tuples the first slot always the away team, and the second the home team.
list_of_dicts = [{"Away_Team":"KC", "Home_Team":"NYY"},
{"Away_Team":"TB", "Home_Team":"MIA"},
{"Away_Team":"TOR", "Home_Team":"BOS"},
]
l = [tuple(l.values()) for l in list_of_dicts]
output:
[('KC', 'NYY'), ('TB', 'MIA'), ('TOR', 'BOS')]
The problem with your proposed solution is iterating through dicts where you don't know the key name is cumbersome, this solution makes the data structure easy to decipher, transform, or manipulate.

Tuple and numpy arrays in python

I have a list of Dictionaries, each consisting of arrays of strings and floats such as below:
Product1 = {
'Name': 'TLSK',
'Name2': 'B1940',
'Tagid': '23456222',
'Cord': np.array(['09:42:23', '9', '-55:52:32', '9']),
'Cord2': np.array([432.34, 222.115]),
'Ref': 'Siegman corp. 234',
'Exp': 22.0,
'CX': np.array([0.00430, 0.00069, 0.00094])
}
I sometimes need access to certain elements of Dictionary for further calculations. What I do is I first merge them as follows:
Products = (Product1, Product2, Product3, ....)
Then I use for loop to store a certain element of each Dictionary as follows
Expall=[]
for i in Products:
exp = i['Exp']
Expall.append(exp)
To me this seems like an inefficient/bad coding and I was wondering if there is a better way to do it. I am coming from IDL language and for instance, in IDL you could have access to that info without a for loop like this. Expall = Products[*]['Exp']
Most of the time, I even have to store the data first and I use pickle in Python to do that. Since I am a bit new to python and I heard few good things about pandas and etc, I wanted to see if there is a more efficient/quicker way to handle all this stuff.

Following could work (List comprehension):
if you have separated Producti,
[product['Exp'] for product in [Product1, Product2]]
if you have Products,
[product['Exp'] for product in Products]

How can I split a list of dictionaries in separate lists of dictionaries based on some condition?

I am new to python, and I am trying to split a list of dictionaries into separate lists of dictionaries based on some condition.
This is how my list looks like this:
[{'username': 'AnastasiadesCY',
'created_at': '2020-12-02 18:58:16',
'id': 1.33421029132062e+18,
'language': 'en',
'contenttype': 'text/plain',
'content': 'Pleased to participate to the international conference in support of the Lebanese people. Cypriot citizens, together with the Government 🇨🇾, have provided significant quantities of material assistance, from the day of the explosion until today.\n\n#Lebanon 🇱🇧'},
{'username': 'AnastasiadesCY',
'created_at': '2020-11-19 18:13:06',
'id': 1.32948788307022e+18,
'language': 'en',
'contenttype': 'text/plain',
'content': '#Cyprus stand ready to support all efforts towards a coordinated approach of vaccination strategies across Europe, that will prove instrumental in our fight against the pandemic.\n\nUnited Against #COVID19 \n\n#EUCO'},...
I would like to split and group all list's elements that have the same username into separate lists of dictionaries. The elements of the list - so each dictionary - are ordered by username.
Is there a way to loop over the dictionaries and append each element to a list until username in "item 1" is equal to username in "item 1 + 1" and so on?
Thank you for your help!

Finding the same thing works the best if we sort the list by it - then all the same names are next to each other.
But even after sorting, we don't need to do such things manually - there are already tools for that. :) - itertools.groupby documentation and a nice explanation how it works
from itertools import groupby
from operator import itemgetter
my_list.sort(key=itemgetter("username"))
result = {}
for username, group in groupby(my_list, key=itemgetter("username")):
result[username] = list(group)
result is a dict with usernames as keys
If you want a list-of-lists, do result = [] and then result.append(list(group)) instead.

A better would be to create a dictionary with username as key and value as list of user attributes
op = defauldict(list)
for user_dic in list_of_userdictss:
op[user_dic.pop('username')].append(user_dic)
op = OrderedDict(sorted(user_dic.items()))

list of dict to separate lists
data = pd.DataFrame(your_list_of_dict)
username_list = data.username.values.tolist()

Extract values from json-file which has no unique markers

A json-file which has unique markers (or [more appropriate] field-names) preceeding the values is (rather) easy to dissect, because you can perform a string search on the unique markers/field-names to find within the string the first and last position of the characters of the value, and with that info you can pinpoint the position of the value, and extract the value.
Have performed that function with various lua-scripts and Python-scripts (also on xml-files).
Now need to extract values from a json-file which does not have unique markers/ field-names, but just a multiple occurrence of "value_type" and "value", preceeding the 'name', respectively the 'value': see below.
{
"software_version": "NRZ-2017-099",
"age":"78",
"sensordatavalues":[
{"value_type":"SDS_P1","value":"4.43"},
{"value_type":"SDS_P2","value":"3.80"},
{"value_type":"temperature","value":"20.10"},
{"value_type":"humidity","value":"44.50"},
{"value_type":"samples","value":"614292"},
{"value_type":"min_micro","value":"233"},
{"value_type":"max_micro","value":"25951"},
{"value_type":"signal","value":"-66"}
]
}
Experience as described above does not provide working solution.
Question: In this json-filelayout, how to directly extract the specific, individual values (preferably by lua-script)?
[Or might XML-parsing provide an easier solution?]

Here is Python to read the JSON file and make it more convenient:
import json
import pprint
with open("/tmp/foo.json") as j:
data = json.load(j)
for sdv in data.pop('sensordatavalues'):
data[sdv['value_type']] = sdv['value']
pprint.pprint(data)
The results:
{'SDS_P1': '4.43',
'SDS_P2': '3.80',
'age': '78',
'humidity': '44.50',
'max_micro': '25951',
'min_micro': '233',
'samples': '614292',
'signal': '-66',
'software_version': 'NRZ-2017-099',
'temperature': '20.10'}

You might want to have a look into filter functions.
E.g. in your example json to get only the dict that contains the value for samples you could go by:
sample_sensordata = list(filter(lambda d: d["value_type"] == "samples", your_json_dict["sensordatavalues"]))
sample_value = sample_sensordata["value"]
To make a dictionary like Ned Batchelder said you could also go with a dict comprehension like this:
sensor_data_dict = {d['value_type']: d['value'] for d in a}
and then get the value you want just by sensor_data_dict['<ValueTypeYouAreLookingFor>']

A little bit late and I'm trying Anvil in which the previous answers didn't work. just for the curious people.
resp = anvil.http.request("http://<ipaddress>/data.json", json=True)
#print(resp) # prints json file
tempdict = resp['sensordatavalues'][2].values()
humiddict = resp['sensordatavalues'][3].values()
temperature = float(list(tempdict)[1])
humidity = float(list(humiddict)[1])
print(temperature)
print(humidity)

Finding a key value by referencing another key value in the same dict level in Python

I'm pretty new to Python, so I'm having a hard time even coming up with the proper jargon to describe my issue.
Basic idea is I have a dict that has the following structure:
myDict =
"SomeMetric":{
"day":[
{"date": "2013-01-01","value": 1234},
{"date": "2013-01-02","value": 5678},
etc...
I want to pull out the "value" where the date is known. So I want:
myDict["SomeMetric"]["day"]["value"] where myDict["SomeMetric"]["day"]["date"] = "2013-01-02"
Is there a nice one-line method for this without iterating through the whole dict as my dict is much larger, and I'm already iterating through it, so I'd rather not do nested iteritems.

Generator expressions to the resque:
next(d['value']
for d in myDict['SomeMetric']['day']
if d['date'] == "2013-01-02")
So, loop over all day dictionaries, and find the first one that matches the date you are looking for. This loop stops as soon as a match is found.

Do you have control over your data structure? It seems to be constructed in such a way that lends itself to sub-optimal lookups.
I'd structure it as such:
data = { 'metrics': { '2013-01-02': 1234, '2013-01-01': 4321 } }
And then your lookup is simply:
data['metrics']['2013-01-02']

Can you change the structure? If you can, you might find it much easier to change the day list to a dictionary which has dates as keys and values as values, so
myDict = {
"SomeMetric":{
"day":{
"2013-01-01": 1234,
"2013-01-02": 5678,
etc...
Then you can just index into it directly with
myDict["SomeMetric"]["day"]["2013-01-02"]