Develop Reference

Extracting data out of Complex JSON

This is the JSON data I am using
In this dataset, I want to extract the value of "yaw" and store it into in camera_loc. My first attempt for the code was as follows:
with open(
"/home/siddhant/catkin_ws/src/view-adaptation/multi_sensors.json"
) as sensors:
multi_sensors = json.load(sensors)
camera_loc = [
multi_sensors["sensors"][0]["yaw"],
multi_sensors["sensors"][1]["yaw"],
multi_sensors["sensors"][2]["yaw"],
multi_sensors["sensors"][3]["yaw"],
multi_sensors["sensors"][4]["yaw"],
multi_sensors["sensors"][5]["yaw"],
]
This gives me the expected result.
But I want to generalize the same for any number of entries in the 'sensors' array.
I tried executing a 'for' loop and extracting the values for the same as follows:
for i in multi_sensors["sensors"]:
camera_loc = []
camera_loc.append(i["yaw"])
However, this method only gives a single value in the camera_loc list which is the last 'yaw' value from the JSON file. I am looking for a better approach or even any modifications to the way I execute the loop so that I can extract all the values of 'yaw' from the JSON file - in this example there are 6 entries but I want to generalize it for 'n' entries that may be created in other cases.
Thank you!

That's because you're defining your camera_loc array inside the loop, meaning every iteration resets it to []. This code should work if you remove the array definition from the loop:
camera_loc = []
for i in multi_sensors["sensors"]:
camera_loc.append(i["yaw"])
This is also a perfect use case for a List Comprehension:
camera_loc = [i["yaw"] for i in multi_sensors["sensors"]]
Both answers results in the same array though, so you may choose whichever you like most.

Search tuple elements within in list

I have a list in Python as
list_data = [('a','b',5),('aa','bb',50)]
and some variables:
a = ('a','b','2')
c = ('aaa','bbb','500')
Now how can I search if a is already there in list_data?
If yes add 2 to the value of a, if not append to list_data?
The result should be as
list_data = [('a','b',7),('aa','bb',50),('aaa','bbb','500')]

Actually, this question is a good way to several demonstrate Pythonic ways of doing things. So lets see what we can do.
In order to check if something is in python list you can just use operator in:
if a in list_data:
do_stuff()
But what you ask is a bit different. You want to do something like a search by multiple keys, if I understand correctly. In this case you can 'trim' your tuple by discarding last entry.
Slicing is handy for this:
value_trimmed = value[:-1]
Now you can make a list of trimmed tuples:
list_trimmed = []
for a in list_data:
list_trimmed.append(a[:-1])
And then search there:
if a[:-1] in list_trimmed:
do_smth()
This list can be constructed in a less verbose way using list_comprehension:
list_trimmed = [item[:-1] for item in list_data]
To find where your item exactly is you can use index() method of list:
list_trimmed.index(a[:-1])
This will return index of a[:-1] first occurrence in list_trimmed or throw if it cant be found. We can avoid explicitly checking if item is in the list, and do the insertion only if the exception is caught.
Your full code will look like this:
list_data = [('a','b',5), ('aa','bb',50)]
values_to_find = [('a','b','2'), ('aaa','bbb','500')]
list_trimmed = [item[:-1] for item in list_data]
for val in values_to_find:
val_trimmed = val[:-1]
try:
ind = list_trimmed.index(val_trimmed)
src_tuple = list_data[ind]
# we can't edit tuple inplace, since they are immutable in python
list_data[ind] = (src_tuple[0], src_tuple[1], src_tuple[2]+2)
except ValueError:
list_data.append(val)
print list_data
Of course, if speed or memory-efficiency is your main concern this code is not very appropriate, but you haven't mentioned these in your question, and that is not what python really about in my opinion.
Edit:
You haven't specified what happens when you check for ('aaa','bbb','500') second time - should we use the updated list and increment matching tuple's last element, or should we stick to the original list and insert another copy?
If we use updated list, it is not clear how to handle incrementing string '500' by 2 (we can convert it to integer, but you should have constructed your query appropriately in the first place).
Or maybe you meant add last element of tuple being searched to the tuple in list if found ? Please edit your question to make it clear.

Can I condense these values with a loop?

I have a set of values that get modified like so:
iskDropped = irrelevant1
iskDestroyed = irrelevant2
iskTotal = irrelevant3
iskDropped = condense_value(int(iskDropped[:-7].replace(',','')))
iskDestroyed = condense_value(int(iskDestroyed[:-7].replace(',','')))
iskTotal = condense_value(int(iskTotal[:-7].replace(',','')))
As you can see, all three lines go through the same changes. (condensed, shortened, and commas removed) before overwriting their original value.
I want to condense those three lines if possible because it feels inefficient.
I was trying something like this:
for value in [iskDropped,iskDestroyed,iskTotal]:
value = condense_value(int(value[:-7].replace(',','')))
which if you changed into a print statement successfully does print the correct values but it does not work in the regard of overwriting / updating the values (iskDropped,iskDestroyed, and iskTotal) that I need to call later in the program.
Is it possible to condense these lines in Python? If so can someone point me in the right direction?

You can do it like this:
iskDropped, iskDestroyed, iskTotal = [condense_value(int(value[:-7].replace(',',''))) for value in [iskDropped, iskDestroyed, iskTotal]]
This works by looping through the list of your 3 variables, performing the condense_value function on each and creates a list of the results, then finally unpacks the list back into the original values.

Nested for loop index out of range

I'm coming up with a rather trivial problem, but since I'm quite new to python, I'm smashing my head to my desk for a while. (Hurts). Though I believe that's more a logical thing to solve...
First I have to say that I'm using the Python SDK for Cinema 4D so I had to change the following code a bit. But here is what I was trying to do and struggling with:
I'm trying to group some polygon selections, which are dynamically generated (based on some rules, not that important).
Here's how it works the mathematical way:
Those selections are based on islands (means, that there are several polygons connected).
Then, those selections have to be grouped and put into a list that I can work with.
Any polygon has its own index, so this one should be rather simple, but like I said before, I'm quite struggling there.
The main problem is easy to explain: I'm trying to access a non existent index in the first loop, resulting in an index out of range error. I tried evaluating the validity first, but no luck. For those who are familiar with Cinema 4D + Python, I will provide some of the original code if anybody wants that. So far, so bad. Here's the simplified and adapted code.
edit: Forgot to mention that the check which causes the error actually should only check for duplicates, so the current selected number will be skipped since it hal already been processed. This is necessary due to computing-heavy calculations.
Really hope, anybody can bump me in the right direction and this code makes sense so far. :)
def myFunc():
sel = [0,1,5,12] # changes with every call of "myFunc", for example to [2,8,4,10,9,1], etc. - list alway differs in count of elements, can even be empty, groups are beeing built from these values
all = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15] # the whole set
groups = [] # list to store indices-lists into
indices = [] # list to store selected indices
count = 0 # number of groups
tmp = [] # temporary list to copy the indices list into before resetting
for i in range(len(all)): # loop through values
if i not in groups[count]: # that's the problematic one; this one actually should check whether "i" is already inside of any list inside the group list, error is simply that I'm trying to check a non existent value
for index, selected in enumerate(sel): # loop through "sel" and return actual indices. "selected" determines, if "index" is selected. boolean.
if not selected: continue # pretty much self-explanatory
indices.append(index) # push selected indices to the list
tmp = indices[:] # clone list
groups.append(tmp) # push the previous generated list to another list to store groups into
indices = [] # empty/reset indices-list
count += 1 # increment count
print groups # debug
myFunc()
edit:
After adding a second list which will be filled by extend, not append that acts as counter, everything worked as expected! The list will be a basic list, pretty simple ;)

groups[count]
When you first call this, groups is an empty list and count is 0. You can't access the thing at spot 0 in groups, because there is nothing there!
Try making
groups = [] to groups = [[]] (i.e. instead of an empty list, a list of lists that only has an empty list).

I'm not sure why you'd want to add the empty list to groups. Perhaps this is better
if i not in groups[count]:
to
if not groups or i not in groups[count]:
You also don't need to copy the list if you're not going to use it for anything else. So you can replace
tmp = indices[:] # clone list
groups.append(tmp) # push the previous generated list to another list to store groups into
indices = [] # empty/reset indices-list
with
groups.append(indices) # push the previous generated list to another list to store groups into
indices = [] # empty/reset indices-list
You may even be able to drop count altogether (you can always use len(groups)). You can also replace the inner loop with a listcomprehension
def myFunc():
sel = [0,1,5,12] # changes with every call of "myFunc", for example to [2,8,4,10,9,1], etc. - list alway differs in count of elements, can even be empty, groups are beeing built from these values
all = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15] # the whole set
groups = [] # list to store indices-lists into
for i in range(len(all)): # loop through values
if not groups or i not in groups[-1]: # look in the latest group
indices = [idx for idx, selected in enumerate(sel) if selected]
groups.append(indices) # push the previous generated list to another list to store groups into
print groups # debug

correct line 11 from:
if i not in groups[count]
to:
if i not in groups:

Multiple values to one key in python

So my main goal is simple, I want multiple values to be returned when using a single key. However I'm getting errors and confusing behavior. I am new to Python, so I fully expect there to be a simple reason for this issue.
I have a list of objects, list which only contains the index of the object. i.e.
1
2
3
4
etc..
and a file containing the groups that each of the objects belong to, listed in the same order. The file is a single value for n lines (n being the length of the list of objects as well.) i.e. the file looks like this:
2
5
2
4
etc..
meaning the first object belongs in group 2, the second object in group 5, the third in group 2 and fourth in group 4. This file will change depending on my input files. I have attempted the two following suggestions (that I could find).
EDIT: my end goal: to have a dictionary with group numbers as the keys and the objects in the groups as the values.
I looked to this StackOverflow question first for help since it is so similar, and ended up with this code:
def createdDict(list, file):
f = open(file, 'r')
d={}
i=0
for line in f:
groupIndex = int(line)
if groupIndex in d:
d[groupIndex].append(list[i])
else:
d[groupIndex] = list[i]
i +=1
print d
f.close()
And this error:
AttributeError: 'Element' object has no attribute 'append'
d[groupIndex] is just a dictionary and its key and groupIndex should also just be an integer.. not an object from a class I created earlier in the script. (Why is this error showing up?)
I then revised my code after coming upon this other question to be like the following, since I thought this was an alternative way to accomplish my task. My code then looked like this:
def createdDict(list, file):
f = open(file, 'r')
d={}
i=0
for line in f:
groupIndex = int(line)
if groupIndex in d:
d.setdefault('groupIndex', []).append(list[i])
else:
d[groupIndex] = list[i]
i +=1
print d
f.close()
This code snippet doesn't end in an error or what I want, but rather (what I believe) are the last objects in the groups... so print d gives me the key and the last object placed in the group (instead of the desired: ALL of the objects in that group) and then terminal randomly spits out groupIndex followed by all of the objects in list.
My question: what exactly am I missing here? People upvoted the answers to the questions I linked, so they are most likely correct and I am likely implementing them incorrectly. I don't need a correction to both procedures, but the most efficient answer to my problem of getting multiple values attached to one key. What would be the most pythonic way to accomplish this task?
EDIT 2: if this helps at all, here is the class that the first method is referencing the error too. I have no idea how it defined any part of this code as a part of this class. I haven't really developed it yet, but I'm all for an answer, so if this helps in locating the error:
class Element(object):
def __init__(self, globalIndex):
self.globalIndex = globalIndex
def GetGlobalIndex (self):
return self.globalIndex
globalIndex is a separate index of objects (Elements). with my current problem, I am taking a list of these Elements (this is the list mentioned earlier) and grouping them into smaller groups based upon my file (also mentioned earlier). Why I thought it shouldn't matter, the list is essentially a counting up of integers... How would it mess with my code?

The base of your problem is in this line:
d[groupIndex] = list[i]
In other words, when a key is not in the dictionary, you add a single value (Element object) under that key. The next time you see that key, you try to append to that single value. But you can't append to single values. You need a container type, such as a list, to append. Python doesn't magically turn your Element object into a list!
The solution is simple. If you want your dictionary's values to be lists, then do that. When you add the first item, append a one-element list:
d[groupIndex] = [list[i]]
Alternatively, you can take a one-item slice of the original list. This will be a list already.
d[groupIndex] = list[i:i+1]
Now the dictionary's values are always lists, and you can append the second and subsequent values to them without error.
As ecatmur points out, you can further simplify this (eliminating the if statement) using d.setdefault(groupIndex, []).append(list[i]). If the key doesn't exist, then the value is taken to be an empty list, and you simply always append the new item. You could also use collections.defaultdict(list).

Just use
d.setdefault(groupIndex, []).append(list[i])
This will check whether groupIndex is in d, so you don't need the if groupIndex in d: line.

from itertools import izip
from collections import defaultdict
dd = defaultdict(list)
with open('filename.txt') as fin:
for idx, line in izip(my_list, fin):
num = int(line)
defaultdict[num].append(idx)
This creates a defaultdict with a default type of list, so you can append without using setdefault. Then reads each element of my_list combined with the corresponding line from the file, converts the line to an integer, then adds to the group (represented by num) the corresponding index.

In your first try, you seem to correctly understand that adding the first element to the dictionary item is a special case, and you cannot append yet, since the dictionary item has no value yet.
In your case you set it to list[i]. However, list[i] is not a list, so you cannot run append on it in later iterations.
I would do something like:
for line in f:
groupIndex = int(line)
try:
blah = d[groupIndex] # to check if it exists
except:
d[groupIndex] = [] # if not, assign empty list
d[groupIndex].append(list[i])
print d
f.close()