Python unexpected behaviour with list.append() - python

I was working with a machine learning model and I tried to append his output in a list for each element in antoher list.
I used a for loop but it seems like the loop jumps the first index and repeats the last ones:
x = [[0.12], [0.36], [0.48]]
print(model.run(x[0])) #this prints [-0.0006]
print(model.run(x[1])) #this prints [-0.0018]
print(model.run(x[2])) #this prints [-0.0024]
out_values = []
for value in x:
out_values.append(model.run(value))
print(out_values) #this should print [[-0.0012], [-0.0018], [-0.0024]]
# but it actually prints [[-0.0018], [-0.0024], [-0.0024]]
It doesn't seem to be a problem with the model.run() output since the first print statements worked perfectly
It must be something related to out_values.append(), because if I run:
x = [[0.12], [0.36], [0.48]]
out_values = []
out_values.append(model.run(x[0]))
out_values.append(model.run(x[1]))
out_values.append(model.run(x[2]))
print(out_values) # the result I get is still [[-0.0018], [-0.0024], [-0.0024]]
In my opinion it wasn't supposed to jump over model.run(x[0]) and repeat model.run(x[2]) twice
Is this documented or supposed to happen? Or am I doing something wrong?

I believe the issue is that model.run is returning a reference to mutable state within the model which is updated on successive run calls -- that's why there's a difference between what you print immediately after the run and what ends up in the list after additional calls. I.e. the value is correct at the time you call append, but it changes after the fact. When you call model.run(x[2]), it's modifying and returning the list that it returned when you called model.run(x[1]), and at some point it also modifies the list that was returned for x[0]. (This is not good behavior IMO -- if model is from an external library, hopefully it's at least documented in the API that you should not keep references to the return value of run! Otherwise it's just plain diabolical.)
To work around this problem so that you can keep each result as it was originally returned, make a copy of each result as you get it:
out_values = [model.run(value).copy() for value in x]

It will not skip any value or repeat any other value twice. Your model.run must be returning those values only. To remove that doubt, can you update the code to print the return value and print that like so:
for value in x:
ans = model.run(value)
print(ans)
out_values.append(ans)
Can you show output for above?

can you use this code instead?
x = (0.12, 0.36, 0.48)
print(x[0]) #this prints [-0.0006]
print(x[1]) #this prints [-0.0018]
print(x[2]) #this prints [-0.0024]
out_values = []
out_values = set(x)
print(out_values)
it'll include all x in the out_values

Related

Deep evaluation of variables

I have a need to use local().update to load some variables from configuration.
The problem is that there are dependencies between them, and i need to evaluate their real values.
For example if run this code:
vars={'x':'z','y':'4','z':'y'}
locals().update(vars)
print (eval(x))
the final result should be x=4, but the assigned value initially is x='y'
def deep(key):
keyChecked = []
while vars[key] in vars.keys() and key not in keyChecked:
keyChecked.append(key)
key = vars[key]
else:
return vars[key]
vars={'x':'z','y':'4','z':'y'}
print(deep('x'))
I think you will get expected answer.

Python: Emptying temporary list used in a loop without losing referenced information

I have following kind of data, which I want to collect to one big list so that each row in the big list has one of these below (c-d and e-f correspond to second and third of these). What I mean is that lista_results[0] includes the first of these and lista_results[1] the second etc.
(Name) (Amount-of-points)
Some name 12
Other name 19
... ...
I have a loop that takes information from other lists and then adds it to result list. The code is along the following lines:
lista_results = []
lista_temp = []
for y in range(0,10):
for x in range(0,10):
lista_temp.append(name_list[y][x], point_list[y][x])
lista_results.append(lista_temp)
#lista_temp[:] = []
Without the emptying, all are appended to same list every time x loop finishes. Basically what I get is:
[0]["Some name",12]
[0]["Other name",19]
[1]["Some name",12]
[1]["Other name",19]
[1][c,1c]
[1][d,2d]
[2]["Some name",12]
[2]["Other name",19]
[2][c,1c]
[2][d,2d]
[2][e,1e]
[2][f,2f]
...
But what I want is:
[0]["Some name",12]
[0]["Other name",19]
[1][c,1c]
[1][d,2d]
[2][e,1e]
[2][f,2f]
...
Which would mean that I would have to somehow empty the lista_temp after appending it to the lista_results. But when I do empty it (using #lista_temp[:] = []), it seems that also the parts that were appended to lista_results are emptied, or at least I get "IndexError" when trying to print out even lista_results[0][0][0]. I assume that bot reference to same list and emptying either empties also the other.
I also tried lista_newtemp=lista_temp and even then I got an IndexError after emptying lista_temp.
What would be a way to empty lista_temp after append without losing previous information already appended?
(Also, my first question here, I hope I did it well. I hope the problem explanation was clear. Feedback appreciated!)
It's really hard to tell from your output samples what data you're working on, but it sounds to me like you don't want to empty the list but create a new list for the next iteration. Something like:
lista_results = []
for y in range(0,10):
lista_temp = [] # we have a new lista_temp for every y
for x in range(0,10):
lista_temp.append(something_here)
lista_results.append(lista_temp)
The behaviour you see is because Python does not generally make implicit copies. Every lista_temp you appended into lista_results is the same object, which in turn kept changing.
It would be more Pythonic to use list comprehesions.
The inner list, lista_temp, can be created like this:
lista_temp = [something_here for x in range(0, 10)]
The outer list, lista_results, can be created like this:
lista_results = [make_lista_temp(y)
for y in range(0, 10)]
You can combine them, and lose the temporary variable lista_temp:
lista_results = [[something_here for x in range(0, 10)]
for y in range(0, 10)]
edit:
Your main problem was that you weren't copying lista_temp, so every element of lista_results points to the same instance that lista_temp points to. You could have copied it like this:
lista_results.append(list(lista_temp))
However, as above, there are better ways.

Python object.append from for loop

I want to append dictionary values to a list from a for loop, though it is only picking up the last value. Please advise, here is my code:
for line in obj:
test = float(line['value'])
print(test)
a = []
a.append(test)
Huh! You have messed up the indentation. Indentation is VERY IMPORTANT in python. a list is outside the for-loop hence it would only have the last value of test. IT should be inside the for-loop.
It should be like this -
a = []
for line in obj:
test = float(line['value'])
print(test)
a.append(test)
print a
Your call to append is outside the loop. Indentation matters in Python.

Why is python dictionary not refilling in while loop?

I've got a database that I loop through, and based on certain conditions, I send the relevant database entries to a dictionary that I created. I then want to pick one randomly from this dictionary, store it in a list, clear the dictionary, and do the whole thing again with incremented variables. FYI, the ['StartNoteNum'], etc. are just column names within the cost database.
What happens though, is that it works fine the first time thorough the loop, but if I try to clear the dictionary anywhere in the code (inside or outside the while loop), then the dictionary never re-fills based on the incremented values even though it should. To confirm that it should re-fill properly, I have set the initial values to be all possible values that it would encounter in the while loop, and each one works the first time through the loop, but fails once it tries to cycle. The error I get is that the random function can't pull from an empty dictionary. Grr... here is the code.
def compute_policy(clean_midi, cost_database):
note = 0 #Setting up starting variables.
total_score = []
current_finger = 1
path = [1]
next_move = {}
while note <= 2:
current_note = clean_midi[note] #get note-pair for scoring
dest_note = clean_midi[note+1]
for each in cost_database: #find all relevant DB entries
if (int(each['StartNoteNum']) == current_note
and int(each['DestNoteNum']) == dest_note
and int(each['StartFing']) == current_finger):
next_move[int(each['DestFing'])] = int(each['Score']) #move relevant entries to separate "bin"
policy_choice = random.choice(next_move.keys()) #choose one at random
total_score.append(next_move[policy_choice]) #track the scores for each choice in a list
path.append(policy_choice) #track the chosen finger
current_finger = policy_choice #update finger variable
note += 1
path.append(current_finger) #append last finger since the loop won't run again
return total_score, path
any help here would be much appreciated. Thanks.
You are trying to use the cost_database iterator twice. After the first time you pass through it is exhausted and thus the second time you try to use it the whole for loop is skipped because it got an empty iterator.
>>> a = xrange(4)
>>> for i in a:
... print(i)
0
1
2
3
4
>>> for i in a:
... print(i)
>>> # Nothing since 'a' is already exhausted.
One possible issue is:
int(each['DestFing']) is always the same, then the same key in the dictionary will get updated and count will stay 1.

Python: problem with list append

Here is my code -
cumulative_nodes_found_list = []
cumulative_nodes_found_total_list = []
no_of_runs = 10
count = 0
while count < no_of_runs:
#My program code
print 'cumulative_nodes_found_list - ' + str(cumulative_nodes_found_list)
cumulative_nodes_found_total_list.insert(count,cumulative_nodes_found_list)
print 'cumulative_nodes_found_total_list - ' + str(cumulative_nodes_found_total_list)
count = count + 1
Here is a part of the output -
#count = 0
cumulative_nodes_found_list - [0.0, 0.4693999, 0.6482, 0.6927999999, 0.7208999999, 0.7561999999, 0.783399999, 0.813999999, 0.8300999999, 0.8498, 0.8621999999]
cumulative_nodes_found_total_list - [[0.0, 0.4693999, 0.6482, 0.6927999999, 0.7208999999, 0.7561999999, 0.783399999, 0.813999999, 0.8300999999, 0.8498, 0.8621999999]]
#count = 1
cumulative_nodes_found_list - [0.0, 0.55979999999999996, 0.66220000000000001, 0.69479999999999997, 0.72040000000000004, 0.75380000000000003, 0.77629999999999999, 0.79679999999999995, 0.82979999999999998, 0.84850000000000003, 0.85760000000000003]
cumulative_nodes_found_total_list -[[0.0, 0.55979999999999996, 0.66220000000000001, 0.69479999999999997, 0.72040000000000004, 0.75380000000000003, 0.77629999999999999, 0.79679999999999995, 0.82979999999999998, 0.84850000000000003, 0.85760000000000003],
[0.0, 0.55979999999999996, 0.66220000000000001, 0.69479999999999997, 0.72040000000000004, 0.75380000000000003, 0.77629999999999999, 0.79679999999999995, 0.82979999999999998, 0.84850000000000003, 0.85760000000000003]]
As the new item is appended the old item is replaced by new item. This trend continues.
Can anyone tell me why this is happening. I have tried using 'append' in place of insert but got the same output. However when I use 'extend' I get the correct output but I need inner items as lists which I dont get with extend.
You need to rebind cumulative_nodes_found_list at the beginning of the loop, instead of just clearing it.
This is psychic debugging at its best, since you're effectively asking "what is wrong with my code, which I'm not going to show to you".
All I can do is assume.
I'm assuming you're re-using the array objects in memory.
In other words, you do something like this:
list1.insert(0, list2)
list2.clear()
list2.append(10)
list2.append(15)
list1.insert(0, list2)
Since list1 points to the same array/list the whole time, and you're adding a reference to the object, and not a copy of it, later changes will make it appear your copy changed.
In other words, the result of the code above is going to be:
[[10, 15], [10, 15]]
regardless of what was in the list before you added it the first time.
Try assigning the changing list a new, empty, object each time you enter the loop body and see if that fixes anything.
You are adding a reference to cumulative_nodes_found_list to the cumulative_nodes_found_total_list, but it's the same reference each time. Move this line into the loop body:
cumulative_nodes_found_list = []
Lists are mutable objects. You're mutating cumulative_nodes_found_list inside your code, so the object added to your total list in the previous run is also mutated, because they are the same object.
Either make a copy to insert in the total:
for count in xrange(no_of_runs):
# ...
cumulative_nodes_found_total_list.append(list(cumulative_nodes_found_list))
... or reset the list on each iteration:
for count in xrange(no_of_runs):
cumulative_nodes_found_list = [] # creates a NEW list for this iteration
# ...
cumulative_nodes_found_total_list.append(cumulative_nodes_found_list)
I believe the problem is in the rest of your program code.
The items in cummulative_nodes_found_list is being replaced in-place each time through the loop.
I assume you're doing something like this:
while count < no_of_runs:
cummulative_nodes_found_list.clear()
#fill up the list with values using whatever program logic you have
cummulative_nodes_found_list.append(1.1)
cummulative_nodes_found_list.append(2.1)
print 'cumulative_nodes_found_list - ' + str(cumulative_nodes_found_list)
cumulative_nodes_found_total_list.insert(count,cumulative_nodes_found_list)
print 'cumulative_nodes_found_total_list - ' + str(cumulative_nodes_found_total_list)
count = count + 1
if this is, infact, what you're doing, then instead of using 'clear()' to clear the list, create a new one:
ie, replace cummulative_nodes_found_list.clear() with
cummulative_nodes_found_list = []
My guess is that you are not assigning the cumulative_nodes_found_list to be a new list each time, but updating its contents instead. So each time around the loop you are adding the same list reference to the total list. Since the reference within the totals list is the same object, when you update this list the next time around the loop, it affects what you hoped was the last loops values.
If you want to append to a list, use mylist.append(item) instead.
Also, if you iterate a fixed number of times it's better to use a for loop:
for i in range(no_of_runs):
# do stuff
The idea is, that range(no_of_runs) generates the list [0, 1, 2, ..., 10] for no_of_runs = 10 and the loop then iterates over its values.
Edit: this doesn't solve the problem. Other answers in this thread do, however. It's just a comment on style.
This method worked for me. Just like you, I was trying to append/insert a list into another list.
cumulative_nodes_found_total_list.insert(count,cumulative_nodes_found_list)
But the old values were being appended by the new values. So instead I tried this -
cumulative_nodes_found_total_list.insert(count,cumulative_nodes_found_list[:])
"Assignment statements in Python do not copy objects, they create
bindings between a target and an object."
Use deepcopy (or copy)

Categories

Resources