Develop Reference

Python: Too Many values to unpack on return

I hope, this is not a duplicate. I know what too many values to unpack means. I am returning two values, and trying to accept two values.
I am providing only a short part of the code, I hope it will be enough.
def test(all the arguments in function_parameters):
// do something
dfData.append([fileToCheck,5,";".join(faceNames),frameTime,";".join(faceDistances),";".join(faceLocations),";".join(gender),str(";".join(age)),str(";".join(expression))])
if len(face_locations) != 0:
keyPointsData.append([fileToCheck,time,str(";".join(encodings)),str(";".join(encodings)),time])
else:
keyPointsData.append([fileToCheck,time,"","",time])
return dfData, keyPointsData
#Start multiprocessing
#Pass variables to the function
function_parameters = zip(
images_to_check,
itertools.repeat(known_names),
itertools.repeat(known_face_encodings),
itertools.repeat(tolerance),
itertools.repeat(processImages),
itertools.repeat(processVideos),
itertools.repeat(fpstoprocess),
itertools.repeat(upsample),
itertools.repeat(algo),
itertools.repeat(onlydetection),
itertools.repeat(saveimagespath),
itertools.repeat(savefullimages),
itertools.repeat(savefaceimage),
itertools.repeat(enablebox),
itertools.repeat(maxarea),
listNumber,
itertools.repeat(totalImages),
itertools.repeat(imageExtensions),
itertools.repeat(videoExtensions),
itertools.repeat(debug),
itertools.repeat(age),
itertools.repeat(gender),
itertools.repeat(expression),
itertools.repeat(keypointsDF)
)
rows,keypointsData = pool.starmap(test, function_parameters)
tdfData and keyPointsData are multidimensional list. I am using Multi threads
I am getting error at this line rows,keypointsData = pool.starmap(test, function_parameters)
Full Error Message
Traceback (most recent call last):
File "face.py", line 829, in <module>
main()
File "face.py", line 702, in main
process_images_in_process_pool()
File "face.py", line 584, in process_images_in_process_pool
rows,keypointsData = pool.starmap(test, function_parameters)
ValueError: too many values to unpack (expected 2)

As per the official docs https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.starmap
starmap takes an iterator to call the function with different input from iterator and then return another iterator with all the output with their corresponding inputs.
So here take an example:
def test(a, b):
return a, b
now calling the function with iterator of different input:
iter1 = zip([1, 2], [3, 4])
list_of_results = pool.starmap(test, iter1)
>>> list_of_results
>>> [(2, 3), (4, 5)]
x, y = pool.starmap(test, iter1) # unpacking will work
But in case of iterator call more times than 2 unpacking with 2 variables will fail:
iter2 = zip([1, 2, 3], [4, 5, 6])
list_of_results = pool.starmap(test, iter2)
>>> list_of_results
>>> [(2, 5), (3, 6), (4, 7)]
x, y = pool.starmap(test, iter1) # unpacking will fail
Therefore first store the result in list_of_results and then iterate over it to use the output values to avoid unpacking issue.
Hope it will clear the doubt and issue

passing each element of a list as an argument to a function

I have a list
try_list = ["[('aaaa', 34), ('bbbb', 1), ('cccc', 1)]", "[('dddd', 4), ('eeee', 1)]"]
Each entry within "" is an element of a list.
When I say
for i in try_list:
print i
I get the individual elements,
[('aaaa', 34), ('bbbb', 1), ('cccc', 1)]
[('dddd', 4), ('eeee', 1)]
But if I try to pass each individual to a function (say func) as
for i in try_list:
item = func(i)
I get an error.
If I test the function func with each element, one by one, it works fine so func is fine. So the error is in passing the lists as arguments, one at a time. What is the correct way of doing this?
Here is func:
def func(my_list):
ans = []
for i,name in enumerate(my_list):
print name[0],name[1]
ans.append(name)
return ans
If I say
func([('aaaa', 34), ('bbbb', 1), ('cccc', 1)])
I get the correct output
aaaa 34
bbbb 1
cccc 1
[('aaaa', 34), ('bbbb', 1), ('cccc', 1)]
But if I say
for i in try_list:
item = func(i)
I get the error
Traceback (most recent call last):
File "", line 2, in
File "", line 4, in func
IndexError: string index out of range

Quite (un)fortunately, the return value of __repr__ for the list [('aaaa', 34)] is the same as the string "[('aaaa', 34)]".
That's why you have print outputting something that looks like a list. But what you have is not a list of lists but a list of strings.
>>> print("[('aaaa', 34)]")
[('aaaa', 34)]
>>>
>>> print([('aaaa', 34)])
[('aaaa', 34)]
You probably want to convert your string into a list before passing as parameter to your function.
You can have a look at ast.literal_eval if you need to safely make the conversion from the string you have into a list.

Accessing attributes of class instances passed as optional arguments

I have two class instances which have list attributes, e.g.:
foo.range = [1,2]
bar.range = [3,4]
I also have a function which takes multiple arguments:
def permutations(*args):
return list(itertools.product(arg.range for arg in args))
I want permutations(foo, bar) to return all permutations of those two (or more) lists (i.e. [(1,3), (1,4) …]) but I actually get [([1, 2],), ([3, 4],)]
Could someone please help me to understand where am I going wrong and how to achieve the result I'm hoping for?

itertools.product expects the iterables as individual arguments(itertools.product(*iterables[, repeat])), so you need to unpack the items to it using the the splat(*) operator:
>>> def permutations(*args):
return list(itertools.product(*(arg.range for arg in args)))
>>> permutations(foo, bar)
[(1, 3), (1, 4), (2, 3), (2, 4)]
Consider this simple function, it will collect all positional arguments passed to it in args:
>>> def func(*args):
print args
print list(args[0])
...
When we pass it a generator expression it is collected as one individual item in the args tuple:
>>> func(x for x in range(5))
(<generator object <genexpr> at 0x7fda2ab4e780>,)
[0, 1, 2, 3, 4]
To fix this we need to unpack our generator expression, also as somethine like *x for x in range(5) is not a valid syntax in Python, we need to add extra parenthesis around the generator expression:
>>> func(*(x for x in range(5)))
(0, 1, 2, 3, 4)
# Failed as expected for args[0]
Traceback (most recent call last):
File "<ipython-input-215-13af4340dad1>", line 1, in <module>
func(*(x for x in range(5)))
File "<ipython-input-213-08941e2a3c06>", line 3, in func
print list(args[0])
TypeError: 'int' object is not iterable

Cant understand some python tuple syntax

I started learning python today and found this very nice code visualization tool pythontutor.com, the problem is that I still don't quite get some of the syntax on the example code.
def listSum(numbers):
if not numbers:
return 0
else:
(f, rest) = numbers
return f + listSum(rest)
myList = (1, (2, (3, None)))
total = listSum(myList)
What does (f, rest) = numbers means?

It's tuple unpacking.
There needs to be 2 items in the tuple when used in this way. More or less will result in an exception, as shown below.
>>> numbers = (1, 2, 3, 4, 5)
>>> (f, rest) = numbers
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: too many values to unpack
>>> numbers = (1, 2)
>>> (f, rest) = numbers
>>> print f
1
>>> print rest
2
>>> numbers = (1)
>>> (f, rest) = numbers
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
>>> numbers = (1,)
>>> (f, rest) = numbers
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: need more than 1 value to unpack
Note that (1) and (1, )are syntactically different, with only the latter being a tuple.
See the Python Doc on Tuples and Sequences for more details.

(f, rest) = numbers
unpacks the tuple. That is, it takes the two values stored in numbers and stores them in f and rest, respectively. Note that the number of variables you unpack into must be the same as the number of values in the tuple, or else an exception will be thrown.

Tupple is a data structure in which you can store multiple items under one name.
Lets say that we have a tupple(t) with two items.
Then t[0] = first_item and t[1] = sencond_item
Another way of accessing the tupple item is:
(f, rest) = numbers
In this syntax numbers (tupple) must have 2 items only otherwise it is an exception
f = numbers[0]
rest = numbers[1]

TypeError : Unhashable type

I am trying to get a list of list of tuples : something like [ [(1,0),(2,0),(3,0)],[(1,1),(2,1),(3,1)....]]
I used this statement
set([(a,b)for a in range(3)]for b in range(3))
But it gives me an error
TypeError: unhashable type: 'list'
I have 2 questions for the Python Guru's:
a) When I look at the Python definition of Hashable -
"An object is hashable if it has a hash value which never changes during its lifetime (it needs a hash() method)"
when I used dir function on the expression above
dir([(a,b)for a in range(3)]for b in range(3))
it seems to say the __hash__ is there. So, why do I get the error?
I was able to get [[(0, 0), (1, 0), (2, 0)], [(0, 1), (1, 1), (2, 1)], [(0, 2), (1, 2), (2, 2)]]
by using the list command :
list(list((a,b) for a in range(3)) for bin range(3))
b)list and set both takes Iterable as parameter. How come one works(list) and another doesn't (set)?

You are creating a set via set(...) call, and set needs hashable items. You can't have set of lists. Because list's arent hashable.
[[(a,b) for a in range(3)] for b in range(3)] is a list. It's not a hashable type. The __hash__ you saw in dir(...) isn't a method, it's just None.
A list comprehension returns a list, you don't need to explicitly use list there, just use:
>>> [[(a,b) for a in range(3)] for b in range(3)]
[[(0, 0), (1, 0), (2, 0)], [(0, 1), (1, 1), (2, 1)], [(0, 2), (1, 2), (2, 2)]]
Try those:
>>> a = {1, 2, 3}
>>> b= [1, 2, 3]
>>> type(a)
<class 'set'>
>>> type(b)
<class 'list'>
>>> {1, 2, []}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
>>> print([].__hash__)
None
>>> [[],[],[]] #list of lists
[[], [], []]
>>> {[], [], []} #set of lists
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

TLDR:
- You can't hash a list, a set, nor a dict to put that into sets
- You can hash a tuple to put it into a set.
Example:
>>> {1, 2, [3, 4]}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
>>> {1, 2, (3, 4)}
set([1, 2, (3, 4)])
Note that hashing is somehow recursive and the above holds true for nested items:
>>> {1, 2, 3, (4, [2, 3])}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
Dict keys also are hashable, so the above holds for dict keys too.

A list is unhashable because its contents can change over its lifetime. You can update an item contained in the list at any time.
A list doesn't use a hash for indexing, so it isn't restricted to hashable items.

The real reason because set does not work is the fact, that it uses the hash function to distinguish different values. This means that sets only allows hashable objects. Why a list is not hashable is already pointed out.

... and so you should do something like this:
set(tuple ((a,b) for a in range(3)) for b in range(3))
... and if needed convert back to list

You'll find that instances of list do not provide a __hash__ --rather, that attribute of each list is actually None (try print [].__hash__). Thus, list is unhashable.
The reason your code works with list and not set is because set constructs a single set of items without duplicates, whereas a list can contain arbitrary data.