Related
I hope, this is not a duplicate. I know what too many values to unpack means. I am returning two values, and trying to accept two values.
I am providing only a short part of the code, I hope it will be enough.
def test(all the arguments in function_parameters):
// do something
dfData.append([fileToCheck,5,";".join(faceNames),frameTime,";".join(faceDistances),";".join(faceLocations),";".join(gender),str(";".join(age)),str(";".join(expression))])
if len(face_locations) != 0:
keyPointsData.append([fileToCheck,time,str(";".join(encodings)),str(";".join(encodings)),time])
else:
keyPointsData.append([fileToCheck,time,"","",time])
return dfData, keyPointsData
#Start multiprocessing
#Pass variables to the function
function_parameters = zip(
images_to_check,
itertools.repeat(known_names),
itertools.repeat(known_face_encodings),
itertools.repeat(tolerance),
itertools.repeat(processImages),
itertools.repeat(processVideos),
itertools.repeat(fpstoprocess),
itertools.repeat(upsample),
itertools.repeat(algo),
itertools.repeat(onlydetection),
itertools.repeat(saveimagespath),
itertools.repeat(savefullimages),
itertools.repeat(savefaceimage),
itertools.repeat(enablebox),
itertools.repeat(maxarea),
listNumber,
itertools.repeat(totalImages),
itertools.repeat(imageExtensions),
itertools.repeat(videoExtensions),
itertools.repeat(debug),
itertools.repeat(age),
itertools.repeat(gender),
itertools.repeat(expression),
itertools.repeat(keypointsDF)
)
rows,keypointsData = pool.starmap(test, function_parameters)
tdfData and keyPointsData are multidimensional list. I am using Multi threads
I am getting error at this line rows,keypointsData = pool.starmap(test, function_parameters)
Full Error Message
Traceback (most recent call last):
File "face.py", line 829, in <module>
main()
File "face.py", line 702, in main
process_images_in_process_pool()
File "face.py", line 584, in process_images_in_process_pool
rows,keypointsData = pool.starmap(test, function_parameters)
ValueError: too many values to unpack (expected 2)
As per the official docs https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.starmap
starmap takes an iterator to call the function with different input from iterator and then return another iterator with all the output with their corresponding inputs.
So here take an example:
def test(a, b):
return a, b
now calling the function with iterator of different input:
iter1 = zip([1, 2], [3, 4])
list_of_results = pool.starmap(test, iter1)
>>> list_of_results
>>> [(2, 3), (4, 5)]
x, y = pool.starmap(test, iter1) # unpacking will work
But in case of iterator call more times than 2 unpacking with 2 variables will fail:
iter2 = zip([1, 2, 3], [4, 5, 6])
list_of_results = pool.starmap(test, iter2)
>>> list_of_results
>>> [(2, 5), (3, 6), (4, 7)]
x, y = pool.starmap(test, iter1) # unpacking will fail
Therefore first store the result in list_of_results and then iterate over it to use the output values to avoid unpacking issue.
Hope it will clear the doubt and issue
i would like to know how to convert a generator with only one element into list.For example if f is a generator with one element,list(f) will raise "not iterable".how to solve this problem ?thanks in advance!
nx.adamic_adar_index(g, (0, 1))
list(nx.adamic_adar_index(g, (0, 1)))
this will raise TypeError: 'int' object is not iterable,the following is OK
nx.adamic_adar_index(g, [(0, 1),(2,3)])
list(nx.adamic_adar_index(g, [(0, 1),(2,3)]))
the result is [(0, 1, 2.3878841007006875), (2, 3, 0.9282626109897467)]
If I understand correctly, this: [(0, 1),(2,3)] is a list of two elements (each being a tuple of two elements) but (0, 1) is a single element, could it be that you need to be a list of a single element, like [(0, 1)]?
I'd guess that:
list(nx.adamic_adar_index(g, [(0, 1)]))
will do what you want.
Could this be it?
The generator must be evaluated to be listed. You can't list the generator itself.
>>> def g():
... yield 1
...
>>> list(g())
[1]
>>> list(g)
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
TypeError: 'function' object is not iterable
type(3,) returns the int type, while
t = 3,
type(t)
returns the tuple type.
Why?
Inside the parentheses that form the function call operator, the comma is not for building tuples, but for separating arguments. Thus, type(3, ) is equivalent to type(3). An additional comma at the end of the argument list is allowed by the grammar. You need an extra pair of parens to build a tuple:
>>> def f(x):
... print x
...
>>> f(3)
3
>>> f(3,)
3
>>> f((3,))
(3,)
The builtin type() is a function, so the comma is being parsed as an argument separator rather than a tuple constructor.
>>> type(3,)
<type 'int'>
>>> type((3,))
<type 'tuple'>
I suspect Python ignores a trailing comma in function arguments:
def f (a):
print a
print type(a)
>>> f(3,)
3
<type 'int'>
Using comma-separated values without parentheses to create a tuple is a trick that doesn't work everywhere. List comprehensions is a good example:
>>> [a,a*2 for a in range(4)]
File "<stdin>", line 1
[a,a*2 for a in range(4)]
^
You have to do this:
>>> [(a,a*2) for a in range(4)]
[(0, 0), (1, 2), (2, 4), (3, 6)]
List unpacking works fine, though, so it's a bit variable where unbounded, comma-separated values are allowed:
>>> [(a,b) for a, b in zip(range(4),range(4))]
[(0, 0), (1, 1), (2, 2), (3, 3)]
In func(bunch_of_args) you are allowed to follow the last arg with a comma, just like in
alist = [1, 2, 3, ]
I am trying to get a list of list of tuples : something like [ [(1,0),(2,0),(3,0)],[(1,1),(2,1),(3,1)....]]
I used this statement
set([(a,b)for a in range(3)]for b in range(3))
But it gives me an error
TypeError: unhashable type: 'list'
I have 2 questions for the Python Guru's:
a) When I look at the Python definition of Hashable -
"An object is hashable if it has a hash value which never changes during its lifetime (it needs a hash() method)"
when I used dir function on the expression above
dir([(a,b)for a in range(3)]for b in range(3))
it seems to say the __hash__ is there. So, why do I get the error?
I was able to get [[(0, 0), (1, 0), (2, 0)], [(0, 1), (1, 1), (2, 1)], [(0, 2), (1, 2), (2, 2)]]
by using the list command :
list(list((a,b) for a in range(3)) for bin range(3))
b)list and set both takes Iterable as parameter. How come one works(list) and another doesn't (set)?
You are creating a set via set(...) call, and set needs hashable items. You can't have set of lists. Because list's arent hashable.
[[(a,b) for a in range(3)] for b in range(3)] is a list. It's not a hashable type. The __hash__ you saw in dir(...) isn't a method, it's just None.
A list comprehension returns a list, you don't need to explicitly use list there, just use:
>>> [[(a,b) for a in range(3)] for b in range(3)]
[[(0, 0), (1, 0), (2, 0)], [(0, 1), (1, 1), (2, 1)], [(0, 2), (1, 2), (2, 2)]]
Try those:
>>> a = {1, 2, 3}
>>> b= [1, 2, 3]
>>> type(a)
<class 'set'>
>>> type(b)
<class 'list'>
>>> {1, 2, []}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
>>> print([].__hash__)
None
>>> [[],[],[]] #list of lists
[[], [], []]
>>> {[], [], []} #set of lists
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
TLDR:
- You can't hash a list, a set, nor a dict to put that into sets
- You can hash a tuple to put it into a set.
Example:
>>> {1, 2, [3, 4]}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
>>> {1, 2, (3, 4)}
set([1, 2, (3, 4)])
Note that hashing is somehow recursive and the above holds true for nested items:
>>> {1, 2, 3, (4, [2, 3])}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
Dict keys also are hashable, so the above holds for dict keys too.
A list is unhashable because its contents can change over its lifetime. You can update an item contained in the list at any time.
A list doesn't use a hash for indexing, so it isn't restricted to hashable items.
The real reason because set does not work is the fact, that it uses the hash function to distinguish different values. This means that sets only allows hashable objects. Why a list is not hashable is already pointed out.
... and so you should do something like this:
set(tuple ((a,b) for a in range(3)) for b in range(3))
... and if needed convert back to list
You'll find that instances of list do not provide a __hash__ --rather, that attribute of each list is actually None (try print [].__hash__). Thus, list is unhashable.
The reason your code works with list and not set is because set constructs a single set of items without duplicates, whereas a list can contain arbitrary data.
s = [1,2,3,4,5,6,7,8,9]
n = 3
list(zip(*[iter(s)]*n)) # returns [(1,2,3),(4,5,6),(7,8,9)]
How does zip(*[iter(s)]*n) work? What would it look like if it was written with more verbose code?
iter() is an iterator over a sequence. [x] * n produces a list containing n quantity of x, i.e. a list of length n, where each element is x. *arg unpacks a sequence into arguments for a function call. Therefore you're passing the same iterator 3 times to zip(), and it pulls an item from the iterator each time.
x = iter([1,2,3,4,5,6,7,8,9])
print(list(zip(x, x, x)))
The other great answers and comments explain well the roles of argument unpacking and zip().
As Ignacio and ujukatzel say, you pass to zip() three references to the same iterator and zip() makes 3-tuples of the integers—in order—from each reference to the iterator:
1,2,3,4,5,6,7,8,9 1,2,3,4,5,6,7,8,9 1,2,3,4,5,6,7,8,9
^ ^ ^
^ ^ ^
^ ^ ^
And since you ask for a more verbose code sample:
chunk_size = 3
L = [1,2,3,4,5,6,7,8,9]
# iterate over L in steps of 3
for start in range(0,len(L),chunk_size): # xrange() in 2.x; range() in 3.x
end = start + chunk_size
print L[start:end] # three-item chunks
Following the values of start and end:
[0:3) #[1,2,3]
[3:6) #[4,5,6]
[6:9) #[7,8,9]
FWIW, you can get the same result with map() with an initial argument of None:
>>> map(None,*[iter(s)]*3)
[(1, 2, 3), (4, 5, 6), (7, 8, 9)]
For more on zip() and map(): http://muffinresearch.co.uk/archives/2007/10/16/python-transposing-lists-with-map-and-zip/
I think one thing that's missed in all the answers (probably obvious to those familiar with iterators) but not so obvious to others is -
Since we have the same iterator, it gets consumed and the remaining elements are used by the zip. So if we simply used the list and not the iter
eg.
l = range(9)
zip(*([l]*3)) # note: not an iter here, the lists are not emptied as we iterate
# output
[(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4), (5, 5, 5), (6, 6, 6), (7, 7, 7), (8, 8, 8)]
Using iterator, pops the values and only keeps remaining available, so for zip once 0 is consumed 1 is available and then 2 and so on. A very subtle thing, but quite clever!!!
iter(s) returns an iterator for s.
[iter(s)]*n makes a list of n times the same iterator for s.
So, when doing zip(*[iter(s)]*n), it extracts an item from all the three iterators from the list in order. Since all the iterators are the same object, it just groups the list in chunks of n.
One word of advice for using zip this way. It will truncate your list if it's length is not evenly divisible. To work around this you could either use itertools.izip_longest if you can accept fill values. Or you could use something like this:
def n_split(iterable, n):
num_extra = len(iterable) % n
zipped = zip(*[iter(iterable)] * n)
return zipped if not num_extra else zipped + [iterable[-num_extra:], ]
Usage:
for ints in n_split(range(1,12), 3):
print ', '.join([str(i) for i in ints])
Prints:
1, 2, 3
4, 5, 6
7, 8, 9
10, 11
Unwinding layers of "cleverness", you may find this equivalent spelling easier to follow:
x = iter(s)
for a, b, c in zip(*([x] * n)):
print(a, b, c)
which is, in turn, equivalent to the even less-clever:
x = iter(accounts_iter)
for a, b, c in zip(x, x, x):
print(a, b, c)
Now it should start to become clear. There is only a single iterator object, x. On each iteration, zip(), under the covers, calls next(x) 3 times, once for each iterator object passed to it. But it's the same iterator object here each time. So it delivers the first 3 next(x) results, and leaves the shared iterator object waiting to deliver its 4th result next. Lather, rinse, repeat.
BTW, I suspect you're parsing *([iter(x)]*n) incorrectly in your head. The trailing *n happens first, and then the prefix * is applied to the n-element list *n created. f(*iterable) is a shortcut for calling f() with a variable number of arguments, one for each object iterable delivers.
I needed to break down each partial step to really internalize how it is working. My notes from the REPL:
>>> # refresher on using list multiples to repeat item
>>> lst = list(range(15))
>>> lst
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
>>> # lst id value
>>> id(lst)
139755081359872
>>> [id(x) for x in [lst]*3]
[139755081359872, 139755081359872, 139755081359872]
# replacing lst with an iterator of lst
# It's the same iterator three times
>>> [id(x) for x in [iter(lst)]*3 ]
[139755085005296, 139755085005296, 139755085005296]
# without starred expression zip would only see single n-item list.
>>> print([iter(lst)]*3)
[<list_iterator object at 0x7f1b440837c0>, <list_iterator object at 0x7f1b440837c0>, <list_iterator object at 0x7f1b440837c0>]
# Must use starred expression to expand n arguments
>>> print(*[iter(lst)]*3)
<list_iterator object at 0x7f1b4418b1f0> <list_iterator object at 0x7f1b4418b1f0> <list_iterator object at 0x7f1b4418b1f0>
# by repeating the same iterator, n-times,
# each pass of zip will call the same iterator.__next__() n times
# this is equivalent to manually calling __next__() until complete
>>> iter_lst = iter(lst)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(0, 1, 2)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(3, 4, 5)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(6, 7, 8)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(9, 10, 11)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(12, 13, 14)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
# all together now!
# continuing with same iterator multiple times in list
>>> print(*[iter(lst)]*3)
<list_iterator object at 0x7f1b4418b1f0> <list_iterator object at 0x7f1b4418b1f0> <list_iterator object at 0x7f1b4418b1f0>
>>> zip(*[iter(lst)]*3)
<zip object at 0x7f1b43f14e00>
>>> list(zip(*[iter(lst)]*3))
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, 10, 11), (12, 13, 14)]
# NOTE: must use list multiples. Explicit listing creates 3 unique iterators
>>> [iter(lst)]*3 == [iter(lst), iter(lst), iter(lst)]
False
>>> list(zip(*[[iter(lst), iter(lst), iter(lst)]))
[(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3), ....
It is probably easier to see what is happening in python interpreter or ipython with n = 2:
In [35]: [iter("ABCDEFGH")]*2
Out[35]: [<iterator at 0x6be4128>, <iterator at 0x6be4128>]
So, we have a list of two iterators which are pointing to the same iterator object. Remember that iter on a object returns an iterator object and in this scenario, it is the same iterator twice due to the *2 python syntactic sugar. Iterators also run only once.
Further, zip takes any number of iterables (sequences are iterables) and creates tuple from i'th element of each of the input sequences. Since both iterators are identical in our case, zip moves the same iterator twice for each 2-element tuple of output.
In [41]: help(zip)
Help on built-in function zip in module __builtin__:
zip(...)
zip(seq1 [, seq2 [...]]) -> [(seq1[0], seq2[0] ...), (...)]
Return a list of tuples, where each tuple contains the i-th element
from each of the argument sequences. The returned list is truncated
in length to the length of the shortest argument sequence.
The unpacking (*) operator ensures that the iterators run to exhaustion which in this case is until there is not enough input to create a 2-element tuple.
This can be extended to any value of n and zip(*[iter(s)]*n) works as described.
x = [1,2,3,4,5,6,7,8,9]
zip(*[iter(x)] * 3)
is the same as:
x = [1,2,3,4,5,6,7,8,9]
iter_var = iter(x)
zip(iter_var,iter_var,iter_var)
Each time zip() gets the next value in iter_var it moves to the next value of x.
Try running next(iter_var) to see how this works.