Related
I have found below code in web, result is tuple of two elements in list, how to understand [iter(list)]*2?
lst = [1,2,3,4,5,6,7,8]
b=zip(*[iter(lst)]*2)
list(b)
[(1, 2), (3, 4), (5, 6), (7, 8)]
------------
[iter(lst)]*2
[<list_iterator at 0x1aff33917f0>, <list_iterator at 0x1aff33917f0>]
I check [iter(lst)]*2, same iterator above, so meaning iter repeat double,
so, if i check num from 2 to 3, result should be [(1, 2, 3), (4, 5, 6),(7,8,NaN)]
but delete 7,8
lst = [1,2,3,4,5,6,7,8]
b=zip(*[iter(lst)]*3)
list(b)
--------------
[(1, 2, 3), (4, 5, 6)]
Quite a tricky construct to explain. I'll give it a shot:
with [iter(lst)] you create a list with with one item. The item is an iterator over a list.
whenever python tries to get an element from this iterator, then the next element of lst is returned until no more element is available.
Just try following:
i = iter(lst)
next(i)
next(i)
the output should look like:
>>> lst = [1,2,3,4,5,6,7,8]
>>> i = iter(lst)
>>> next(i)
1
>>> next(i)
2
>>> next(i)
3
>>> next(i)
4
>>> next(i)
5
>>> next(i)
6
>>> next(i)
7
>>> next(i)
8
>>> next(i)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Now you create a list that contains twice exactly the same iterator.
You do this with
itlst = [iter(lst)] * 2
try out following:
itlst1 = [iter(lst)] * 2
itlst2 = [iter(lst), iter(lst)]
print(itlst1)
print(itlst2)
The result will look something like:
>>> itlst1 = [iter(lst)] * 2
>>> itlst2 = [iter(lst), iter(lst)]
>>> print(itlst1)
[<list_iterator object at 0x7f9251172b00>, <list_iterator object at 0x7f9251172b00>]
>>> print(itlst2)
[<list_iterator object at 0x7f9251172b70>, <list_iterator object at 0x7f9251172ba8>]
What is important to notice is, that itlst1 is a list containing twice the same iterator, whereas itlst2 contains two different iterators.
to illustrate try to type:
next(itlst1[0])
next(itlst1[1])
next(itlst1[0])
next(itlst1[1])
and compare it with:
next(itlst2[0])
next(itlst2[1])
next(itlst2[0])
next(itlst2[1])
The result is:
>>> next(itlst1[0])
1
>>> next(itlst1[1])
2
>>> next(itlst1[0])
3
>>> next(itlst1[1])
4
>>>
>>> next(itlst2[0])
1
>>> next(itlst2[1])
1
>>> next(itlst2[0])
2
>>> next(itlst2[1])
2
Now to the zip() function ( https://docs.python.org/3/library/functions.html#zip ):
Try following:
i = iter(lst)
list(zip(i, i))
zip() with two parameters.
Whenver you try to get the next element from zip it will do following:
get one value from the iterable that is the first parameter
get one value from the iterable that is the second parameter
return a tuple with these two values.
list(zip(xxx)) will do this repeatedly and store the result in a list.
The result will be:
>>> i = iter(lst)
>>> list(zip(i, i))
[(1, 2), (3, 4), (5, 6), (7, 8)]
The next trick being used is the * that is used to use the first element as first parameter to a function call, the second element as second parameter and so forth) What does ** (double star/asterisk) and * (star/asterisk) do for parameters?
so writing:
itlst1 = [iter(lst)] * 2
list(zip(*itlst1))
is in this case identical to
i = iter(lst)
itlst1 = [i] * 2
list(zip(itlst1[0], itlst1[1]))
which is identical to
list(zip(i, i))
which I explained already.
Hope this explains most of the above tricks.
iter(lst) turns a list into an iterator. Iterators let you step lazily through an iterable by calling next() until the iterator runs out of items.
[iter(lst)] puts the iterator into a single-element list.
[iter(lst)] * 2 makes 2 copies of the iterator in the list, giving
it = iter(lst)
[it, it]
Both list elements are aliases of the same underlying iterator object, so whenever next() is called on either of the iterators as zip exhausts them, successive elements are yielded.
*[...] unpacks the list of the two copies of the same iterator into the arguments for zip. This creates a zip object that lets you iterate through tuples of elements from each of its arguments.
list(...) iterates through the zip object and copies the elements into a list. Since both zipped iterators point to the same underlying iterator, we get the sequential elements seen in your output.
Without using the iterator alias, you'd get
>>> list(zip(iter(lst), iter(lst)))
[(1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8)]
A similar way to write list(zip(*[iter(lst)] * 2)) is list(zip(lst[::2], lst[1::2])), which seems a bit less magical (if much less performant).
The explanation for
>>> list(zip(*[iter(lst)] * 3))
[(1, 2, 3), (4, 5, 6)]
omitting elements is that the first time the zip object tries to yield a None result on any of the argument iterables, it stops and does not generate a tuple. You can use itertools.zip_longest to match your expected behavior, more or less:
>>> list(zip_longest(*[iter(lst)] * 3))
[(1, 2, 3), (4, 5, 6), (7, 8, None)]
See the canonical answer List of lists changes reflected across sublists unexpectedly if the [...] * 2 aliasing behavior is surprising.
I hope, this is not a duplicate. I know what too many values to unpack means. I am returning two values, and trying to accept two values.
I am providing only a short part of the code, I hope it will be enough.
def test(all the arguments in function_parameters):
// do something
dfData.append([fileToCheck,5,";".join(faceNames),frameTime,";".join(faceDistances),";".join(faceLocations),";".join(gender),str(";".join(age)),str(";".join(expression))])
if len(face_locations) != 0:
keyPointsData.append([fileToCheck,time,str(";".join(encodings)),str(";".join(encodings)),time])
else:
keyPointsData.append([fileToCheck,time,"","",time])
return dfData, keyPointsData
#Start multiprocessing
#Pass variables to the function
function_parameters = zip(
images_to_check,
itertools.repeat(known_names),
itertools.repeat(known_face_encodings),
itertools.repeat(tolerance),
itertools.repeat(processImages),
itertools.repeat(processVideos),
itertools.repeat(fpstoprocess),
itertools.repeat(upsample),
itertools.repeat(algo),
itertools.repeat(onlydetection),
itertools.repeat(saveimagespath),
itertools.repeat(savefullimages),
itertools.repeat(savefaceimage),
itertools.repeat(enablebox),
itertools.repeat(maxarea),
listNumber,
itertools.repeat(totalImages),
itertools.repeat(imageExtensions),
itertools.repeat(videoExtensions),
itertools.repeat(debug),
itertools.repeat(age),
itertools.repeat(gender),
itertools.repeat(expression),
itertools.repeat(keypointsDF)
)
rows,keypointsData = pool.starmap(test, function_parameters)
tdfData and keyPointsData are multidimensional list. I am using Multi threads
I am getting error at this line rows,keypointsData = pool.starmap(test, function_parameters)
Full Error Message
Traceback (most recent call last):
File "face.py", line 829, in <module>
main()
File "face.py", line 702, in main
process_images_in_process_pool()
File "face.py", line 584, in process_images_in_process_pool
rows,keypointsData = pool.starmap(test, function_parameters)
ValueError: too many values to unpack (expected 2)
As per the official docs https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.starmap
starmap takes an iterator to call the function with different input from iterator and then return another iterator with all the output with their corresponding inputs.
So here take an example:
def test(a, b):
return a, b
now calling the function with iterator of different input:
iter1 = zip([1, 2], [3, 4])
list_of_results = pool.starmap(test, iter1)
>>> list_of_results
>>> [(2, 3), (4, 5)]
x, y = pool.starmap(test, iter1) # unpacking will work
But in case of iterator call more times than 2 unpacking with 2 variables will fail:
iter2 = zip([1, 2, 3], [4, 5, 6])
list_of_results = pool.starmap(test, iter2)
>>> list_of_results
>>> [(2, 5), (3, 6), (4, 7)]
x, y = pool.starmap(test, iter1) # unpacking will fail
Therefore first store the result in list_of_results and then iterate over it to use the output values to avoid unpacking issue.
Hope it will clear the doubt and issue
I have a list of tuples as below in python:
Index Value
0 (1,2,3)
1 (2,5,4)
2 (3,3,3)
How can I select rows from this in which the second value is less than or equal 2?
EDIT:
Basically, the data is in the form of [(1,2,3), (2,5,4), (3,3,3)....]
You could slice the tuple by using apply:
df[df['Value'].apply(lambda x: x[1] <= 2)]
Seems, it was a list of tuples and not a DF:
To return a list back:
data = [item for item in [(1,2,3), (2,5,4), (3,3,3)] if item[1] <= 2]
# [(1, 2, 3)]
To return a series instead:
pd.Series(data)
#0 (1, 2, 3)
#dtype: object
df[df["Value"].str[1] <= 2]
You can read this for more details - http://pandas.pydata.org/pandas-docs/stable/text.html#indexing-with-str
just to put this out there. you should read mcve. your question was/is confusing because it looks as though you've got 2 good answers and your not satisfied. then you edited your question to be clearer, but just barely.
ok, now that i've got my editorial out of the way.
setup
this is what i'm assuming i'm working with
# list of tuples
lot = [(1, 2, 3), (2, 5, 4), (3, 3, 3)]
desired output
i think. btw, nothing to do with pandas at all
[(a, b, c) for a, b, c in lot if b <= 2]
[(1, 2, 3)]
with pandas
however, since you did tag this pandas
s = pd.Series(lot)
#TrigonaMinima's answer, give them credit if this works for you.
s[s.str[1].le(2)]
0 (1, 2, 3)
dtype: object
import numpy as np
x = [1,2,3,4,5,6,7,8,9]
y = [11,12,13,14,15,16,17,18,19]
I have a list of tuples as follows:
xy = [(x,y) for x,y in zip(x,y)]
Now I wanted to randomly select 3 position/index of the tuples in the list.
random_indices = np.random.choice(len(xy),3,replace=False)
Here I applied the indices to return the list of SELECTED tuples:
selected_xy = xy[random_indices]
print selected_xy
BUT I got the following error:
Traceback (most recent call last):
File "D:/test.py", line 11, in <module>
selected_xy = xy[random_indices]
TypeError: only integer arrays with one element can be converted to an index
My goal is to randomly select the tuples from the list, the expected result should look LIKE as follows:
[(1,11),(3,13),(4,14)]
What is the best way of doing it?
Convert xy to a NumPy array, lists only support indexing with integers:
>>> xy = np.array([(a, b) for a, b in zip(x, y)])
>>> random_indices = np.random.choice(len(xy), 3,replace=False)
>>> xy[random_indices]
array([[ 7, 17],
[ 2, 12],
[ 4, 14]])
In Python you can do this using either operator.itemgetter or loop over indices in a list comprehension and fetch the item at that index:
>>> from operator import itemgetter
>>> xy = [(a, b) for a, b in zip(x, y)]
>>> random_indices = np.random.choice(len(xy), 3, replace=False)
>>> itemgetter(*random_indices)(xy)
((2, 12), (1, 11), (4, 14))
s = [1,2,3,4,5,6,7,8,9]
n = 3
list(zip(*[iter(s)]*n)) # returns [(1,2,3),(4,5,6),(7,8,9)]
How does zip(*[iter(s)]*n) work? What would it look like if it was written with more verbose code?
iter() is an iterator over a sequence. [x] * n produces a list containing n quantity of x, i.e. a list of length n, where each element is x. *arg unpacks a sequence into arguments for a function call. Therefore you're passing the same iterator 3 times to zip(), and it pulls an item from the iterator each time.
x = iter([1,2,3,4,5,6,7,8,9])
print(list(zip(x, x, x)))
The other great answers and comments explain well the roles of argument unpacking and zip().
As Ignacio and ujukatzel say, you pass to zip() three references to the same iterator and zip() makes 3-tuples of the integers—in order—from each reference to the iterator:
1,2,3,4,5,6,7,8,9 1,2,3,4,5,6,7,8,9 1,2,3,4,5,6,7,8,9
^ ^ ^
^ ^ ^
^ ^ ^
And since you ask for a more verbose code sample:
chunk_size = 3
L = [1,2,3,4,5,6,7,8,9]
# iterate over L in steps of 3
for start in range(0,len(L),chunk_size): # xrange() in 2.x; range() in 3.x
end = start + chunk_size
print L[start:end] # three-item chunks
Following the values of start and end:
[0:3) #[1,2,3]
[3:6) #[4,5,6]
[6:9) #[7,8,9]
FWIW, you can get the same result with map() with an initial argument of None:
>>> map(None,*[iter(s)]*3)
[(1, 2, 3), (4, 5, 6), (7, 8, 9)]
For more on zip() and map(): http://muffinresearch.co.uk/archives/2007/10/16/python-transposing-lists-with-map-and-zip/
I think one thing that's missed in all the answers (probably obvious to those familiar with iterators) but not so obvious to others is -
Since we have the same iterator, it gets consumed and the remaining elements are used by the zip. So if we simply used the list and not the iter
eg.
l = range(9)
zip(*([l]*3)) # note: not an iter here, the lists are not emptied as we iterate
# output
[(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4), (5, 5, 5), (6, 6, 6), (7, 7, 7), (8, 8, 8)]
Using iterator, pops the values and only keeps remaining available, so for zip once 0 is consumed 1 is available and then 2 and so on. A very subtle thing, but quite clever!!!
iter(s) returns an iterator for s.
[iter(s)]*n makes a list of n times the same iterator for s.
So, when doing zip(*[iter(s)]*n), it extracts an item from all the three iterators from the list in order. Since all the iterators are the same object, it just groups the list in chunks of n.
One word of advice for using zip this way. It will truncate your list if it's length is not evenly divisible. To work around this you could either use itertools.izip_longest if you can accept fill values. Or you could use something like this:
def n_split(iterable, n):
num_extra = len(iterable) % n
zipped = zip(*[iter(iterable)] * n)
return zipped if not num_extra else zipped + [iterable[-num_extra:], ]
Usage:
for ints in n_split(range(1,12), 3):
print ', '.join([str(i) for i in ints])
Prints:
1, 2, 3
4, 5, 6
7, 8, 9
10, 11
Unwinding layers of "cleverness", you may find this equivalent spelling easier to follow:
x = iter(s)
for a, b, c in zip(*([x] * n)):
print(a, b, c)
which is, in turn, equivalent to the even less-clever:
x = iter(accounts_iter)
for a, b, c in zip(x, x, x):
print(a, b, c)
Now it should start to become clear. There is only a single iterator object, x. On each iteration, zip(), under the covers, calls next(x) 3 times, once for each iterator object passed to it. But it's the same iterator object here each time. So it delivers the first 3 next(x) results, and leaves the shared iterator object waiting to deliver its 4th result next. Lather, rinse, repeat.
BTW, I suspect you're parsing *([iter(x)]*n) incorrectly in your head. The trailing *n happens first, and then the prefix * is applied to the n-element list *n created. f(*iterable) is a shortcut for calling f() with a variable number of arguments, one for each object iterable delivers.
I needed to break down each partial step to really internalize how it is working. My notes from the REPL:
>>> # refresher on using list multiples to repeat item
>>> lst = list(range(15))
>>> lst
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
>>> # lst id value
>>> id(lst)
139755081359872
>>> [id(x) for x in [lst]*3]
[139755081359872, 139755081359872, 139755081359872]
# replacing lst with an iterator of lst
# It's the same iterator three times
>>> [id(x) for x in [iter(lst)]*3 ]
[139755085005296, 139755085005296, 139755085005296]
# without starred expression zip would only see single n-item list.
>>> print([iter(lst)]*3)
[<list_iterator object at 0x7f1b440837c0>, <list_iterator object at 0x7f1b440837c0>, <list_iterator object at 0x7f1b440837c0>]
# Must use starred expression to expand n arguments
>>> print(*[iter(lst)]*3)
<list_iterator object at 0x7f1b4418b1f0> <list_iterator object at 0x7f1b4418b1f0> <list_iterator object at 0x7f1b4418b1f0>
# by repeating the same iterator, n-times,
# each pass of zip will call the same iterator.__next__() n times
# this is equivalent to manually calling __next__() until complete
>>> iter_lst = iter(lst)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(0, 1, 2)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(3, 4, 5)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(6, 7, 8)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(9, 10, 11)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
(12, 13, 14)
>>> ((iter_lst.__next__(), iter_lst.__next__(), iter_lst.__next__()))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
# all together now!
# continuing with same iterator multiple times in list
>>> print(*[iter(lst)]*3)
<list_iterator object at 0x7f1b4418b1f0> <list_iterator object at 0x7f1b4418b1f0> <list_iterator object at 0x7f1b4418b1f0>
>>> zip(*[iter(lst)]*3)
<zip object at 0x7f1b43f14e00>
>>> list(zip(*[iter(lst)]*3))
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, 10, 11), (12, 13, 14)]
# NOTE: must use list multiples. Explicit listing creates 3 unique iterators
>>> [iter(lst)]*3 == [iter(lst), iter(lst), iter(lst)]
False
>>> list(zip(*[[iter(lst), iter(lst), iter(lst)]))
[(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3), ....
It is probably easier to see what is happening in python interpreter or ipython with n = 2:
In [35]: [iter("ABCDEFGH")]*2
Out[35]: [<iterator at 0x6be4128>, <iterator at 0x6be4128>]
So, we have a list of two iterators which are pointing to the same iterator object. Remember that iter on a object returns an iterator object and in this scenario, it is the same iterator twice due to the *2 python syntactic sugar. Iterators also run only once.
Further, zip takes any number of iterables (sequences are iterables) and creates tuple from i'th element of each of the input sequences. Since both iterators are identical in our case, zip moves the same iterator twice for each 2-element tuple of output.
In [41]: help(zip)
Help on built-in function zip in module __builtin__:
zip(...)
zip(seq1 [, seq2 [...]]) -> [(seq1[0], seq2[0] ...), (...)]
Return a list of tuples, where each tuple contains the i-th element
from each of the argument sequences. The returned list is truncated
in length to the length of the shortest argument sequence.
The unpacking (*) operator ensures that the iterators run to exhaustion which in this case is until there is not enough input to create a 2-element tuple.
This can be extended to any value of n and zip(*[iter(s)]*n) works as described.
x = [1,2,3,4,5,6,7,8,9]
zip(*[iter(x)] * 3)
is the same as:
x = [1,2,3,4,5,6,7,8,9]
iter_var = iter(x)
zip(iter_var,iter_var,iter_var)
Each time zip() gets the next value in iter_var it moves to the next value of x.
Try running next(iter_var) to see how this works.