I need to iterate over a circular list, possibly many times, each time starting with the last visited item.
The use case is a connection pool. A client asks for connection, an iterator checks if pointed-to connection is available and returns it, otherwise loops until it finds one that is available.
How can I do this neatly in Python?
If you instead need an immediately created list of the results up to a certain length, rather than iterating on demand: see Repeat list to max number of elements for general techniques, and How to replicate array to specific length array for Numpy-specific techniques.
Use itertools.cycle, that's its exact purpose:
from itertools import cycle
lst = ['a', 'b', 'c']
pool = cycle(lst)
for item in pool:
print item,
Output:
a b c a b c ...
(Loops forever, obviously)
In order to manually advance the iterator and pull values from it one by one, simply call next(pool):
>>> next(pool)
'a'
>>> next(pool)
'b'
The correct answer is to use itertools.cycle. But, let's assume that library function doesn't exist. How would you implement it?
Use a generator:
def circular():
while True:
for connection in ['a', 'b', 'c']:
yield connection
Then, you can either use a for statement to iterate infinitely, or you can call next() to get the single next value from the generator iterator:
connections = circular()
next(connections) # 'a'
next(connections) # 'b'
next(connections) # 'c'
next(connections) # 'a'
next(connections) # 'b'
next(connections) # 'c'
next(connections) # 'a'
#....
Or you can do like this:
conn = ['a', 'b', 'c', 'd', 'e', 'f']
conn_len = len(conn)
index = 0
while True:
print(conn[index])
index = (index + 1) % conn_len
prints a b c d e f a b c... forever
you can accomplish this with append(pop()) loop:
l = ['a','b','c','d']
while True:
print l[0]
l.append(l.pop(0))
or for i in range() loop:
l = ['a','b','c','d']
ll = len(l)
while True:
for i in range(ll):
print l[i]
or simply:
l = ['a','b','c','d']
while True:
for i in l:
print i
all of which print:
>>>
a
b
c
d
a
b
c
d
...etc.
of the three I'd be prone to the append(pop()) approach as a function
servers = ['a','b','c','d']
def rotate_servers(servers):
servers.append(servers.pop(0))
return servers
while True:
servers = rotate_servers(servers)
print servers[0]
If you wish to cycle n times, implement the ncycles itertools recipe:
from itertools import chain, repeat
def ncycles(iterable, n):
"Returns the sequence elements n times"
return chain.from_iterable(repeat(tuple(iterable), n))
list(ncycles(["a", "b", "c"], 3))
# ['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c']
You need a custom iterator -- I'll adapt the iterator from this answer.
from itertools import cycle
class ConnectionPool():
def __init__(self, ...):
# whatever is appropriate here to initilize
# your data
self.pool = cycle([blah, blah, etc])
def __iter__(self):
return self
def __next__(self):
for connection in self.pool:
if connection.is_available: # or however you spell it
return connection
In order to avoid infinite loop, I have used length of array to iterate only until size of list is double.You can implement your own pre condition .Idea is to avoid infinite loop.
#Implement Circular Linked List
from itertools import cycle
list=[1,2,3,4,5]
lstlength=len(list)*2
print(lstlength)
pool=cycle(list)
i=0
#To avoid infinite loop break when you have iterated twice size of the list
for items in pool:
print(items)
if i >lstlength:
break
i += 1
class A(object):
def __init__(self, l):
self.strt = 0
self.end = len(l)
self.d = l
def __iter__(self):
return self
def __next__(self):
val = None
if self.strt>=self.end:
self.strt=0
val = self.d[self.strt]
self.strt += 1
return val
a= A([8,9,7,66])
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
For those, who may be interested.
To loop forward or backward starting from given index:
def loop_fwd(arr, index):
while True:
arr_index = index % len(arr)
yield arr_index, arr[arr_index]
index += 1
def loop_bcw(arr, index):
while True:
arr_index = index % len(arr)
yield arr_index, arr[arr_index]
index -= 1
forward_it = loop_fwd([1,2,3,4,5], 3)
backward_it = loop_bcw([1,2,3,4,5], 3)
print('forward:')
for i in range(10):
print(next(forward_it))
print('backward:')
for i in range(10):
print(next(backward_it))
Related
In python, How do I do something like:
for car in cars:
# Skip first and last, do work for rest
To skip the first element in Python you can simply write
for car in cars[1:]:
# Do What Ever you want
or to skip the last elem
for car in cars[:-1]:
# Do What Ever you want
You can use this concept for any sequence (not for any iterable though).
The other answers only work for a sequence.
For any iterable, to skip the first item:
itercars = iter(cars)
next(itercars)
for car in itercars:
# do work
If you want to skip the last, you could do:
itercars = iter(cars)
# add 'next(itercars)' here if you also want to skip the first
prev = next(itercars)
for car in itercars:
# do work on 'prev' not 'car'
# at end of loop:
prev = car
# now you can do whatever you want to do to the last one on 'prev'
The best way to skip the first item(s) is:
from itertools import islice
for car in islice(cars, 1, None):
pass
# do something
islice in this case is invoked with a start-point of 1, and an end point of None, signifying the end of the iterable.
To be able to skip items from the end of an iterable, you need to know its length (always possible for a list, but not necessarily for everything you can iterate on). for example, islice(cars, 1, len(cars)-1) will skip the first and last items in cars.
Here is a more general generator function that skips any number of items from the beginning and end of an iterable:
def skip(iterable, at_start=0, at_end=0):
it = iter(iterable)
for x in itertools.islice(it, at_start):
pass
queue = collections.deque(itertools.islice(it, at_end))
for x in it:
queue.append(x)
yield queue.popleft()
Example usage:
>>> list(skip(range(10), at_start=2, at_end=2))
[2, 3, 4, 5, 6, 7]
This code skips the first and the last element of the list:
for item in list_name[1:-1]:
#...do whatever
Here's my preferred choice. It doesn't require adding on much to the loop, and uses nothing but built in tools.
Go from:
for item in my_items:
do_something(item)
to:
for i, item in enumerate(my_items):
if i == 0:
continue
do_something(item)
Example:
mylist=['one','two','three','four','five']
for i in mylist[1:]:
print(i)
In python index start from 0, We can use slicing operator to make manipulations in iteration.
for i in range(1,-1):
Well, your syntax isn't really Python to begin with.
Iterations in Python are over he contents of containers (well, technically it's over iterators), with a syntax for item in container. In this case, the container is the cars list, but you want to skip the first and last elements, so that means cars[1:-1] (python lists are zero-based, negative numbers count from the end, and : is slicing syntax.
So you want
for c in cars[1:-1]:
do something with c
Based on #SvenMarnach 's Answer, but bit simpler and without using deque
>>> def skip(iterable, at_start=0, at_end=0):
it = iter(iterable)
it = itertools.islice(it, at_start, None)
it, it1 = itertools.tee(it)
it1 = itertools.islice(it1, at_end, None)
return (next(it) for _ in it1)
>>> list(skip(range(10), at_start=2, at_end=2))
[2, 3, 4, 5, 6, 7]
>>> list(skip(range(10), at_start=2, at_end=5))
[2, 3, 4]
Also Note, based on my timeit result, this is marginally faster than the deque solution
>>> iterable=xrange(1000)
>>> stmt1="""
def skip(iterable, at_start=0, at_end=0):
it = iter(iterable)
it = itertools.islice(it, at_start, None)
it, it1 = itertools.tee(it)
it1 = itertools.islice(it1, at_end, None)
return (next(it) for _ in it1)
list(skip(iterable,2,2))
"""
>>> stmt2="""
def skip(iterable, at_start=0, at_end=0):
it = iter(iterable)
for x in itertools.islice(it, at_start):
pass
queue = collections.deque(itertools.islice(it, at_end))
for x in it:
queue.append(x)
yield queue.popleft()
list(skip(iterable,2,2))
"""
>>> timeit.timeit(stmt = stmt1, setup='from __main__ import iterable, skip, itertools', number = 10000)
2.0313770640908047
>>> timeit.timeit(stmt = stmt2, setup='from __main__ import iterable, skip, itertools, collections', number = 10000)
2.9903135454296716
An alternative method:
for idx, car in enumerate(cars):
# Skip first line.
if not idx:
continue
# Skip last line.
if idx + 1 == len(cars):
continue
# Real code here.
print car
The more_itertools project extends itertools.islice to handle negative indices.
Example
import more_itertools as mit
iterable = 'ABCDEFGH'
list(mit.islice_extended(iterable, 1, -1))
# Out: ['B', 'C', 'D', 'E', 'F', 'G']
Therefore, you can elegantly apply it slice elements between the first and last items of an iterable:
for car in mit.islice_extended(cars, 1, -1):
# do something
Similar to #maninthecomputer 's answer, when you need to skip the first iteration of a loop based on an int (self._model.columnCount() in my case):
for col in range(self._model.columnCount()):
if col == 0:
continue
Put more simply:
test_int = 3
for col in range(test_int):
if col == 0:
continue
print(col)
Provides output:
1
2
3
Good solution for support of itertools.chain is to use itertools.islice in order to take a slice of an iterable:
your_input_list = ['list', 'of', 'things']
for i, variant in list(itertools.islice(enumerate(some_function_that_will_output_itertools_chain(your_input_list)), 1, None)):
"""
# No need for unnecessary conditions like this:
if i == 0:
continue
"""
variant = list(variant) # (optional) converting back to list
print(variant)
I do it like this, even though it looks like a hack it works every time:
ls_of_things = ['apple', 'car', 'truck', 'bike', 'banana']
first = 0
last = len(ls_of_things)
for items in ls_of_things:
if first == 0
first = first + 1
pass
elif first == last - 1:
break
else:
do_stuff
first = first + 1
pass
I need to iterate over a circular list, possibly many times, each time starting with the last visited item.
The use case is a connection pool. A client asks for connection, an iterator checks if pointed-to connection is available and returns it, otherwise loops until it finds one that is available.
How can I do this neatly in Python?
If you instead need an immediately created list of the results up to a certain length, rather than iterating on demand: see Repeat list to max number of elements for general techniques, and How to replicate array to specific length array for Numpy-specific techniques.
Use itertools.cycle, that's its exact purpose:
from itertools import cycle
lst = ['a', 'b', 'c']
pool = cycle(lst)
for item in pool:
print item,
Output:
a b c a b c ...
(Loops forever, obviously)
In order to manually advance the iterator and pull values from it one by one, simply call next(pool):
>>> next(pool)
'a'
>>> next(pool)
'b'
The correct answer is to use itertools.cycle. But, let's assume that library function doesn't exist. How would you implement it?
Use a generator:
def circular():
while True:
for connection in ['a', 'b', 'c']:
yield connection
Then, you can either use a for statement to iterate infinitely, or you can call next() to get the single next value from the generator iterator:
connections = circular()
next(connections) # 'a'
next(connections) # 'b'
next(connections) # 'c'
next(connections) # 'a'
next(connections) # 'b'
next(connections) # 'c'
next(connections) # 'a'
#....
Or you can do like this:
conn = ['a', 'b', 'c', 'd', 'e', 'f']
conn_len = len(conn)
index = 0
while True:
print(conn[index])
index = (index + 1) % conn_len
prints a b c d e f a b c... forever
you can accomplish this with append(pop()) loop:
l = ['a','b','c','d']
while True:
print l[0]
l.append(l.pop(0))
or for i in range() loop:
l = ['a','b','c','d']
ll = len(l)
while True:
for i in range(ll):
print l[i]
or simply:
l = ['a','b','c','d']
while True:
for i in l:
print i
all of which print:
>>>
a
b
c
d
a
b
c
d
...etc.
of the three I'd be prone to the append(pop()) approach as a function
servers = ['a','b','c','d']
def rotate_servers(servers):
servers.append(servers.pop(0))
return servers
while True:
servers = rotate_servers(servers)
print servers[0]
If you wish to cycle n times, implement the ncycles itertools recipe:
from itertools import chain, repeat
def ncycles(iterable, n):
"Returns the sequence elements n times"
return chain.from_iterable(repeat(tuple(iterable), n))
list(ncycles(["a", "b", "c"], 3))
# ['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c']
You need a custom iterator -- I'll adapt the iterator from this answer.
from itertools import cycle
class ConnectionPool():
def __init__(self, ...):
# whatever is appropriate here to initilize
# your data
self.pool = cycle([blah, blah, etc])
def __iter__(self):
return self
def __next__(self):
for connection in self.pool:
if connection.is_available: # or however you spell it
return connection
In order to avoid infinite loop, I have used length of array to iterate only until size of list is double.You can implement your own pre condition .Idea is to avoid infinite loop.
#Implement Circular Linked List
from itertools import cycle
list=[1,2,3,4,5]
lstlength=len(list)*2
print(lstlength)
pool=cycle(list)
i=0
#To avoid infinite loop break when you have iterated twice size of the list
for items in pool:
print(items)
if i >lstlength:
break
i += 1
class A(object):
def __init__(self, l):
self.strt = 0
self.end = len(l)
self.d = l
def __iter__(self):
return self
def __next__(self):
val = None
if self.strt>=self.end:
self.strt=0
val = self.d[self.strt]
self.strt += 1
return val
a= A([8,9,7,66])
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
print(next(a))
For those, who may be interested.
To loop forward or backward starting from given index:
def loop_fwd(arr, index):
while True:
arr_index = index % len(arr)
yield arr_index, arr[arr_index]
index += 1
def loop_bcw(arr, index):
while True:
arr_index = index % len(arr)
yield arr_index, arr[arr_index]
index -= 1
forward_it = loop_fwd([1,2,3,4,5], 3)
backward_it = loop_bcw([1,2,3,4,5], 3)
print('forward:')
for i in range(10):
print(next(forward_it))
print('backward:')
for i in range(10):
print(next(backward_it))
I’m having trouble writing a generator function that takes an iterable and one more parameter which is an integer x. It outputs every value except for the last x values. It doesn’t know how to count how many values the iterable outputs.
I don’t know how to do this using a while loop as well as iter. I also need to use a comprehension that creates a list to store x values at most.
Lets say we call :
for i in func_function(“abcdefghijk”,5):
print(i,end =”)
It should print abcdef.
Here's what I've tried:
def func_function(iterable, x):
while True:
l = []
for x in iter(iterable):
if len(x) == x:
yield x
The trick is to turn this from lookahead into lookbehind.
I'd do this by iterating over the input and maintaining a window of the most recent n elements:
def except_last_n(iterable, n):
last_n = []
for val in iterable:
last_n.append(val)
if len(last_n) > n:
yield last_n.pop(0)
for val in except_last_n(range(10), 3):
print(val)
Rewriting this as a while loop and iter is left as exercise for the reader.
def except_last_n(iterable, n):
last_n = [val for val in iterable]
if len(last_n) > n:
yield last_n.pop(0)
from collections import deque
def drop_last_few(iterable, x=5):
it = iter(iterable)
data = deque(maxlen=x)
data.extend([next(it) for i in range(x)])
for val in it:
yield data[0]
data.append(val)
This uses a double-ended queue as storage to cache at most x elements. Demo:
>>> print(*drop_last_few("abcdefghijk", 5))
a b c d e f
Strings are sliceable:
def func_function(iterable, x):
yield from iterable[:-x]
print(func_function("asdfgkjbewqrfgkjb",8))
k = list(func_function("asdfgkjbewqrfgkjb",8))
print(k) # ['a', 's', 'd', 'f', 'g', 'k', 'j', 'b', 'e']
The while loop, iter and l=[] are not needed...
In python, How do I do something like:
for car in cars:
# Skip first and last, do work for rest
To skip the first element in Python you can simply write
for car in cars[1:]:
# Do What Ever you want
or to skip the last elem
for car in cars[:-1]:
# Do What Ever you want
You can use this concept for any sequence (not for any iterable though).
The other answers only work for a sequence.
For any iterable, to skip the first item:
itercars = iter(cars)
next(itercars)
for car in itercars:
# do work
If you want to skip the last, you could do:
itercars = iter(cars)
# add 'next(itercars)' here if you also want to skip the first
prev = next(itercars)
for car in itercars:
# do work on 'prev' not 'car'
# at end of loop:
prev = car
# now you can do whatever you want to do to the last one on 'prev'
The best way to skip the first item(s) is:
from itertools import islice
for car in islice(cars, 1, None):
pass
# do something
islice in this case is invoked with a start-point of 1, and an end point of None, signifying the end of the iterable.
To be able to skip items from the end of an iterable, you need to know its length (always possible for a list, but not necessarily for everything you can iterate on). for example, islice(cars, 1, len(cars)-1) will skip the first and last items in cars.
Here is a more general generator function that skips any number of items from the beginning and end of an iterable:
def skip(iterable, at_start=0, at_end=0):
it = iter(iterable)
for x in itertools.islice(it, at_start):
pass
queue = collections.deque(itertools.islice(it, at_end))
for x in it:
queue.append(x)
yield queue.popleft()
Example usage:
>>> list(skip(range(10), at_start=2, at_end=2))
[2, 3, 4, 5, 6, 7]
This code skips the first and the last element of the list:
for item in list_name[1:-1]:
#...do whatever
Here's my preferred choice. It doesn't require adding on much to the loop, and uses nothing but built in tools.
Go from:
for item in my_items:
do_something(item)
to:
for i, item in enumerate(my_items):
if i == 0:
continue
do_something(item)
Example:
mylist=['one','two','three','four','five']
for i in mylist[1:]:
print(i)
In python index start from 0, We can use slicing operator to make manipulations in iteration.
for i in range(1,-1):
Well, your syntax isn't really Python to begin with.
Iterations in Python are over he contents of containers (well, technically it's over iterators), with a syntax for item in container. In this case, the container is the cars list, but you want to skip the first and last elements, so that means cars[1:-1] (python lists are zero-based, negative numbers count from the end, and : is slicing syntax.
So you want
for c in cars[1:-1]:
do something with c
Based on #SvenMarnach 's Answer, but bit simpler and without using deque
>>> def skip(iterable, at_start=0, at_end=0):
it = iter(iterable)
it = itertools.islice(it, at_start, None)
it, it1 = itertools.tee(it)
it1 = itertools.islice(it1, at_end, None)
return (next(it) for _ in it1)
>>> list(skip(range(10), at_start=2, at_end=2))
[2, 3, 4, 5, 6, 7]
>>> list(skip(range(10), at_start=2, at_end=5))
[2, 3, 4]
Also Note, based on my timeit result, this is marginally faster than the deque solution
>>> iterable=xrange(1000)
>>> stmt1="""
def skip(iterable, at_start=0, at_end=0):
it = iter(iterable)
it = itertools.islice(it, at_start, None)
it, it1 = itertools.tee(it)
it1 = itertools.islice(it1, at_end, None)
return (next(it) for _ in it1)
list(skip(iterable,2,2))
"""
>>> stmt2="""
def skip(iterable, at_start=0, at_end=0):
it = iter(iterable)
for x in itertools.islice(it, at_start):
pass
queue = collections.deque(itertools.islice(it, at_end))
for x in it:
queue.append(x)
yield queue.popleft()
list(skip(iterable,2,2))
"""
>>> timeit.timeit(stmt = stmt1, setup='from __main__ import iterable, skip, itertools', number = 10000)
2.0313770640908047
>>> timeit.timeit(stmt = stmt2, setup='from __main__ import iterable, skip, itertools, collections', number = 10000)
2.9903135454296716
An alternative method:
for idx, car in enumerate(cars):
# Skip first line.
if not idx:
continue
# Skip last line.
if idx + 1 == len(cars):
continue
# Real code here.
print car
The more_itertools project extends itertools.islice to handle negative indices.
Example
import more_itertools as mit
iterable = 'ABCDEFGH'
list(mit.islice_extended(iterable, 1, -1))
# Out: ['B', 'C', 'D', 'E', 'F', 'G']
Therefore, you can elegantly apply it slice elements between the first and last items of an iterable:
for car in mit.islice_extended(cars, 1, -1):
# do something
Similar to #maninthecomputer 's answer, when you need to skip the first iteration of a loop based on an int (self._model.columnCount() in my case):
for col in range(self._model.columnCount()):
if col == 0:
continue
Put more simply:
test_int = 3
for col in range(test_int):
if col == 0:
continue
print(col)
Provides output:
1
2
3
Good solution for support of itertools.chain is to use itertools.islice in order to take a slice of an iterable:
your_input_list = ['list', 'of', 'things']
for i, variant in list(itertools.islice(enumerate(some_function_that_will_output_itertools_chain(your_input_list)), 1, None)):
"""
# No need for unnecessary conditions like this:
if i == 0:
continue
"""
variant = list(variant) # (optional) converting back to list
print(variant)
I do it like this, even though it looks like a hack it works every time:
ls_of_things = ['apple', 'car', 'truck', 'bike', 'banana']
first = 0
last = len(ls_of_things)
for items in ls_of_things:
if first == 0
first = first + 1
pass
elif first == last - 1:
break
else:
do_stuff
first = first + 1
pass
I have two lists that contain many of the same items, including duplicate items. I want to check which items in the first list are not in the second list. For example, I might have one list like this:
l1 = ['a', 'b', 'c', 'b', 'c']
and one list like this:
l2 = ['a', 'b', 'c', 'b']
Comparing these two lists I would want to return a third list like this:
l3 = ['c']
I am currently using some terrible code that I made a while ago that I'm fairly certain doesn't even work properly shown below.
def list_difference(l1,l2):
for i in range(0, len(l1)):
for j in range(0, len(l2)):
if l1[i] == l1[j]:
l1[i] = 'damn'
l2[j] = 'damn'
l3 = []
for item in l1:
if item!='damn':
l3.append(item)
return l3
How can I better accomplish this task?
You didn't specify if the order matters. If it does not, you can do this in >= Python 2.7:
l1 = ['a', 'b', 'c', 'b', 'c']
l2 = ['a', 'b', 'c', 'b']
from collections import Counter
c1 = Counter(l1)
c2 = Counter(l2)
diff = c1-c2
print list(diff.elements())
Create Counters for both lists, then subtract one from the other.
from collections import Counter
a = [1,2,3,1,2]
b = [1,2,3,1]
c = Counter(a)
c.subtract(Counter(b))
To take into account both duplicates and the order of elements:
from collections import Counter
def list_difference(a, b):
count = Counter(a) # count items in a
count.subtract(b) # subtract items that are in b
diff = []
for x in a:
if count[x] > 0:
count[x] -= 1
diff.append(x)
return diff
Example
print(list_difference("z y z x v x y x u".split(), "x y z w z".split()))
# -> ['y', 'x', 'v', 'x', 'u']
Python 2.5 version:
from collections import defaultdict
def list_difference25(a, b):
# count items in a
count = defaultdict(int) # item -> number of occurrences
for x in a:
count[x] += 1
# subtract items that are in b
for x in b:
count[x] -= 1
diff = []
for x in a:
if count[x] > 0:
count[x] -= 1
diff.append(x)
return diff
Counters are new in Python 2.7.
For a general solution to substract a from b:
def list_difference(b, a):
c = list(b)
for item in a:
try:
c.remove(item)
except ValueError:
pass #or maybe you want to keep a values here
return c
you can try this
list(filter(lambda x:l1.remove(x),li2))
print(l1)
Try this one:
from collections import Counter
from typing import Sequence
def duplicates_difference(a: Sequence, b: Sequence) -> Counter:
"""
>>> duplicates_difference([1,2],[1,2,2,3])
Counter({2: 1, 3: 1})
"""
shorter, longer = sorted([a, b], key=len)
return Counter(longer) - Counter(shorter)