Develop Reference

How do I set limits for my range function in python?

Say I am iterating through a list. I want to check if the list's neighbours (i-1 and i+1) contain a certain element. How do I do this without running into "list index out of range" problem?
Example:
list = [1, 0, 1, 0, 1, 0]
for i, j in enumerate(list):
elements = 0
for m in range(i-1,i+2):
if list[m] == 1:
elements += 1
print(list[i], elements)
How do I set boundaries for the range function, so that it doesn't go below 0 and above len(list)?

If you want to iterate for all elements in the target list, one solution is to check the value of second for loop:
_list = [1, 0, 1, 0, 1, 0]
elements = 0
for i, j in enumerate(_list):
for m in range(max(i-1, 0), min(i+2, len(_list))):
if _list[m] == 1:
elements += 1

Try slicing list from the top and bottom
list = [1, 0, 1, 0, 1, 0]
elements = 0
# slice list and start from second element and finish at the penultimate element
for i, j in enumerate(list[1:-1], 1):
for m in range(i-1,i+2):
if list[m] == 1:
elements += 1
or since you don't use list items in the outer loop, loop over range
elements = 0
# start from the second index and finish at the penultimate index
for i in range(1, len(list)-1):
for m in range(i-1,i+2):
if list[m] == 1:
elements += 1

Sounds like you want to use a window function. I got this somewhere here and have been using it over the years:
from typing import Generator
from itertools import islice
def window(seq, n: int = 2) -> Generator:
"""
Returns a sliding window (of width n) over data from the iterable
"""
it = iter(seq)
result = tuple(islice(it, n))
if len(result) == n:
yield result
for elem in it:
result = result[1:] + (elem,)
yield result
mylist = [1, 0, 1, 0, 1, 0, 5]
for chunk in window(mylist, n=3):
print(chunk)
This will give you :
(1, 0, 1)
(0, 1, 0)
(1, 0, 1)
(0, 1, 0)
(1, 0, 5)
Where you can compare the contents of the resulting 'window' however you like.

Simple method to find 0,0,1 in that order, within a given List

I am trying to write a simple function to find if 0,0,1 occurs in a list, in that order.
It should return True or False.
The list can contain any number of numbers.
For the function ZeroZeroOne examples would be as follows:
>> ZeroZeroOne( [0,0,1] )
>> True
>> ZeroZeroOne( [1,0,0] )
>> False
# there are 2s in between but the following does have 0,0,1 occurring and in correct order
>> ZeroZeroOne( [0,2,2,2,2,0,1] )
>> True
I have this function:
def ZeroZeroOne(nums):
FoundIt = False
#quick return if defo not possible
if (nums.count(0) < 2) and (nums.count(1) == 0):
return FoundIt
n = len(nums)
for x in range(n-2):
if nums[x] == 0:
for i,z in enumerate(nums[(x+1):]):
if z==0 and z!=1:
for j,q in enumerate(nums[(i+1):]):
if q==1 and q!=0:
FoundIt=True
return FoundIt
Why does the function return True for this list [0, 1, 0, 2, 1]?
Moreover....
This function seems overly-complex for a seemingly simple problem.
Is there a correct approach to this problem in Python - a canonical or Pythonic approach?
Or is ones approach simply opinion-based?

You can trivially modify the ordered subsequence test from this answer for an elegant solution:
def ZeroZeroOne(arr):
test = iter(a for a in arr if a in (0, 1))
return all(z in test for z in (0, 0, 1))
I realize now that you don't want to accept 0, 1 0, 1.
You can use itertools.tee to check for a match:
def ZeroZeroOne(arr):
e = itertools.tee((a for a in arr if a in (0, 1)), 3)
# move second iterator forward one
next(e[1])
# move third iterator forward two
next(e[2])
next(e[2])
return (0, 0, 1) in zip(*e)
The nice thing about using tee in this case is that it effectively maintains a rolling buffer of the last three elements for you. You don't need to make a new slice or loop over indices it anything like that.
Just for fun, here's a more general solution in pure python. It accepts any iterable for arr and template:
def contains_template(arr, template):
template = tuple(template)
unique = set(template)
filtered = (a for a in arr if a in unique)
e = itertools.tee(filtered, len(template))
for n, it in enumerate(e):
for _ in range(n):
next(it)
return template in zip(*e)
While itertools.tee is a nice way to maintain a rolling buffer, you can implement the same thing using a list (or more efficiently, collections.deque):
def contains_template(arr, template):
template = list(template)
unique = set(template)
filtered = (a for a in arr if a in unique)
buffer = [next(filtered) for _ in range(len(template) - 1)]
buffer.insert(0, None)
for e in filtered:
buffer.pop(0)
buffer.append(e)
if template == buffer:
return True
return False
Finally, here is the really simple solution, without a rolling buffer:
def contains_template(arr, template):
template = list(template)
n = len(template)
unique = set(template)
filtered = [a for a in arr if a in unique]
return any(filtered[i:i + n] == template for i in range(len(filtered) - n))

You can also do it with a recursive function :
def check(seq, liste, i=0, j=0):
if i >= len(seq):
return True
if j >= len(liste):
return False
if seq[i] == liste[j]:
return check(seq, liste, i + 1, j + 1)
elif liste[j] in seq:
# look for the last index you can restart from
for k in range(i - 1, -1, -1):
if seq[k] == liste[j]:
if seq[:k] == seq[i - k:i]:
ind = k
break
else:
ind = 0
return check(seq, liste, ind, j + (not i))
else:
return check(seq, liste, i, j + 1)
# seq = [0,0,1] for ZeroZeroOne
print(check([0, 0, 1], [0, 0, 0, 0, 1])) # True
print(check([0, 0, 1], [0, 200, 0, 0, 101, 1])) # True
print(check([0, 2, 2, 0, 1], [0, 2, 0, 4, 2, 5, 2, 0, 3, 1])) # True
print(check([0, 2, 2, 0, 1], [0, 2, 4, 2, 5, 2, 0, 3, 1])) # False

You can achieve this with a single loop - O(n) time complexity. Since it is for this specific case. Try the code below.
def ZeroZeroOne(nums):
found_pattern = []
for num in nums:
if num == 1:
found_pattern.append(1)
if len(found_pattern) == 3:
return True
else:
found_pattern = []
elif num == 0 and len(found_pattern) < 2:
found_pattern.append(0)
return False
print(ZeroZeroOne([0, 0, 1]))
print(ZeroZeroOne([0, 1, 0, 2, 1]))
print(ZeroZeroOne([0, 2, 0, 1]))
print(ZeroZeroOne([0, 0, 0, 1]))
print(ZeroZeroOne([0, 2, 2, 2, 2, 0, 1]))
But I think you can generalize this as well if required. Probably you need to look in to how grep works and modify it for your use case if you want a generic approach.

I think this does what you want :)
def ZeroZeroOne(arr):
dropped = [x for x in arr if x==0 or x==1]
slices = [dropped[i:i+3] for i in range(len(dropped)-2)]
if [0,0,1] in slices: return True
else: return False

def ZeroZeroOne(nums):
filtered_nums = [x for x in nums if x in [0,1]]
return '*'.join([str(x) for x in [0,0,1]) in '*'.join([str(x) for x in filtered_nums])

Finding longest run in a list

Given a list of data, I'm trying to create a new list in which the value at position i is the length of the longest run starting from position i in the original list. For instance, given
x_list = [1, 1, 2, 3, 3, 3]
Should return:
run_list = [2, 1, 1, 3, 2, 1]
My solution:
freq_list = []
current = x_list[0]
count = 0
for num in x_list:
if num == current:
count += 1
else:
freq_list.append((current,count))
current = num
count = 1
freq_list.append((current,count))
run_list = []
for i in freq_list:
z = i[1]
while z > 0:
run_list.append(z)
z -= 1
Firstly I create a list freq_list of tuples, where every tuple's first element is the element from x_list, and where the second element is the number of the total run.
In this case:
freq_list = [(1, 2), (2, 1), (3, 3)]
Having this, I create a new list and append appropriate values.
However, I was wondering if there is a shorter way/another way to do this?

Here's a simple solution that iterates over the list backwards and increments a counter each time a number is repeated:
last_num = None
result = []
for num in reversed(x_list):
if num != last_num:
# if the number changed, reset the counter to 1
counter = 1
last_num = num
else:
# if the number is the same, increment the counter
counter += 1
result.append(counter)
# reverse the result
result = list(reversed(result))
Result:
[2, 1, 1, 3, 2, 1]

This is possible using itertools:
from itertools import groupby, chain
x_list = [1, 1, 2, 3, 3, 3]
gen = (range(len(list(j)), 0, -1) for _, j in groupby(x_list))
res = list(chain.from_iterable(gen))
Result
[2, 1, 1, 3, 2, 1]
Explanation
First use itertools.groupby to group identical items in your list.
For each item in your groupby, create a range object which counts backwards from the length of the number of consecutive items to 1.
Turn this all into a generator to avoid building a list of lists.
Use itertools.chain to chain the ranges from the generator.
Performance note
Performance will be inferior to #Aran-Fey's solution. Although itertools.groupby is O(n), it makes heavy use of expensive __next__ calls. These do not scale as well as iteration in simple for loops. See itertools docs for groupby pseudo-code.
If performance is your main concern, stick with the for loop.

You are performing a reverse cumulative count on contiguous groups. We can create a Numpy cumulative count function with
import numpy as np
def cumcount(a):
a = np.asarray(a)
b = np.append(False, a[:-1] != a[1:])
c = b.cumsum()
r = np.arange(len(a))
return r - np.append(0, np.flatnonzero(b))[c] + 1
and then generate our result with
a = np.array(x_list)
cumcount(a[::-1])[::-1]
array([2, 1, 1, 3, 2, 1])

I would use a generator for this kind of task because it avoids building the resulting list incrementally and can be used lazily if one wanted:
def gen(iterable): # you have to think about a better name :-)
iterable = iter(iterable)
# Get the first element, in case that fails
# we can stop right now.
try:
last_seen = next(iterable)
except StopIteration:
return
count = 1
# Go through the remaining items
for item in iterable:
if item == last_seen:
count += 1
else:
# The consecutive run finished, return the
# desired values for the run and then reset
# counter and the new item for the next run.
yield from range(count, 0, -1)
count = 1
last_seen = item
# Return the result for the last run
yield from range(count, 0, -1)
This will also work if the input cannot be reversed (certain generators/iterators cannot be reversed):
>>> x_list = (i for i in range(10)) # it's a generator despite the variable name :-)
>>> ... arans solution ...
TypeError: 'generator' object is not reversible
>>> list(gen((i for i in range(10))))
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
And it works for your input:
>>> x_list = [1, 1, 2, 3, 3, 3]
>>> list(gen(x_list))
[2, 1, 1, 3, 2, 1]
This can actually be made simpler by using itertools.groupby:
import itertools
def gen(iterable):
for _, group in itertools.groupby(iterable):
length = sum(1 for _ in group) # or len(list(group))
yield from range(length, 0, -1)
>>> x_list = [1, 1, 2, 3, 3, 3]
>>> list(gen(x_list))
[2, 1, 1, 3, 2, 1]
I also did some benchmarks and according to these Aran-Feys solution is the fastest except for long lists where piRSquareds solution wins:
This was my benchmarking setup if you want to confirm the results:
from itertools import groupby, chain
import numpy as np
def gen1(iterable):
iterable = iter(iterable)
try:
last_seen = next(iterable)
except StopIteration:
return
count = 1
for item in iterable:
if item == last_seen:
count += 1
else:
yield from range(count, 0, -1)
count = 1
last_seen = item
yield from range(count, 0, -1)
def gen2(iterable):
for _, group in groupby(iterable):
length = sum(1 for _ in group)
yield from range(length, 0, -1)
def mseifert1(iterable):
return list(gen1(iterable))
def mseifert2(iterable):
return list(gen2(iterable))
def aran(x_list):
last_num = None
result = []
for num in reversed(x_list):
if num != last_num:
counter = 1
last_num = num
else:
counter += 1
result.append(counter)
return list(reversed(result))
def jpp(x_list):
gen = (range(len(list(j)), 0, -1) for _, j in groupby(x_list))
res = list(chain.from_iterable(gen))
return res
def cumcount(a):
a = np.asarray(a)
b = np.append(False, a[:-1] != a[1:])
c = b.cumsum()
r = np.arange(len(a))
return r - np.append(0, np.flatnonzero(b))[c] + 1
def pirsquared(x_list):
a = np.array(x_list)
return cumcount(a[::-1])[::-1]
from simple_benchmark import benchmark
import random
funcs = [mseifert1, mseifert2, aran, jpp, pirsquared]
args = {2**i: [random.randint(0, 5) for _ in range(2**i)] for i in range(1, 20)}
bench = benchmark(funcs, args, "list size")
%matplotlib notebook
bench.plot()
Python 3.6.5, NumPy 1.14

Here's a simple iterative approach to achieve it using collections.Counter:
from collections import Counter
x_list = [1, 1, 2, 3, 3, 3]
x_counter, run_list = Counter(x_list), []
for x in x_list:
run_list.append(x_counter[x])
x_counter[x] -= 1
which will return you run_list as:
[2, 1, 1, 3, 2, 1]
As an alternative, here's one-liner to achieve this using list comprehension with enumerate but it is not performance efficient due to iterative usage of list.index(..):
>>> [x_list[i:].count(x) for i, x in enumerate(x_list)]
[2, 1, 1, 3, 2, 1]

You can count the consecutive equal items and then add a countdown from count-of-items to 1 to the result:
def runs(p):
old = p[0]
n = 0
q = []
for x in p:
if x == old:
n += 1
else:
q.extend(range(n, 0, -1))
n = 1
old = x
q.extend(range(n, 0, -1))
return q
(A couple of minutes later) Oh, that's the same as MSeifert's code but without the iterable aspect. This version seems to be almost as fast as the method shown by Aran-Fey.

Python: forward fill the data in a list

I have a list named x, I would like to fill the zero data with previous value, which means:
x = [x[t]=x[t-1] if x[t] == 0.0 for t in range(1,len(x)-2)]
But it displayed: SyntaxError: invalid syntax
I'm wondering where is wrong with my code? Thanks a lot.

It's your assignment x[t] = x[t-1]. Instead just use a for loop:
for t in range(1, len(x)-1):
if x[t] == 0:
x[t] = x[t-1]
Although it would probably be considered more Pythonic to use enumerate to do this:
for idx, val in enumerate(x):
if idx==0: continue # skip the first element
if val == 0:
x[idx] = x[idx-1]
# DEMO
In [1]: x = [1,0,3,0,4,0,5,0]
In [2]: for idx,val in enumerate(x):
...: if idx==0: continue
...: if val == 0:
...: x[idx] = x[idx-1]
...:
In [3]: x
Out[3]: [1, 1, 3, 3, 4, 4, 5, 5]
You could also make this work with a list comp by implementing a pairwise iterator
from itertools import tee
def pairwise(iterable):
a,b = tee(iterable)
next(b) # advance one iterator
return zip(a,b)
x = [x[0]] + [val if val else lastval for lastval,val in pairwise(x)]
We need to specifically add the first element since the pairwise iterator skips it. Alternatively we could define pairwise differently, e.g.
def pairwise(iterable):
iterable = itertools.chain([None], iterable)
a,b = itertools.tee(iterable)
next(b)
return zip(a,b)
x = [val if val else lastval for lastval,val in pairwise(x)]
# ta-da!

Here's a list comprehension to do what you require:
x = [xi if xi or i==0 else x[i-1]
for i, xi in enumerate(x)]

For full forward and backward filling (backwards in case non found before), the following will give you a filled element even if the element before it is zero:
# ∨ ∨ ∨ ∨ ∨ ∨
x = [ 0, 1, 2, 0, 3, 5, 0, 0, 0, 9, 0 ]
y = []
for i,e in enumerate(x):
if e == 0:
# search left, get first non zero
for left_e in reversed(x[:i]):
if left_e != 0:
e = left_e
break
# backward fill if all elements on the left are zeros
if e == 0:
# search right, get first non zero
for right_e in x[i+1:]:
if right_e != 0:
e = right_e
break
y.append(e)
print(y)
# [1, 1, 2, 2, 3, 5, 5, 5, 5, 9, 9]
If you want forward filling with looking only at one the previous element then Alex's answers suffice.
You can also use a simpler method next():
x = [ 0, 1, 2, 0, 3, 5, 0, 0, 0, 9, 0 ]
y = []
for i,e in enumerate(x):
if e == 0:
e = next((item for item in x[i:] if item != 0), e)
e = next((item for item in reversed(x[:i]) if item != 0), e)
y.append(e)
print(y)
# [1, 1, 2, 2, 3, 5, 5, 5, 5, 9, 9]

fill in list in multiple steps

Lets assume you have a list with y poisitions (0 for sake of this question). If y = 10:
[0,0,0,0,0,0,0,0,0,0]
You want to fill adjacent positions up to a given value x and append it to an empty list. If x = 4:
[[1,1,1,1,0,0,0,0,0,0], [0,1,1,1,1,0,0,0,0,0], [0,0,1,1,1,1,0,0,0,0], ... , [0,0,0,0,0,0,1,1,1,1]]
I made that occur through this function:
def permutations(number=4, limit=10):
perms = []
if type(number) == int:
a = -1
b = a + number
while b < limit:
a+=1
b = a + number
start = [0 for x in range(limit)]
for i in range(a, b):
start[i] = 1
perms.append(start)
This is fine, but if I want to do the same thing, but pass a tuple instead of an integer I'd like the output to be:
if number = (4,3):
[[1,1,1,1,0,1,1,1,0,0], [1,1,1,1,0,0,1,1,1,0], [1,1,1,1,0,0,0,1,1,1],
[0,1,1,1,1,0,1,1,1,0], [0,1,1,1,1,0,0,1,1,1],
[0,0,1,1,1,1,0,1,1,1]]
The 0 between the two groupings of 1's is necessary the first value of the tuple corresponds to the number of 1's in the first grouping, and the second value of the tuple corresponds to the number of 1's in the second grouping. Ideally this function would work with tuples that have more than 2 values.
This idea is a little challenging to get across so please let me know if you need any clarification.
Thank you for your help!

The simplest approach I can think of is to generate all possible combinations of 1 and 0, and filter out all of the ones that don't have the right grouping lengths.
import itertools
def permutations(tup, limit=10):
for candidate in itertools.product([0,1], repeat=limit):
segment_lengths = [len(list(b)) for a,b in itertools.groupby(candidate) if a == 1]
if tup == tuple(segment_lengths):
yield candidate
for seq in permutations((4, 3), 10):
print seq
Result:
(0, 0, 1, 1, 1, 1, 0, 1, 1, 1)
(0, 1, 1, 1, 1, 0, 0, 1, 1, 1)
(0, 1, 1, 1, 1, 0, 1, 1, 1, 0)
(1, 1, 1, 1, 0, 0, 0, 1, 1, 1)
(1, 1, 1, 1, 0, 0, 1, 1, 1, 0)
(1, 1, 1, 1, 0, 1, 1, 1, 0, 0)
Note that this is very slow for large values of limit - it has to evaluate 2^limit candidate sequences. Not bad for limit = 10; only 1024 candidates need to be evaluated. But it quickly grows into the millions and beyond for larger limits.
Edit: Inspired by user2097159's excellent comment, here's an approach with better run time.
import itertools
"""Finds all non-negative integer sequences whose sum equals `total`, and who have `size` elements."""
def possible_sums(total, size):
if total == 0:
yield [0]*size
return
if size == 1:
yield [total]
return
for i in range(total+1):
left = [i]
for right in possible_sums(total-i, size-1):
yield left + right
"""
combines two lists a and b in order like:
[a[0], b[0], a[1], b[1]...]
"""
def interleave(a,b):
result = []
for pair in itertools.izip_longest(a,b):
for item in pair:
if item is not None:
result.append(item)
return result
"""flattens a list of lists into a one dimensional list"""
def flatten(seq):
return [x for item in seq for x in item]
def permutations(tup, limit):
one_segments = [[1]*size for size in tup]
for i in range(len(tup)-1):
one_segments[i].append(0)
remaining_zeroes = limit - sum(tup) - len(tup) + 1
assert remaining_zeroes >= 0, "not enough room to separate ranges!"
for gap_sizes in possible_sums(remaining_zeroes, len(tup)+1):
zero_segments = [[0]*size for size in gap_sizes]
yield flatten(interleave(zero_segments, one_segments))
for seq in permutations((4, 3), 10):
print seq

You can generate all list recursively.
F(tup, limit) =
[1, 1, ...1, 0] combine with all solutions of F(tup[1:], limit - len(tup[1]) - 1)
[0, 1 ,1 , ... 1, 0] combine with all solutions of F(tup[1:], limit - len(tup[1]) - 2)
.
.
.
if tup is empty return a list of zero
if sum(tup) + len(tup) - 1 > limit, return an empty list since there is no solution.
e.g. permutations((4,3,2), 10) shall return []
Otherwise, enumerating how many prefix zero there will be:
Generate prefix list which is [0, 0, 0 .. 0, 1, 1, ... 1, 0] The number of 1s is the value of first item in the tuple. Append additional 0 if it's not the last item of the tuple.
Call the function recursively for the rest element in the tuple to solve the similar sub-problem
Combine the prefix list with each solution of the sub-problem
Here is the code:
def permutations(tup, limit=100):
if len(tup) <= 0:
return [[0] * limit]
minimum_len = sum(tup) + len(tup) - 1
if minimum_len > limit:
return []
perms = []
for prefix_zero in range(0, limit - minimum_len + 1):
prefix = [0] * prefix_zero + [1] * tup[0]
if len(tup) > 1:
prefix += [0]
suffix_list = permutations(tup[1:], limit - len(prefix))
perms += [prefix + suffix for suffix in suffix_list] #combine the solutions
return perms

This solution creates all permutations of blocks of ones (a list defined by each entry in the tuple) with blocks of zeros (lists of length one) for the extra padding.
import itertools as it
spec = (1,2,3)
nBlocks = len(spec)
nZeros = 5
totalSize = sum(spec) + nZeros+1-nBlocks
blocks = [[1,]*s + [0,] for s in spec]
zeros = [[0,],]*(nZeros+1-nBlocks)
a = list(it.permutations(blocks + zeros, nZeros+1))
b = [list(it.chain.from_iterable(l))[:-1] for l in a]
for l in b:
print l

Without using itertools.
My shot at this, should be fairly quick, but uses a recursive generator (python recursion depth limit, here I come...).
# simple test case
seqs = (1, 2, 3)
length = 10
# '0' spots count
zeros = length - (sum(seqs))
# partitions count
partitions = len(seqs) + 1
# first and last can partitions have 0 zeros
# so use a flag when we call the function or check if it's the last partition
def generate_gaps(zeros_left, partition, first=False):
"""
:param zeros_left: how many zeros we can still use
:param partition: what partition is this
:param first: is this the first gap
:return: all possible gaps
"""
for gap in range((0 if first or partition == 0 else 1), zeros_left + 1):
if partition == 0:
if (zeros_left - gap) == 0:
yield [gap]
else:
for rest in generate_gaps(zeros_left - gap, partition - 1):
yield [gap] + rest
for gaps in generate_gaps(zeros, partitions - 1, True):
print "using gaps: " + str(gaps)
# merge lists
# zip gaps (0's) and sequences (1's) - all but last gap (added to result)
gaps_seqs = zip(gaps, seqs)
# expand everything... magic (could be done explicitly trivially).
result = sum(map(lambda x: [0] * x[0] + [1] * x[1], gaps_seqs)
# last gap (truncated from zip)
result = result + [[0] * gaps[-1]], [])

A simple non-recursive generator solution without itertools:
def fill_sequence(sequence, size):
n_slots = size - len(sequence)
for start in xrange(n_slots + 1):
yield [0]*start + sequence + [0]*(n_slots - start)
def all_placements(inner_sizes, outer_size):
x, y = inner_sizes
for margin in xrange(1, outer_size - sum(block_sizes) + 1):
sequence = [1]*x + [0]*margin + [1]*y
for p in fill_sequence(sequence, outer_size):
yield p
So that:
>>> list(all_placements((4,3), 10))
[[1, 1, 1, 1, 0, 1, 1, 1, 0, 0],
[0, 1, 1, 1, 1, 0, 1, 1, 1, 0],
[0, 0, 1, 1, 1, 1, 0, 1, 1, 1],
[1, 1, 1, 1, 0, 0, 1, 1, 1, 0],
[0, 1, 1, 1, 1, 0, 0, 1, 1, 1],
[1, 1, 1, 1, 0, 0, 0, 1, 1, 1]]
The idea is quite simple. Suppose you fix the number of zeros between your two blocks of ones, call it the margin. This gives you a 4 + margin + 3 sequence. You can easily place this sequence in the larger list of zeros using the approach you took in your post. Then simply iteratively increase the margin, yielding all possible placements.