Develop Reference

How to improve efficiency and time-complexity of AIO Cloud Cover solution

Sorry for the noob question, but is there a less time expensive method to iterate through the input list, as upon submission I receive timeout errors. I tried changing the method of checking for the minimum answer by appending to a list and using min function, but as expected that didn't help at all.
Input:
6 3
3
6
4
2
5
Solution:
with open("cloudin.txt", "r") as input_file:
n, covered = map(int, input_file.readline().split())
ls = [None for i in range(100005)]
for i in range(n-1):
ls[i] = int(input_file.readline().strip())
ans = 1000000001
file = open("cloudout.txt", "w")
for i in range(n-covered):
a = 0
for j in range(covered):
a += ls[i+j]
if a < ans:
ans = a
file.write(str(ans))
output:
11
https://orac2.info/problem/aio18cloud/
Note: Blue + White indicates timeout

The core logic of your code is contained in these lines:
ans = 1000000001
for i in range(n-covered):
a = 0
for j in range(covered):
a += ls[i+j]
if a < ans:
ans = a
Let's break down what this code actually does. For each closed interval (i.e. including the endpoints) [left, right] from the list [0, covered-1], [1, covered], [2, covered+1], ..., [n-covered-1, n-2] (that is, all closed intervals containing exactly covered elements and that are subintervals of [0, n-2]), you are computing the range sum ls[left] + ls[left+1] + ... + ls[right]. Then you set ans to the minimum such range sum.
Currently, that nested loop takes O((n-covered)*covered)) steps, which is O(n^2) if covered is n/2, for example. You want a way to compute that range sum in constant time, eliminating the nested loop, to make the runtime O(n).
The easiest way to do this is with a prefix sum array. In Python, itertools.accumulate() is the standard/simplest way to generate those. To see how this helps:
Original Sum: ls[left] + ls[left+1] + ... + ls[right]
can be rewritten as the difference of prefix sums
(ls[0] + ls[1] + ... + ls[right])
- (ls[0] + ls[1] + ... + ls[left-1])
which is prefix_sum(0, right) - prefix_sum(0, left-1)
where are intervals are written in inclusive notation.
Pulling this into a separate range_sum() function, you can rewrite the original core logic block as:
prefix_sums = list(itertools.accumulate(ls, initial=0))
def range_sum(left: int, right: int) -> int:
"""Given indices left and right, returns the sum of values of
ls in the inclusive interval [left, right].
Equivalent to sum(ls[left : right+1])"""
return prefix_sums[right+1] - prefix_sums[left]
ans = 1000000001
for i in range(n - covered):
a = range_sum(left=i, right=i+covered-1)
if a < ans:
ans = a
The trickiest part of prefix sum arrays is just avoiding off-by-one errors in indexes. Notice that our prefix sum array of the length-n array ls has n+1 elements, since it starts with the empty initial prefix sum of 0, and so we add 1 to array accesses to prefix_sums compared to our formula.
Also, it's possible there may be an off-by-one error in your original code, as the value ls[n-1] is never accessed or used for anything after being set?

How to add the last two elements in a list and add the sum to existing list

I am learning Python and using it to work thru a challenge found in Project Euler. Unfortunately, I cannot seem to get around this problem.
The problem:
Even Fibonacci numbers
Each new term in the Fibonacci sequence is generated by adding the
previous two terms. By starting with 1 and 2, the first 10 terms will
be:
1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ...
By considering the terms in the Fibonacci sequence whose values do not
exceed four million, find the sum of the even-valued terms.
I created a for loop that adds the second to last element and the last element from the list x:
x = [1,2]
for i in x:
second_to_last = x[-2]
running_sum = i + second_to_last
If you run the above, you get 3. I am looking to add this new element back to the original list, x, and repeat the process. However, each time I try to use the append() function, the program crashes and keeps on running without stopping. I tried to use a while loop to stop this, but that was a complete failure. Why am I not able to add or append() the new element (running_sum) back to the original list (x)?
UPDATE:
I did arrive at the solution (4613732), but I the work to getting there did not seem efficient. Here is my solution:
while len(x) in range(1,32):
for i in x:
second_to_last = x[-2]
running_sum = i + second_to_last
x.append(running_sum)
print(x)
new_x = []
for i in x:
if i%2 == 0:
new_x.append(i)
sum(new_x)
I did have to check the range to see visually whether I did not exceed 4 million. But as I said, the process I took was not efficient.

If you keep adding elements to a list while iterating over that list, the iteration will never finish.
You will need some other criterion to abort the loop - for example, in this case
if running_sum > 4000000:
break
would work.
(Note that you don't strictly speaking need a list at all here; I'd suggest experimenting a bit with it.)

Here are two different ways to solve this. One of them builds the whole list, then sums the even elements. The other one only keeps the last two elements, without making the whole list.
fib = [1,2]
while fib[-1] < 4000000:
fib.append(fib[-2]+fib[-1])
# Get rid of the last one, since it was over the limit.
fib.pop(-1)
print( sum(i for i in fib if i % 2 == 0) )
fib = (1,2)
sumx = 2
while True:
nxt = fib[0]+fib[1]
if nxt >= 4000000:
break
if nxt % 2 == 0:
sumx += nxt
fib = (fib[1],nxt)
print(sumx)

I don't answer your question about list modification but the solution for your problem:
def sum_even_number_fibonacci(limit):
n0 = 0 # Since we don't care about index (n-th), we can use n0 = 0 or 1
n1 = 1
even_number_sum = 0
while n1 <= limit:
if n1 % 2 == 0:
even_number_sum += n1
n2 = n0 + n1
# Only store the last two number of the Fibonacci sequence to calculate the next one
n0 = n1
n1 = n2
return even_number_sum
sum_even_number_fibonacci(4_000_000)

Nth 1 in a sequence [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 5 years ago.
Improve this question
A array of length t has all elements initialized by 1 .Now we can perform two types of queries on the array
to replace the element at ith index to 0 .This query is denoted by 0 index
find and print an integer denoting the index of the kth 1 in array A on a new line; if no such index exists print -1.This query is denoted by 1 k
Now suppose for array of length t=4 all its elements at the beginning are [1,1,1,1] now for query 0 2 the array becomes [1,0,1,1] and for query 1 3 the output comes out to be 4
I have used a brute force approach but how to make the code more efficient?
n,q=4,2
arr=[1]*4
for i in range(q):
a,b=map(int,input().split())
if a==0:
arr[b-1]=0
else:
flag=True
count=0
target=b
for i,j in enumerate(arr):
if j ==1:
count+=1
if count==target:
print(i+1)
flag=False
break
if flag:
print(-1)
I have also tried to first append all the indexes of 1 in a list and then do binary search but pop 0 changes the indices due to which the code fails
def binary_search(low,high,b):
while(low<=high):
mid=((high+low)//2)
#print(mid)
if mid+1==b:
print(stack[mid]+1)
return
elif mid+1>b:
high=mid-1
else:
low=mid+1
n=int(input())
q=int(input())
stack=list(range(n))
for i in range(q):
a,b=map(int,input().split())
if a==0:
stack.pop(b-1)
print(stack)
else:
if len(stack)<b:
print(-1)
continue
else:
low=0
high=len(stack)-1
binary_search(low,high,b)

You could build a binary tree where each node gives you the number of ones that are below and at the left of it. So if n is 7, that tree would initially look like this (the actual list with all ones is shown below it):
4
/ \
2 2
/ \ / \
1 1 1 1
----------------
1 1 1 1 1 1 1 -
Setting the array element at index 4 (zero-based) to 0, would change that tree to:
4
/ \
2 1*
/ \ / \
1 1 0* 1
----------------
1 1 1 1 0*1 1 -
Putting a 0 thus represents a O(log(n)) time complexity.
Counting the number of ones can then also be done in the same time complexity by summing up the node values while descending down the tree in the right direction.
Here is Python code you could use. It represents the tree in a list in breadth-first order. I have not gone to great lengths to further optimise the code, but it has the above time complexities:
class Ones:
def __init__(self, n): # O(n)
self.lst = [1] * n
self.one_count = n
self.tree = []
self.size = 1 << (n-1).bit_length()
at_left = self.size // 2
width = 1
while width <= at_left:
self.tree.extend([at_left//width] * width)
width *= 2
def clear_index(self, i): # O(logn)
if i >= len(self.lst) or self.lst[i] == 0:
return
self.one_count -= 1
self.lst[i] = 0
# Update tree
j = 0
bit = self.size >> 1
while bit >= 1:
go_right = (i & bit) > 0
if not go_right:
self.tree[j] -= 1
j = j*2 + 1 + go_right
bit >>= 1
def get_index_of_ith_one(self, num_ones): # O(logn)
if num_ones <= 0 or num_ones > self.one_count:
return -1
j = 0
k = 0
bit = self.size >> 1
while bit >= 1:
go_right = num_ones > self.tree[j]
if go_right:
k |= bit
num_ones -= self.tree[j]
j = j*2 + 1 + go_right
bit >>= 1
return k
def is_consistent(self): # Only for debugging
# Check that list can be derived by calling get_index_of_ith_one for all i
lst = [0] * len(self.lst)
for i in range(1, self.one_count+1):
lst[self.get_index_of_ith_one(i)] = 1
return lst == self.lst
# Example use
ones = Ones(12)
print('tree', ones.tree)
ones.clear_index(5)
ones.clear_index(2)
ones.clear_index(1)
ones.clear_index(10)
print('tree', ones.tree)
print('lst', ones.lst)
print('consistent = ', ones.is_consistent())
Be aware that this treats indexes as zero-based, while the method get_index_of_ith_one expects an argument that is at least 1 (but it returns a zero-based index).
It should be easy to adapt to your needs.
Complexity
Creation: O(n)
Clear at index: O(logn)
Get index of one: O(logn)
Space complexity: O(n)

Let's start with some general tricks:
Check if the n-th element is too big for the list before iterating. If you also keep a "counter" that stores the number of zeros, you could even check if nth >= len(the_list) - number_of_zeros (not sure if >= is correct here, it seems like the example uses 1-based indices so I could be off-by-one). That way you save time whenever too big values are used.
Use more efficient functions.
So instead of input you could use sys.stdin.readline (note that it will include the trailing newline).
And, even though it's probably not useful in this context, the built-in bisect module would be better than the binary_search function you created.
You could also use for _ in itertools.repeat(None, q) instead of for i in range(q), that's a bit faster and you don't need that index.
Then you can use some more specialized facts about the problem to improve the code:
You only store zeros and ones, so you can use if j to check for ones and if not j to check for zeros. These will be a bit faster than manual comparisons especially in when you do that in a loop.
Every time you look for the nth 1, you could create a temporary dictionary (or a list) that contains the encountered ns + index. Then re-use that dict for subsequent queries (dict-lookup and list-random-access is O(1) while your search is O(n)). You could even expand it if you have subsequent queries without change in-between.
However if a change happens you either need to discard that dictionary (or list) or update it.
A few nitpicks:
The variable names are not very descriptive, you could use for index, item in enumerate(arr): instead of i and j.
You use a list, so arr is a misleading variable name.
You have two i variables.
But don't get me wrong. It's a very good attempt and the fact that you use enumerate instead of a range is great and shows that you already write pythonic code.

Consider something akin to the interval tree:
root node covers the entire array
children nodes cover left and right halves of the parent range respectively
each node holds the number of ones in its range
Both replace and search queries could be completed in logarithmic time.

Refactored with less lines, so more efficient in terms of line count but run time probably the same O(n).
n,q=4,2
arr=[1]*4
for i in range(q):
query, target = map(int,input('query target: ').split())
if query == 0:
arr[target-1] = 0
else:
count=0
items = enumerate(arr, 1)
try:
while count < target:
index, item = next(items)
count += item
except StopIteration as e:
index = -1
print(index)
Assumes arr contains ONLY ones and zeroes - you don't have to check if an item is one before you add it to count, adding zero has no affect.
No flags to check, just keep calling next on the enumerate object (items) till you reach your target or the end of arr.
For runtime efficiency, using an external library but basically the same process (algorithm):
import numpy as np
for i in range(q):
query, target = map(int,input('query target: ').split())
if query == 0:
arr[target-1] = 0
else:
index = -1
a = np.array(arr).cumsum() == target
if np.any(a):
index = np.argmax(a) + 1
print(index)

Python: How can I make my implementation of bubble sort more time efficient?

Here is my code - a bubble sort algorithm for sorting list elements in asc order:
foo = [7, 0, 3, 4, -1]
cnt = 0
for i in foo:
for i in range(len(foo)-1):
if foo[cnt] > foo[cnt + 1]:
temp = foo[cnt]
c[cnt] = c[cnt + 1]
c[cnt + 1] = temp
cnt = cnt + 1
cnt = 0
I've been revising my code, but it is still too inefficient for an online judge. Some help would be greatly appreciated!

Early Exit BubbleSort
The first loop has no bearing on what happens inside
The second loop does all the heavy lifting. You can get rid of count by using enumerate
To swap elements, use the pythonic swap - a, b = b, a.
As per this comment, make use of an early exit. If there are no swaps to be made at any point in the inner loop, that means the list is sorted, and no further iteration is necessary. This is the intuition behind changed.
By definition, after the ith iteration of the outer loop, the last i elements will have been sorted, so you can further reduce the constant factor associated with the algorithm.
foo = [7, 0, 3, 4, -1]
for i in range(len(foo)):
changed = False
for j, x in enumerate(foo[:-i-1]):
if x > foo[j + 1]:
foo[j], foo[j + 1] = foo[j + 1], foo[j]
changed = True
if not changed:
break
print(foo)
[-1, 0, 3, 4, 7]
Note that none of these optimisations change the asymptotic (Big-O) complexity of BubbleSort (which remains O(N ** 2)), instead, only reduces the constant factors associated.

One easy optimization is to start second loop from i+1 index:
for i in range(0, len(foo)):
for j in range(i+1, len(foo)):
if (foo[i] > foo[j]):
temp = foo[i]
foo[i] = foo[j]
foo[j] = temp
Since you already sorted everything up to index i there is no need to iterate over it again. This can save you more than 50% of comparisons - in this case it's 10 versus 25 in your original algorithm.

You need to understand the big Oh notation in order to understand how efficient your algorithm is in terms of usage of computational resources independent of computer architecture or clock rate. It basically helps you analyze the worst case running time or memory usage of your algorithm as the size of the input increases.
In summary, the running time of your algorithm will fall into one of these categories (from fastest to slowest);
O(1): Constant time. Pronounced (Oh of 1). The fastest time.
O(lg n): Logarithmic time. Pronounced (Oh of log n). Faster than linear time.
Traditionally, it is the fastest time bound for search.
O(n): Linear time. Pronounced (Oh of n, n is the size of your input e.g size of
an array). Usually something when you need to examine every single bit of
your input.
O(nlgn): The fastest time we can currently achieve when performing a sort on a
list of elements.
O(n**2): Oh of n squared. Quadratic time. Often this is the bound when we have
nested loops.
O(2**n): Really, REALLY big! A number raised to the power of n is slower than
n raised to any power.
In your case, you are using nested loops which is O(n2). The code i have written uses a single while loop and has a growth complexity of O(n) which is faster than O(n2). I haven't really tried it on a very large array but in your case it seems to work. Try it and let me know if it works as expected.
k = [7, 0, 3, 4, -1]
n = len(k)
i = 0
count = 0
while count < n**2: # assuming we wouldn't go through the loop more than n squared times
if i == n - 1:
i = 0
count += 1
swapped = False
elif k[i] > k[i+1]:
temp = k[i]
k[i] = k[i+1]
k[i+1] = temp
i+=1
swapped = True
elif swapped == False:
i += 1
elif swapped == True and i < n - 1:
i += 1
Note: In the example list (k), we only need to loop through the list three times in order for it to be sorted in ascending order. So if you change the while loop to this line of code while count < 4:, it would still work.

Shuffling a list with maximum distance travelled [duplicate]

I have tried to ask this question before, but have never been able to word it correctly. I hope I have it right this time:
I have a list of unique elements. I want to shuffle this list to produce a new list. However, I would like to constrain the shuffle, such that each element's new position is at most d away from its original position in the list.
So for example:
L = [1,2,3,4]
d = 2
answer = magicFunction(L, d)
Now, one possible outcome could be:
>>> print(answer)
[3,1,2,4]
Notice that 3 has moved two indices, 1 and 2 have moved one index, and 4 has not moved at all. Thus, this is a valid shuffle, per my previous definition. The following snippet of code can be used to validate this:
old = {e:i for i,e in enumerate(L)}
new = {e:i for i,e in enumerate(answer)}
valid = all(abs(i-new[e])<=d for e,i in old.items())
Now, I could easily just generate all possible permutations of L, filter for the valid ones, and pick one at random. But that doesn't seem very elegant. Does anyone have any other ideas about how to accomplish this?

This is going to be long and dry.
I have a solution that produces a uniform distribution. It requires O(len(L) * d**d) time and space for precomputation, then performs shuffles in O(len(L)*d) time1. If a uniform distribution is not required, the precomputation is unnecessary, and the shuffle time can be reduced to O(len(L)) due to faster random choices; I have not implemented the non-uniform distribution. Both steps of this algorithm are substantially faster than brute force, but they're still not as good as I'd like them to be. Also, while the concept should work, I have not tested my implementation as thoroughly as I'd like.
Suppose we iterate over L from the front, choosing a position for each element as we come to it. Define the lag as the distance between the next element to place and the first unfilled position. Every time we place an element, the lag grows by at most one, since the index of the next element is now one higher, but the index of the first unfilled position cannot become lower.
Whenever the lag is d, we are forced to place the next element in the first unfilled position, even though there may be other empty spots within a distance of d. If we do so, the lag cannot grow beyond d, we will always have a spot to put each element, and we will generate a valid shuffle of the list. Thus, we have a general idea of how to generate shuffles; however, if we make our choices uniformly at random, the overall distribution will not be uniform. For example, with len(L) == 3 and d == 1, there are 3 possible shuffles (one for each position of the middle element), but if we choose the position of the first element uniformly, one shuffle becomes twice as likely as either of the others.
If we want a uniform distribution over valid shuffles, we need to make a weighted random choice for the position of each element, where the weight of a position is based on the number of possible shuffles if we choose that position. Done naively, this would require us to generate all possible shuffles to count them, which would take O(d**len(L)) time. However, the number of possible shuffles remaining after any step of the algorithm depends only on which spots we've filled, not what order they were filled in. For any pattern of filled or unfilled spots, the number of possible shuffles is the sum of the number of possible shuffles for each possible placement of the next element. At any step, there are at most d possible positions to place the next element, and there are O(d**d) possible patterns of unfilled spots (since any spot further than d behind the current element must be full, and any spot d or further ahead must be empty). We can use this to generate a Markov chain of size O(len(L) * d**d), taking O(len(L) * d**d) time to do so, and then use this Markov chain to perform shuffles in O(len(L)*d) time.
Example code (currently not quite O(len(L)*d) due to inefficient Markov chain representation):
import random
# states are (k, filled_spots) tuples, where k is the index of the next
# element to place, and filled_spots is a tuple of booleans
# of length 2*d, representing whether each index from k-d to
# k+d-1 has an element in it. We pretend indices outside the array are
# full, for ease of representation.
def _successors(n, d, state):
'''Yield all legal next filled_spots and the move that takes you there.
Doesn't handle k=n.'''
k, filled_spots = state
next_k = k+1
# If k+d is a valid index, this represents the empty spot there.
possible_next_spot = (False,) if k + d < n else (True,)
if not filled_spots[0]:
# Must use that position.
yield k-d, filled_spots[1:] + possible_next_spot
else:
# Can fill any empty spot within a distance d.
shifted_filled_spots = list(filled_spots[1:] + possible_next_spot)
for i, filled in enumerate(shifted_filled_spots):
if not filled:
successor_state = shifted_filled_spots[:]
successor_state[i] = True
yield next_k-d+i, tuple(successor_state)
# next_k instead of k in that index computation, because
# i is indexing relative to shifted_filled_spots instead
# of filled_spots
def _markov_chain(n, d):
'''Precompute a table of weights for generating shuffles.
_markov_chain(n, d) produces a table that can be fed to
_distance_limited_shuffle to permute lists of length n in such a way that
no list element moves a distance of more than d from its initial spot,
and all permutations satisfying this condition are equally likely.
This is expensive.
'''
if d >= n - 1:
# We don't need the table, and generating a table for d >= n
# complicates the indexing a bit. It's too complicated already.
return None
table = {}
termination_state = (n, (d*2 * (True,)))
table[termination_state] = 1
def possible_shuffles(state):
try:
return table[state]
except KeyError:
k, _ = state
count = table[state] = sum(
possible_shuffles((k+1, next_filled_spots))
for (_, next_filled_spots) in _successors(n, d, state)
)
return count
initial_state = (0, (d*(True,) + d*(False,)))
possible_shuffles(initial_state)
return table
def _distance_limited_shuffle(l, d, table):
# Generate an index into the set of all permutations, then use the
# markov chain to efficiently find which permutation we picked.
n = len(l)
if d >= n - 1:
random.shuffle(l)
return
permutation = [None]*n
state = (0, (d*(True,) + d*(False,)))
permutations_to_skip = random.randrange(table[state])
for i, item in enumerate(l):
for placement_index, new_filled_spots in _successors(n, d, state):
new_state = (i+1, new_filled_spots)
if table[new_state] <= permutations_to_skip:
permutations_to_skip -= table[new_state]
else:
state = new_state
permutation[placement_index] = item
break
return permutation
class Shuffler(object):
def __init__(self, n, d):
self.n = n
self.d = d
self.table = _markov_chain(n, d)
def shuffled(self, l):
if len(l) != self.n:
raise ValueError('Wrong input size')
return _distance_limited_shuffle(l, self.d, self.table)
__call__ = shuffled
1We could use a tree-based weighted random choice algorithm to improve the shuffle time to O(len(L)*log(d)), but since the table becomes so huge for even moderately large d, this doesn't seem worthwhile. Also, the factors of d**d in the bounds are overestimates, but the actual factors are still at least exponential in d.

In short, the list that should be shuffled gets ordered by the sum of index and a random number.
import random
xs = range(20) # list that should be shuffled
d = 5 # distance
[x for i,x in sorted(enumerate(xs), key= lambda (i,x): i+(d+1)*random.random())]
Out:
[1, 4, 3, 0, 2, 6, 7, 5, 8, 9, 10, 11, 12, 14, 13, 15, 19, 16, 18, 17]
Thats basically it. But this looks a little bit overwhelming, therefore...
The algorithm in more detail
To understand this better, consider this alternative implementation of an ordinary, random shuffle:
import random
sorted(range(10), key = lambda x: random.random())
Out:
[2, 6, 5, 0, 9, 1, 3, 8, 7, 4]
In order to constrain the distance, we have to implement a alternative sort key function that depends on the index of an element. The function sort_criterion is responsible for that.
import random
def exclusive_uniform(a, b):
"returns a random value in the interval [a, b)"
return a+(b-a)*random.random()
def distance_constrained_shuffle(sequence, distance,
randmoveforward = exclusive_uniform):
def sort_criterion(enumerate_tuple):
"""
returns the index plus a random offset,
such that the result can overtake at most 'distance' elements
"""
indx, value = enumerate_tuple
return indx + randmoveforward(0, distance+1)
# get enumerated, shuffled list
enumerated_result = sorted(enumerate(sequence), key = sort_criterion)
# remove enumeration
result = [x for i, x in enumerated_result]
return result
With the argument randmoveforward you can pass a random number generator with a different probability density function (pdf) to modify the distance distribution.
The remainder is testing and evaluation of the distance distribution.
Test function
Here is an implementation of the test function. The validatefunction is actually taken from the OP, but I removed the creation of one of the dictionaries for performance reasons.
def test(num_cases = 10, distance = 3, sequence = range(1000)):
def validate(d, lst, answer):
#old = {e:i for i,e in enumerate(lst)}
new = {e:i for i,e in enumerate(answer)}
return all(abs(i-new[e])<=d for i,e in enumerate(lst))
#return all(abs(i-new[e])<=d for e,i in old.iteritems())
for _ in range(num_cases):
result = distance_constrained_shuffle(sequence, distance)
if not validate(distance, sequence, result):
print "Constraint violated. ", result
break
else:
print "No constraint violations"
test()
Out:
No constraint violations
Distance distribution
I am not sure whether there is a way to make the distance uniform distributed, but here is a function to validate the distribution.
def distance_distribution(maxdistance = 3, sequence = range(3000)):
from collections import Counter
def count_distances(lst, answer):
new = {e:i for i,e in enumerate(answer)}
return Counter(i-new[e] for i,e in enumerate(lst))
answer = distance_constrained_shuffle(sequence, maxdistance)
counter = count_distances(sequence, answer)
sequence_length = float(len(sequence))
distances = range(-maxdistance, maxdistance+1)
return distances, [counter[d]/sequence_length for d in distances]
distance_distribution()
Out:
([-3, -2, -1, 0, 1, 2, 3],
[0.01,
0.076,
0.22166666666666668,
0.379,
0.22933333333333333,
0.07766666666666666,
0.006333333333333333])
Or for a case with greater maximum distance:
distance_distribution(maxdistance=9, sequence=range(100*1000))

This is a very difficult problem, but it turns out there is a solution in the academic literature, in an influential paper by Mark Jerrum, Alistair Sinclair, and Eric Vigoda, A Polynomial-Time Approximation Algorithm for the Permanent of a Matrix with Nonnegative Entries, Journal of the ACM, Vol. 51, No. 4, July 2004, pp. 671–697. http://www.cc.gatech.edu/~vigoda/Permanent.pdf.
Here is the general idea: first write down two copies of the numbers in the array that you want to permute. Say
1 1
2 2
3 3
4 4
Now connect a node on the left to a node on the right if mapping from the number on the left to the position on the right is allowed by the restrictions in place. So if d=1 then 1 on the left connects to 1 and 2 on the right, 2 on the left connects to 1, 2, 3 on the right, 3 on the left connects to 2, 3, 4 on the right, and 4 on the left connects to 3, 4 on the right.
1 - 1
X
2 - 2
X
3 - 3
X
4 - 4
The resulting graph is bipartite. A valid permutation corresponds a perfect matching in the bipartite graph. A perfect matching, if it exists, can be found in O(VE) time (or somewhat better, for more advanced algorithms).
Now the problem becomes one of generating a uniformly distributed random perfect matching. I believe that can be done, approximately anyway. Uniformity of the distribution is the really hard part.
What does this have to do with permanents? Consider a matrix representation of our bipartite graph, where a 1 means an edge and a 0 means no edge:
1 1 0 0
1 1 1 0
0 1 1 1
0 0 1 1
The permanent of the matrix is like the determinant, except there are no negative signs in the definition. So we take exactly one element from each row and column, multiply them together, and add up over all choices of row and column. The terms of the permanent correspond to permutations; the term is 0 if any factor is 0, in other words if the permutation is not valid according to the matrix/bipartite graph representation; the term is 1 if all factors are 1, in other words if the permutation is valid according to the restrictions. In summary, the permanent of the matrix counts all permutations satisfying the restriction represented by the matrix/bipartite graph.
It turns out that unlike calculating determinants, which can be accomplished in O(n^3) time, calculating permanents is #P-complete so finding an exact answer is not feasible in general. However, if we can estimate the number of valid permutations, we can estimate the permanent. Jerrum et. al. approached the problem of counting valid permutations by generating valid permutations uniformly (within a certain error, which can be controlled); an estimate of the value of the permanent can be obtained by a fairly elaborate procedure (section 5 of the paper referenced) but we don't need that to answer the question at hand.
The running time of Jerrum's algorithm to calculate the permanent is O(n^11) (ignoring logarithmic factors). I can't immediately tell from the paper the running time of the part of the algorithm that uniformly generates bipartite matchings, but it appears to be over O(n^9). However, another paper reduces the running time for the permanent to O(n^7): http://www.cc.gatech.edu/fac/vigoda/FasterPermanent_SODA.pdf; in that paper they claim that it is now possible to get a good estimate of a permanent of a 100x100 0-1 matrix. So it should be possible to (almost) uniformly generate restricted permutations for lists of 100 elements.
There may be further improvements, but I got tired of looking.
If you want an implementation, I would start with the O(n^11) version in Jerrum's paper, and then take a look at the improvements if the original algorithm is not fast enough.
There is pseudo-code in Jerrum's paper, but I haven't tried it so I can't say how far the pseudo-code is from an actual implementation. My feeling is it isn't too far. Maybe I'll give it a try if there's interest.

I am not sure how good it is, but maybe something like:
create a list of same length than initial list L; each element of this list should be a list of indices of allowed initial indices to be moved here; for instance [[0,1,2],[0,1,2,3],[0,1,2,3],[1,2,3]] if I understand correctly your example;
take the smallest sublist (or any of the smallest sublists if several lists share the same length);
pick a random element in it with random.choice, this element is the index of the element in the initial list to be mapped to the current location (use another list for building your new list);
remove the randomly chosen element from all sublists
For instance:
L = [ "A", "B", "C", "D" ]
i = [[0,1,2],[0,1,2,3],[0,1,2,3],[1,2,3]]
# I take [0,1,2] and pick randomly 1 inside
# I remove the value '1' from all sublists and since
# the first sublist has already been handled I set it to None
# (and my result will look as [ "B", None, None, None ]
i = [None,[0,2,3],[0,2,3],[2,3]]
# I take the last sublist and pick randomly 3 inside
# result will be ["B", None, None, "D" ]
i = [None,[0,2], [0,2], None]
etc.
I haven't tried it however. Regards.

My idea is to generate permutations by moving at most d steps by generating d random permutations which move at most 1 step and chaining them together.
We can generate permutations which move at most 1 step quickly by the following recursive procedure: consider a permutation of {1,2,3,...,n}. The last item, n, can move either 0 or 1 place. If it moves 0 places, n is fixed, and we have reduced the problem to generating a permutation of {1,2,...,n-1} in which every item moves at most one place.
On the other hand, if n moves 1 place, it must occupy position n-1. Then n-1 must occupy position n (if any smaller number occupies position n, it will have moved by more than 1 place). In other words, we must have a swap of n and n-1, and after swapping we have reduced the problem to finding such a permutation of the remainder of the array {1,...,n-2}.
Such permutations can be constructed in O(n) time, clearly.
Those two choices should be selected with weighted probabilities. Since I don't know the weights (though I have a theory, see below) maybe the choice should be 50-50 ... but see below.
A more accurate estimate of the weights might be as follows: note that the number of such permutations follows a recursion that is the same as the Fibonacci sequence: f(n) = f(n-1) + f(n-2). We have f(1) = 1 and f(2) = 2 ({1,2} goes to {1,2} or {2,1}), so the numbers really are the Fibonacci numbers. So my guess for the probability of choosing n fixed vs. swapping n and n-1 would be f(n-1)/f(n) vs. f(n-2)/f(n). Since the ratio of consecutive Fibonacci numbers quickly approaches the Golden Ratio, a reasonable approximation to the probabilities is to leave n fixed 61% of the time and swap n and n-1 39% of the time.
To construct permutations where items move at most d places, we just repeat the process d times. The running time is O(nd).
Here is an outline of an algorithm.
arr = {1,2,...,n};
for (i = 0; i < d; i++) {
j = n-1;
while (j > 0) {
u = random uniform in interval (0,1)
if (u < 0.61) { // related to golden ratio phi; more decimals may help
j -= 1;
} else {
swap items at positions j and j-1 of arr // 0-based indexing
j -= 2;
}
}
}
Since each pass moves items at most 1 place from their start, d passes will move items at most d places. The only question is the uniform distribution of the permutations. It would probably be a long proof, if it's even true, so I suggest assembling empirical evidence for various n's and d's. Probably to prove the statement, we would have to switch from using the golden ratio approximation to f(n-1)/f(n-2) in place of 0.61.
There might even be some weird reason why some permutations might be missed by this procedure, but I'm pretty sure that doesn't happen. Just in case, though, it would be helpful to have a complete inventory of such permutations for some values of n and d to check the correctness of my proposed algorithm.
Update
I found an off-by-one error in my "pseudocode", and I corrected it. Then I implemented in Java to get a sense of the distribution. Code is below. The distribution is far from uniform, I think because there are many ways of getting restricted permutations with short max distances (move forward, move back vs. move back, move forward, for example) but few ways of getting long distances (move forward, move forward). I can't think of a way to fix the uniformity issue with this method.
import java.util.Random;
import java.util.Map;
import java.util.TreeMap;
class RestrictedPermutations {
private static Random rng = new Random();
public static void rPermute(Integer[] a, int d) {
for (int i = 0; i < d; i++) {
int j = a.length-1;
while (j > 0) {
double u = rng.nextDouble();
if (u < 0.61) { // related to golden ratio phi; more decimals may help
j -= 1;
} else {
int t = a[j];
a[j] = a[j-1];
a[j-1] = t;
j -= 2;
}
}
}
}
public static void main(String[] args) {
int numTests = Integer.parseInt(args[0]);
int d = 2;
Map<String,Integer> count = new TreeMap<String,Integer>();
for (int t = 0; t < numTests; t++) {
Integer[] a = {1,2,3,4,5};
rPermute(a,d);
// convert a to String for storage in Map
String s = "(";
for (int i = 0; i < a.length-1; i++) {
s += a[i] + ",";
}
s += a[a.length-1] + ")";
int c = count.containsKey(s) ? count.get(s) : 0;
count.put(s,c+1);
}
for (String k : count.keySet()) {
System.out.println(k + ": " + count.get(k));
}
}
}

Here are two sketches in Python; one swap-based, the other non-swap-based. In the first, the idea is to keep track of where the indexes have moved and test if the next swap would be valid. An additional variable is added for the number of swaps to make.
from random import randint
def swap(a,b,L):
L[a], L[b] = L[b], L[a]
def magicFunction(L,d,numSwaps):
n = len(L)
new = list(range(0,n))
for i in xrange(0,numSwaps):
x = randint(0,n-1)
y = randint(max(0,x - d),min(n - 1,x + d))
while abs(new[x] - y) > d or abs(new[y] - x) > d:
y = randint(max(0,x - d),min(n - 1,x + d))
swap(x,y,new)
swap(x,y,L)
return L
print(magicFunction([1,2,3,4],2,3)) # [2, 1, 4, 3]
print(magicFunction([1,2,3,4,5,6,7,8,9],2,4)) # [2, 3, 1, 5, 4, 6, 8, 7, 9]
Using print(collections.Counter(tuple(magicFunction([0, 1, 2], 1, 1)) for i in xrange(1000))) we find that the identity permutation comes up heavy with this code (the reason why is left as an exercise for the reader).
Alternatively, we can think about it as looking for a permutation matrix with interval restrictions, where abs(i - j) <= d where M(i,j) would equal 1. We can construct a one-off random path by picking a random j for each row from those still available. x's in the following example represent matrix cells that would invalidate the solution (northwest to southeast diagonal would represent the identity permutation), restrictions represent how many is are still available for each j. (Adapted from my previous version to choose both the next i and the next j randomly, inspired by user2357112's answer):
n = 5, d = 2
Start:
0 0 0 x x
0 0 0 0 x
0 0 0 0 0
x 0 0 0 0
x x 0 0 0
restrictions = [3,4,5,4,3] # how many i's are still available for each j
1.
0 0 1 x x # random choice
0 0 0 0 x
0 0 0 0 0
x 0 0 0 0
x x 0 0 0
restrictions = [2,3,0,4,3] # update restrictions in the neighborhood of (i ± d)
2.
0 0 1 x x
0 0 0 0 x
0 0 0 0 0
x 0 0 0 0
x x 0 1 0 # random choice
restrictions = [2,3,0,0,2] # update restrictions in the neighborhood of (i ± d)
3.
0 0 1 x x
0 0 0 0 x
0 1 0 0 0 # random choice
x 0 0 0 0
x x 0 1 0
restrictions = [1,0,0,0,2] # update restrictions in the neighborhood of (i ± d)
only one choice for j = 0 so it must be chosen
4.
0 0 1 x x
1 0 0 0 x # dictated choice
0 1 0 0 0
x 0 0 0 0
x x 0 1 0
restrictions = [0,0,0,0,2] # update restrictions in the neighborhood of (i ± d)
Solution:
0 0 1 x x
1 0 0 0 x
0 1 0 0 0
x 0 0 0 1 # dictated choice
x x 0 1 0
[2,0,1,4,3]
Python code (adapted from my previous version to choose both the next i and the next j randomly, inspired by user2357112's answer):
from random import randint,choice
import collections
def magicFunction(L,d):
n = len(L)
restrictions = [None] * n
restrict = -1
solution = [None] * n
for i in xrange(0,n):
restrictions[i] = abs(max(0,i - d) - min(n - 1,i + d)) + 1
while True:
availableIs = filter(lambda x: solution[x] == None,[i for i in xrange(n)]) if restrict == -1 else filter(lambda x: solution[x] == None,[j for j in xrange(max(0,restrict - d),min(n,restrict + d + 1))])
if not availableIs:
L = [L[i] for i in solution]
return L
i = choice(availableIs)
availableJs = filter(lambda x: restrictions[x] <> 0,[j for j in xrange(max(0,i - d),min(n,i + d + 1))])
nextJ = restrict if restrict != -1 else choice(availableJs)
restrict = -1
solution[i] = nextJ
restrictions[ nextJ ] = 0
for j in xrange(max(0,i - d),min(n,i + d + 1)):
if j == nextJ or restrictions[j] == 0:
continue
restrictions[j] = restrictions[j] - 1
if restrictions[j] == 1:
restrict = j
print(collections.Counter(tuple(magicFunction([0, 1, 2], 1)) for i in xrange(1000)))
Using print(collections.Counter(tuple(magicFunction([0, 1, 2], 1)) for i in xrange(1000))) we find that the identity permutation comes up light with this code (why is left as an exercise for the reader).

Here's an adaptation of #גלעד ברקן's code that takes only one pass through the list (in random order) and swaps only once (using a random choice of possible positions):
from random import choice, shuffle
def magicFunction(L, d):
n = len(L)
swapped = [0] * n # 0: position not swapped, 1: position was swapped
positions = list(xrange(0,n)) # list of positions: 0..n-1
shuffle(positions) # randomize positions
for x in positions:
if swapped[x]: # only swap an item once
continue
# find all possible positions to swap
possible = [i for i in xrange(max(0, x - d), min(n, x + d)) if not swapped[i]]
if not possible:
continue
y = choice(possible) # choose another possible position at random
if x != y:
L[y], L[x] = L[x], L[y] # swap with that position
swapped[x] = swapped[y] = 1 # mark both positions as swapped
return L
Here is a refinement of the above code that simply finds all possible adjacent positions and chooses one:
from random import choice
def magicFunction(L, d):
n = len(L)
positions = list(xrange(0, n)) # list of positions: 0..n-1
for x in xrange(0, n):
# find all possible positions to swap
possible = [i for i in xrange(max(0, x - d), min(n, x + d)) if abs(positions[i] - x) <= d]
if not possible:
continue
y = choice(possible) # choose another possible position at random
if x != y:
L[y], L[x] = L[x], L[y] # swap with that position
positions[x] = y
positions[y] = x
return L