Develop Reference

Fast way to find the maximum of all elements before i in a list in Python

I have an array, X, which I want to make monotonic. Specifically, I want to do
y = x.copy()
for i in range(1, len(x)):
y[i] = np.max(x[:i])
This is extremely slow for large arrays, but it feels like there should be a more efficient way of doing this. How can this operation be sped up?

The OP implementation is very inefficient because it does not use the information acquired on the previous iteration, resulting in O(n²) complexity.
def max_acc_OP(arr):
result = np.empty_like(arr)
for i in range(len(arr)):
result[i] = np.max(arr[:i + 1])
return result
Note that I fixed the OP code (which was otherwise throwing a ValueError: zero-size array to reduction operation maximum which has no identity) by allowing to get the largest value among those up to position i included.
It is easy to adapt that so that values at position i are excluded, but it leaves the first value of the result undefined, and it would never use the last value of the input. The first value of the result can be taken to be equal to the first value of the input, e.g.:
def max_acc2_OP(arr):
result = np.empty_like(arr)
result[0] = arr[0] # uses first value of input
for i in range(1, len(arr) + 1):
result[i] = np.max(arr[:i])
return result
It is equally easy to have similar adaptations for the code below, and I do not think it is particularly relevant to cover both cases of the value at position i included and excluded. Henceforth, only the "included" case is covered.
Back to the efficiency of the solotion, if you keep track of the current maximum and use that to fill your output array instead of re-computing the maximum for all value up to i at each iteration, you can easily get to O(n) complexity:
def max_acc(arr):
result = np.empty_like(arr)
curr_max = arr[0]
for i, x in enumerate(arr):
if x > curr_max:
curr_max = x
result[i] = curr_max
return result
However, this is still relatively slow because of the explicit looping.
Luckily, one can either rewrite this in vectorized form combining np.fmax() (or np.maximum() -- depending on how you need NaNs to be handled) and np.ufunc.accumulate():
np.fmax.accumulate()
# or
np.maximum.accumulate()
or, accelerating the solution above with Numba:
max_acc_nb = nb.njit(max_acc)
Some timings on relatively large inputs are provided below:
n = 10000
arr = np.random.randint(0, n, n)
%timeit -n 4 -r 4 max_acc_OP(arr)
# 97.5 ms ± 14.2 ms per loop (mean ± std. dev. of 4 runs, 4 loops each)
%timeit -n 4 -r 4 np.fmax.accumulate(arr)
# 112 µs ± 134 µs per loop (mean ± std. dev. of 4 runs, 4 loops each)
%timeit -n 4 -r 4 np.maximum.accumulate(arr)
# 88.4 µs ± 107 µs per loop (mean ± std. dev. of 4 runs, 4 loops each)
%timeit -n 4 -r 4 max_acc(arr)
# 2.32 ms ± 146 µs per loop (mean ± std. dev. of 4 runs, 4 loops each)
%timeit -n 4 -r 4 max_acc_nb(arr)
# 9.11 µs ± 3.01 µs per loop (mean ± std. dev. of 4 runs, 4 loops each)
indicating that max_acc() is already much faster than max_acc_OP(), but np.maximum.accumulate() / np.fmax.accumulate() is even faster, and max_acc_nb() comes out as the fastest. As always, it is important to take these kind of numbers with a grain of salt.

I think it will work faster to just keep track of the maximum rather than calculating it each time for each sub-array
y = x.copy()
_max = y[0]
for i in range(1, len(x)):
y[i] = _max
_max = max(x[i], _max)

you can use list comprehension for it. but you need to start your loop from 1 not from 0. either you can use like that if you want loop from 0.
y=[np.max(x[:i+1]) for i in range(len(x))]
or like that
y=[np.max(x[:i]) for i in range(1,len(x)+1)]

Formatting the arrangement of two arrays in order of one in Numpy Python

I am trying to order the zeroes and ones in arrangement of the order. The expected output is what I am trying to get to. Without using a list comprehension preferably.
import numpy as np
order = np.array([0,1,0,1,0])
zeroes= np.array([10,55, 30])
ones = np.array([3,8])
Expected Output
[10, 3, 55, 8, 30]

How about this (no Python loops: 750x faster than a list comprehension, when tested on 200k elements):
# note: updated version: faster and more robust to faulty input
def altcat(zeroes, ones, order):
i0 = np.nonzero(order == 0)[0][:len(zeroes)]
i1 = np.nonzero(order == 1)[0][:len(ones)]
z = np.zeros_like(order, dtype=zeroes.dtype)
z[i0] = zeroes[:len(i0)]
z[i1] = ones[:len(i1)]
return z
On your example:
>>> altcat(zeroes=np.array([10,55, 30]), ones=np.array([3,8]),
... order=np.array([0,1,0,1,0]))
array([10, 3, 55, 8, 30])
Speed
# set up
n = 200_000
np.random.seed(0)
order = np.random.randint(0, 2, size=n)
n1 = order.sum()
n0 = n - n1
ones = np.random.randint(100, size=n1)
zeroes = np.random.randint(100, size=n0)
# for comparison, a method proposed elsewhere, based on lists
def altcat_list(zeroes, ones, order):
zeroes = list(zeroes)
ones = list(ones)
return [zeroes.pop(0) if i == 0 else ones.pop(0) for i in order]
Test:
a = %timeit -o altcat(zeroes, ones, order)
# 2.38 ms ± 573 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)
b = %timeit -o altcat_list(zeroes, ones, order)
# 1.84 s ± 1.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
b.average / a.average
# 773.59
Note: I initially tried with n = 1_000_000, but while altcat does that in 12.4ms, the list-based version would take forever and I had to stop it.
It seems that the list-based method is worse than O(n) (100K: 0.4s; 200K: 1.84s; 400K: 10.4s).
Addendum
If you really want to do it with a list comprehension and not in pure numpy, then at least consider this:
def altcat_list_mod(zeroes, ones, order):
it = [iter(zeroes), iter(ones)]
return [next(it[i]) for i in order]
That's faster than altcat_list(), but still almost 25x slower than altcat():
# on 200k elements
c = %timeit -o altcat_list_mod(zeroes, ones, order)
# 60 ms ± 24.3 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
c.average / a.average
# 24.93

More optimized math function using NumPy

for my class I need to write more optimized math function using NumPy. Problem is, when using NumPy my solutions are slower when native Python.
function which cubes all the elements of an array and sum them
Python:
def cube(x):
result = 0
for i in range(len(x)):
result += x[i] ** 3
return result
My, using NumPy (15-30% slower):
def cube(x):
it = numpy.nditer([x, None])
for a, b in it:
b[...] = a*a*a
return numpy.sum(it.operands[1])
Some random calculation function
Python:
def calc(x):
m = sum(x) / len(x)
result = 0
for i in range(len(x)):
result += (x[i] - m)**4
return result / len(x)
NumPy (>10x slower):
def calc(x):
m = numpy.mean(x)
result = 0
for i in range(len(x)):
result += numpy.power((x[i] - m), 4)
return result / len(x)
I don't know how to approatch this, so far I have tried random functions from NumPy

To elaborate on what has been said in comments:
Numpy's power comes from being able to do all the looping in fast c/fortran rather than slow Python looping. For example, if you have an array x and you want to calculate the square of every value in that array, you could do
y = []
for value in x:
y.append(value**2)
or even (with a list comprehension)
y = [value**2 for value in x]
but it will be much faster if you can do all the looping inside numpy with
y = x**2
(assuming x is already a numpy array).
So for your examples, the proper way to do it in numpy would be
1.
def sum_of_cubes(x):
result = 0
for i in range(len(x)):
result += x[i] ** 3
return result
def sum_of_cubes_numpy(x):
return (x**3).sum()
def calc(x):
m = sum(x) / len(x)
result = 0
for i in range(len(x)):
result += (x[i] - m)**4
return result / len(x)
def calc_numpy(x):
m = numpy.mean(x) # or just x.mean()
return numpy.sum((x - m)**4) / len(x)
Note that I've assumed that the input x is already a numpy array, not a regular Python list: if you have a list lst, you can create an array from it with arr = numpy.array(lst).

In [337]: def cube(x):
...: result = 0
...: for i in range(len(x)):
...: result += x[i] ** 3
...: return result
...:
nditer is not a good numpy iterator, at least not when used in Python level code. It's really just a stepping stone toward writing compiled code. It's docs need a better disclaimer.
In [338]: def cube1(x):
...: it = numpy.nditer([x, None])
...: for a, b in it:
...: b[...] = a*a*a
...: return numpy.sum(it.operands[1])
...:
In [339]: cube(list(range(10)))
Out[339]: 2025
In [340]: cube1(list(range(10)))
Out[340]: 2025
In [341]: cube1(np.arange(10))
Out[341]: 2025
A more direct numpy iteration:
In [342]: def cube2(x):
...: it = [a*a*a for a in x]
...: return numpy.sum(it)
...:
The better whole-array code. Just as sum can work with the whole array, the power also applies the whole.
In [343]: def cube3(x):
...: return numpy.sum(x**3)
...:
In [344]: cube2(np.arange(10))
Out[344]: 2025
In [345]: cube3(np.arange(10))
Out[345]: 2025
Doing some timings:
The list reference:
In [346]: timeit cube(list(range(1000)))
438 µs ± 9.87 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
The slow nditer:
In [348]: timeit cube1(np.arange(1000))
2.8 ms ± 5.65 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
The partial numpy:
In [349]: timeit cube2(np.arange(1000))
520 µs ± 20 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
I can improve its time by passing a list instead of an array. Iteration on lists is faster.
In [352]: timeit cube2(list(range(1000)))
229 µs ± 9.53 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
But the time for a 'pure' numpy version blows all of those out of the water:
In [350]: timeit cube3(np.arange(1000))
23.6 µs ± 128 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
The general rule is that numpy methods applied to a numpy array are fastest. But if you must loop, it's usually better to use lists.
Sometimes the pure numpy approach creates very large temporary array. Then memory management complexities can reduce performance. In such cases a modest of number of iterations on a complex task may be best.

How to remove the last occurrence of an item from a list?

The list.remove() function serves to remove the first time an item appears in a list. Is there a built-in function to remove the last time? For instance if I have a list, say:
X = ['sf', 'cc', 'ch', 'sc', 'sh', 'ch']
and I want to remove the last 'ch' from the list, is there a better method than what I'm currently doing, which is:
X.reverse()
X.remove('ch')
X.reverse()
I will soon also have to worry about cases where the item being removed is potentially not in the list. So methods that do not throw errors in this case would be preferred.

if 'ch' in X:
X.reverse()
X.remove('ch')
X.reverse()
The most pythonic way would be to do a try: except around remove:
X.reverse()
try:
X.remove('ch')
except:
pass
X.reverse()
As per your comment on speed, both of these methods are O(N), as x in list and list.reverse() are both O(N), so there's not much between them. If you expect the element to usually be there, you can save the x in list check by using try: catch, however if you expect it to usually not be there, you can save the 2 reverse()s by checking for membership first.

There's really nothing wrong with your code at all. It works, it's clear why it works, it's hard to get wrong or misunderstand.
Yes, you could make it faster, but only by a constant factor. (Your algorithm does a two reverses, for N steps each, and one remove, which is N-1 steps, so O(N). And since your data aren't sorted or anything that would help us find a value faster, it's obvious that the ideal algorithm would also be O(N).) And at the cost of making it more complicated.
The obvious probably-faster way to do it is to just manually iterate from the end until we find a value, then delete that value. That also avoids having to deal with the ValueError. Using enumerate might help… but getting it right (without copying the whole thing) may be tricky.
So, let's compare these to your existing code, both wrapped it in a try/except, and in an if:
def f_rev_ex(xs, s):
xs.reverse()
try:
xs.remove(s)
except ValueError:
pass
xs.reverse()
def f_rev_if(xs, s):
if s in xs:
xs.reverse()
xs.remove(s)
xs.reverse()
def f_for(xs, s):
for i in range(len(xs)-1, -1, -1):
if s == xs[i]:
del xs[i]
break
def f_enum(xs, s):
for i, x in reversed(list(enumerate(xs))):
if x == s:
del xs[i]
break
For a list as tiny as yours, the test isn't even worth running, so I invented my own random data (in real life you have to know your data, of course):
In [58]: xs = [random.choice(string.ascii_lowercase) for _ in range(10000)]
In [59]: %timeit y = x[:]; f_rev_ex(y, 'a')
10000 loops, best of 3: 34.7 µs per loop
In [60]: %timeit y = x[:]; f_rev_if(y, 'a')
10000 loops, best of 3: 35.1 µs per loop
In [61]: %timeit y = x[:]; f_for(y, 'a')
10000 loops, best of 3: 26.6 µs per loop
In [62]: %timeit y = x[:]; f_enum(y, 'a')
1000 loops, best of 3: 604 µs per loop
Well, that last one wasn't a very good idea… but the other one is about 25% faster than what we started with. So we've saved a whole 9 microseconds, on data 4 orders of magnitude larger than your actual data. It's up to you whether that's worth the less-readable, easier-to-screw up code. (And I'm not going to show you my enumerate-based implementation without copying, because I got it wrong. :P)

Produce a reversed list, preserving the original indexes and remove the first instance you find.
X = ['sf', 'cc', 'ch', 'sc', 'sh', 'ch']
print X
for i, e in reversed(list(enumerate(X))):
if e == 'ch':
del X[i]
break
print X
If it doesn't find the string it leaves the list untouched.

Without reverse() and similar to one answer above:
def RightRemove(alist, x):
for i in range(len(alist), 0, -1): # from end to begin
if alist[i-1] == x: # element x exists
alist.pop(i-1) # remove it
break # return

Well first you can check if the item is in the list using a if in statement. Then you can reverse the list and remove the element.
if "ch" in X:
X.reverse()
X.remove("ch")
X.reverse()

Yet another answer...
def remove_last_occurrence(lst, element):
'''
Removes the last occurrence of a given element in a list (modifies list in-place).
:return bool:
True if the element was found and False otherwise.
'''
for i, s in enumerate(reversed(lst)):
if s == element:
del lst[len(lst) - 1 - i]
return True
return False

Yet another one ..
def remove_last_occurrence_one_liner(lst, element):
"""
Removes the last occurrence of a given element in a list (modifies list in-place).
Raises same exception than lst.index(element) if element can not be found.
"""
del lst[len(lst) - lst[::-1].index(element) - 1]
But it does not beat the for loop from abarnert
x = [random.choice(string.ascii_lowercase) for _ in range(10000)]
%timeit y = x[:]; f_rev_ex(y, 'a')
34.3 µs ± 219 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit y = x[:]; f_rev_if(y, 'a')
34.9 µs ± 195 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit y = x[:]; f_for(y, 'a')
26.9 µs ± 109 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit y = x[:]; f_enum(y, 'a')
699 µs ± 4.86 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit y = x[:]; remove_last_occurrence_one_liner(y, 'a')
49 µs ± 375 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Inverting permutations in Python

I'm new to programming, and I'm trying to write a Python function to find the inverse of a permutation on {1,2,3,...,n} using the following code:
def inv(str):
result = []
i = list(str).index(min(list(str)))
while min(list(str)) < len(list(str)) + 1:
list(str)[i : i + 1] = [len(list(str)) + 1]
result.append(i + 1)
return result
However, when I try to use the function, inv('<mypermutation>') returns []. Am I missing something? Is Python skipping over my while loop for some syntactical reason I don't understand? None of my google and stackoverflow searches on topics I think of are returning anything helpful.

Other answers are correct, but for what it's worth, there's a much more performant alternative using numpy:
inverse_perm = np.argsort(permutation)
EDIT: and the fourth function below is even faster.
Timing code:
def invert_permutation_list_scan(p):
return [p.index(l) for l in range(len(p))]
def invert_permutation_list_comp(permutation):
return [i for i, j in sorted(enumerate(permutation), key=lambda i_j: i_j[1])]
def invert_permutation_numpy(permutation):
return np.argsort(permutation)
def invert_permutation_numpy2(permutation):
inv = np.empty_like(permutation)
inv[permutation] = np.arange(len(inv), dtype=inv.dtype)
return inv
x = np.random.randn(1000)
perm = np.argsort(x)
permlist = list(perm)
assert np.array_equal(invert_permutation_list_scan(permlist), invert_permutation_numpy(perm))
assert np.array_equal(invert_permutation_list_comp(perm), invert_permutation_numpy(perm))
assert np.array_equal(invert_permutation_list_comp(perm), invert_permutation_numpy2(perm))
%timeit invert_permutation_list_scan(permlist)
%timeit invert_permutation_list_comp(perm)
%timeit invert_permutation_numpy(perm)
%timeit invert_permutation_numpy2(perm)
Results:
82.2 ms ± 7.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
479 µs ± 9.19 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
18 µs ± 1.17 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)
4.22 µs ± 388 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

If you only want the inverse permutation, you can use
def inv(perm):
inverse = [0] * len(perm)
for i, p in enumerate(perm):
inverse[p] = i
return inverse
perm = [3, 0, 2, 1]
print(inv(perm))
for i in perm:
print(inv(perm)[i])
[1, 3, 2, 0]
0
1
2
3

I believe the best way to invert a permutation perm is
pinv = sorted(range(len(perm)), key=perm.__getitem__)
This avoids repeated calls to .index() (as in the answer by SeF), which may not be very efficient (quadratic time complexity, while sorting should only take O(n log n)).
Note, however, that this yields as a result a permutation of {0,1,...n-1}, regardless of whether the input was a permutation of {0,1,...,n-1} or of {1,2,...,n} (the latter is what is stated in the question). If the output is supposed to be a permutation of {1,2,...,n}, each element of the result has to be increased by one, for example, like this:
pinv = [i+1 for i in sorted(range(len(perm)), key=perm.__getitem__)]

Correct me if I have this wrong, but I think the problem with my code comes when I change str to a list: str is a string, and list(str) is a list of string elements. However, since string elements can't be numerically compared to numbers, the code fails to produce a result (other than []).

A "functional style" version:
def invert_permutation(permutation):
return [i for i, j in sorted(enumerate(permutation), key=lambda (_, j): j)]
Basically, sorting the indices i of the permutation by their values j in the permutation yields the desired inverse.
p = [2, 1, 5, 0, 4, 3]
invert_permutation(p)
# [3, 1, 0, 5, 4, 2]
# inverse of inverse = identity
invert_permutation(invert_permutation(p)) == p
# True

Just since no one has recommended it here yet, I think it should be mentioned that SymPy has an entire combinatorics module, with a Permutation class:
from sympy.combinatorics import Permutation
o = [3, 0, 2, 1]
p = Permutation(o)
inv = p.__invert__()
print(inv.array_form) # [1, 3, 2, 0]
Using the SymPy class gives you access to a whole lot of other useful methods, such as comparison between equivalent permutations with ==.
You can read the sympy.combinatorics.Permutation source code here.
Other than that, I would recommend the answer on this page using np.arange and argsort.

Maybe there is a shorter way:
def invert(p):
return [p.index(l) for l in range(len(p))]
so that:
perm = [3, 0, 2, 1]; print(invert(perm))
returns
[1,3,2,0]