Related
a = [1, 3, 6, -2, 4, 5, 8, -3, 9,
2, -5, -7, -9, 3, 6, -7, -6, 2]
I want to do like:
a = [1, 3, 6, 4, 5, 8, 9, 2,
-5, -7, -9, 3, 6, -7, -6, 2]
which deletes only 4th and 8th elements, which are single negative elements between two positive elements.
import numpy as np
a = [1, 3, 6, -2, 4, 5, 8, -3, 9,
2, -5, -7, -9, 3, 6, -7, -6, 2]
for i in range(len(a)):
if a[i] < 0 and a[i - 1] > 0 and a[i + 1] > 0:
np.delete(a[i])
print(a)
This did not work. Can I know where I have to fix?
Because you ask about numpy in the subject line and also attempt to use np.delete() in your code, I assume you intend for a to be a numpy array.
Here is a way to do what your question asks using vectorized operations in numpy:
import numpy as np
a = np.array([1,3,6,-2,4,5,8,-3,9,2,-5,-7,-9, 3, 6, -7, -6, 2])
b = np.concatenate([a[1:], [np.NaN]])
c = np.concatenate([[np.NaN], a[:-1]])
d = (a<0)&(b>0)&(c>0)
print(a[~d])
Output:
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
What we've done is to shift a one to the left with NaN fill on the right (b) and one to the right with NaN fill on the left (c), then to create a boolean mask d using vectorized compare and boolean operators <, > and & which is True only where we want to delete single negative values sandwiched between positives. Finally, we use the ~ operator to flip the boolean value of the mask and use it to filter out the unneeded negative values in a.
UPDATE: Based on benchmarking of several possible strategies for answering your question (see below), the conclusion is that the following solution appears to be the most performative in answering OP's question (credit to #Kelly Bundy for suggesting this in a comment):
a = np.concatenate((a[:1], a[1:-1][(a[1:-1]>=0)|(a[2:]<=0)|(a[:-2]<=0)], a[-1:]))
UPDATE: Here are some timeit() comparisons of several variations on answers given for this question using NumPy 1.22.2.
The fastest of the 8 strategies is:
a = np.concatenate([a[:1], a[1:-1][(a[1:-1]>=0)|(a[2:]<=0)|(a[:-2]<=0)], a[-1:]])
A close second is:
a = a[np.concatenate([[True], ~((a[1:-1]<0)&(a[2:]>0)&(a[:-2]>0)), [True]])]
The strategies using np.r_(), either with np.delete() or with a boolean mask and [] syntax, are about twice as slow as the fastest.
The strategy using numpy.roll() is about 3 times as slow as the fastest. Note: As highlighted by in a comment by #Kelly Bundy, the roll() strategy in the benchmark does not give a correct answer to this question in all cases (though for the particular input example it happens to). I have nevertheless included it in the benchmark because the performance of roll() relative to concatenate() and r_() may be of general interest beyond the narrow context of this question.
Results:
foo_1 output:
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
foo_2 output:
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
foo_3 output:
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
foo_4 output:
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
foo_5 output:
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
foo_6 output:
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
foo_7 output:
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
foo_8 output:
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
Timeit results:
foo_1 ran in 1.2354546000715346e-05 seconds using 100000 iterations
foo_2 ran in 1.0962473000399769e-05 seconds using 100000 iterations
foo_3 ran in 7.733614000026136e-06 seconds using 100000 iterations
foo_4 ran in 7.751871000509709e-06 seconds using 100000 iterations
foo_5 ran in 5.856722998432815e-06 seconds using 100000 iterations
foo_6 ran in 7.5727709988132115e-06 seconds using 100000 iterations
foo_7 ran in 1.7790602000895887e-05 seconds using 100000 iterations
foo_8 ran in 5.435103999916464e-06 seconds using 100000 iterations
Code that generated the results:
import numpy as np
a = np.array([1,3,6,-2,4,5,8,-3,9,2,-5,-7,-9, 3, 6, -7, -6, 2])
from timeit import timeit
def foo_1(a):
a = a if a.shape[0] < 2 else np.delete(a, np.r_[False, (a[1:-1] < 0) & (a[:-2] > 0) & (a[2:] > 0), False])
return a
def foo_2(a):
a = a if a.shape[0] < 2 else a[np.r_[True, ~((a[1:-1] < 0) & (a[:-2] > 0) & (a[2:] > 0)), True]]
return a
def foo_3(a):
b = np.concatenate([a[1:], [np.NaN]])
c = np.concatenate([[np.NaN], a[:-1]])
d = (a<0)&(b>0)&(c>0)
a = a[~d]
return a
def foo_4(a):
a = a[~((a<0)&(np.concatenate([a[1:], [np.NaN]])>0)&(np.concatenate([[np.NaN], a[:-1]])>0))]
return a
def foo_5(a):
a = a if a.shape[0] < 2 else a[np.concatenate([[True], ~((a[1:-1]<0)&(a[2:]>0)&(a[:-2]>0)), [True]])]
return a
def foo_6(a):
a = a if a.shape[0] < 2 else np.delete(a, np.concatenate([[False], (a[1:-1]<0)&(a[2:]>0)&(a[:-2]>0), [False]]))
return a
def foo_7(a):
mask_bad = (
(a < 0) & # the value is < 0 AND
(np.roll(a,1) >= 0) & # the value to the right is >= 0
(np.roll(a,-1) >= 0) # the value to the left is >= 0
)
mask_good = ~mask_bad
a = a[mask_good]
return a
def foo_8(a):
a = np.concatenate([a[:1], a[1:-1][(a[1:-1]>=0)|(a[2:]<=0)|(a[:-2]<=0)], a[-1:]])
return a
foo_count = 8
for foo in ['foo_' + str(i + 1) for i in range(foo_count)]:
print(f'{foo} output:')
print(eval(f"{foo}(a)"))
n = 100000
print(f'Timeit results:')
for foo in ['foo_' + str(i + 1) for i in range(foo_count)]:
t = timeit(f"{foo}(a)", setup=f"from __main__ import a, {foo}", number=n) / n
print(f'{foo} ran in {t} seconds using {n} iterations')
A solution that handles edges correctly and doesn't create an unholy number of temporary arrays:
a = np.delete(a, np.r_[False, (a[1:-1] < 0) & (a[:-2] > 0) & (a[2:] > 0), False])
Alternatively, you can create the positive rather than the negative mask
a = a[np.r_[True, (a[1:-1] >= 0) | (a[:-2] <= 0) | (a[2:] <= 0), True]]
Since np.concatenate is faster than np.r_, you could rephrase the masks as
np.concatenate(([False], (a[1:-1] < 0) & (a[:-2] > 0) & (a[2:] > 0), [False])
and
np.concatenate(([True], (a[1:-1] >= 0) | (a[:-2] <= 0) | (a[2:] <= 0), [True]))
In some cases, you might get extra mileage out of applying np.where(...)[0] or np.flatnonzero to the mask. This works sometimes because it avoids having to recompute the size of the number of masked elements twice.
Your conditional logic
if a[i] < 0 and a[i - 1] > 0 and a[i + 1] > 0
seems sound and readable to me. But it would have issues with the boundary cases:
[1, 2, -3] -> IndexError: list index out of range
[-1, 2, 3] -> [2, 3]
Handling it properly could be as simple as skipping the first and last element of you list with
for i in range(1, len(a) - 1)
Test
import numpy as np
def del_neg_between_pos(a):
delete_idx = []
for i in range(1, len(a) - 1):
if a[i] < 0 and a[i - 1] > 0 and a[i + 1] > 0:
delete_idx.append(i)
return np.delete(a, delete_idx)
if __name__ == "__main__":
a1 = [1, 3, 6, -2, 4, 5, 8, -3, 9, 2, -5, -7, -9, 3, 6, -7, -6, 2]
a2 = [1, 2, -3]
a3 = [-1, 2, 3]
for a in [a1, a2, a3]:
print(del_neg_between_pos(a))
Output
[ 1 3 6 4 5 8 9 2 -5 -7 -9 3 6 -7 -6 2]
[ 1 2 -3]
[-1 2 3]
A one-liner is
a[1:-1] = [a[i] for i in range(1, len(a) - 1) if not (a[i] < 0 and a[i-1] > 0 and a[i+1] > 0)]
The above assigns elements from the list a that are not negative and preceded and followed by a positive number to a slicing of a.
Output (from printing a)
[1, 3, 6, 4, 5, 8, 9, 2, -5, -7, -9, 3, 6, -7, -6, 2]
Timings
The below timings compare my approach above to the fastest approach in #constantstranger's answer:
a = np.concatenate([a[:1], a[1:-1][(a[1:-1]>=0)|(a[2:]<=0)|(a[:-2]<=0)], a[-1:]])
My suggested approach is obviously optimized for the case where you want both the input and output to be a list. However, even in suboptimal (for my approach) input/output configurations, for this input, my approach appears to be faster than the numpy approach.
Input/Output Configuration 1
Input is a list (as in your question).
Output is a numpy array.
In [3]: %%timeit
...: a = [1, 3, 6, -2, 4, 5, 8, -3, 9, 2, -5, -7, -9, 3, 6, -7, -6, 2]
...: a[1:-1] = [a[i] for i in range(1, len(a) - 1) if not (a[i] < 0 and a[i-
...: 1] > 0 and a[i+1] > 0)]
...: a = np.array(a)
...:
...:
3.08 µs ± 17.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
In [5]: %%timeit
...: a = np.array([1, 3, 6, -2, 4, 5, 8, -3, 9, 2, -5, -7, -9, 3, 6, -7, -6,
...: 2])
...: a = np.concatenate([a[:1], a[1:-1][(a[1:-1]>=0)|(a[2:]<=0)|(a[:-2]<=0)]
...: , a[-1:]])
...:
...:
6.66 µs ± 16.1 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
Input/Output Configuration 2
Input and output is a numpy array (as assumed in other answers).
The input
a = np.array([1, 3, 6, -2, 4, 5, 8, -3, 9, 2, -5, -7, -9, 3, 6, -7, -6, 2])
Timings:
In [3]: %%timeit
...: b = a.tolist()
...: b[1:-1] = [b[i] for i in range(1, len(b) - 1) if not (b[i] < 0 and b[i-
...: 1] > 0 and b[i+1] > 0)]
...: b = np.array(b)
...:
...:
3.1 µs ± 10.7 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
In [5]: %%timeit
...: b = np.concatenate([a[:1], a[1:-1][(a[1:-1]>=0)|(a[2:]<=0)|(a[:-2]<=0)]
...: , a[-1:]])
...:
...:
4.8 µs ± 13.9 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
Remarks
The above holds for this specific input. A larger input size may have different results (particularly due to the conversion between types). I would be happy to provide timings that vary the input size (presented graphically). However, it would be useful to know whether you want the input or output to be a list or a numpy array.
I have a dataframe test which is as below
Student_Id Math Physical Arts Class Sub_Class
0 id_1 6 7 9 A x
1 id_2 9 7 1 A y
2 id_3 3 5 5 C x
3 id_4 6 8 9 A x
4 id_5 6 7 10 B z
5 id_6 9 5 10 B z
6 id_7 3 5 6 C x
7 id_8 3 4 6 C x
8 id_9 6 8 9 A x
9 id_10 6 7 10 B z
10 id_11 9 5 10 B z
11 id_12 3 5 6 C x
There are two arrays as listed in the My Code section: arr_list and array_top.
I want to create a new column such that it loops through each row of the dataframe and then update the value from the arrays as below:
for index, row in test.iterrows():
test.loc[index,'Highest_Score'] = arr_list [index][array_top [index]]
This looping takes too much of time for a bigger set. Is there a faster way to do this?
My Code
import pandas as pd
import numpy as np
#Ceate dataframe
data = [
["id_1",6,7,9, "A", "x"],
["id_2",9,7,1, "A","y" ],
["id_3",3,5,5, "C", "x"],
["id_4",6,8,9, "A","x" ],
["id_5",6,7,10, "B", "z"],
["id_6",9,5,10,"B", "z"],
["id_7",3,5,6, "C", "x"],
["id_8",3,4,6, "C", "x"],
["id_9",6,8,9, "A","x" ],
["id_10",6,7,10, "B", "z"],
["id_11",9,5,10,"B", "z"],
["id_12",3,5,6, "C", "x"]
]
test = pd.DataFrame(data, columns = ['Student_Id', 'Math', 'Physical','Arts', 'Class', 'Sub_Class'])
#Create two arrays which are of same length as the test data
arr_list = np.array([[1, 2, 3], [4, 5, 6], [4, 5, 6], [4, 5, 6], [4, 5, 6], [4, 5, 6], [1, 2, 3], [4, 5, 6], [4, 5, 6], [4, 5, 6], [4, 5, 6], [4, 5, 6]])
array_top = np.array([[0],[1],[1],[2],[1], [0], [0],[1],[1],[2],[1], [0]])
#Create the column Highest_Scoe
for index, row in test.iterrows():
test.loc[index,'Highest_Score'] = arr_list [index][array_top [index]]
Looping through the arrays first to create your new column, then just assigning it to the dataframe will be much faster than looping through each row of the dataframe
71.7 µs vs 2.77 ms (a.k.a. 39 times faster) by my time trial
In [95]: %%timeit
...: new_test['Highest_Score'] = [arr_list[r][c][0] for r,c in enumerate(array_top)]
...:
...:
71.7 µs ± 1.7 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [96]: %%timeit
...: for index, row in test.iterrows():
...: test.loc[index,'Highest_Score'] = arr_list [index][array_top [index]]
...:
2.77 ms ± 49.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
As a general rule with adding new data to a pandas DataFrame, you want to do all of the looping and compiling outside of pandas, and then assign all of the data all at once
I have large 1D NumPy array a of any comparable dtype, some of its elements may be repeated.
How do I find sorting indexes ix that will stable-sort (stability in a sense described here) a by frequencies of values in descending/ascending orders?
I want to find fastest and simplest way to do this. Maybe there is existing standard numpy function to do that.
There is another related question here but it was asking specifically to remove arrays duplicates, i.e. output only unique sorted values, I need all values of original array including duplicates.
I've coded my first trial to do the task, but it is not the fastest (uses Python's loop) and probably not shortest/simplest possible form. This python loop can be very expensive if repeating of equal elements is not high and array is huge. Also would be nice to have short function for doing this all if available in NumPy (e.g. imaginary np.argsort_by_freq()).
Try it online!
import numpy as np
np.random.seed(1)
hi, n, desc = 7, 24, True
a = np.random.choice(np.arange(hi), (n,), p = (
lambda p = np.random.random((hi,)): p / p.sum()
)())
us, cs = np.unique(a, return_counts = True)
af = np.zeros(n, dtype = np.int64)
for u, c in zip(us, cs):
af[a == u] = c
if desc:
ix = np.argsort(-af, kind = 'stable') # Descending sort
else:
ix = np.argsort(af, kind = 'stable') # Ascending sort
print('rows: i_col(0) / original_a(1) / freqs(2) / sorted_a(3)')
print(' / sorted_freqs(4) / sorting_ix(5)')
print(np.stack((
np.arange(n), a, af, a[ix], af[ix], ix,
), 0))
outputs:
rows: i_col(0) / original_a(1) / freqs(2) / sorted_a(3)
/ sorted_freqs(4) / sorting_ix(5)
[[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]
[ 1 1 1 1 3 0 5 0 3 1 1 0 0 4 6 1 3 5 5 0 0 0 5 0]
[ 7 7 7 7 3 8 4 8 3 7 7 8 8 1 1 7 3 4 4 8 8 8 4 8]
[ 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 5 5 5 5 3 3 3 4 6]
[ 8 8 8 8 8 8 8 8 7 7 7 7 7 7 7 4 4 4 4 3 3 3 1 1]
[ 5 7 11 12 19 20 21 23 0 1 2 3 9 10 15 6 17 18 22 4 8 16 13 14]]
I might be missing something, but it seems that with a Counter you can then sort the indexes of each element according to the count of that element's value, using the element value and then the index to break ties. For example:
from collections import Counter
a = [ 1, 1, 1, 1, 3, 0, 5, 0, 3, 1, 1, 0, 0, 4, 6, 1, 3, 5, 5, 0, 0, 0, 5, 0]
counts = Counter(a)
t = [(counts[v], v, i) for i, v in enumerate(a)]
t.sort()
print([v[2] for v in t])
t.sort(reverse=True)
print([v[2] for v in t])
Output:
[13, 14, 4, 8, 16, 6, 17, 18, 22, 0, 1, 2, 3, 9, 10, 15, 5, 7, 11, 12, 19, 20, 21, 23]
[23, 21, 20, 19, 12, 11, 7, 5, 15, 10, 9, 3, 2, 1, 0, 22, 18, 17, 6, 16, 8, 4, 14, 13]
If you want to maintain ascending order of indexes with groups with equal counts, you can just use a lambda function for the descending sort:
t.sort(key = lambda x:(-x[0],-x[1],x[2]))
print([v[2] for v in t])
Output:
[5, 7, 11, 12, 19, 20, 21, 23, 0, 1, 2, 3, 9, 10, 15, 6, 17, 18, 22, 4, 8, 16, 14, 13]
If you want to maintain the ordering of elements in the order that they originally appeared in the array if their counts are the same, then rather than sort on the values, sort on the index of their first occurrence in the array:
a = [ 1, 1, 1, 1, 3, 0, 5, 0, 3, 1, 1, 0, 0, 4, 6, 1, 3, 5, 5, 0, 0, 0, 5, 0]
counts = Counter(a)
idxs = {}
t = []
for i, v in enumerate(a):
if not v in idxs:
idxs[v] = i
t.append((counts[v], idxs[v], i))
t.sort()
print([v[2] for v in t])
t.sort(key = lambda x:(-x[0],x[1],x[2]))
print([v[2] for v in t])
Output:
[13, 14, 4, 8, 16, 6, 17, 18, 22, 0, 1, 2, 3, 9, 10, 15, 5, 7, 11, 12, 19, 20, 21, 23]
[5, 7, 11, 12, 19, 20, 21, 23, 0, 1, 2, 3, 9, 10, 15, 6, 17, 18, 22, 4, 8, 16, 13, 14]
To sort according to count, and then position in the array, you don't need the value or the first index at all:
from collections import Counter
a = [ 1, 1, 1, 1, 3, 0, 5, 0, 3, 1, 1, 0, 0, 4, 6, 1, 3, 5, 5, 0, 0, 0, 5, 0]
counts = Counter(a)
t = [(counts[v], i) for i, v in enumerate(a)]
t.sort()
print([v[1] for v in t])
t.sort(key = lambda x:(-x[0],x[1]))
print([v[1] for v in t])
This produces the same output as the prior code for the sample data, for your string array:
a = ['g', 'g', 'c', 'f', 'd', 'd', 'g', 'a', 'a', 'a', 'f', 'f', 'f',
'g', 'f', 'c', 'f', 'a', 'e', 'b', 'g', 'd', 'c', 'b', 'f' ]
This produces the output:
[18, 19, 23, 2, 4, 5, 15, 21, 22, 7, 8, 9, 17, 0, 1, 6, 13, 20, 3, 10, 11, 12, 14, 16, 24]
[3, 10, 11, 12, 14, 16, 24, 0, 1, 6, 13, 20, 7, 8, 9, 17, 2, 4, 5, 15, 21, 22, 19, 23, 18]
I just figured myself probably very fast solution for any dtype using just numpy functions without python looping, it works in O(N log N) time. Used numpy functions: np.unique, np.argsort and array indexing.
Although wasn't asked in original question, I implemented extra flag equal_order_by_val if it is False then array elements with same frequencies are sorted as equal stable range, meaning that there could be c d d c d c output like in outputs dumps below, because this is the order as elements go in original array for equal frequency. When flag is True such elements are in addition sorted by value of original array, resulting in c c c d d d. In other words in case of False we sort stably just by key freq, and when it is True we sort by (freq, value) for ascending order and by (-freq, value) for descending order.
Try it online!
import string, math
import numpy as np
np.random.seed(0)
# Generating input data
hi, n, desc = 7, 25, True
letters = np.array(list(string.ascii_letters), dtype = np.object_)[:hi]
a = np.random.choice(letters, (n,), p = (
lambda p = np.random.random((letters.size,)): p / p.sum()
)())
for equal_order_by_val in [False, True]:
# Solving task
us, ui, cs = np.unique(a, return_inverse = True, return_counts = True)
af = cs[ui]
sort_key = -af if desc else af
if equal_order_by_val:
shift_bits = max(1, math.ceil(math.log(us.size) / math.log(2)))
sort_key = ((sort_key.astype(np.int64) << shift_bits) +
np.arange(us.size, dtype = np.int64)[ui])
ix = np.argsort(sort_key, kind = 'stable') # Do sorting itself
# Printing results
print('\nequal_order_by_val:', equal_order_by_val)
for name, val in [
('i_col', np.arange(n)), ('original_a', a),
('freqs', af), ('sorted_a', a[ix]),
('sorted_freqs', af[ix]), ('sorting_ix', ix),
]:
print(name.rjust(12), ' '.join([str(e).rjust(2) for e in val]))
outputs:
equal_order_by_val: False
i_col 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
original_a g g c f d d g a a a f f f g f c f a e b g d c b f
freqs 5 5 3 7 3 3 5 4 4 4 7 7 7 5 7 3 7 4 1 2 5 3 3 2 7
sorted_a f f f f f f f g g g g g a a a a c d d c d c b b e
sorted_freqs 7 7 7 7 7 7 7 5 5 5 5 5 4 4 4 4 3 3 3 3 3 3 2 2 1
sorting_ix 3 10 11 12 14 16 24 0 1 6 13 20 7 8 9 17 2 4 5 15 21 22 19 23 18
equal_order_by_val: True
i_col 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
original_a g g c f d d g a a a f f f g f c f a e b g d c b f
freqs 5 5 3 7 3 3 5 4 4 4 7 7 7 5 7 3 7 4 1 2 5 3 3 2 7
sorted_a f f f f f f f g g g g g a a a a c c c d d d b b e
sorted_freqs 7 7 7 7 7 7 7 5 5 5 5 5 4 4 4 4 3 3 3 3 3 3 2 2 1
sorting_ix 3 10 11 12 14 16 24 0 1 6 13 20 7 8 9 17 2 15 22 4 5 21 19 23 18
I have array and need max of rolling difference with dynamic window.
a = np.array([8, 18, 5,15,12])
print (a)
[ 8 18 5 15 12]
So first I create difference by itself:
b = a - a[:, None]
print (b)
[[ 0 10 -3 7 4]
[-10 0 -13 -3 -6]
[ 3 13 0 10 7]
[ -7 3 -10 0 -3]
[ -4 6 -7 3 0]]
Then replace upper triangle matrix to 0:
c = np.tril(b)
print (c)
[[ 0 0 0 0 0]
[-10 0 0 0 0]
[ 3 13 0 0 0]
[ -7 3 -10 0 0]
[ -4 6 -7 3 0]]
Last need max values per diagonal, so it means:
max([0,0,0,0,0]) = 0
max([-10,13,-10,3]) = 13
max([3,3,-7]) = 3
max([-7,6]) = 6
max([-4]) = -4
So expected output is:
[0, 13, 3, 6, -4]
What is some nice vectorized solution? Or is possible some another way for expected output?
Use ndarray.diagonal
v = [max(c.diagonal(-i)) for i in range(b.shape[0])]
print(v) # [0, 13, 3, 6, -4]
Not sure exactly how efficient this is considering the advanced indexing involved, but this is one way to do that:
import numpy as np
a = np.array([8, 18, 5, 15, 12])
b = a[:, None] - a
# Fill lower triangle with largest negative
b[np.tril_indices(len(a))] = np.iinfo(b.dtype).min # np.finfo for float
# Put diagonals as rows
s = b.strides[1]
diags = np.ndarray((len(a) - 1, len(a) - 1), b.dtype, b, offset=s, strides=(s, (len(a) + 1) * s))
# Get maximum from each row and add initial zero
c = np.r_[0, diags.max(1)]
print(c)
# [ 0 13 3 6 -4]
EDIT:
Another alternative, which may not be what you were looking for though, is just using Numba, for example like this:
import numpy as np
import numba as nb
def max_window_diffs_jdehesa(a):
a = np.asarray(a)
dtinf = np.iinfo(b.dtype) if np.issubdtype(b.dtype, np.integer) else np.finfo(b.dtype)
out = np.full_like(a, dtinf.min)
_pwise_diffs(a, out)
return out
#nb.njit(parallel=True)
def _pwise_diffs(a, out):
out[0] = 0
for w in nb.prange(1, len(a)):
for i in range(len(a) - w):
out[w] = max(a[i] - a[i + w], out[w])
a = np.array([8, 18, 5, 15, 12])
print(max_window_diffs(a))
# [ 0 13 3 6 -4]
Comparing these methods to the original:
import numpy as np
import numba as nb
def max_window_diffs_orig(a):
a = np.asarray(a)
b = a - a[:, None]
out = np.zeros(len(a), b.dtype)
out[-1] = b[-1, 0]
for i in range(1, len(a) - 1):
out[i] = np.diag(b, -i).max()
return out
def max_window_diffs_jdehesa_np(a):
a = np.asarray(a)
b = a[:, None] - a
dtinf = np.iinfo(b.dtype) if np.issubdtype(b.dtype, np.integer) else np.finfo(b.dtype)
b[np.tril_indices(len(a))] = dtinf.min
s = b.strides[1]
diags = np.ndarray((len(a) - 1, len(a) - 1), b.dtype, b, offset=s, strides=(s, (len(a) + 1) * s))
return np.concatenate([[0], diags.max(1)])
def max_window_diffs_jdehesa_nb(a):
a = np.asarray(a)
dtinf = np.iinfo(b.dtype) if np.issubdtype(b.dtype, np.integer) else np.finfo(b.dtype)
out = np.full_like(a, dtinf.min)
_pwise_diffs(a, out)
return out
#nb.njit(parallel=True)
def _pwise_diffs(a, out):
out[0] = 0
for w in nb.prange(1, len(a)):
for i in range(len(a) - w):
out[w] = max(a[i] - a[i + w], out[w])
np.random.seed(0)
a = np.random.randint(0, 100, size=100)
r = max_window_diffs_orig(a)
print((max_window_diffs_jdehesa_np(a) == r).all())
# True
print((max_window_diffs_jdehesa_nb(a) == r).all())
# True
%timeit max_window_diffs_orig(a)
# 348 µs ± 986 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit max_window_diffs_jdehesa_np(a)
# 91.7 µs ± 1.3 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit max_window_diffs_jdehesa_nb(a)
# 19.7 µs ± 88.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
np.random.seed(0)
a = np.random.randint(0, 100, size=10000)
%timeit max_window_diffs_orig(a)
# 651 ms ± 26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit max_window_diffs_jdehesa_np(a)
# 1.61 s ± 6.19 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit max_window_diffs_jdehesa_nb(a)
# 22 ms ± 967 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
The first one may be a bit better for smaller arrays, but doesn't work well for bigger ones. Numba on the other hand is pretty good in all cases.
You can use numpy.diagonal:
a = np.array([8, 18, 5,15,12])
b = a - a[:, None]
c = np.tril(b)
for i in range(b.shape[0]):
print(max(c.diagonal(-i)))
Output:
0
13
3
6
-4
Here's a vectorized solution with strides -
from skimage.util import view_as_windows
n = len(a)
z = np.zeros(n-1,dtype=a.dtype)
p = np.concatenate((a,z))
s = view_as_windows(p,n)
mask = np.tri(n,k=-1,dtype=bool)[:,::-1]
v = s[0]-s
out = np.where(mask,v.min()-1,v).max(1)
With one-loop for memory-efficiency -
n = len(a)
out = [max(a[:-i+n]-a[i:]) for i in range(n)]
Use np.max in place of max for better use of array-memory.
You can abuse the fact that reshaping non-square arrays of shape (N+1, N) to (N, N+1) will make diagonals appear as columns
from scipy.linalg import toeplitz
a = toeplitz([1,2,3,4], [1,4,3])
# array([[1, 4, 3],
# [2, 1, 4],
# [3, 2, 1],
# [4, 3, 2]])
a.reshape(3, 4)
# array([[1, 4, 3, 2],
# [1, 4, 3, 2],
# [1, 4, 3, 2]])
Which you can then use like (note that I've swapped the sign and set the lower triangle to zero)
smallv = -10000 # replace this with np.nan if you have floats
a = np.array([8, 18, 5,15,12])
b = a[:, None] - a
b[np.tril_indices(len(b), -1)] = smallv
d = np.vstack((b, np.full(len(b), smallv)))
d.reshape(len(d) - 1, -1).max(0)[:-1]
# array([ 0, 13, 3, 6, -4])
I have a list object,i want to know that how many numbers are in a particular interval?The code is as follows
a = [1, 7, 4, 7, 4, 8, 5, 2, 17, 8, 3, 12, 9, 6, 28]
interval = 3
a = list(map(lambda x:int(x/interval),a))
for i in range(min(a),max(a)+1):
print(i*interval,(i+1)*interval,':',a.count(i))
Output
0 3 : 2
3 6 : 4
6 9 : 5
9 12 : 1
12 15 : 1
15 18 : 1
18 21 : 0
21 24 : 0
24 27 : 0
27 30 : 1
Is there a simple way to get this information?The simpler the better
Now that we're talking about performance, I'd like to offer my numpy solution using bincount:
import numpy as np
interval = 3
a = [1, 7, 4, 7, 4, 8, 5, 2, 17, 8, 3, 12, 9, 6, 28]
l = max(a) // interval + 1
b = np.bincount(a, minlength=l*interval).reshape((l,interval)).sum(axis=1)
(minlength is necessary just to be able to reshape if max(a) isn't a multiple of interval)
With the lables taken from Erfan's answer we get:
rnge = range(0, max(a) + interval + 1, interval)
lables = [f'[{i}-{j})' for i, j in zip(rnge[:-1], rnge[1:])]
for l,b in zip(lables,b):
print(l,b)
[0-3) 2
[3-6) 4
[6-9) 5
[9-12) 1
[12-15) 1
[15-18) 1
[18-21) 0
[21-24) 0
[24-27) 0
[27-30) 1
This is much faster than the pandas solution.
Performance and scaling comparison
In order to assess the scaling capability, I just replaced a = [1, ..., 28] * n and timed the execution (without imports and printing) for n = 1, 10, 100, 1K, 10K and 100K:
(python 3.7.3 on win32 / pandas 0.24.2 / numpy 1.16.2)
Pandas solution with pd.cut and groupby
s = pd.Series(a)
bins = pd.cut(s, range(0, s.max() + interval, interval), right=False)
s.groupby(bins).count()
[0, 3) 2
[3, 6) 4
[6, 9) 5
[9, 12) 1
[12, 15) 1
[15, 18) 1
[18, 21) 0
[21, 24) 0
[24, 27) 0
[27, 30) 1
dtype: int64
To get cleaner bins results, we can use this method from linked answer:
s = pd.Series(a)
rnge = range(0, s.max() + interval, interval)
labels = [f'{i}-{j}' for i, j in zip(rnge[:-1], rnge[1:])]
bins = pd.cut(s, range(0, s.max() + interval, interval), right=False, labels=labels)
s.groupby(bins).count()
0-3 2
3-6 4
6-9 5
9-12 1
12-15 1
15-18 1
18-21 0
21-24 0
24-27 0
27-30 1
dtype: int64
You can do it in one line using a dictionary comprehension :
a = [1, 7, 4, 7, 4, 8, 5, 2, 17, 8, 3, 12, 9, 6, 28]
{"[{};{}[".format(x, x+3) : len( [y for y in a if y >= x and y < x+3] )
for x in range(min(a), max(a), 3)}
Output :
{'[1;4[': 3,
'[4;7[': 4,
'[7;10[': 5,
'[10;13[': 1,
'[13;16[': 0,
'[16;19[': 1,
'[19;22[': 0,
'[22;25[': 0,
'[25;28[': 0}
Performance comparison :
Pandas solution with pd.cut and groupby : 8.51 ms ± 32 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Dictionary comprehension : 19.7 µs ± 37.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Using np.bincount : 22.4 µs ± 263 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)