Pretend I have a pandas Series that consists of 0s and 1s, but this can work with numpy arrays or any iterable. I would like to create a formula that would take an array and an input n and then return a new series that contains 1s at the nth indices leading up to every time that there is at least a single 1 in the original series. Here is an example:
array = np.array([0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1])
> preceding_indices_function(array, 2)
np.array([0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1])
For each time there is a 1 in the input array, the two indices preceding it are filled in with 1 regardless of whether there is a 0 or 1 in that index in the original array.
I would really appreciate some help on this. Thanks!
Use a convolution with np.convolve:
N = 2
# craft a custom kernel
kernel = np.ones(2*N+1)
kernel[-N:] = 0
# array([1, 1, 1, 0, 0])
out = (np.convolve(array, kernel, mode='same') != 0).astype(int)
Output:
array([0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1])
Unless you don't want to use numpy, mozway's transpose is the best solution.
But since several iterations have been given, I add my itertools based solution
[a or b or c for a,b,c in itertools.zip_longest(array, array[1:], array[2:], fillvalue=0)]
zip_longest is the same as classical zip, but if the iterators have different "lengths", the number of iteration is the one of the longest, and finished iterators will return None. Unless you add a fillvalue parameter to zip_longest.
So, here itertools.zip_longest(array, array[1:], array[2:], fillvalue=0) gives a sequence of triplets (a,b,c), of 3 subsequent elements (a being the current element, b the next, c the one after, b and c being 0 if there isn't any next element or element after the next).
So from there, a simple comprehension build a list of [a or b or c] that is 1 if a, or b or c is 1, 0 else.
import numpy as np
array = np.array([0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1])
array = np.array([a or array[idx+1] or array[idx+2] for idx, a in enumerate(array[:-2])] + [array[-2] or array[-1]] + [array[-1]])
this function works if a is a list, should work with other iterables as well:
def preceding_indices_function(array, n):
for i in range(len(a)):
if array[i] == 1:
for j in range(n):
if i-j-1 >= 0:
array[i-j-1] = 1
return array
I got a solution that is similar to the other one but slightly simpler in my opinion:
>>> [1 if (array[i+1] == 1 or array[i+2] == 1) else x for i,x in enumerate(array) if i < len(array) - 2]
[0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1]
Related
Is there is a vectorized way to change all concurrent 1s that are within offset of the first 1 into 0s (transform A into B)? I'm currently trying to do this on a numpy array with over 1 million items where speed is critical.
The 1s represent a signal trigger and the 0s represent no trigger. For example: Given an offset of 5, whenever there is a 1, the following 5 items must be 0 (to remove signal concurrency).
Example 1:
offset = 3
A = np.array([1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0])
B = np.array([1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0])
Example 2:
offset = 2
A = np.array([1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0])
B = np.array([1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0])
From the comments, it seems that, the question is not just related to use NumPy and …, and the main objective is to speed up the code. Since, you are using the partial solution, mentioned by JohanC, (Which needs much more considerations for this question), I suggest the following methods:
def com_():
n = 1
for i in range(1, len(A)+1):
if A[n-1] == 1:
A[n:n+offset] = 0
n += offset + 1
else:
n += 1
if n > len(A):
break
#nb.jit(forceobj=True)
def com_fast():
B = A.tolist()
n = 1
while n < len(B):
if B[n-1] == 1:
for i in range(offset):
if n+i < len(B):
B[n+i] = 0
n += offset + 1
else:
n += 1
The first method is using A in the form of NumPy array and loops. The second one uses an input in the form of list and loops, and is accelerated by numba as it is mentioned by hpaulj in the comments.
Using the same inputs (1,000,000 in length) for the methods, and running on Google Colab TPU:
1000 loops, best of 5: 153 ms per loop # for com_()
1000 loops, best of 5: 10.2 ms per loop # for com_fast()
Which, I think, will show acceptable performance times with that large data.
I think, this question could not be solved just by NumPy, or if so, It will be very difficult and need to think about it a lot (I have tried and I achieved good results, but finally needs to loops). My guess is that, using numba and libraries like that, could have similar results (in runtime) and, so, it does not need to use just NumPy.
I got a list and I want to find the first element index by expression, for example, the list is
list_A = [None, None, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0]
I want to find the index number when the 0 changes to 1 for the first time, which is list_A[8]?
you can do it like this:
>>> list_A = [None, None, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0]
>>> for index, value in enumerate(list_A):
... if index > 0 and list_A[index - 1] == 0 and list_A[index] == 1:
... print(index)
... break
...
8
>>>
you can read about enumerate here
Problem
I want to identify when I've encountered a true value and maintain that value for the rest of the array... for a particular bin. From a Numpy perspective it would be like a combination of numpy.logical_or.accumulate and numpy.logical_or.at.
Example
Consider the truth values in a, the bins in b and the expected output in c.
I've used 0 for False and 1 for True then converted to bool in order to align the array values.
a = np.array([0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0]).astype(bool)
b = np.array([0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 2, 3, 3, 0, 1, 2, 3])
# zeros ↕ ↕ ↕ ↕ ↕ ↕ ↕
# ones ↕ ↕ ↕ ↕ ↕
# twos ↕ ↕
# threes ↕ ↕ ↕
c = np.array([0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1]).astype(bool)
# ╰─────╯ ↑ ↑ ↑ ↑
# zero bin no True yet │ │ │ two never had a True
# one bin first True │ three bin first True
# zero bin first True
What I've Tried
I can loop through each value and track whether the associated bin has seen a True value yet.
tracker = np.zeros(4, bool)
result = np.zeros(len(b), bool)
for i, (truth, bin_) in enumerate(zip(a, b)):
tracker[bin_] |= truth
result[i] = tracker[bin_]
result * 1
array([0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1])
But I was hoping for a O(n) time Numpy solution. I have the option of using a JIT wrapper like Numba but I'd rather keep it just Numpy.
O(n) solution
def cumulative_linear_seen(seen, bins):
"""
Tracks whether or not a value has been observed as
True in a 1D array, and marks all future values as
True for these each individual value.
Parameters
----------
seen: ndarray
One-hot array marking an occurence of a value
bins: ndarray
Array of bins to which occurences belong
Returns
-------
One-hot array indicating if the corresponding bin has
been observed at a point in time
"""
# zero indexing won't work with logical and, need to 1-index
one_up = bins + 1
# Next step is finding where each unique value is seen
occ = np.flatnonzero(a)
v_obs = one_up[a]
# We can fill another mapping array with these occurences.
# then map by corresponding index
i_obs = np.full(one_up.max() + 1, seen.shape[0] + 1)
i_obs[v_obs] = occ
# Finally, we create the map and compare to an array of
# indices from the original seen array
seen_idx = i_obs[one_up]
return (seen_idx <= np.arange(seen_idx.shape[0])).astype(int)
array([0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1])
PiR's contribution
Based on insights above
r = np.arange(len(b))
one_hot = np.eye(b.max() + 1, dtype=bool)[b]
np.logical_or.accumulate(one_hot & a[:, None], axis=0)[r, b] * 1
array([0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1])
Older attempts
Just to get things started, here is a solution that, while vectorized, is not O(n). I believe an O(n) solution similar to this exists, I'll work on the complexity :-)
Attempt 1
q = b + 1
u = sparse.csr_matrix(
(a, q, np.arange(a.shape[0] + 1)), (a.shape[0], q.max()+1)
)
m = np.maximum.accumulate(u.A) * np.arange(u.shape[1])
r = np.where(m[:, 1:] == 0, np.nan, m[:, 1:])
(r == q[:, None]).any(1).view(np.int8)
array([0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1], dtype=int8)
Attempt 2
q = b + 1
m = np.logical_and(a, q)
r = np.flatnonzero(u)
t = q[m]
f = np.zeros((a.shape[0], q.max()))
f[r, t-1] = 1
v = np.maximum.accumulate(f) * np.arange(1, q.max()+1)
(v == q[:, None]).any(1).view(np.int8)
array([0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1], dtype=int8)
How can I find the amount of consecutive 1s (or any other value) in each row for of the following numpy array? I need a pure numpy solution.
array([[0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0],
[0, 0, 1, 0, 0, 1, 2, 0, 0, 1, 1, 1],
[0, 0, 0, 4, 1, 0, 0, 0, 0, 1, 1, 0]])
There are two parts to my question, first: what is the maximum number of 1s in a row? Should be
array([2,3,2])
in the example case.
And second, what is the index of the start of the first set of multiple consecutive 1s in a row? For the example case this would be
array([3,9,9])
In this example I put 2 consecutive 1s in a row. But it should be possible to change that to 5 consecutive 1s in a row, this is important.
A similar question was answered using np.unique, but it only works for one row and not an array with multiple rows as the result would have different lengths.
Here's a vectorized approach based on differentiation -
import numpy as np
import pandas as pd
# Append zeros columns at either sides of counts
append1 = np.zeros((counts.shape[0],1),dtype=int)
counts_ext = np.column_stack((append1,counts,append1))
# Get start and stop indices with 1s as triggers
diffs = np.diff((counts_ext==1).astype(int),axis=1)
starts = np.argwhere(diffs == 1)
stops = np.argwhere(diffs == -1)
# Get intervals using differences between start and stop indices
start_stop = np.column_stack((starts[:,0], stops[:,1] - starts[:,1]))
# Get indices corresponding to max. interval lens and thus lens themselves
SS_df = pd.DataFrame(start_stop)
out = start_stop[SS_df.groupby([0],sort=False)[1].idxmax(),1]
Sample input, output -
Original sample case :
In [574]: counts
Out[574]:
array([[0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0],
[0, 0, 1, 0, 0, 1, 2, 0, 0, 1, 1, 1],
[0, 0, 0, 4, 1, 0, 0, 0, 0, 1, 1, 0]])
In [575]: out
Out[575]: array([2, 3, 2], dtype=int64)
Modified case :
In [577]: counts
Out[577]:
array([[0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0],
[0, 0, 1, 0, 0, 1, 2, 0, 1, 1, 1, 1],
[0, 0, 0, 4, 1, 1, 1, 1, 1, 0, 1, 0]])
In [578]: out
Out[578]: array([2, 4, 5], dtype=int64)
Here's a Pure NumPy version that is identical to the previous until we have start, stop. Here's the full implementation -
# Append zeros columns at either sides of counts
append1 = np.zeros((counts.shape[0],1),dtype=int)
counts_ext = np.column_stack((append1,counts,append1))
# Get start and stop indices with 1s as triggers
diffs = np.diff((counts_ext==1).astype(int),axis=1)
starts = np.argwhere(diffs == 1)
stops = np.argwhere(diffs == -1)
# Get intervals using differences between start and stop indices
intvs = stops[:,1] - starts[:,1]
# Store intervals as a 2D array for further vectorized ops to make.
c = np.bincount(starts[:,0])
mask = np.arange(c.max()) < c[:,None]
intvs2D = mask.astype(float)
intvs2D[mask] = intvs
# Get max along each row as final output
out = intvs2D.max(1)
I think one problem that is very similar is to check if between the sorted rows the element wise difference is a certain amount. Here if there is a difference of 1 between 5 consecutive would be as follows. It can also be done for difference of 0 for two cards:
cardAmount=cards[0,:].size
has4=cards[:,np.arange(0,cardAmount-4)]-cards[:,np.arange(cardAmount-3,cardAmount)]
isStraight=np.any(has4 == 4, axis=1)
I have a binary matrix which I create by NumPy. The matrix has 6 rows and 8 columns.
array([[1, 0, 1, 1, 1, 0, 1, 1],
[1, 1, 1, 1, 1, 1, 0, 0],
[0, 0, 1, 0, 0, 1, 1, 1],
[1, 0, 1, 1, 0, 1, 1, 0],
[0, 1, 0, 0, 1, 0, 1, 1],
[0, 1, 0, 1, 1, 1, 0, 0]])
First column is the sign of a number.
Example:
1, 0, 1, 1, 1, 0, 1, 1 -> 1 0111011 -> -59
When I used int(str, base=2) as a result I received value 187, and the value should be -59.
>>> int(''.join(map(str, array[0])), 2)
>>> 187
How can I convert the string into the signed integer?
Pyhton doesn't know that the first bit is supposed to represent the sign (compare with bin(-59)), so you have to handle that yourself, for example, if A contains the array:
num = int(''.join(map(str, A[0,1:])), 2)
if A[0,0]:
num *= -1
Here's a more Numpy-ish way to do it, for the whole array at once:
num = np.packbits(A).astype(np.int8)
num[num<0] = -128 - num[num<0]
Finally, a code-golf version:
(A[:,:0:-1]<<range(7)).sum(1)*(1-2*A[:,0])
You could split each row a sign and value variable. Then if sign is negative multiply the value by -1.
row = array[0]
sign, value = row[0], row[1:]
int(''.join(map(str, value)), 2) if sign == 0 else int(''.join(map(str, value)), 2) * -1
First of all, it looks like NumPy array rather than NumPy matrix.
There are a couple options I can think of. Pretty straight forward way will look like that:
def rowToSignedDec(arr, row):
res = int(''.join(str(x) for x in arr[row][1:].tolist()),2)
if arr[row][0] == 1:
return -res
else:
return res
print rowToSignedDec(arr, 0)
-59
That one is clearly not the most efficient one and neither the shortest one-liner:
int(''.join(str(x) for x in arr[0][1:].tolist()),2) - 2*int(arr[0][0])*int(''.join(str(x) for x in arr[0][1:].tolist()),2)
Where arr is the above-mentioned array.