Related
I have a numpy array of shape [batch_size, timesteps_per_samples, width, height], where width and height refer to a 2D grid. The values in this array can be interpreted as an elevation at a certain location that changes over time.
I want to know the elevation over time for various paths within this array. Therefore i have a second array of shape [batch_size, paths_per_batch_sample, timesteps_per_path, coordinates] (coordinates = 2, for x and y in the 2D plane).
The resulting array should be of shape [batch_size, paths_per_batch_sample, timesteps_per_path] containing the elevation over time for each sample within the batch.
The following two examples work. The first one is very slow and just serves for understanding what I am trying to do. I think the second one does what I want but I have no idea why this works nor if it may crash under certain circumstances.
Code for the problem setup:
import numpy as np
batch_size=32
paths_per_batch_sample=10
timesteps_per_path=4
width=64
height=64
elevation = np.arange(0, batch_size*timesteps_per_path*width*height, 1)
elevation = elevation.reshape(batch_size, timesteps_per_path, width, height)
paths = np.random.randint(0, high=width-1, size=(batch_size, paths_per_batch_sample, timesteps_per_path, 2))
range_batch = range(batch_size)
range_paths = range(paths_per_batch_sample)
range_timesteps = range(timesteps_per_path)
The following code works but is very slow:
elevation_per_time = np.zeros((batch_size, paths_per_batch_sample, timesteps_per_path))
for s in range_batch:
for k in range_paths:
for t in range_timesteps:
x_co, y_co = paths[s,k,t,:].astype(int)
elevation_per_time[s,k,t] = elevation[s,t,x_co,y_co]
The following code works (even fast) but I can't understand why and how o.0
elevation_per_time_fast = elevation[
:,
range_timesteps,
paths[:, :, range_timesteps, 0].astype(int),
paths[:, :, range_timesteps, 1].astype(int),
][range_batch, range_batch, :, :]
Prove that the results are equal
check = (elevation_per_time == elevation_per_time_fast)
print(np.all(check))
Can somebody explain how I can slice an nd-array by multiple other arrays?
Especially, I don't understand how the numpy knows that 'range_timesteps' has to run in step (for the index in axis 1,2,3).
Thanks in advance!
Lets take a quick look at slicing numpy array first:
a = np.arange(0,9,1).reshape([3,3])
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
Numpy has 2 ways of slicing array, full sections start:stop and by index from a list [index1, index2 ...]. The output will still be an array with the shape of your slice:
a[0:2,:]
array([[0, 1, 2],
[3, 4, 5]])
a[:,[0,2]]
array([[0, 2],
[3, 5],
[6, 8]])
The second part is that since you get a returned array with the same amount of dimensions you can easily stack any number of slices as long as you dont try to directly access an index outside of the array.
a[:][:][:][:][:][:][:][[0,2]][:,[0,2]]
array([[0, 2],
[6, 8]])
I have a question related to finding maxima or more preciseley discontinuities in a numpy array?
My exemplary data looks for example like this
a = np.array([3,4,5,8,7,6,5,4,1])
In general, I am interested in every maximum/jump in the data. For array a, I want to detect the 8 since it is a maximum (growing numbers on the left side and decreasing numbers on the right) and the value of 4, since the data drops after this value. Until now, I have used scipy.signal.argrelextrema
with np.greater to detect maxima, but I am not able to detect these jumps/discontinuities. For the data I am looking at, only a jump towards smaller values can occur not the opposite. Is there an easy pythonic way to detect these jumps?
Let's try this:
threshold = 1
a = np.array([3, 4, 5, 8, 7, 6, 5, 4, 1])
discontinuities_idx = np.where(abs(np.diff(a))>threshold)[0] + 1
np.diff(a) gives the difference between every component of a:
>>> array([ 1, 1, 3, -1, -1, -1, -1, -3])
From then np.where(abs(np.diff(a))>threshold)[0] is applied to find where detected discontinuities are (above user specified threshold in terms of absolute difference). Finally, you may add +1 to compensate for n=1 difference idx if needed (see np.diff kwargs) depending on which side of the discontinuities you need to be.
>>> discontinuities_idx
>>> array([3, 8])
>>> a[discontinuities_idx]
>>> array([8, 1])
It sounds like mathemathical analysis where you need to define some conditions like a'(x)>0 or a'(x)<0. So you can mask them:
a = np.array([3,4,5,8,7,8,6,5,4,9,2,9,9,7])
mask1 = np.diff(a) > 0
mask2 = np.diff(a) < 0
>>> np.flatnonzero(mask1[:-1] & mask2[1:]) + 1
array([3, 5, 9], dtype=int64)
It returns indices of items where maxima is met.
You can try this:
import numpy as np
import math
a = np.array([3,4,5,8,7,6,5,4,1])
MaxJump = np.diff(a)
print(MaxJump)
print(len(MaxJump))
MaxJump1 = []
for i in range (len(MaxJump)):
MaxJump1.append(math.fabs(MaxJump[i]))
print(MaxJump1)
MaxJump3 = np.max(MaxJump1)
print(MaxJump3)
I have following numpy arrays:
whole = np.array(
[1, 0, 3, 0, 6]
)
sparse = np.array(
[9, 8]
)
Now I want to replace every zero in the whole array in chronological order with the items in the sparse array. In the example my desired array would look like:
merged = np.array(
[1, 9, 3, 8, 6]
)
I could write a small algorithm by myself to fix this but if someone knows a time efficient way to solve this I would be very grateful for you help!
Do you assume that sparse has the same length as there is zeros in whole ?
If so, you can do:
import numpy as np
from copy import copy
whole = np.array([1, 0, 3, 0, 6])
sparse = np.array([9, 8])
merge = copy(whole)
merge[whole == 0] = sparse
if the lengths mismatch, you have to restrict to the correct length using len(...) and slicing.
I am using numpy in python. I have a 1D(nx1) array and a 2D(nxm) array. I used argsort to get a indice of the 1D array. Now I want to use that indice to sort my 2D(nxm) array's colum.
I want to know how to do it?
For example:
>>>array1d = np.array([1, 3, 0])
>>>array2d = np.array([[1,2,3],[4,5,6]])
>>>array1d_indice = np.argsort(array1d)
array([2, 0, 1], dtype=int64)
I want use array1d_indice to sord array2d colum to get:
[[3, 1, 2],
[6, 4, 5]]
Or anyway easier to achieve this is welcome
If what you mean is that you want the columns sorted based on the vector, then you use argsort on the vector:
vi = np.argsort(vector)
then to arrange the columns of array in the right order,
sorted = array[:, tuple(vi)]
to get rows, switch around the order of : and tuple(vi)
I'm trying to convert a piece of MATLAB code, and this is a line I'm struggling with:
f = 0
wlab = reshape(bsxfun(#times,cat(3,1-f,f/2,f/2),lab),[],3)
I've come up with
wlab = lab*(np.concatenate((3,1-f,f/2,f/2)))
How do I reshape it now?
Not going to do it for your code, but more as a general knowledge:
bsxfun is a function that fills a gap in MATLAB that python doesn't need to fill: broadcasting.
Broadcasting is a thing where if a matrix that is being multiplied/added/whatever similar is not the same size as the other one being used, the matrix will be repeated.
So in python, if you have a 3D matrix A and you want to multiply every 2D slice of it with a matrix B that is 2D, you dont need anything else, python will broadcast B for you, it will repeat the matrix again and again. A*B will suffice. However, in MATLAB that will raise an error Matrix dimension mismatch. To overcome that, you'd use bsxfun as bsxfun(#times,A,B) and this will broadcast (repeat) B over the 3rd dimension of A.
This means that converting bsxfun to python generally requires nothing.
MATLAB
reshape(x,[],3)
is the equivalent of numpy
np.reshape(x,(-1,3))
the [] and -1 are place holders for 'fill in the correct shape here'.
===============
I just tried the MATLAB expression is Octave - it's on a different machine, so I'll just summarize the action.
For lab=1:6 (6 elements) the bsxfun produces a (1,6,3) matrix; the reshape turns it into (6,3), i.e. just removes the first dimension. The cat produces a (1,1,3) matrix.
np.reshape(np.array([1-f,f/2,f/2])[None,None,:]*lab[None,:,None],(-1,3))
For lab with shape (n,m), the bsxfun produces a (n,m,3) matrix; the reshape would make it (n*m,3)
So for a 2d lab, the numpy needs to be
np.array([1-f,f/2,f/2])[None,None,:]*lab[:,:,None]
(In MATLAB the lab will always be 2d (or larger), so this 2nd case it closer to its action even if n is 1).
=======================
np.array([1-f,f/2,f/2])*lab[...,None]
would handle any shaped lab
If I make the Octave lab (4,2,3), the `bsxfun is also (4,2,3)
The matching numpy expression would be
In [94]: (np.array([1-f,f/2,f/2])*lab).shape
Out[94]: (4, 2, 3)
numpy adds dimensions to the start of the (3,) array to match the dimensions of lab, effectively
(np.array([1-f,f/2,f/2])[None,None,:]*lab) # for 3d lab
If f=0, then the array is [1,0,0], so this has the effect of zeroing values on the last dimension of lab. In effect, changing the 'color'.
It is equivalent to
import numpy as np
wlab = np.kron([1-f,f/2,f/2],lab.reshape(-1,1))
In Python, if you use numpy you do not need to do any broadcasting, as this is done automatically for you.
For instance, looking at the following code should make it clearer:
>>> import numpy as np
>>> a = np.array([[1, 2, 3], [3, 4, 5], [6, 7, 8], [9, 10, 100]])
>>> b = np.array([1, 2, 3])
>>>
>>> a
array([[ 1, 2, 3],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 100]])
>>> b
array([1, 2, 3])
>>>
>>> a - b
array([[ 0, 0, 0],
[ 2, 2, 2],
[ 5, 5, 5],
[ 8, 8, 97]])
>>>