I have several two-dimensional data arrays loaded into NumPy arrays, all of which have identical dimensions. These shared dimensions are 2606x and 1228y.
I am interested in computing calculations between the first two arrays (a1 & a2) across a moving window, using a window sized 2x by 2y, with the resultant calculation then applied to the third array. Specifically, the workflow would be:
Finding the the maximum & minimum value of a1 at the piece of this moving window
Selecting the corresponding array indices of these values
Extracting the values at these indices of a2
Cast the calculation result to each of the indices in the third array (a3) inside the moving window.
I know that this process involves the following pieces of code to obtain the values I require:
idx1 = np.where(a1 == a1.max())
idx2 = np.where(a1 == a1.min())
val1 = a2[idx1[1], idx1[2]]
val2 = a2[idx2[1], idx2[2]]
What additional code is required to perform this moving window along the identically sized arrays?
Since your array shape is divisible by your window size, you can use numpy.reshape to split your array up into little windows such that your original array shape of (2606, 1228) becomes (2606/2, 2, 1228/2, 2).
If numpy.argmin accepted sequences of axes, this would be easier, but since it only accepts a single axis (or None but we don't want that), we need to compress the two window axes into a single axes. To do that, we use numpy.moveaxis to make the shape (2606/2, 1228/2, 2, 2) and then numpy.reshape again to flatten the last two axes into (2606/2, 1228/2, 4).
With that headache over with, we can then use numpy.argmin and numpy.argmax on the last axis to compute the indices you're interested in and use advanced indexing to write the corresponding value of a2 to a3. After that, we just have to undo the reshape and moveaxis operations that were done to a3.
import numpy as np
shape = (4, 6)
a1 = np.random.random(shape)
a2 = np.random.random(shape)
a3 = np.zeros(shape)
win_x = 2
win_y = 2
shape_new = (shape[0] // win_x, win_x, shape[1] // win_y, win_y)
a1_r = np.moveaxis(a1.reshape(shape_new), 1, 2).reshape(*shape_new[::2], -1)
a2_r = np.moveaxis(a2.reshape(shape_new), 1, 2).reshape(*shape_new[::2], -1)
a3_r = np.moveaxis(a3.reshape(shape_new), 1, 2).reshape(*shape_new[::2], -1)
index_x, index_y = np.indices(shape_new[::2])
index_min = np.argmin(a1_r, axis=-1)
index_max = np.argmax(a1_r, axis=-1)
a3_r[index_x, index_y, index_min] = a2_r[index_x, index_y, index_min]
a3_r[index_x, index_y, index_max] = a2_r[index_x, index_y, index_max]
a3 = np.moveaxis(a3_r.reshape(*shape_new[::2], win_x, win_y), 2, 1).reshape(shape)
print(a1)
print()
print(a2)
print()
print(a3)
Outputs
[[0.54885307 0.74457945 0.84943538 0.14139329 0.68678556 0.03460323]
[0.74031057 0.5499962 0.03148748 0.13936734 0.05006111 0.88850868]
[0.97789608 0.13262023 0.76350358 0.74640822 0.7918286 0.80675845]
[0.35784598 0.20918229 0.82880072 0.06051794 0.0825886 0.6398353 ]]
[[0.66176657 0.10120202 0.15306892 0.05963046 0.79057051 0.08837686]
[0.78550049 0.09918834 0.00213652 0.61053454 0.42966757 0.25952916]
[0.00387273 0.78247644 0.65549303 0.39351233 0.11002493 0.55652453]
[0.06047582 0.87997514 0.60820023 0.06705212 0.34581512 0.93504438]]
[[0.66176657 0.10120202 0.15306892 0. 0. 0.08837686]
[0. 0. 0.00213652 0. 0. 0.25952916]
[0.00387273 0.78247644 0. 0. 0. 0.55652453]
[0. 0. 0.60820023 0.06705212 0.34581512 0. ]]
Related
This question already has answers here:
Matrix multiply two 1-D numpy arrays
(3 answers)
Closed 4 months ago.
I have for example two arrays, a and b. Array a has a length of 3. Array b an arbitrary length.
I would like to do the following with a numpy approach:
temp_res = 0
for i in range(3):
tem_res += a[i] * b
a can be treated as a vector of scalar values for multiplication. Basically I want to have a Matrix with 3 rows which has the same length as b and are multiplied with a's value at the corresponding index. However, because of the different shapes, I do not say any how to this without any loop (or list comprehension).
How can the example above implemented with purely numpy (and without any python loop)? I already checked out the documentation, but same shape is always a condition.
you need to read about numpy broadcasting, putting 1 in the first dimension of b will force broadcasting on it, a reshape only changes the stride but doesn't make a copy of the data.
tem_res = a * b.reshape([1,-1])
this can also be written this way in case b was larger than 2D
tem_res = a * b[None,:]
Example:
import numpy as np
a = np.ones([3,4]) # 3x4 array of ones
b = np.zeros([4]) # 1D array 4 elements of zeros
c = a * b.reshape([1,-1]) # b.reshape is now 1x4, it can be multipled by 3x4
print(c) # confirm it is 3x4 array
[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]]
I want to remove elements from a numpy vector that are closer than a distance d. (I don't want any pair in the array or list that have a smaller distance between them than d but don't want to remove the pair completely otherwise.
for example if my array is:
array([[0. ],
[0.9486833],
[1.8973666],
[2.8460498],
[0.9486833]], dtype=float32)
All I need is to remove either the element with the index 1 or 4 not both of them.
I also need the indices of the elements from the original array that remain in the latent one.
Since the original array is in tensorflow 2.0, I will be happier if conversion to numpy is not needed like above. Because of speed also I prefer not to use another package and stay with numpy or scipy.
Thanks.
Here's a solution, using only a list. Note that this modifies the original list, so if you want to keep the original, copy.deepcopy it.
THRESHOLD = 0.1
def wrangle(l):
for i in range(len(l)):
for j in range(len(l)-1, i, -1):
if abs(l[i] - l[j]) < THRESHOLD:
l.pop(j)
using numpy:
import numpy as np
a = np.array([[0. ],
[0.9486833],
[1.8973666],
[2.8460498],
[0.9486833]])
threshold = 1.0
# The indices of the items smaller than a certain threshold, but larger than 0.
smaller_than = np.flatnonzero(np.logical_and(a < threshold, a > 0))
# Get the first index smaller than threshold
first_index = smaller_than[0]
# Recreate array without this index (bit cumbersome)
new_array = a[np.arange(len(a)) != first_index]
I'm pretty sure this is really easy to recreate in tensorflow, but I don't know how.
If your array is really only 1-d you can flatten it and do something like this:
a=tf.constant(np.array([[0. ],
[0.9486833],
[1.8973666],
[2.8460498],
[0.9486833]], dtype=np.float32))
d = 0.1
flat_a = tf.reshape(a,[-1]) # flatten
a1 = tf.expand_dims(flat_a, 1)
a2 = tf.expand_dims(flat_a, 0)
distance_map = tf.math.abs(a1-a2)
too_small = tf.cast(tf.math.less(dist_map, d), tf.int32)
# 1 at indices i,j if the distance between elements at i and j is less than d, 0 otherwise
upper_triangular_part = tf.linalg.band_part(too_small, 0, -1) - tf.linalg.band_part(too_small, 0,0)
remove = tf.reduce_sum(upper_triangular_part, axis=0)
remove = tf.cast(tf.math.greater(remove, 0), tf.float32)
# 1. at indices where the element should be removed, 0. otherwise
output = flat_a - remove * flat_a
You can access the indices through the remove tensor. If you need the extra dimension you can just use tf.expand_dims at the end of this.
I'm trying to insert a mini array into a larger array without resizing, so changing the values of the larger array with the mini array.
Have a mini array, xx.
Have a larger array, XX
Every Y elements, replace the next elements with mini array values.
All the way till the end.
I've tried to do it through indexing (code can be found below).
mesh_array = np.zeros(shape=(100,100), dtype=np.uint8)
mini_square = np.ones(shape=(2,2), dtype=np.uint8)
flattened_array = np.ravel(mesh_array)
flattened_minisquare = np.ravel(mini_square)
flattened_array[1:-1:10] = flattened_minisquare
Expected result is that every 10 elements, it will replace the following ones with the flattened_minisquare values.
[0,0,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0,1,1,1,1,0...]
The error message that I get:
"ValueError: could not broadcast input array from shape (4) into shape (1000)"
There may be better ways, but one way is to approach this problem as follows:
import numpy as np
mesh_array = np.zeros(shape=(100,100), dtype=np.uint8)
mini_square = np.ones(shape=(2,2), dtype=np.uint8)
flattened_array = np.ravel(mesh_array)
flattened_minisquare = np.ravel(mini_square)
Now, we can construct array which would correspond to the positions where you wish to fill the minisquare with the remaining values as zeros. Note that it would, if the given output is correct, be an array of length 13 in this case. 9 elements of original array + 4 from the minisquare
stepsize = 10
temp = np.zeros(stepsize + len(flattened_minisquare) - 1)
temp[-len(flattened_minisquare):] = flattened_minisquare
We also create a mask for the values which are not filled by the minisquare.
mask = np.copy(temp)
mask[-len(flattened_minisquare):] = np.ones_like(flattened_minisquare)
mask = ~mask.astype(bool)
Now, just use np.resize to expand both the mask and the temp array, and then finally use the mask to fill the values back from the old array.
out = np.resize(temp, len(flattened_array))
final_mask = np.resize(mask, len(flattened_array))
out[final_mask] = flattened_array[final_mask]
print(out)
#[0. 0. 0. ... 0. 0. 0.]
I have some 3D and 4D arrays that I want to mutate. Indexing the arrays looks like:
3D: array[time][x][y]
4D: array[time][z][x][y]
I want to do multiple types of data mutation and have been writing a lot of for-in loops. Examples of data mutation:
Replace 0. values with NaN
Create cumulative array (time = 0, x = x0, y = y0 -> time = 1, x = x0 + x1, y = y0 + y1) etc.
Use 2 arrays and combine them (sqrt(xa * xa + xb * xb)))
Numpy can be used for example to do (1) using:
for i in range(len(data)):
if len(data[i].shape) == 3:
for z in range(len(data[i]):
data[i][z][data[i][z] == 0.] = np.NaN
else:
data[i][data[i] == 0.] = np.NaN
But I have to write for loops every time I encounter such a problem and have to write the data mutation 2 times. Once for the 4D array and once for the 3D array. For-loops is a feature to overcome, not having to write if else statements and writing the mutation twice is almost a must. If I somehow or someone else changes the first part but forgets to change the second part the code becomes bugged.
Is there a way to for example enumerate the [z][x][y] but if the array is only [x][y] just do it once as if there was a single z-index like [0][x][y]?
I have a program in numpy utf8, which allows me to calculate the coordinates of a parabolic shot from the ground. I need to create a function which returns the coordinates (#1), create the different arrays of values to work with (#2), and finally use the function to generate the different coordinates for each pack of values
#1
def coordenadas(v,a,t,g=9.81):
rad=deg2rad(a)
x=v*cos(a)*t
y=v*sin(a)*t-(1./2.)*g*(t**2)
xy=array([x,y]).T
return xy
#2
v=arange(50,100,10) #m/s
adegree=arange(45,90,5) #degrees
a=deg2rad(adegree) #rads
t=linspace(0,10,50) #segundos
#3
v.shape=(5,1,1)
a.shape=(1,9,1)
t.shape=(1,1,50)
#5
XY=coordenadas(v,a,t,g=9.81)
print shape(XY)
print XY
#4
My question is that shape(XY) returns
(50L, 9L, 5L, 2L)
And XY (only a bit, is too long)
[[[[ 0. 0. ]
[ 0. 0. ]
[ 0. 0. ]
[ 0. 0. ]
[ 0. 0. ]]
And more boxes of this shape
What this really means(big boxes, boxes, small boxes, rows, columns) ???
(50L, 9L, 5L, 2L) means a 4D array.
You can visualize as a 50x9 matrix and each cell of this matrix contains a 5x2 matrix
Numpy arrays are basically matrices, where each box [] represents the start of a new dimension. As an easy example the matrix
11
23
could be written in numpy as:
a = numpy.array([[1,1],[2,3]])
which then would be printed as
array([[1, 1],
[2, 3]])
As this is a two-dimensional matrix, the outer "box" marks the edges of the matrix, whereas the inner boxes are the rows of the matrix with the , separating the entries. Calling a.shape without an argument gives the shape of the 2x2 matrix:
(2, 2)
Calling the shape method with argument reshapes the matrix given to the shape defined in the argument. But to further help you with the code:
1
Your function definition seems to be totally fine, except I don't see a reason, why you export x and y in an array, rather than just returning two different values.
2
The initialization of your arrays seem to be fine as well.
3
There is totally no reason to reshape the arrays you just created, just leave them as they are.
4
You have to call the function separately with each set of values to create the coordinates. Do that by using an itteration over the arrays you just created.