I work with raster images and the module rasterio to imprt them as numpy arrays. I would like to cut a portion of size (1000, 1000) out of the middle of each (to avoid the out-of-bound masks of the image).
image = np.random.random_sample((2000, 2000))
s = image.shape
mid = [round(x / 2) for x in s] # middle point of both axes
margins = [[y + x for y in [-500, 500]] for x in mid] # 1000 range around every middle point
The result is a list of 2 lists, for the cut range on each axis. But this is where I stump: range() doesn't accept lists, and I'm attempting the following brute force method:
cut_image = image[range(margins[0][0], margins[0][1]), range(margins[1][0], margins[1][1])]
However:
cut_image.shape
## (1000,)
Slicing an array loses dimension information which is exactly what I don't want.
Consider me confused.
Looking for a more tasteful solution.
As the other answer points it out, you're not really slicing your array, but using indexing on it.
If you want to slice your array (and you're right, that's more elegant than using list of indices) , you'll be happier with slices. That objects represents the start:end:step syntax.
In your case,
import numpy as np
WND = 50
image = np.random.random_sample((200, 300))
s = image.shape
mid = [round(x / 2) for x in s] # middle point of both axes
margins = [[y + x for y in [-WND, WND]] for x in mid] # 1000 range around every middle point
# array[slice(start, end)] -> array[start:end]
x_slice = slice(margins[0][0], margins[0][1])
y_slice = slice(margins[1][0], margins[1][1])
print(x_slice, y_slice)
# slice(50, 150, None) slice(100, 200, None)
cut_image = image[x_slice, y_slice]
print(cut_image.shape)
# (100,100)
Indexing ?
You might wonder what was happening in your question that resulted in only 1000 elements instead of the expected 1000*1000.
Here is a simpler example of indexing with lists on different dimensions
# n and n2 have identical values
n = a[[i0, i1, i2],[j0, j1, j2]]
n2 = np.array([a[i0, j0], a[i1, j1], a[i2, j2]]
This being clarified, you'll understand that instead of taking a block matrix, your code only returns the diagonal coefficients of that block matrix :)
The issue here is that what you're doing is known as integer indexing, instead of slice indexing. The bahaviour changes and may seem counterintuitive when not acquainted with it. You can check the docs for more details.
Here's how you could do it with basic slicing:
# center coordinates of the image
x_0, y_0 = np.asarray(image.shape)//2
# slice taken from the center point
out = image[x_0-x_0//2:x_0+x_0//2, y_0-y_0//2:y_0+y_0//2]
print(out.shape)
# (1000, 1000)
Related
Good evening,
I'm trying to learn NumPy and have written a simple Linear transformation that applies to an image using for loops:
import numpy as np
M = np.array([
[width, 0],
[0, height]
])
T = np.array([
[1, 3],
[0, 1]
])
def transform_image(M, T):
T_rel_M = abs(M # T)
new_img = np.zeros(T_rel_M.sum(axis=1).astype("int")).T
for i in range(0, 440):
for j in range(0, 440):
x = np.array([j, i])
coords = (T # x)
x = coords[0]
y = coords[1]
new_img[y, -x] = image[i, -j]
return new_img
plt.imshow(transform_image(M, T))
It does what I want and spits out a transformation that is correct, except that I think there is a way to do this without the loops.
I tried doing some stuff with meshgrid but I couldn't figure out how to get the pixels from the image in the same way I do it in the loop (using i and j). I think I figured out how to apply the transformation but then getting the pixels from the image in the correct spots wouldn't work.
Any ideas?
EDIT:
Great help with below solutions, lezaf's solution was very similar to what I tried before, the only step missing that I couldn't figure out was assigning the pixels from the old to the new image. I made some changes to the code to exclude transposing, and also added a astype("int") so it works with float values in the T matrix:
def transform_image(M, T):
T_rel_M = abs(M # T)
new_img = np.zeros(T_rel_M.sum(axis=1).astype("int")).T
x_combs = np.array(np.meshgrid(np.arange(width), np.arange(height))).reshape(2,-1)
coords = (T # x_combs).astype("int")
new_img[coords[1, :], -coords[0, :]] = image[x_combs[1, :], -x_combs[0, :]]
return new_img
A more efficient solution is the following:
def transform_image(M, T):
T_rel_M = abs(M # T)
new_img = np.zeros(T_rel_M.sum(axis=1).astype("int")).T
# This one replaces the double for-loop
x_combs = np.array(np.meshgrid(np.arange(440), np.arange(440))).T.reshape(-1,2)
# Calculate the new coordinates
coords = (T#x_combs.T)
# Apply changes to new_img
new_img[coords[1, :], -coords[0, :]] = image[x_combs[:, 1], -x_combs[:,0]]
I updated my solution removing the for-loop, so now is a lot more straightforward.
After this change, the time of the optimized code is 50 ms compared to the initial 3.06 s of the code in question.
There seems to have some confusions between width/height, x/y, ... so not 100% my code won't need adaptation. But I think, the main idea is the one you are looking for
def transform_image(M, T):
T_rel_M = abs(M # T)
j,i=np.meshgrid(range(width), range(height))
ji=np.array((j.flatten(), i.flatten()))
coords = (T#ji).astype(int)
new_img=np.zeros((coords[1].max()+1, coords[0].max()+1), dtype=np.uint8)
new_img[coords[1], coords[0]] = image.flatten()
The main idea here is to build a set of coordinates of the input image with meshgrid. I don't want a 2d-array of coordinates. Just a list of coordinates (a list of pairs i,j). Hence the flatten. So ji is a huge 2×N array, N being the number of pixels (so width×height).
coords is the transformation of all those coordinates.
Since your original code seemed to have some inconsistency with size (the rotated image did not fit in the new_img), I choose the easy way to compute the size of new_img, and just compute the max of those coordinates (a bit overkill: the max of the four corners would be enough)
And then, I use this set of coordinates as indexes for new_img, to which I affect the matching image, that is image flatten
So, no for loop at all.
(Note that I've dropped the -x thing also. Just because I struggled to understand. I could have putted it back now that I have a working solution. But I am not 100% sure if it wasn't there because you also tried/errored some strange adjustment. But anyway, I think what you were looking for is how to use meshgrid to create a set of coordinates and process them without loop. Even if you may need to adapt my solution, you have it: flatten the coordinates of meshgrid, transform them with a matrix multiplication, and use them as index for places of all pixels of the original image)
Edit : variant
def transform_image(M, T):
T_rel_M = abs(M # T)
ji=np.array(np.meshgrid(range(width), range(height)))
coords = np.einsum('ik,kjl', T, ji).astype(int)
new_img=np.zeros((max(coords[1,0,-1],coords[1,-1,0], coords[1,-1,-1])+1, max(coords[0,0,-1], coords[0,-1,0], coords[0,-1,-1])+1), dtype=np.uint8)
new_img[coords[1].flatten(), coords[0].flatten()] = image.flatten()
return new_img
The idea is the same. But instead of flattening directly ji original coordinates, I keep them as is. Then use einsum to perform a matrix multiplication on a 3D array (which returns also a 2d 2×width×height arrays, whose each [:,j,i] value is just the transformation of [j,i]. So, it is just the same as previous #, except that it works even if, instead of having a 2×N set of coordinates we have a 2×width×height one).
Which has 2 advantages
Apparently it is sensibly faster to create ji than way
It allows the usage of just corners to find the size of the new image, as I've mentioned before (that was more difficult when coords was flatten from its creation).
Timing
Solution
Timing
Yours
4.5 s
lezaf's
3.2 s
This one
49 ms
The variant
41 ms
I wrote the following function, which takes as inputs three 1D array (namely int_array, x, and y) and a number lim. The output is a number as well.
def integrate_to_lim(int_array, x, y, lim):
if lim >= np.max(x):
res = 0.0
if lim <= np.min(x):
res = int_array[0]
else:
index = np.argmax(x > lim) # To find the first element of x larger than lim
partial = int_array[index]
slope = (y[index-1] - y[index]) / (x[index-1] - x[index])
rest = (x[index] - lim) * (y[index] + (lim - x[index]) * slope / 2.0)
res = partial + rest
return res
Basically, outside form the limit cases lim>=np.max(x) and lim<=np.min(x), the idea is that the function finds the index of the first value of the array x larger than lim and then uses it to make some simple calculations.
In my case, however lim can also be a fairly big 2D array (shape ~2000 times ~1000 elements)
I would like to rewrite it such that it makes the same calculations for the case that lim is a 2D array.
Obviously, the output should also be a 2D array of the same shape of lim.
I am having a real struggle figuring out how to vectorize it.
I would like to stick only to the numpy package.
PS I want to vectorize my function because efficiency is important and as I understand using for loops is not a good choice in this regard.
Edit: my attempt
I was not aware of the function np.take, which made the task way easier.
Here is my brutal attempt that seems to work (suggestions on how to clean up or to make the code faster are more than welcome).
def integrate_to_lim_vect(int_array, x, y, lim_mat):
lim_mat = np.asarray(lim_mat) # Make sure that it is an array
shape_3d = list(lim_mat.shape) + [1]
x_3d = np.ones(shape_3d) * x # 3 dimensional version of x
lim_3d = np.expand_dims(lim_mat, axis=2) * np.ones(x_3d.shape) # also 3d
# I use np.argmax on the 3d matrices (is there a simpler way?)
index_mat = np.argmax(x_3d > lim_3d, axis=2)
# Silly calculations
partial = np.take(int_array, index_mat)
y1_mat = np.take(y, index_mat)
y2_mat = np.take(y, index_mat - 1)
x1_mat = np.take(x, index_mat)
x2_mat = np.take(x, index_mat - 1)
slope = (y1_mat - y2_mat) / (x1_mat - x2_mat)
rest = (x1_mat - lim_mat) * (y1_mat + (lim_mat - x1_mat) * slope / 2.0)
res = partial + rest
# Make the cases with np.select
condlist = [lim_mat >= np.max(x), lim_mat <= np.min(x)]
choicelist = [0.0, int_array[0]] # Shoud these options be a 2d matrix?
output = np.select(condlist, choicelist, default=res)
return output
I am aware that if the limit is larger than the maximum value in the array np.argmax returns the index zero (leading to wrong results). This is why I used np.select to check and correct for these cases.
Is it necessary to define the three dimensional matrices x_3d and lim_3d, or there is a simpler way to find the 2D matrix of the indices index_mat?
Suggestions, especially to improve the way I expanded the dimension of the arrays, are welcome.
I think you can solve this using two tricks. First, a 2d array can be easily flattened to a 1d array, and then your answers can be converted back into a 2d array with reshape.
Next, your use of argmax suggests that your array is sorted. Then you can find your full set of indices using digitize. Thus instead of a single index, you will get a complete array of indices. All the calculations you are doing are intrinsically supported as array operations in numpy, so that should not cause any problems.
You will have to specifically look at the limiting cases. If those are rare enough, then it might be okay to let the answers be derived by the default formula (they will be garbage values), and then replace them with the actual values you desire.
Currently, I have a 4d array, say,
arr = np.arange(48).reshape((2,2,3,4))
I want to apply a function that takes a 2d array as input to each 2d array sliced from arr. I have searched and read this question, which is exactly what I want.
The function I'm using is im2col_sliding_broadcasting() which I get from here. It takes a 2d array and list of 2 elements as input and returns a 2d array. In my case: it takes 3x4 2d array and a list [2, 2] and returns 4x6 2d array.
I considered using apply_along_axis() but as said it only accepts 1d function as parameter. I can't apply im2col function this way.
I want an output that has the shape as 2x2x4x6. Surely I can achieve this with for loop, but I heard that it's too time expensive:
import numpy as np
def im2col_sliding_broadcasting(A, BSZ, stepsize=1):
# source: https://stackoverflow.com/a/30110497/10666066
# Parameters
M, N = A.shape
col_extent = N - BSZ[1] + 1
row_extent = M - BSZ[0] + 1
# Get Starting block indices
start_idx = np.arange(BSZ[0])[:, None]*N + np.arange(BSZ[1])
# Get offsetted indices across the height and width of input array
offset_idx = np.arange(row_extent)[:, None]*N + np.arange(col_extent)
# Get all actual indices & index into input array for final output
return np.take(A, start_idx.ravel()[:, None] + offset_idx.ravel()[::stepsize])
arr = np.arange(48).reshape((2,2,3,4))
output = np.empty([2,2,4,6])
for i in range(2):
for j in range(2):
temp = im2col_sliding_broadcasting(arr[i, j], [2,2])
output[i, j] = temp
Since my arr in fact is a 10000x3x64x64 array. So my question is: Is there another way to do this more efficiently ?
We can leverage np.lib.stride_tricks.as_strided based scikit-image's view_as_windows to get sliding windows. More info on use of as_strided based view_as_windows.
from skimage.util.shape import view_as_windows
W1,W2 = 2,2 # window size
# create sliding windows along last two axes1
w = view_as_windows(arr,(1,1,W1,W2))[...,0,0,:,:]
# Merge the window axes (tha last two axes) and
# merge the axes along which those windows were created (3rd and 4th axes)
outshp = arr.shape[:-2] + (W1*W2,) + ((arr.shape[-2]-W1+1)*(arr.shape[-1]-W2+1),)
out = w.transpose(0,1,4,5,2,3).reshape(outshp)
The last step forces a copy. So, skip it if possible.
PREREQUISITE
import numpy as np
import pandas as pd
INPUT1:boolean 2d array (a sample array as below)
x = np.array(
[[False,False,False,False,True],
[True,False,False,False,False],
[False,False,True,False,True],
[False,True,True,False,False],
[False,False,False,False,False]])
INPUT2:1D Range values (a sample as below)
y=np.array([1,2,3,4])
EXPECTED OUTPUT:2D ndarray
[[0,0,0,0,1],
[1,0,0,0,2],
[2,0,1,0,1],
[3,1,1,0,2],
[4,2,2,0,3]]
I want to set a range value(vertical vector) for each True in 2d ndarray(INPUT1) efficiently. Is there some useful APIs or solutions for this purpose?
Unfortunately I couldn't come up with an elegant solution, so I came up with multiple inelegant ones. The two main approaches I could think of are
brute-force looping over each True value and assigning slices, and
using a single indexed assignment to replace the necessary values.
It turns out that the time complexity of these approaches is non-trivial, so depending on the size of your array either can be faster.
Using your example input:
import numpy as np
x = np.array(
[[False,False,False,False,True],
[True,False,False,False,False],
[False,False,True,False,True],
[False,True,True,False,False],
[False,False,False,False,False]])
y = np.array([1,2,3,4])
refout = np.array([[0,0,0,0,1],
[1,0,0,0,2],
[2,0,1,0,1],
[3,1,1,0,2],
[4,2,2,0,3]])
# alternative input with arbitrary size:
# N = 100; x = np.random.rand(N,N) < 0.2; y = np.arange(1,N)
def looping_clip(x, y):
"""Loop over Trues, use clipped slices"""
nmax = x.shape[0]
n = y.size
# initialize output
out = np.zeros_like(x, dtype=y.dtype)
# loop over True values
for i,j in zip(*x.nonzero()):
# truncate right-hand side where necessary
out[i:i+n, j] = y[:nmax-i]
return out
def looping_expand(x, y):
"""Loop over Trues, use an expanded buffer"""
n = y.size
nmax,mmax = x.shape
ivals,jvals = x.nonzero()
# initialize buffed-up output
out = np.zeros((nmax + max(n + ivals.max() - nmax,0), mmax), dtype=y.dtype)
# loop over True values
for i,j in zip(ivals, jvals):
# slice will always be complete, i.e. of length y.size
out[i:i+n, j] = y
return out[:nmax, :].copy() # rather not return a view to an auxiliary array
def index_2d(x, y):
"""Assign directly with 2d indices, use an expanded buffer"""
n = y.size
nmax,mmax = x.shape
ivals,jvals = x.nonzero()
# initialize buffed-up output
out = np.zeros((nmax + max(n + ivals.max() - nmax,0), mmax), dtype=y.dtype)
# now we can safely index for each "(ivals:ivals+n, jvals)" so to speak
upped_ivals = ivals[:,None] + np.arange(n) # shape (ntrues, n)
upped_jvals = jvals.repeat(y.size).reshape(-1, n) # shape (ntrues, n)
out[upped_ivals, upped_jvals] = y # right-hand size of shape (n,) broadcasts
return out[:nmax, :].copy() # rather not return a view to an auxiliary array
def index_1d(x,y):
"""Assign using linear indices, use an expanded buffer"""
n = y.size
nmax,mmax = x.shape
ivals,jvals = x.nonzero()
# initialize buffed-up output
out = np.zeros((nmax + max(n + ivals.max() - nmax,0), mmax), dtype=y.dtype)
# grab linear indices corresponding to Trues in a buffed-up array
inds = np.ravel_multi_index((ivals, jvals), out.shape)
# now all we need to do is start stepping along rows for each item and assign y
upped_inds = inds[:,None] + mmax*np.arange(n) # shape (ntrues, n)
out.flat[upped_inds] = y # y of shape (n,) broadcasts to (ntrues, n)
return out[:nmax, :].copy() # rather not return a view to an auxiliary array
# check that the results are correct
print(all([np.array_equal(refout, looping_clip(x,y)),
np.array_equal(refout, looping_expand(x,y)),
np.array_equal(refout, index_2d(x,y)),
np.array_equal(refout, index_1d(x,y))]))
I tried to document each function, but here's a synopsis:
looping_clip loops over every True value in the input and assigns to a corresponding slice in the output. We take care on the right-hand side to shorten the assigned array for when part of the slice would go beyond the edge of the array along the first dimension.
looping_expand loops over every True value in the input and assigns to a corresponding full slice in the output after allocating a padded output array ensuring that every slice will be full. We do more work when allocating a larger output array, but we don't have to shorten the right-hand side on assignment. We could omit the .copy() call in the last step, but I prefer not to return a nontrivially strided array (i.e. a view to an auxiliary array rather than a proper copy) as this might lead to obscure surprises for the user.
index_2d computes the 2d indices of every value to be assigned to, and assumes that duplicate indices will be handled in order. This is not guaranteed! (More on this a bit later.)
index_1d does the same using linearized indices and indexing into the flatiter of the output.
Here are the timings of the above methods using random arrays (see the commented line near the start):
What we can see is that for small and large arrays the looping versions are faster, but for linear sizes between roughly 10 and 150 the indexing versions are better. The reason I didn't go to higher sizes is that the indexing cases start to use a lot of memory, and I didn't want to have to worry about this messing with timings.
Just to make the above worse, note that the indexing versions assume that duplicate indices in a fancy indexing scenario are handled in order, so when True values are handled which are "lower" in the array, previous values will be overwritten as per your requirements. There's only one problem: this is not guaranteed:
For advanced assignments, there is in general no guarantee for the iteration order. This means that if an element is set more than once, it is not possible to predict the final result.
This doesn't sounds very encouraging. While in my experiments it seems that the indices are handled in order (according to C order), this can also be coincidence, or an implementation detail. So if you want to use the indexing versions, make sure that on your specific version and specific dimensions and shapes this still holds true.
We can make the assignment safer by getting rid of duplicate indices ourselves. For this we can make use of this answer by Divakar on a corresponding question:
def index_1d_safe(x,y):
"""Same as index_1d but use Divakar's safe solution for reducing duplicates"""
n = y.size
nmax,mmax = x.shape
ivals,jvals = x.nonzero()
# initialize buffed-up output
out = np.zeros((nmax + max(n + ivals.max() - nmax,0), mmax), dtype=y.dtype)
# grab linear indices corresponding to Trues in a buffed-up array
inds = np.ravel_multi_index((ivals, jvals), out.shape)
# now all we need to do is start stepping along rows for each item and assign y
upped_inds = inds[:,None] + mmax*np.arange(n) # shape (ntrues, n)
# now comes https://stackoverflow.com/a/44672126
# need additional step: flatten upped_inds and corresponding y values for selection
upped_flat_inds = upped_inds.ravel() # shape (ntrues, n) -> (ntrues*n,)
y_vals = np.broadcast_to(y, upped_inds.shape).ravel() # shape (ntrues, n) -> (ntrues*n,)
sidx = upped_flat_inds.argsort(kind='mergesort')
sindex = upped_flat_inds[sidx]
idx = sidx[np.r_[np.flatnonzero(sindex[1:] != sindex[:-1]), upped_flat_inds.size-1]]
out.flat[upped_flat_inds[idx]] = y_vals[idx]
return out[:nmax, :].copy() # rather not return a view to an auxiliary array
This still reproduces your expected output. The problem is that now the function takes much longer to finish:
Bummer. Considering how my indexing versions are only faster for an intermediate array size and how their faster versions are not guaranteed to work, perhaps it's simplest to just use one of the looping versions. This is not to say, of course, that there aren't any optimal vectorized solutions that I missed.
considering I have a 3D histogram or for simplicity a 3D numpy array of shape (X,Y,Z)
import numpy as np
array = np.random.random((100,100,100))
What is the best way, using numpy or scipy to obtain array's values' indexes of which satisfy a sphere conditions?
(index_x**2 + index_y**2 + index_z**2) <= radius**2
Obvioulsy, in the later condition, the array center is (0, 0, 0). In general the condition will be
((index_x-center_x)**2 + (index_y-center_y)**2 +(index_z-center_z)**2) <= radius**2
The problem is easy to solve using simply a python loop, but I need that to be optimized.
many thanks for your help
You can first efficiently get the indexes with ogrid() and then obtain the indexes that satisfy your condition with nonzero().
Getting the indexes can be obtained with nonzero() like so:
indexes = numpy.transpose((x**2+y**2+z**2 <= radius**2).nonzero()) # transpose() might be unnecessary: it depends on your needs
where the indexes arrays are obtained efficiently with ogrid():
x, y, z = numpy.ogrid[:100, :100, :100]
or, for an arbitrary shape for your input data array:
x, y, z = ogrid[tuple(slice(None, dim) for dim in data.shape)]
Just for making #EOL nice approach more general, one can define a center within the shape of the array
array = np.random.random((100,100,100))
center = (30,10,25)
radius = 5.0
x, y, z = np.ogrid[-center[0]:array.shape[0]-center[0],-center[1] :array.shape[1]-center[1], -center[2]:array.shape[2]-center[2]]
indexes = numpy.transpose((x**2+y**2+z**2 <= radius**2).nonzero())