Currently writing a custom data generator for Keras and I'm looking to randomly flip a numpy input array during training for data augmentation purposes.
X: input array of shape (batchsize, y_dim, x_dim)
to_flip: boolean array of shape (batchsize, X_rank) that describes which axes to flip
For example:
X = [[[0, 1],
[2, 3]],
[[4, 5],
[6, 7]]]
to_flip = [[0, 1],
[1, 0]]
flipped_X = [[[1, 0],
[3, 2]],
[[6, 7],
[4, 5]]]
or any combination of axes (both/no axes flipped).
I couldn't figure out how to use either np.flip or advanced indexing (e.g. X[::-1]) by themselves to get this effect. Ideally, I'd be able to find a solution that could be applied to the whole batch without explicit loops, because I believe that such a solution would be faster.
The best solution I have currently is:
def flip_axes(an_input, axes_to_flip):
axis_inds_to_flip = np.where(axes_to_flip.reshape(-1))[0]
return np.flip(an_input, axis_inds_to_flip)[None, ...]
flipped_X = np.concatenate([flip_axes(an_input, axes_to_flip)
for an_input, axes_to_flip
in zip(X, to_flip)], axis=0)
Anyone have any thoughts? Would this indeed be quicker than what I've written here?
edit: I've revised my solution so that it executes and I've added my motivation for the question.
You can do it using python indexing as follows,
flipped_X = []
for sample, flip in zip(X, to_flip):
flipped = [*sample] # Generate a copy of x
if flip[0]: # if flip[0] is 1
flipped = flipped[::-1] # Flip matrix vertically
if flip[1]: # if flip[1] is 1
flipped = [elem[::-1] for elem in flipped] # Flip elements to flip horizontally
flipped_X.append(flipX) # Finally append to result
print(flipped_X)
Will output:
[[[1, 0], [3, 2]], [[6, 7], [4, 5]]]
EDIT: Using numpy you can achieve the same result with np.flip(m, axis), where m will be each matrix in X and axis will be a tuple of the indexes to flip. For example:
to_flip = [[0,0], [1,0], [0,1], [1,1]]
so the indexes will be
[[], [0], [1], [0,1]]
# Get indexes to flip at each sample
flip_idxes = map(lambda flip: [idx for idx, x in enumerate(flip) if x], to_flip)
# Flip samples
flipped_X = np.array([np.flip(sample, indexes) for sample, indexes in zip(X, flip_idxes)])
Related
I am using the following example to understand the working of axis in 3D arrays in Numpy.
a = np.array([[9],[9],[8]])
b = np.array([[1],[4],[6]])
print(np.stack([a,b],axis=0)
>>>
array([[[9],
[9],
[8]],
[[1],
[4],
[6]]])
print(np.stack([a,b],axis=1)
>>>
array([[[9],
[1]],
[[9],
[4]],
[[8],
[6]]])
print(np.stack([a,b],axis=2)
>>>
array([[[9, 1]],
[[9, 4]],
[[8, 6]]])
I am able to understand how axis=0 and axis=1 work. Can anyone explain how axis=2 works with pictorial representation as it is done for 2D arrays?
For reference
print(np.stack([a,b],axis=0.shape) #(2,3,1)
print(np.stack([a,b],axis=1.shape) #(3,2,1)
print(np.stack([a,b],axis=2.shape) #(3,1,2)
You can see 3D numpy arrays as data cube.
Let's suppose we have an np.array A.
(z, y, x) = np.shape(A). You notice that the z dimensions corresponds to the indices 0.
Your array A is simply z 2d array of dimensions (y, x) that you stack together.
It explains why A[0,:,:] is a 2d array.
Axis = 2 simply points out that you consider 2d arrays stacked on the z direction!
I have a test case that reshaping the array changes the result of plt.imshow:
import numpy as np
import matplotlib.pyplot as plt
from skimage import io
file_raw_path = "8258792/Fig5_ColorfulCell_raw.tif"
im = io.imread(file_raw_path)
im= np.max(im, axis=0)
im_reshaped = im.reshape((im.shape[1],im.shape[2],im.shape[0]))
for i in range(im.shape[0]):
plt.imshow(im[i],cmap='gray')
plt.show()
for i in range(im_reshaped.shape[2]):
plt.imshow(im_reshaped[...,i],cmap='gray')
plt.show()
The first loop shows these images:
And the second loop shows this image (of course 5 times the same thing...):
Any idea why this is happening?!
np.reshape() doesn't move any data around; it just changes where the axes "wrap around". You can think about it as first flattening the input array, then wrapping the data across the axes to fit the new shape.
>>> arr = np.arange(6).reshape(2, 3)
array([[0, 1, 2],
[3, 4, 5]])
>>> arr.reshape(3, 2)
array([[0, 1],
[2, 3],
[4, 5]])
>>> arr.
If you read across left-to-right, top-to-bottom, all the numbers are in the same order.
You probably want np.transpose() and friends, which (essentially) shuffle the data around to change the order of the axes, so that im[i, j, k] == im.transpose(1, 2, 0)[j, k, i] (note, it doesn't actually move any data, it just looks like that). For your use case, np.moveaxis(im, 0, -1) will do the same thing, and is a bit easier to read ("move axis 0 to the end").
>>> arr.transpose(1, 0)
array([[0, 3],
[1, 4],
[2, 5]])
I have a numpy array of shape [batch_size, timesteps_per_samples, width, height], where width and height refer to a 2D grid. The values in this array can be interpreted as an elevation at a certain location that changes over time.
I want to know the elevation over time for various paths within this array. Therefore i have a second array of shape [batch_size, paths_per_batch_sample, timesteps_per_path, coordinates] (coordinates = 2, for x and y in the 2D plane).
The resulting array should be of shape [batch_size, paths_per_batch_sample, timesteps_per_path] containing the elevation over time for each sample within the batch.
The following two examples work. The first one is very slow and just serves for understanding what I am trying to do. I think the second one does what I want but I have no idea why this works nor if it may crash under certain circumstances.
Code for the problem setup:
import numpy as np
batch_size=32
paths_per_batch_sample=10
timesteps_per_path=4
width=64
height=64
elevation = np.arange(0, batch_size*timesteps_per_path*width*height, 1)
elevation = elevation.reshape(batch_size, timesteps_per_path, width, height)
paths = np.random.randint(0, high=width-1, size=(batch_size, paths_per_batch_sample, timesteps_per_path, 2))
range_batch = range(batch_size)
range_paths = range(paths_per_batch_sample)
range_timesteps = range(timesteps_per_path)
The following code works but is very slow:
elevation_per_time = np.zeros((batch_size, paths_per_batch_sample, timesteps_per_path))
for s in range_batch:
for k in range_paths:
for t in range_timesteps:
x_co, y_co = paths[s,k,t,:].astype(int)
elevation_per_time[s,k,t] = elevation[s,t,x_co,y_co]
The following code works (even fast) but I can't understand why and how o.0
elevation_per_time_fast = elevation[
:,
range_timesteps,
paths[:, :, range_timesteps, 0].astype(int),
paths[:, :, range_timesteps, 1].astype(int),
][range_batch, range_batch, :, :]
Prove that the results are equal
check = (elevation_per_time == elevation_per_time_fast)
print(np.all(check))
Can somebody explain how I can slice an nd-array by multiple other arrays?
Especially, I don't understand how the numpy knows that 'range_timesteps' has to run in step (for the index in axis 1,2,3).
Thanks in advance!
Lets take a quick look at slicing numpy array first:
a = np.arange(0,9,1).reshape([3,3])
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
Numpy has 2 ways of slicing array, full sections start:stop and by index from a list [index1, index2 ...]. The output will still be an array with the shape of your slice:
a[0:2,:]
array([[0, 1, 2],
[3, 4, 5]])
a[:,[0,2]]
array([[0, 2],
[3, 5],
[6, 8]])
The second part is that since you get a returned array with the same amount of dimensions you can easily stack any number of slices as long as you dont try to directly access an index outside of the array.
a[:][:][:][:][:][:][:][[0,2]][:,[0,2]]
array([[0, 2],
[6, 8]])
I am trying to do the following.
I have two 2D arrays, X and Y. Each is 100x100 elements. I want to linearize/flatten them into 10,000x1 columns and then concatenate them so I am left with a single matrix that is 10,000x2. In MATLAB I could do the following
BigMatrix = [X(:) Y(:)]
I want do the same thing in Python. After playing around with it for a bit I've been able to replicate the MATLAB result, albeit in quite a verbose manner, shown below. Is there a better, more succinct way to accomplish this?
BigMatrix = np.concatenate(
(X.reshape((10000,1), order = 'F'),
Y.reshape((10000,1), order = 'F')),
axis=1)
There are multiple ways to achieve what you want, and what you have is perfectly valid. However here are some other methods that you might find more "succinct."
Using np.ndarray.flatten
Return a copy of the array collapsed into one dimension.
You can also specify whether to treat it as column major or row major.
In order to get the result you want (a m x 2 matrix, with each flattened matrix as a column) you can then use numpy.column_stack
BigMatrix = np.column_stack([X.flatten(order = 'F'), Y.flatten(order = 'F')])
Or if you are looking for really succinct, as #ssp mentioned, you can use numpy indexing routines (which basically give special behavior to slices). There are two for concatenation, one for each axis. r_ is for row-wise (the first axis) and c_ is for column-wise (the second axis) so to get your m x 2 matrix you can do:
BigMatrix = np.c_[X.flatten(order = 'F'), Y.flatten(order = 'F')]
Performance?
As far as performance goes, you might be better off with your original code, as #hpaulj suggests. Here is a simple timing of the three methods, where each method is done 1 million times for your size of 100x100 matrices.
from timeit import timeit
print("c_ w/ flatten", timeit(
setup="import numpy as np\nX=np.random.standard_normal((100,100))\nY=np.random.standard_normal((100,100))",
stmt="Z=np.c_[X.flatten(order='F'), Y.flatten(order='F')]"
))
print("column_stack w/ flatten", timeit(
setup="import numpy as np\nX=np.random.standard_normal((100,100))\nY=np.random.standard_normal((100,100))",
stmt="Z=np.column_stack((X.flatten(order='F'), Y.flatten(order='F')))"
))
print("concatenate w/ reshape", timeit(
setup="import numpy as np\nX=np.random.standard_normal((100,100))\nY=np.random.standard_normal((100,100))",
stmt="Z=np.concatenate((X.reshape((10000,1),order='F'), Y.reshape((10000,1),order='F')), axis=1)"
))
and we get
c_ w/ flatten 44.47710300699691
column_stack w/ flatten 29.201319813000737
concatenate w/ reshape 27.67507728200144
Surprisingly, the column_stack and flatten is comparable, while the index routine is significantly slower.
(If there is anything I missed with this performance analysis, let me know. I am not a performance guru).
With a small 2 array:
In [404]: x = np.arange(4).reshape(2,2)
reshape with order F is the most direct equivalent of the MATLAB (:) indexing, producing a (n,1) array. (Is x(:).' the syntax for a (1,n) matrix?)
In [405]: x1 = x.reshape((4,1),order='F')
In [406]: x
Out[406]:
array([[0, 1],
[2, 3]])
In [407]: x1
Out[407]:
array([[0],
[2],
[1],
[3]])
Joining two such 'column vectors' is easy:
In [408]: np.concatenate((x1,x1), axis=1)
Out[408]:
array([[0, 0],
[2, 2],
[1, 1],
[3, 3]])
np.stack is a version of concatenate that creates a new dimension and joins on that. With axis=0 it's the same as np.array((x,x))
In [409]: np.stack((x,x), axis=2)
Out[409]:
array([[[0, 0],
[1, 1]],
[[2, 2],
[3, 3]]])
A order F reshape creates the 2 column array as before:
In [411]: np.stack((x,x), axis=2).reshape((-1,2),order='F')
Out[411]:
array([[0, 0],
[2, 2],
[1, 1],
[3, 3]])
or using the default order:
In [412]: np.stack((x,x), axis=2).reshape((-1,2))
Out[412]:
array([[0, 0],
[1, 1],
[2, 2],
[3, 3]])
numpy is a Python package, using functions, indexing and methods. It doesn't alter or add to the basic Python syntax.
It may be a stupid question but I couldn't find a similar question asked(for now).
For example, I define as function called f(x,y)
def f(x, y):
return x+y
Now I want to output a 2D numpy array, the value of an element is equal to its indices summed, for example, if I want a 2x2 array:
arr = [[0, 1],
[1, 2]]
If I want a 3x3 array, then the output should be:
arr = [[0, 1, 2],
[1, 2, 3],
[2, 3, 4]]
It's not efficient to assign the values one by one, especially if the array size is large, say 10000*10000, which is also a waste of the quick speed of numpy. Although it sounds quite basic but I can't think of a simple and quick solution to it. What is the most common and efficient way to do it?
By the way, the summing indices just an example. I hope that the method can also be generalized to arbitrary functions like, say,
def f(x,y):
return np.cos(x)+np.sin(y)
Or even to higher dimensional arrays, like 4x4 arrays.
You can use numpy.indices, which returns an array representing the indices of a grid; you'll just need to sum along the 0 axis:
>>> a = np.random.random((2,2))
>>> np.indices(a.shape).sum(axis=0) # array([[0, 1], [1, 2]])
>>> a = np.random.random((3,3))
>>> np.indices((3,3)).sum(axis=0) #array([[0, 1, 2], [1, 2, 3], [2, 3, 4]])