Develop Reference

NumPy + PyTorch Tensor assignment

lets assume we have a tensor representing an image of the shape (910, 270, 1) which assigned a number (some index) to each pixel with width=910 and height=270.
We also have a numpy array of size (N, 3) which maps a 3-tuple to an index.
I now want to create a new numpy array of shape (920, 270, 3) which has a 3-tuple based on the original tensor index and the mapping-3-tuple-numpy array. How do I do this assignment without for loops and other consuming iterations?
This would look simething like:
color_image = np.zeros((self._w, self._h, 3), dtype=np.int32)
self._colors = np.array(N,3) # this is already present
indexed_image = torch.tensor(920,270,1) # this is already present
#how do I assign it to this numpy array?
color_image[indexed_image.w, indexed_image.h] = self._colors[indexed_image.flatten()]

Assuming you have _colors, and indexed_image. Something that ressembles to:
>>> indexed_image = torch.randint(0, 10, (920, 270, 1))
>>> _colors = np.random.randint(0, 255, (N, 3))
A common way of converting a dense map to a RGB map is to loop over the label set:
>>> _colors = torch.FloatTensor(_colors)
>>> rgb = torch.zeros(indexed_image.shape[:-1] + (3,))
>>> for lbl in range(N):
... rgb[lbl == indexed_image[...,0]] = _colors[lbl]

Python make montaage of images stored in a 4D numpy array

I have a stack of images stored in a 4D array, e.g. [0, 0, :, :] is the image at the location (0, 0). Now I want to make a montage of the images and store them in a 2D array and do something with the images, then I want to transfer the montage back to a 4D array. How can I manage this with numpy? Following is a schematic of what I want to do. It is shown with a 3D array, but I think you can get the idea.

The first part of the operation can be carried out using np.block. You would need to convert to a non-array sequence type for the outer dimensions:
l = [list(x) for x in arr]
montage = np.block(l)
Alternatively, you can just arrange your dimensions the way you like first, then reshape. The key is to remember that later dimensions get raveled together. So if you have an array with (A, B) elements, each of which is an (M, N) image, the result should be an (A * M, B * N) image. You want the original image pixels from each row to stay contiguous, but the rows to be concatenated. So transpose and reshape like this:
a, b, m, n = arr.shape
montage = arr.transpose(0, 2, 1, 3).reshape(a * m, b * n)
You can reshape back using the inverse operation fairly easily:
stack = montage.reshape(a, m, b, n).transpose(0, 2, 1, 3)

This is actually the default behavior of np.reshape(). Just calculate how wide/tall the collage image will be, and then call np.reshape. reshape again will reverse it.
import numpy as np
# placeholder data -- 4 images that are 5x5
image = np.arange(4 * 5 * 5 * 3).reshape(4, 5, 5, 3)
# 2x2 grid of images
collage = image.reshape(10, 10, 3)
result = collage.reshape(4, 5, 5, 3)
assert np.array_equal(image, result)
Edit: I misunderstood the question. I assumed that the 4D array was a 1D-list of NxMx3 RGB images. If, instead, it is a 2D grid of 2D (single channel) images, I can't think of a clever way to do it with numpy operations. But, it shouldn't be to slow to just use a python for-loop.
(assuming row-major order)
# rows = number of rows in image grid
# cols = number of cols in image grid
# width = width of each image
# height = height of each image
rows, cols, height, width = images.shape
collage = np.empty(rows * height, cols * width, dtype=images.dytpe)
for i in range(rows):
for j in range(cols):
y = i * height
x = j * range
collage[y:y+height, x:x+width] = images[i, j]
Then to reverse just flip it:
result = np.empty(rows, cols, width, height, dtype=collage.dytpe)
for i in range(rows):
for j in range(cols):
y = i * height
x = j * range
images[i, j, :, :] = collage[y:y+height, x:x+width]

How can I use a 3d numpy array of indices to retrieve the corresponding values i a 4d array?

I have a 4d numpy array temperature of data with the measured temperature at points x,y,z and time t. Assuming I have an array indices with the indices where the first instance of a condition is met, say temperature < 0, how do I extract a 3d array with the first temperatures satisfying this condition? That is I'm looking for the equivalent of numpy's 1d version (import numpy as np tacitly assumed)
>>> temperatures = np.arange(10,-10,-1)
>>> ind = np.argmax(temperatures < 0)
>>> T = temperature[ind]
I have tried the analogous
In [1]: temperatures = np.random.random((11,8,5,200)) * 1000
In [2]: temperatures.shape
Out[2]: (11, 8, 5, 200)
In [3]: indices= np.argmax(temperatures > 900,axis=3)
In [4]: indices.shape
Out[4]: (11, 8, 5)
In [5]: T = temperatures[:,:,:,indices]
In [6]: T.shape
Out[6]: (11, 8, 5, 11, 8, 5)
However, the dimensions if Tis 6.
I could of course do it with a for loop:
indices = np.argmax(temperatures > 900,axis=3)
x,y,z = temperatures.shape[:-1]
T = np.zeros((x,y,z))
for indx in range(x):
for indy in range(y):
for indz in range(z):
T[indx,indy,indz] = temperatures[indx,indy,indz,indices[indx,indy,indz]]
but I'm looking for something fore elegant and more pythonic. Is there someone more skilled with numpy out there who can help me out on this?
P.S. For the sake of clarity, I'm not just looking for the temperature at these points given by indices, I'm also looking for other quantities in arrays of the same shape as temperature, e.g. the time derivative. Also, in reality the arrays are much larger then this minimal example.

Numpy advanced indexing does always work:
import numpy as np
temperatures = np.random.random((11,8,5, 200)) * 1000
indices = np.argmax(temperatures > 900, axis=3)
x, y, z = temperatures.shape[:-1]
T = temperatures[np.arange(x)[:, np.newaxis, np.newaxis],
np.arange(y)[np.newaxis, :, np.newaxis],
np.arange(z)[np.newaxis, np.newaxis, :],
indices]
As jdehesa pointed out this can be made more concise:
x, y, z = np.ogrid[:x, :y, :z]
T = temperatures[x, y, z, i]

I think you need:
axis = 3
indices = np.argmax(temperatures > 900, axis=axis)
result = np.take_along_axis(temperatures, np.expand_dims(indices, axis), axis)
result = result.squeeze(axis)

numpy pad with zeros creates 2d array instead of desired 1d

I am trying to pad a 1d numpy array with zeros.
Here is my code
v = np.random.rand(100, 1)
pad_size = 100
v = np.pad(v, (pad_size, 0), 'constant')
result is 200x101 array, whose last column is [0,0,0,... <v>], (leading 100 zeros),
and all 1st 100 columns are zeros.
How to get my desired array
[0,0,0,..0,<v>]
of size (len(v)+pad_size, 1)?

The pad output is 2D because the pad input was 2D. You made a 2D array with rand for some reason:
v = np.random.rand(100, 1)
If you wanted a 1D array, you should have made a 1D array:
v = np.random.rand(100)
If you wanted a 1-column 2D array, then you're using pad incorrectly. The second argument should be ((100, 0), (0, 0)): padding 100 elements before in the first axis, 0 elements after in the first axis, 0 elements before in the second axis, 0 elements after in the second axis:
v = np.random.rand(100, 1)
pad_size = 100
v = np.pad(v, ((pad_size, 0), (0, 0)), 'constant')
For a 1-row 2D array, you would need to adjust both the rand call and the pad call:
v = np.random.rand(1, 100)
pad_size = 100
v = np.pad(v, ((0, 0), (pad_size, 0)), 'constant')

np.hstack((np.zeros((200, 100)), your v))
np.concatenate((np.zeros((200, 100)), your v), axis=1)
may be your desire this:

Python: speed up matrix coordinate mapping through iteration

I'm trying to get this code to run as fast as possible and at the moment is very inefficient.
I have a 4D matrix of scalar data. The 4 dimensions correspond to latitude, longitude, altitude and time. The data is stored in a numpy array and its shape is (5,5,30,2).
In 4 different lists I am keeping the "map" for each axis, storing what value corresponds to each index. For example, the map arrays could look like:
mapLatitude = [45.,45.2,45.4,45.6,45.8]
mapLongitude = [-10.8,-10.6,-10.4,-10.2,-10.]
mapAltitude = [0,50,100,150,...,1450]
mapTime = [1345673,1345674]
This means that in the data matrix, the data point at location 0,1,3,0 corresponds to
Lat = 45, Lon = -10.6, Alt = 150, Time = 1345673.
Now, I need to generate a new array containing the coordinates of each point in my data matrix.
So far, this is what I've written:
import numpy as np
# data = np.array([<all data>])
coordinateMatrix = [
(mapLatitude[index[0]],
mapLongitude[index[1]],
mapAltitude[index[2]],
mapTime[index[3]] ) for index in numpy.ndindex(data.shape) ]
This works, but takes quite a long time, especially when the data matrix increases in size (I need to use this with matrices with a shape like (100,100,150,30) ).
If it helps, I need to generate this coordinateMatrix to feed it to scipy.interpolate.NearestNDInterpolator .
Any suggestions on how to speed this up?
Thank you very much!

If you turn your lists into ndarray's you can use broadcasting as follows:
coords = np.zeros((5, 5, 30, 2, 4))
coords[..., 0] = np.array(mapLatitude).reshape(5, 1, 1, 1)
coords[..., 1] = np.array(mapLongitude).reshape(1, 5, 1, 1)
coords[..., 2] = np.array(mapAltitude).reshape(1, 1, 30, 1)
coords[..., 3] = np.array(mapTime).reshape(1, 1, 1, 2)
For more general inputs something like this should work:
def makeCoordinateMatrix(*coords) :
dims = len(coords)
coords = [np.array(a) for a in coords]
shapes = tuple([len(a) for a in coords])
ret = np.zeros(shapes + (dims,))
for j, a in enumerate(coords) :
ret[..., j] = a.reshape((len(a),) + (1,) * (dims - j - 1))
return ret
coordinateMatrix = makeCoordinateMatrix(mapLatitude, mapLongitude,
mapAltitude, mapTime)