Develop Reference

How can I use a 3d numpy array of indices to retrieve the corresponding values i a 4d array?

I have a 4d numpy array temperature of data with the measured temperature at points x,y,z and time t. Assuming I have an array indices with the indices where the first instance of a condition is met, say temperature < 0, how do I extract a 3d array with the first temperatures satisfying this condition? That is I'm looking for the equivalent of numpy's 1d version (import numpy as np tacitly assumed)
>>> temperatures = np.arange(10,-10,-1)
>>> ind = np.argmax(temperatures < 0)
>>> T = temperature[ind]
I have tried the analogous
In [1]: temperatures = np.random.random((11,8,5,200)) * 1000
In [2]: temperatures.shape
Out[2]: (11, 8, 5, 200)
In [3]: indices= np.argmax(temperatures > 900,axis=3)
In [4]: indices.shape
Out[4]: (11, 8, 5)
In [5]: T = temperatures[:,:,:,indices]
In [6]: T.shape
Out[6]: (11, 8, 5, 11, 8, 5)
However, the dimensions if Tis 6.
I could of course do it with a for loop:
indices = np.argmax(temperatures > 900,axis=3)
x,y,z = temperatures.shape[:-1]
T = np.zeros((x,y,z))
for indx in range(x):
for indy in range(y):
for indz in range(z):
T[indx,indy,indz] = temperatures[indx,indy,indz,indices[indx,indy,indz]]
but I'm looking for something fore elegant and more pythonic. Is there someone more skilled with numpy out there who can help me out on this?
P.S. For the sake of clarity, I'm not just looking for the temperature at these points given by indices, I'm also looking for other quantities in arrays of the same shape as temperature, e.g. the time derivative. Also, in reality the arrays are much larger then this minimal example.

Numpy advanced indexing does always work:
import numpy as np
temperatures = np.random.random((11,8,5, 200)) * 1000
indices = np.argmax(temperatures > 900, axis=3)
x, y, z = temperatures.shape[:-1]
T = temperatures[np.arange(x)[:, np.newaxis, np.newaxis],
np.arange(y)[np.newaxis, :, np.newaxis],
np.arange(z)[np.newaxis, np.newaxis, :],
indices]
As jdehesa pointed out this can be made more concise:
x, y, z = np.ogrid[:x, :y, :z]
T = temperatures[x, y, z, i]

I think you need:
axis = 3
indices = np.argmax(temperatures > 900, axis=axis)
result = np.take_along_axis(temperatures, np.expand_dims(indices, axis), axis)
result = result.squeeze(axis)

Matrix and Tensor Multiplication with Numpy

I am trying to do a matrix multiplication with a tensor, but I'm uncertain how to do it with Numpy. I have been trying to work with np.tensordot(), but I haven't been able to do so
In a simpler way, if we were to do matrix multiplications and we had a vector v (Nx1) and a matrix S (NxN), we can do the operation
v^T S v => (1xN)(NxN)(Nx1) => a number
v = np.ones((3,1))
S = np.ones((3,3))
y = v.T.dot(S).dot(v)
y.shape = (1) or ()
Now, I'd like to do the following:
Let matrix M (3x5) and tensor Z (5x3x3) so that I can have
M^T Z M
where (M^T Z) yields a (5x3) matrix, and M^T Z M yields a (1x5) vector
M = np.ones((3,5))
Z = np.ones((5,3,3))
Y = <?> M.T * Z * M <?>
Y.shape = (5,) or (1,5)
Does anyone know how to do this using Numpy without using Tensorflow?

I think this computes what you want:
import numpy as np
M = np.ones((3, 5))
Z = np.ones((5, 3, 3))
# Multiply (5, 1, 3) x (5, 3, 3) x (5, 3, 1)
result = (M.T[:, np.newaxis] # Z # M.T[:, :, np.newaxis]).squeeze()
print(result)
Output:
[9. 9. 9. 9. 9.]
I have used the # operator for convenience, but you can replace it with np.matmul if you like it more or are using an older Python version:
result = np.matmul(np.matmul(M.T[:, np.newaxis], Z), M.T[:, :, np.newaxis]).squeeze()

Generating data from meshgrid data (Numpy)

I'd like to ask how to generate corresponding values from a meshgrid. I have a function "foo" that takes one 1D array with the length of 2, and returns some real number.
import numpy as np
def foo(X):
#this function takes a vector, e.g., np.array([2,3]), and returns a real number.
return sum(X)**np.sin( sum(X) );
x = np.arange(-2, 1, 1) # points in the x axis
y = np.arange( 3, 8, 1) # points in the y axis
X, Y = np.meshgrid(x, y) # X, Y : grid
I generate X and Y grids using meshgrid.
Then, how can I generate corresponding Z values using "foo" function, in order to plot them in 3D, e.g., plotting using plot_surface function with X,Y,Z values?
Here the question is how to generate Z values, which has the same shape to X and Y, using "foo" function. Since my "foo" function only takes an 1D array, I do not know how I can uses this function with X and Y to generate corresponding Z values.

Stack your two numpy arrays in "depth" using np.dstack, and then modify your foo function, so that it operates on only the last axis of your stacked array. This is easily done using np.sum with parameter axis=-1, instead of using the builtin sum:
import numpy as np
def foo(xy):
return np.sum(xy, axis=-1) ** np.sin(np.sum(xy, axis=-1))
x = np.arange(-2, 1, 1) # points in the x axis
y = np.arange( 3, 8, 1) # points in the y axis
X, Y = np.meshgrid(x, y) # X, Y : grid
XY = np.dstack((X, Y))
And now, you should get:
>>> XY.shape
(5, 3, 2)
>>> foo(XY)
array([[ 1. , 1.87813065, 1.1677002 ],
[ 1.87813065, 1.1677002 , 0.35023496],
[ 1.1677002 , 0.35023496, 0.2136686 ],
[ 0.35023496, 0.2136686 , 0.60613935],
[ 0.2136686 , 0.60613935, 3.59102217]])
If you want to achieve the same effect, but without modifying foo, then you can use np.apply_along_axis, which should do exactly what you need:
>>> np.apply_along_axis(foo, -1, XY)
array([[ 1. , 1.87813065, 1.1677002 ],
[ 1.87813065, 1.1677002 , 0.35023496],
[ 1.1677002 , 0.35023496, 0.2136686 ],
[ 0.35023496, 0.2136686 , 0.60613935],
[ 0.2136686 , 0.60613935, 3.59102217]])

Python, Numpy, method of adding arrays together in multiple dimensions (broadcasting)

I have a number of different size arrays with a common index.
For example,
Arr1 = np.arange(0, 1000, 1).reshape(100, 10)
Arr2 = np.arange(0, 500, 1).reshape(100,5)
Arr1.shape = (100, 10)
Arr2.shape = (100, 5)
I want to add these together into a new Array, Arr3 which is three dimensional. e.g.
Arr3 = Arr1 + Arr2
Arr3.shape = (100, 10, 5)
Note, in this instance the values should allign e.g.
Arr3[10, 3, 2] = Arr1[10, 3] + Arr2[10, 2]
I have been attempting to use the following method
test = Arr1.copy()
test = test[:, np.newaxis] + Arr2
Now, I've been able to make this work when adding two square matrices together.
m = np.arange(0, 100, 1)
[x, y] = np.meshgrid(x, y)
x.shape = (100, 100)
test44 = x.copy()
test44 = test44[:, np.newaxis] + x
test44.shape = (100, 100, 100)
test44[4, 3, 2] = 4
x[4, 2] = 2
x[3, 2] = 2
However, in my actual program I will not have square matrices for this issue.
In addition this method is extremely memory intensive as evidenced when you begin moving up the number of dimensions as follows.
test44 = test44[:, :, np.newaxis] + x
test44.shape = (100, 100, 100, 100)
# Note this next command will fail with a memory error on my computer.
test44 = test44[:, :, :, np.newaxis] + x
So my question has two parts:
Is it possible to create a 3D array from two differently shaped 2D array with a common "shared" axis.
Is such a method extensible at higher order dimensions?
Any assistance is greatly appreciated.

Yes what you're trying to do is called broadcasting, it's done automatically by numpy if the inputs have the right shapes. Try this:
Arr1 = Arr1.reshape((100, 10, 1))
Arr2 = Arr2.reshape((100, 1, 5))
Arr3 = Arr1 + Arr2
I've found this to be a really good introduction to broadcasting which should show you how to extend this kind of behavior to n dimensions.

Numpy meshgrid in 3D

Numpy's meshgrid is very useful for converting two vectors to a coordinate grid. What is the easiest way to extend this to three dimensions? So given three vectors x, y, and z, construct 3x3D arrays (instead of 2x2D arrays) which can be used as coordinates.

Numpy (as of 1.8 I think) now supports higher that 2D generation of position grids with meshgrid. One important addition which really helped me is the ability to chose the indexing order (either xy or ij for Cartesian or matrix indexing respectively), which I verified with the following example:
import numpy as np
x_ = np.linspace(0., 1., 10)
y_ = np.linspace(1., 2., 20)
z_ = np.linspace(3., 4., 30)
x, y, z = np.meshgrid(x_, y_, z_, indexing='ij')
assert np.all(x[:,0,0] == x_)
assert np.all(y[0,:,0] == y_)
assert np.all(z[0,0,:] == z_)

Here is the source code of meshgrid:
def meshgrid(x,y):
"""
Return coordinate matrices from two coordinate vectors.
Parameters
----------
x, y : ndarray
Two 1-D arrays representing the x and y coordinates of a grid.
Returns
-------
X, Y : ndarray
For vectors `x`, `y` with lengths ``Nx=len(x)`` and ``Ny=len(y)``,
return `X`, `Y` where `X` and `Y` are ``(Ny, Nx)`` shaped arrays
with the elements of `x` and y repeated to fill the matrix along
the first dimension for `x`, the second for `y`.
See Also
--------
index_tricks.mgrid : Construct a multi-dimensional "meshgrid"
using indexing notation.
index_tricks.ogrid : Construct an open multi-dimensional "meshgrid"
using indexing notation.
Examples
--------
>>> X, Y = np.meshgrid([1,2,3], [4,5,6,7])
>>> X
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
>>> Y
array([[4, 4, 4],
[5, 5, 5],
[6, 6, 6],
[7, 7, 7]])
`meshgrid` is very useful to evaluate functions on a grid.
>>> x = np.arange(-5, 5, 0.1)
>>> y = np.arange(-5, 5, 0.1)
>>> xx, yy = np.meshgrid(x, y)
>>> z = np.sin(xx**2+yy**2)/(xx**2+yy**2)
"""
x = asarray(x)
y = asarray(y)
numRows, numCols = len(y), len(x) # yes, reversed
x = x.reshape(1,numCols)
X = x.repeat(numRows, axis=0)
y = y.reshape(numRows,1)
Y = y.repeat(numCols, axis=1)
return X, Y
It is fairly simple to understand. I extended the pattern to an arbitrary number of dimensions, but this code is by no means optimized (and not thoroughly error-checked either), but you get what you pay for. Hope it helps:
def meshgrid2(*arrs):
arrs = tuple(reversed(arrs)) #edit
lens = map(len, arrs)
dim = len(arrs)
sz = 1
for s in lens:
sz*=s
ans = []
for i, arr in enumerate(arrs):
slc = [1]*dim
slc[i] = lens[i]
arr2 = asarray(arr).reshape(slc)
for j, sz in enumerate(lens):
if j!=i:
arr2 = arr2.repeat(sz, axis=j)
ans.append(arr2)
return tuple(ans)

Can you show us how you are using np.meshgrid? There is a very good chance that you really don't need meshgrid because numpy broadcasting can do the same thing without generating a repetitive array.
For example,
import numpy as np
x=np.arange(2)
y=np.arange(3)
[X,Y] = np.meshgrid(x,y)
S=X+Y
print(S.shape)
# (3, 2)
# Note that meshgrid associates y with the 0-axis, and x with the 1-axis.
print(S)
# [[0 1]
# [1 2]
# [2 3]]
s=np.empty((3,2))
print(s.shape)
# (3, 2)
# x.shape is (2,).
# y.shape is (3,).
# x's shape is broadcasted to (3,2)
# y varies along the 0-axis, so to get its shape broadcasted, we first upgrade it to
# have shape (3,1), using np.newaxis. Arrays of shape (3,1) can be broadcasted to
# arrays of shape (3,2).
s=x+y[:,np.newaxis]
print(s)
# [[0 1]
# [1 2]
# [2 3]]
The point is that S=X+Y can and should be replaced by s=x+y[:,np.newaxis] because
the latter does not require (possibly large) repetitive arrays to be formed. It also generalizes to higher dimensions (more axes) easily. You just add np.newaxis where needed to effect broadcasting as necessary.
See http://www.scipy.org/EricsBroadcastingDoc for more on numpy broadcasting.

i think what you want is
X, Y, Z = numpy.mgrid[-10:10:100j, -10:10:100j, -10:10:100j]
for example.

Here is a multidimensional version of meshgrid that I wrote:
def ndmesh(*args):
args = map(np.asarray,args)
return np.broadcast_arrays(*[x[(slice(None),)+(None,)*i] for i, x in enumerate(args)])
Note that the returned arrays are views of the original array data, so changing the original arrays will affect the coordinate arrays.

Instead of writing a new function, numpy.ix_ should do what you want.
Here is an example from the documentation:
>>> ixgrid = np.ix_([0,1], [2,4])
>>> ixgrid
(array([[0],
[1]]), array([[2, 4]]))
>>> ixgrid[0].shape, ixgrid[1].shape
((2, 1), (1, 2))'

You can achieve that by changing the order:
import numpy as np
xx = np.array([1,2,3,4])
yy = np.array([5,6,7])
zz = np.array([9,10])
y, z, x = np.meshgrid(yy, zz, xx)