Develop Reference

Broadcasting a function to a 3D array Python

I tried understanding numpy broadcasting with 3d arrays but I think the OP there is asking something slightly different.
I have a 3D numpy array like so -
IQ = np.array([
[[1,2],
[3,4]],
[[5,6],
[7,8]]
], dtype = 'float64')
The shape of this array is (2,2,2). I want to apply a function to each 1x2 array in this 3D matrix like so -
def func(IQ):
I = IQ[0]
Q = IQ[1]
amp = np.power((np.power(I,2) + np.power(Q, 2)),1/2)
phase = math.atan(Q/I)
return [amp, phase]
As you can see, I want to apply my function to each 1x2 array and replace it with the return value of my function. The output is a 3D array with the same dimensions. Is there a way to broadcast this function to each 1x2 array in my original 3D array? Currently I am using loops which becomes very slow as the 3D array increases in dimensions.
Currently I am doing this -
#IQ is defined from above
for i in range(IQ.shape[0]):
for j in range(IQ.shape[1]):
I = IQ[i,j,0]
Q = IQ[i,j,1]
amp = np.power((np.power(I,2) + np.power(Q, 2)),1/2)
phase = math.atan(Q/I)
IQ[i,j,0] = amp
IQ[i,j,1] = phase
And the returned 3D array is -
[[[ 2.23606798 1.10714872]
[ 5. 0.92729522]]
[[ 7.81024968 0.87605805]
[10.63014581 0.85196633]]]

One way is to slice the arrays to extract the I and Q values, perform the computations using normal broadcasting, and then stick the values back together:
>>> Is, Qs = IQ[...,0], IQ[...,1]
>>> np.stack(((Is**2 + Qs**2) ** 0.5, np.arctan2(Qs, Is)), axis=-1)
array([[[ 2.23606798, 1.10714872],
[ 5. , 0.92729522]],
[[ 7.81024968, 0.87605805],
[10.63014581, 0.85196633]]])

It can be done using arrays:
# sort of sum of squares along axis 2, ie (IQ[..., 0]**2 + IQ[..., 1]**2 + ...)**0.5
amp = np.sqrt(np.square(IQ).sum(axis=2))
amp
>>> array([[ 2.23606798, 5. ],
[ 7.81024968, 10.63014581]])
# and phase is arctan for each component in each matrix
phase = np.arctan2(IQ[..., 1], IQ[..., 0])
phase
>>> array([[1.10714872, 0.92729522],
[0.87605805, 0.85196633]])
# then combine the arrays to 3d
np.stack([amp, phase], axis=2)
>>> array([[[ 2.23606798, 1.10714872],
[ 5. , 0.92729522]],
[[ 7.81024968, 0.87605805],
[10.63014581, 0.85196633]]])

I = IQ[..., 0]
Q = IQ[..., 1]
amp = np.linalg.norm(IQ, axis= 2)
phase = np.arctan(Q/I)
IQ[..., 0] = amp
IQ[..., 1] = phase
IQ
>> [[[ 2.23606798, 1.10714872],
[ 5. , 0.92729522]],
[[ 7.81024968, 0.87605805],
[10.63014581, 0.85196633]]]

Remove zero vectors from a matrix in TensorFlow

Just like the question says, I'm trying to remove all zeros vectors (i.e [0, 0, 0, 0]) from a tensor.
Given:
array([[ 0. , 0. , 0. , 0. ],
[ 0.19999981, 0.5 , 0. , 0. ],
[ 0.4000001 , 0.29999995, 0.10000002, 0. ],
...,
[-0.5999999 , 0. , -0.0999999 , -0.20000005],
[-0.29999971, -0.4000001 , -0.30000019, -0.5 ],
[ 0. , 0. , 0. , 0. ]], dtype=float32)
I had tried the following code (inspired by this SO):
x = tf.placeholder(tf.float32, shape=(10000, 4))
zeros_vector = tf.zeros(shape=(1, 4), dtype=tf.float32)
bool_mask = tf.not_equal(x, zero_vector)
omit_zeros = tf.boolean_mask(x, bool_mask)
But bool_mask seem also to be of shape (10000, 4), like it was comparing every element in the x tensor to zero, and not rows.
I thought about using tf.reduce_sum where an entire row is zero, but that will omit also rows like [1, -1, 0, 0] and I don't want that.
Ideas?

One possible way would be to sum over the absolute values of the row, in this way it will not omit rows like [1, -1, 0, 0] and then compare it with a zero vector. You can do something like this:
intermediate_tensor = reduce_sum(tf.abs(x), 1)
zero_vector = tf.zeros(shape=(1,1), dtype=tf.float32)
bool_mask = tf.not_equal(intermediate_tensor, zero_vector)
omit_zeros = tf.boolean_mask(x, bool_mask)

I tried solution by Rudresh Panchal and it doesn't work for me. Maybe due versions change.
I found tipo in the first row: reduce_sum(tf.abs(x), 1) -> tf.reduce_sum(tf.abs(x), 1).
Also, bool_mask has rank 2 instead of rank 1, which is required:
tensor: N-D tensor.
mask: K-D boolean tensor, K <= N and K must be known statically. In other words, the shape of bool_mask must be for example [6] not [1,6]. tf.squeeze works well to reduce dimension.
Corrected code which works for me:
intermediate_tensor = tf.reduce_sum(tf.abs(x), 1)
zero_vector = tf.zeros(shape=(1,1), dtype=tf.float32)
bool_mask = tf.squeeze(tf.not_equal(intermediate_tensor, zero_vector))
omit_zeros = tf.boolean_mask(x, bool_mask)

Just cast the tensor to tf.bool and use it as a boolean mask:
boolean_mask = tf.cast(x, dtype=tf.bool)
no_zeros = tf.boolean_mask(x, boolean_mask, axis=0)

Making a multidimensional list of vectors

I am quite new to Python so bear with me. I am writing a program to calculate some physical quantity, let's call it A. A is a function of several variables, let's call them x, y, z. So I have three nested loops to calculate A for the values of x, y, z that I am interested in.
for x in xs:
for y in ys:
for z in zs:
A[x, y, z] = function_calculating_value(x,y,z)
Now, the problem is that A[x,y,z] is two-dimensional array containing both the mean value and the variance so that A[x,y,z] = [mean, variance]. From other languages I am used to initializing A using function similar to np.zeros(). How do I do that here? What is the easiest way to achieve what I want, and how do I access the mean and variance easily for a given (x,y,z)?
(the end goal is to be able to plot the mean with the variance as error bars, so if there is an even more elegant way of doing this, I appreciate that as well)
thanks in advance!

You can create and manipulate your multi-dimensional array with numpy
# Generate a random 4d array that has nx = 3, ny = 3, and nz = 3, with each 3D point having 2 values
mdarray = np.random.random( size = (3,3,3,2) )
# The overall shape of the 4d array
mdarray
Out[66]:
array([[[[ 0.80091246, 0.28476668],
[ 0.94264747, 0.27247111],
[ 0.64503087, 0.13722768]],
[[ 0.21371798, 0.41006764],
[ 0.79783723, 0.02537987],
[ 0.80658387, 0.43464532]],
[[ 0.04566927, 0.74836831],
[ 0.8280196 , 0.90288647],
[ 0.59271082, 0.65910184]]],
[[[ 0.82533798, 0.29075978],
[ 0.76496127, 0.1308289 ],
[ 0.22767752, 0.01865939]],
[[ 0.76849458, 0.7934015 ],
[ 0.93313128, 0.88436557],
[ 0.06897508, 0.00307739]],
[[ 0.15975812, 0.00792386],
[ 0.40292818, 0.21209199],
[ 0.48805502, 0.71974702]]],
[[[ 0.66522525, 0.49797465],
[ 0.29369336, 0.68743839],
[ 0.46411967, 0.69547356]],
[[ 0.50339875, 0.66423777],
[ 0.80520751, 0.88115054],
[ 0.08296022, 0.69467829]],
[[ 0.76572574, 0.45332754],
[ 0.87982243, 0.15773385],
[ 0.5762041 , 0.91268172]]]])
# Both values for this specific sample at x = 0, y = 1 and z = 2
mdarray[0,1,2]
Out[67]: array([ 0.80658387, 0.43464532])
mdarray[0,1,2,0] # mean only at the same point
Out[68]: 0.8065838666297338
mdarray[0,1,2,1] # variance only at the same point
Out[69]: 0.43464532443865489
You can also get only the means or the variance values separately by slicing the array:
mean = mdarray[:,:,:,0]
variance = mdarray[:,:,:,1]
mean
Out[74]:
array([[[ 0.80091246, 0.94264747, 0.64503087],
[ 0.21371798, 0.79783723, 0.80658387],
[ 0.04566927, 0.8280196 , 0.59271082]],
[[ 0.82533798, 0.76496127, 0.22767752],
[ 0.76849458, 0.93313128, 0.06897508],
[ 0.15975812, 0.40292818, 0.48805502]],
[[ 0.66522525, 0.29369336, 0.46411967],
[ 0.50339875, 0.80520751, 0.08296022],
[ 0.76572574, 0.87982243, 0.5762041 ]]])
I'm still unsure how I would have preferred to plot this data, will think about this a bit and update this answer.

index 2d numpy.array with 2d numpy.array

I have an N-by-2 numpy array of 2d coordinates named coords, and another 2d numpy array named plane. What I want to do is like
for x,y in coords:
plane[x,y] = 0
but without for loop to improve efficiency. How to do this with vectorized code? Which function or method in numpy to use?

You can try plane[coords.T[0], coords.T[1]] = 0 Not sure this is what you want. For example:
Let,
plane = np.random.random((5,5))
coords = np.array([ [2,3], [1,2], [1,3] ])
Then,
plane[coords.T[0], coords.T[1]] = 0
will give:
array([[ 0.41981685, 0.4584495 , 0.47734686, 0.23959934, 0.82641475],
[ 0.64888387, 0.44788871, 0. , 0. , 0.298522 ],
[ 0.22764842, 0.06700281, 0.04856316, 0. , 0.70494825],
[ 0.18404081, 0.27090759, 0.23387404, 0.02314846, 0.3712009 ],
[ 0.28215705, 0.12886813, 0.62971 , 0.9059715 , 0.74247202]])

Numpy: Avoiding nested loops to operate on matrix-valued images

I am a beginner at python and numpy and I need to compute the matrix logarithm for each "pixel" (i.e. x,y position) of a matrix-valued image of dimension NxMx3x3. 3x3 is the dimensions of the matrix at each pixel.
The function I have written so far is the following:
def logm_img(im):
from scipy import linalg
dimx = im.shape[0]
dimy = im.shape[1]
res = zeros_like(im)
for x in range(dimx):
for y in range(dimy):
res[x, y, :, :] = linalg.logm(asmatrix(im[x,y,:,:]))
return res
Is it ok?
Is there a way to avoid the two nested loops ?

Numpy can do that. Just call numpy.log:
>>> import numpy
>>> a = numpy.array(range(100)).reshape(10, 10)
>>> b = numpy.log(a)
__main__:1: RuntimeWarning: divide by zero encountered in log
>>> b
array([[ -inf, 0. , 0.69314718, 1.09861229, 1.38629436,
1.60943791, 1.79175947, 1.94591015, 2.07944154, 2.19722458],
[ 2.30258509, 2.39789527, 2.48490665, 2.56494936, 2.63905733,
2.7080502 , 2.77258872, 2.83321334, 2.89037176, 2.94443898],
[ 2.99573227, 3.04452244, 3.09104245, 3.13549422, 3.17805383,
3.21887582, 3.25809654, 3.29583687, 3.33220451, 3.36729583],
[ 3.40119738, 3.4339872 , 3.4657359 , 3.49650756, 3.52636052,
3.55534806, 3.58351894, 3.61091791, 3.63758616, 3.66356165],
[ 3.68887945, 3.71357207, 3.73766962, 3.76120012, 3.78418963,
3.80666249, 3.8286414 , 3.8501476 , 3.87120101, 3.8918203 ],
[ 3.91202301, 3.93182563, 3.95124372, 3.97029191, 3.98898405,
4.00733319, 4.02535169, 4.04305127, 4.06044301, 4.07753744],
[ 4.09434456, 4.11087386, 4.12713439, 4.14313473, 4.15888308,
4.17438727, 4.18965474, 4.20469262, 4.21950771, 4.2341065 ],
[ 4.24849524, 4.26267988, 4.27666612, 4.29045944, 4.30406509,
4.31748811, 4.33073334, 4.34380542, 4.35670883, 4.36944785],
[ 4.38202663, 4.39444915, 4.40671925, 4.41884061, 4.4308168 ,
4.44265126, 4.4543473 , 4.46590812, 4.47733681, 4.48863637],
[ 4.49980967, 4.51085951, 4.52178858, 4.53259949, 4.54329478,
4.55387689, 4.56434819, 4.57471098, 4.58496748, 4.59511985]])