For a numpy array of dimension n, I'd like to apply np.nanmax() to n-1 dimensions producing a 1 dimensional array of maxima, ignoring all values set to np.nan.
q = np.arange(5*4*3.).reshape(3,4,5) % (42+1)
q[q%5==0] = np.nan
producing:
array([[[ nan, 1., 2., 3., 4.],
[ nan, 6., 7., 8., 9.],
[ nan, 11., 12., 13., 14.],
[ nan, 16., 17., 18., 19.]],
[[ nan, 21., 22., 23., 24.],
[ nan, 26., 27., 28., 29.],
[ nan, 31., 32., 33., 34.],
[ nan, 36., 37., 38., 39.]],
[[ nan, 41., 42., nan, 1.],
[ 2., 3., 4., nan, 6.],
[ 7., 8., 9., nan, 11.],
[ 12., 13., 14., nan, 16.]]])
If I know ahead of time that I want to use the last axis as the remaining dimension, I can use the -1 feature in .reshape() and do this:
np.nanmax(q.reshape(-1, q.shape[-1]), axis=0)
which produces the result I want:
array([ 12., 41., 42., 38., 39.])
However, suppose I don't know ahead of time to which one of the axes that I don't want to apply the maximum? Suppose I started with n=4 dimensions, and wanted it to apply to all axes except the mth axis, which could be 0, 1, 2, or 3? Would have to actually use a conditional if-elif-else ?
Is there something that would work like a hypothetical exeptaxis=m?
The axis argument of nanmax can be a tuple of axes over which the maximum is computed. In your case, you want that tuple to contain all the axes except m. Here's one way you could do that:
In [62]: x
Out[62]:
array([[[[ 4., 3., nan, nan],
[ 0., 2., 2., nan],
[ 4., 5., nan, 3.],
[ 2., 0., 3., 1.]],
[[ 2., 0., 0., 1.],
[ nan, 3., 0., nan],
[ 0., 1., nan, 2.],
[ 5., 4., 0., 1.]],
[[ 4., 0., 2., 0.],
[ 4., 0., 4., 5.],
[ 3., 4., 1., 0.],
[ 5., 3., 4., 3.]]],
[[[ 2., nan, 6., 4.],
[ 3., 1., 2., nan],
[ 5., 4., 1., 0.],
[ 2., 6., 0., nan]],
[[ 4., 1., 4., 2.],
[ nan, 1., 5., 5.],
[ 2., 0., 1., 1.],
[ 6., 3., 6., 5.]],
[[ 1., 0., 0., 1.],
[ 1., nan, 2., nan],
[ 3., 4., 0., 5.],
[ 1., 6., 2., 3.]]]])
In [63]: m = 0
In [64]: np.nanmax(x, axis=tuple(i for i in range(x.ndim) if i != m))
Out[64]: array([ 5., 6.])
Related
This is the standard batch matrix multiplication:
import torch
a = torch.arange(12, dtype=torch.float).view(2,3,2)
b = torch.arange(12, dtype=torch.float).view(2,3,2) - 1
c = a.matmul(b.transpose(-1,-2))
a,b,c
>>
(tensor([[[ 0., 1.],
[ 2., 3.],
[ 4., 5.]],
[[ 6., 7.],
[ 8., 9.],
[10., 11.]]]),
tensor([[[-1., 0.],
[ 1., 2.],
[ 3., 4.]],
[[ 5., 6.],
[ 7., 8.],
[ 9., 10.]]]),
tensor([[[ 0., 2., 4.],
[ -2., 8., 18.],
[ -4., 14., 32.]],
[[ 72., 98., 124.],
[ 94., 128., 162.],
[116., 158., 200.]]]))
This is the one that I have:
e = a.view(6,2)
f = b.view(6,2)
g = e.matmul(f.transpose(-1,-2))
e,f,g
>>
(tensor([[ 0., 1.],
[ 2., 3.],
[ 4., 5.],
[ 6., 7.],
[ 8., 9.],
[10., 11.]]),
tensor([[-1., 0.],
[ 1., 2.],
[ 3., 4.],
[ 5., 6.],
[ 7., 8.],
[ 9., 10.]]),
tensor([[ 0., 2., 4., 6., 8., 10.],
[ -2., 8., 18., 28., 38., 48.],
[ -4., 14., 32., 50., 68., 86.],
[ -6., 20., 46., 72., 98., 124.],
[ -8., 26., 60., 94., 128., 162.],
[-10., 32., 74., 116., 158., 200.]]))
It's obvious that g covers c. I want to know if there is an efficient way to retrieve/slice c from g. Note that such retrieving/slicing method should generalize well to any shape of a and b.
Got it. We can just slice g with fancy indexing. We just extract the matrix multiplication result within the same batch:
g = g.view(2,3,2,3)
res = g[range(2),:,range(2),:]
res
I want to change an array such as
array([[ 7., 3., 14., 1., 9., 17.],
[ 7., 3., 14., 1., 9., 17.],
[ 7., 3., 14., 1., 9., 17.],
[ 7., 3., 14., 1., 9., 17.]])
To
array([[[7., 3.],
[7., 3.],
[7., 3.],
[7., 3.]],
[[14., 1.],
[14., 1.],
[14., 1.],
[14., 1.]],
[[9., 17.],
[9., 17.],
[9., 17.],
[9., 17.]]])
I thought I would manage with reshape, but none of the order statements work in this case. Currently, I am doing it this way
np.vstack([mat[:, i-2:i] for i in range(2, mat.shape[1]+1, 2)]).reshape(3,-1,2)
And I was wondering if there is a better way to do it
reshape and then swapaxes:
import numpy as np
a = np.array(
[[ 7., 3., 14., 1., 9., 17.],
[ 7., 3., 14., 1., 9., 17.],
[ 7., 3., 14., 1., 9., 17.],
[ 7., 3., 14., 1., 9., 17.]])
a.reshape((a.shape[0], -1, 2)).swapaxes(0, 1)
I have a numpy 2-d array which I divided in several numpy 2-d blocks. All blocks have the same shape. On these blocks I performed K-means segementation using the scikit-learn module. The edges of each block are overlapping (each block has one row/column overlap with the adjacent block). What I want is to give the overlapping segments in two adjacent blocks the same value. My current code can be downloaded here.
Image of the blocks and their position in the original image:
Blocks in python code
blockNW=np.array([[ 0., 0., 0., 0., 5.],
[ 0., 0., 4., 5., 5.],
[ 0., 4., 4., 5., 2.],
[ 0., 4., 5., 5., 2.],
[ 5., 5., 2., 2., 2.]])
blockNE=np.array([[ 1., 18., 18., 18., 6.],
[ 1., 18., 7., 6., 6.],
[ 3., 7., 7., 7., 6.],
[ 3., 3., 3., 7., 7.],
[ 3., 3., 7., 7., 7.]])
blockSW=np.array([[ 8., 8., 8., 10., 10.],
[ 8., 8., 9., 10., 10.],
[ 8., 8., 9., 9., 10.],
[ 8., 8., 8., 9., 10.],
[ 8., 8., 9., 9., 11.]])
blockSE=np.array([[ 12., 12., 12., 12., 12.],
[ 12., 12., 12., 12., 13.],
[ 12., 12., 12., 13., 13.],
[ 12., 12., 13., 13., 13.],
[ 12., 13., 13., 13., 13.]])
blocksStacked=np.array([blockNW,blockNE,blockSW,blockSE])
What I want is to connect the overlapping segments. For this I would like to use as few for-loops as possible, because they are slowing down the code. My current steps are:
import math
import numpy as np
from scipy import ndimage,stats
n_blocks,blocksize = np.shape(blocksStacked)[0],np.shape(blocksStacked)[1]
# shape of original image
out_shp = (8,8)
# horizontal and vertical blocks
horizontal_blocks=math.ceil(out_shp[1]/float(blocksize))
vertical_blocks=math.ceil(out_shp[0]/float(blocksize))
# numpy 2_d array in the shape of the image with an unique ID for each block
blockindex=np.arange(horizontal_blocks*vertical_blocks).reshape(-1,horizontal_blocks)
Block index
def find_neighbours(values,neighbourslist):
'''function to find the index of neighbouring blocks'''
mode=stats.mode(values)
if mode.count>1:
values=np.delete(values,np.where(values==mode[0]))
else:
values=np.delete(values,np.where(values==np.median(values)))
neighbourslist.append(values)
return 0
#Locate overlapping rows and columns per block
neighbourlist=[]
kernel=np.array([[0,1,0],[1,1,1],[0,1,0]],dtype='uint8')
_ =ndimage.generic_filter(blockindex, find_neighbours, footprint=kernel,extra_arguments=(neighbourlist,))
#output (block 0 has neighbours 1 and 2, etc.):
>>> neighbourlist
[array([ 1., 2.]), array([ 0., 3.]), array([ 0., 3.]), array([ 1., 2.])]
Now the next step could be is to loop through all blocks and neighbors and select the overlapping rows or columns (If possible I would also like to remove these loops).
# First I create masks to select overlapping rows or columns:
upmask=np.ones((blocksize,blocksize),dtype=bool)
upmask[1:,:]=0
downmask=np.ones((blocksize,blocksize),dtype=bool)
downmask[:-1,:]=0
rightmask=np.ones((blocksize,blocksize),dtype=bool)
rightmask[:,:-1]=0
leftmask=np.ones((blocksize,blocksize),dtype=bool)
leftmask[:,1:]=0
# Now loop through all blocks and neighbours and select the overlapping rows/columsn
for i in range(n_blocks):
n_neighbours = len(neighbourlist[i])
block=blocksStacked[i,:,:]
for j in range(n_neighbours):
neighborindex=neighbourlist[i][j]
block_neighbour=blocksStacked[neighborindex,:,:]
if i+1==neighborindex:
blockvals=block[rightmask]
neighbourvals=block_neighbour[leftmask]
elif i-1==neighborindex:
blockvals=block[leftmask]
neighbourvals=block_neighbour[rightmask]
elif i+horizontal_blocks==neighborindex:
blockvals=block[downmask]
neighbourvals=block_neighbour[upmask]
elif i-horizontal_blocks==neighborindex:
blockvals=block[upmask]
neighbourvals=block_neighbour[downmask]
In each loop I end up with two numpy 1d arrays representing the overlapping columns or rows. For the first loop I will end up with:
>>> blockvals
array([5., 5., 2., 2., 2.])
>>> neighbourvals
array([1., 1., 3., 3., 3.])
I want to relabel the values of the overlapping segments to the values of the segments in the block which is not a neighbour:
blockNW=np.array([[ 0., 0., 0., 0., 5.],
[ 0., 0., 4., 5., 5.],
[ 0., 4., 4., 5., 2.],
[ 0., 4., 5., 5., 2.],
[ 5., 5., 2., 2., 2.]])
blockNE=np.array([[ 5., 18., 18., 18., 6.],
[ 5., 18., 7., 6., 6.],
[ 2., 7., 7., 7., 6.],
[ 2., 2., 2., 7., 7.],
[ 2., 2., 7., 7., 7.]])
Any idea on how to detect and relabel these overlapping segments?
Also my code looks a bit too cumbersome, any ideas on how to improve my code?
A few remarks:
Some segments will not overlap for 100%, so it should be possible to set a threshold. For example is segments are overlapping for more than 70% they should be relabeled
The output shape of the function should be similar to the shape of the stacked blocks
The desired output will look like this:
EDIT
With for-loops the code to solve the question would look something like this:
from scipy.stats import itemfreq
# Locate and re-label overlapping segments
for k in range(len(np.unique(blockvals))):
#Iterate over each value in the overlapping row/column of the block
blockval=np.unique(blockvals)[k]
#count of blockval
block_val_count=len(blockvals[np.where(blockvals==blockval)])
#Select values in neighbour on the same location
overlap=neighbourvals[np.where(blockvals==blockval)]
overlapfreq=itemfreq(overlap)
#select neighboring value which overlaps the most
neighval_overlap_count= np.max(overlapfreq[:,1])
neighval=overlapfreq[np.where(overlapfreq[:,1]==neighval_overlap_count),0][0]
# count occurence of selected neighboring value
neigh_val_count=len(neighbourvals[np.where(neighbourvals==neighval)])
#If overlap is more than 70% relabel the neigboring value to the value in the block
thresh=0.7
if (neighval_overlap_count/float(neigh_val_count)>=thresh) and (neighval_overlap_count/float(block_val_count)>=thresh):
blocksStacked[neighborindex,:,:,][np.where(blocksStacked[neighborindex,:,:]==neighval)]=blockval
#output
>>> blocksStacked
array([[[ 0., 0., 0., 0., 5.],
[ 0., 0., 4., 5., 5.],
[ 0., 4., 4., 5., 2.],
[ 0., 4., 5., 5., 2.],
[ 5., 5., 2., 2., 2.]],
[[ 5., 18., 18., 18., 6.],
[ 5., 18., 7., 6., 6.],
[ 2., 7., 7., 7., 6.],
[ 2., 2., 2., 7., 7.],
[ 2., 2., 7., 7., 7.]],
[[ 8., 8., 8., 10., 10.],
[ 8., 8., 9., 10., 10.],
[ 8., 8., 9., 9., 10.],
[ 8., 8., 8., 9., 10.],
[ 8., 8., 9., 9., 11.]],
[[ 10., 10., 10., 10., 10.],
[ 10., 10., 10., 10., 13.],
[ 10., 10., 10., 13., 13.],
[ 10., 10., 13., 13., 13.],
[ 10., 13., 13., 13., 13.]]])
I have a five 100x100 arrays, A, and I want to multiply each matrix by a value from an array of length five, B. I wish to multiply the first matrix in A by the first value in B and the second matrix by the second value in B, etc. Am I able to do this?
Actually the answer has been provided by gboffi in his comment. Yet I want to elaborate that answer, giving a concrete example with code:
import numpy as np
#example data, all arrays of ones 100x100
A1 = A2 = A3 =A4 = A5 = np.ones((100, 100))
#example array containing the factor for each matrix
B = np.array([1, 2, 3, 4, 5])
#create an array containing all matrices
A = np.array([A1, A2, A3, A4, A5])
A*B[:,None,None]
The result then looks like this:
array([[[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
...,
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.]],
[[ 2., 2., 2., ..., 2., 2., 2.],
[ 2., 2., 2., ..., 2., 2., 2.],
[ 2., 2., 2., ..., 2., 2., 2.],
...,
[ 2., 2., 2., ..., 2., 2., 2.],
[ 2., 2., 2., ..., 2., 2., 2.],
[ 2., 2., 2., ..., 2., 2., 2.]],
[[ 3., 3., 3., ..., 3., 3., 3.],
[ 3., 3., 3., ..., 3., 3., 3.],
[ 3., 3., 3., ..., 3., 3., 3.],
...,
[ 3., 3., 3., ..., 3., 3., 3.],
[ 3., 3., 3., ..., 3., 3., 3.],
[ 3., 3., 3., ..., 3., 3., 3.]],
[[ 4., 4., 4., ..., 4., 4., 4.],
[ 4., 4., 4., ..., 4., 4., 4.],
[ 4., 4., 4., ..., 4., 4., 4.],
...,
[ 4., 4., 4., ..., 4., 4., 4.],
[ 4., 4., 4., ..., 4., 4., 4.],
[ 4., 4., 4., ..., 4., 4., 4.]],
[[ 5., 5., 5., ..., 5., 5., 5.],
[ 5., 5., 5., ..., 5., 5., 5.],
[ 5., 5., 5., ..., 5., 5., 5.],
...,
[ 5., 5., 5., ..., 5., 5., 5.],
[ 5., 5., 5., ..., 5., 5., 5.],
[ 5., 5., 5., ..., 5., 5., 5.]]])
I have an array:
a = array([
[ nan, 2., 3., 2., 5., 3.],
[ nan, 4., 3., 2., 5., 4.],
[ nan, 2., 1., 2., 3., 2.]
])
And I make a filled contour with:
plt.contourf(a)
So, I'll have it:
Nothing happens when I do plt.axis('tight'), but I want to hide boundary NaN values. How to do it easy?
You can set the min and max xlim using nanmin and nanmax:
import numpy as np
a = np.array([
[ np.nan, 2., 3., 2., 5., 3.],
[ np.nan, 4., 3., 2., 5., 4.],
[ np.nan, 2., 1., 2., 3., 2.]
])
import pylab as plt
xmax= np.nanmax(a)
xmin=np.nanmin(a)
plt.xlim(xmin,xmax)
plt.contourf(a)
plt.show()
If the array has the NaNs in a column like in your example, you can do the following way:
import matplotlib.pyplot as plt
a = array([
[ nan, 2., 3., 2., 5., 3.],
[ nan, 4., 3., 2., 5., 4.],
[ nan, 2., 1., 2., 3., 2.]
])
b = np.delete(a,0,1)
plt.contourf(b)
Well..
If I consider columns os NaNs in the begin and end, I tried that and it worked:
x = np.arange(0,a.shape[1])
plt.xlim([x[~np.isnan(a[0,:])][0],x[~np.isnan(a[0,:])][-1]])