I have this huge tensor from which I just want to keep selected tensors.
Background - first contains coordinates of quadrilaterals being predicted.
np.shape(coords_detached) = (15969, 8)
coords.shape() = torch.Size([15969, 8])
The second, contains same coordinates but filtered after selection using NMS, for this discussion just say I select 9 rows from above tensor. 8 coordinates + 1 confidence score
But NMS is being done in numpy so I detach the tensors.
coords_nms = torch.tensor(nms_coords_, dtype=torch.float32)
coords_nms.shape() = torch.Size([9, 9])
So now I want to select just these 9 rows from the original tensor, coz it had the gradient information that gets lost during detach() and numpy nms.
I tried this :
s = torch.ones_like(nms_coords_)
s *=-1
nms_coords = torch.where(coords == coords_nms[:,:-1], coords, s)
nms_coords = nms_coords[nms_coords>=0]
nms_coords.reshape(-1, 8)
to iterate through coords and match value coords_nms and just store those. but it needs same dimension at axis=0
The iterative loop would be the following but how to do it using tensor notation :
poo = []
for x in coords:
for z in nms_coords_:
if sum(x[:] == z[:-1]) == 8 :
poo.append(z[:-1])
Related
I'm trying to enumerate through a 2D numpy array of shape (512, 512), which holds the pixel values of an image. So basically it's an array representing width and height in pixel values for the image. I'm trying to enumerate through each element to output: [y_index, x_index, pixel_value]. I need these 3 output values to be stored in an array (either the existing one or a new one, whichever is more efficient to execute).
So the input array would have a shape of (512, 512), and if I'm not mistaken, the output array would have an array shape (262144, 3). 262144 is the total number of pixels in a 512x512 matrix. And 3 because there are 3 columns, for 3 pieces of data that I need to extract: pixel value, y coordinate, x coordinate. So basically I want to have an array of pixel values and their y, x coordinates.
In the code below, I used ndenumerate to enumerate through the numpy array (img) of pixel values (512, 512). But I'm struggling on how to store the output values in an array. I created coordinates array to store the output values, but my attempt at it with the last line is clearly incorrect to achieve the desired effect. So how to solve this?
img = as_np_array[:, :, 0]
img = img.reshape(512, 512)
coordinates = np.empty([262,144, 3])
for index, x in np.ndenumerate(img):
coordinates = np.append(coordinates, [[x], index[0], index[1]])
Also the other challenge I'm facing is, to execute this code, my Intel Core i7 2.7GHz (4 Cores) processor takes about 7-10 minutes (possibly more at times) to execute. Is there a more efficient code that can execute faster?
Any help would be greatly appreciated.
You could use numpy.indices to do this. What you want ultimately is image_data with y, x indices and the corresponding pixels (px). There are three columns in image_data.
row, col = np.indices(img.shape)
y, x, px = row.flatten(), col.flatten(), img.flatten()
image_data = np.array([y, x, px]).T
Detailed Example:
img = np.arange(20).reshape(5, 4)
def process_image_data(img):
row, col = np.indices(img.shape)
return (row.flatten(), col.flatten(), img.flatten())
y, x, px = process_image_data(img)
The solution that worked for me is this:
with open('img_pixel_coor1.csv', 'w', newline='') as f:
headernames = ['y_coord', 'x_coord', 'pixel_value']
thewriter = csv.DictWriter(f, fieldnames=headernames)
thewriter.writeheader()
for index, pix in np.ndenumerate(img):
thewriter.writerow({'y_coord' : index[0], 'x_coord' : index[1], 'pixel_value' : pix})
I am trying to run a PCA analysis over an dataset representing the 3 bands of an image. The dataset is of size (300000,3) being pixels and 3bands.I find the Eigen values and pairs which are then put into a tuple called eig_pairs. I then calculate the variance to determine how many bands to use for PCA.
I determine that I wish to use 2 bands.
My eig_pairs shape is a list of tuples of size 3.
Following this tutorial I says I need to reshape everything by reducing from original dimension space (3) to how every many a dimension equal to the number of dimensions I wish to use (2). Their example goes for 7 to 4 as shown here:
matrix_w = np.hstack((eig_pairs[0][1].reshape(7,1),
eig_pairs[1][1].reshape(7,1),
eig_pairs[2][1].reshape(7,1),
eig_pairs[3][1].reshape(7,1)))
Following this logic I changed my own to:
matrix_w = np.hstack((eig_pairs0.reshape(3,1),
eig_pairs1.reshape(3,1)))
However I get the error ValueError: shapes (3131892,3) and (2,3) not aligned: 3 (dim 1) != 2 (dim 0)
#read in image
img = cv2.imread('/Volumes/EXTERNAL/Stitched-Photos-for-Chris/p7_0015_20161005-949am-75m-pass-1.jpg.png',1)
row,col = img.shape[:2]
b,g,r = cv2.split(img)
# Pandas dataset
# samples = 3000000, featuress = 3
dataSet = pd.DataFrame({'bBnad':b.flat[:],'gBnad':g.flat[:],'rBnad':r.flat[:]})
print(dataSet.head())
# Standardize the data
X = dataSet.values
X_std = StandardScaler().fit_transform(X) #converts data from unit8 to float64
#Calculating Eigenvectors and eigenvalues of Covariance matrix
meanVec = np.mean(X_std, axis=0)
covarianceMatx = np.cov(X_std.T)
eigVals, eigVecs = np.linalg.eig(covarianceMatx)
# Create a list of (eigenvalue, eigenvector) tuples
eig_pairs = [ (np.abs(eigVals[i]),eigVecs[:,i]) for i in range(len(eigVals))]
# Sort from high to low
eig_pairs.sort(key = lambda x: x[0], reverse= True)
# Determine how many PC going to choose for new feature subspace via
# the explained variance measure which is calculated from eigen vals
# The explained variance tells us how much information (variance) can
# be attributed to each of the principal components
tot = sum(eigVals)
var_exp = [(i / tot)*100 for i in sorted(eigVals, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
#convert 3 dimension space to 2 dimensional space therefore getting a 2x3 matrix W
matrix_w = np.hstack((eig_pairs[0][1].reshape(3,1),
eig_pairs[1][1].reshape(3,1)))
Appreciate any help.
I'm trying to create a method for evaluating co-ordinates for a project that's due in about a week.
Assuming that I'm working in a 3D cartesian co-ordinate system - whose values are stored as rows in a numpy array. I am trying to read if 'z' (n[i, 2]) values exist given the corresponding, predetermined 'x' (n[i,0]) and 'y' (n[i,1]) values.
In the case where the values that are assigned are scalars, I am content to think that:
# Given that n is some numpy array
x, y = 2,3
out = []
for i in range(0,n.shape[0]):
if n[i, 0] == x and n[i,1] == y:
out.append(n[i,2])
However, where the sorrow comes in is having to check if the values in another numpy array are in the original numpy array 'n'.
# Given that n is the numpy array that is to be searched
# Given that x contains the 'search elements'
out = []
for i in range(0,n.shape[0]):
for j in range(0, x.shape[0]):
if n[i, 0] == x[j,0] and n[i,1] == x[j,1]:
out.append(n[i,2])
The issue with doing it this way is that the 'n' matrix in my application may well be in excess of 100 000 lines long.
Is there a more efficient way of performing this function?
This might be more efficient than nested loops:
out = []
for row in x:
idx = np.equal(n[:,:2], row).all(1)
out.extend(n[idx,2].tolist())
Note this assumes that x is of shape (?, 2). Otherwise, if it has more than two columns, just change row to row[:2] in the loop body.
Numpythonic solution without loops.
This solution works in case the x and y coordinates are non-negative.
import numpy as np
# Using a for x and b for n, to avoid confusion with x,y coordinates and array names
a = np.array([[1,2],[3,4]])
b = np.array([[1,2,10],[1,2,11],[3,4,12],[5,6,13],[3,4,14]])
# Adjust the shapes by taking the z coordinate as 0 in a and take the dot product with b transposed
a = np.insert(a,2,0,axis=1)
dot_product = np.dot(a,b.T)
# Reshape a**2 to check the dot product values corresponding to exact values in the x, y coordinates
sum_reshaped = np.sum(a**2,axis=1).reshape(a.shape[0],1)
# Match for values for indivisual elements in a. Can be used if you want z coordinates corresponding to some x, y separately
indivisual_indices = ( dot_product == np.tile(sum_reshaped,b.shape[0]) )
# Take OR of column values and take z if atleast one x,y present
indices = np.any(indivisual_indices, axis=0)
print(b[:,2][indices]) # prints [10 11 12 14]
I'm quite new to programming in general, but I could not figure this problem out until now.
I've got a two-dimensional numpy array mask, lets say mask.shape is (3800,3500)which is filled with 0s and 1s representing a spatial resolution of a 2D image, where a 1 represents a visible pixel and 0 represents background.
I've got a second two-dimensional array data of data.shape is (909,x) where x is exactly the amount of 1s in the first array. I now want to replace each 1 in the first array with a vector of length 909 from the second array. Resulting in a final 3D array of shape(3800,3500,909) which is basically a 2D x by y image where select pixels have a spectrum of 909 values in z direction.
I tried
mask_vector = mask.flatten
ones = np.ones((909,1))
mask_909 = mask_vector.dot(ones) #results in a 13300000 by 909 2d array
count = 0
for i in mask_vector:
if i == 1:
mask_909[i,:] = data[:,count]
count += 1
result = mask_909.reshape((3800,3500,909))
This results in a viable 3D array giving a 2D picture when doing plt.imshow(result.mean(axis=2))
But the values are still only 1s and 0s not the wanted spectral data in z direction.
I also tried using np.where but broadcasting fails as the two 2D arrays have clearly different shapes.
Has anybody got a solution? I am sure that there must be an easy way...
Basically, you simply need to use np.where to locate the 1s in your mask array. Then initialize your result array to zero and replace the third dimension with your data using the outputs of np.where:
import numpy as np
m, n, k = 380, 350, 91
mask = np.round(np.random.rand(m, n))
x = np.sum(mask == 1)
data = np.random.rand(k, x)
result = np.zeros((m, n, k))
row, col = np.where(mask == 1)
result[row,col] = data.transpose()
I would like to use a generic filter to calculate the mean of values within a given window (or kernel), for values that fulfill a couple of conditions. I expected the following code to produce a mean filter of the first array in a 3-layer window, using the other two arrays to mask values from the mean calculation.
from scipy import ndimage
import numpy as np
#some test data
tstArr = np.random.rand(3,7,7)
tstArr = tstArr*10
tstArr = np.int_(tstArr)
tstArr[1] = tstArr[1]*100
tstArr[2] = tstArr[2] *1000
#mean function
def testFun(tstData,processLayer,nLayers,kernelSize):
funData= tstData.reshape((nLayers,kernelSize,kernelSize))
meanLayer = funData[processLayer]
maskedData = meanLayer[(funData[1]>1)&(funData[2]<9000)]
returnMean = np.mean(maskedData)
return returnMean
#number of layers in the array
nLayers = np.shape(tstArr)[0]
#window size
kernelSize = 5
#create a sampling window of 5x5 elements from each array
footprnt = np.ones((nLayers,kernelSize,kernelSize),dtype = np.int)
# calculate the mean of the first layer in the array (other two are for masking)
processLayer = 0
tstOut = ndimage.generic_filter(tstArr, testFun, footprint=footprnt, extra_arguments = (processLayer,nLayers,kernelSize))
I thought this would yield a 7x7 array of masked mean values from the first layer in the input array. The output is a 3x7x7 array, and I don't understand what the values represent. I'm not sure how to produce the "masked" mean-filtered array, or how to interpret the output as given.
Your code produce a mean filter of the first array in a 3-layer window, using the over two arrays to mask values from the mean calculation. You will find the result in tstOut[1].
What is going on ? When you call ndimage.generic_filter with tstArr of shape (3, 7, 7) and footprint=np.ones((3, 5, 5)) then for all i from 0 to 2, for all j from 0 to 6 and for all k from 0 to 6, testFun is called with the subarray of tstArr centered in (i, j, k) and of shape (3, 5, 5) (the array is reflected at the boundary to supply missing values).
In the end:
tstOut[0] is the mean filter of tstArr[0] with tstArr[0] and tstArr[1] as masks
tstOut[1] is the mean filter of tstArr[0] with tstArr[1] and tstArr[2] as masks
tstOut[2] is the mean filter of tstArr[1] with tstArr[2] and tstArr[2] as masks
Again, the wanted result is in tstOut[1].
I hope this will help you.