I have Pytorch 2d tensor with normal distribution.
Is there a fast way to nullify top 10% max values of this tensor using Python?
I see two possible ways here:
Flatten tensor to 1d and just sort it
Non-vectorized way using some native Python operators (for-if)
But neither of these looks fast enough.
So, what is the fastest way to set X max values of a tensor to zero?
Well, it seems that Pytorch has a useful operator torch.quantile() that helps here a lot.
The solution (for 1d tensor):
import torch
x = torch.randn(100)
y = torch.tensor(0.) #new value to assign
split_val = torch.quantile(x, 0.9)
x = torch.where(x < split_val, x, y)
Related
I would like to form a loss function in Tensorflow that relies on a matrix containing all combinations of (squared) Euclidean distances for a set of embeddings. In numpy, like this:
# E is (batch_size,N,32)
N=100
D = np.zeros((batch_size,N,N))
for x in range(N):
for y in range(N):
D[:,x,y] = np.sum(np.square(E[:,x,:]-E[:,y,:]),axis=1)
How can I code this in Tensorflow/Keras without using the nested for loop, or no for loops at all?
This should do:
D = tf.reduce_sum((E[:, None, :] - E[:, :, None])**2, axis=-1)
D will be (batch_size, N, N). This also works in numpy (obviously use np.sum), so you could use that to check equivalence to the loop version just to be sure.
This solution works via broacasting: None is used to insert axes such that a size-N axis is matched against a size-1 axis, and the latter is broadcast (repeated) to match the former. This results in all elements being compared to all others (per batch element). It's a little hard to describe in text and also difficult to visualize since we are dealing with four-dimensional tensors here...
I am generating a series of Gaussian arrays given a x vector of length (1400), and arrays for the sigma, center, amplitude (amp), all with length (100). I thought the best way to speed this up would be to use numpy and list comprehension:
g = np.sum([(amp[i]*np.exp(-0.5*(x - (center[i]))**2/(sigma[i])**2)) for i in range(len(center))],axis=0)
Each row is a gaussian along a vector x, and then I sum the columns into a single array of length x.
But this doesn't seem to speed things up at all. I think there is a faster way to do this while avoiding the for loop but I can't quite figure out how.
You should use vectorized computation instead of comprehension so the loops are all performed at c speed.
In order to do so you have to reshape x to be a column vector. For example you could do x = x.reshape((1400,1)).
Then you can operate directly on the arrays, like this:
v=(amp*np.exp(-0.5*(x - (center))**2/(sigma)**2
Then you obtain an array of shape (1400,100) which you can sum up to a vector by np.sum(v, axe=1)
You should try to vectorize all the operations. IMHO the most efficient to first converts your input data to numpy arrays (if they were plain Python lists) and then let numpy process the computations:
np_amp = np.array(amp)
np_center = np.array(center)
np_sigma = np.array(sigma)
g = np.sum((np_amp*np.exp(-0.5*(x - (np_center))**2/(np_sigma)**2)),axis=0)
I want to find the covariance of a 10304*280 matrix (i.e 280 variable and each have 10304 subjects) and I am using the following numpy function to find this.
cov = numpy.cov(matrix)
I am expected 208*280 matrix as a result but it returned 10304*10304 matrix.
As suggested in the previous answer, you can change your memory layout.
An easy way to do this in 2d is simply transposing the matrix:
import numpy as np
r = np.random.rand(100, 10)
np.cov(r).shape # is (100,100)
np.cov(r.T).shape # is (10,10)
But you can also specify a rowvar flag. Read about it here:
import numpy as np
r = np.random.rand(100, 10)
np.cov(r).shape # is (100,100)
np.cov(r, rowvar=False).shape # is (10,10)
I think especially for large matrices this might be more performant, since you avoid the swapping/transposing of axes.
UPDATE:
I thought about this and wondered if the algorithm is actually different depending on rowvar == True or rowvar == False. Well, as it turns out, if you change the rowvar flag, numpy simply transposes the array itself :P.
Look here.
So, in terms of performance, nothing will change between the two versions.
here is what numpy.cov(m, y=None..) document says
m : array_like A 1-D or 2-D array containing multiple variables and
observations. Each row of m represents a variable, and each column a
single observation of all those variables...
So if your matrix contains 280 variable with 10304 samples for each, it suppose to be 280*10304 matrix instead of 10304*280 one. The simple solution would be same as others suggesting.
swap_matrix = numpy.swapaxis(matrix)
cov = numpy.cov(swap_matrix)
I have a matrix thing that looks like this:
thing.shape
(8070829, 2)
and I want to scale all elements by some scalingfactor = np.iinfo(np.int16).max/thing.max() to normalize the values. Right now I am iterating over all elements which works, but is really slow:
for j, sample in enumerate(thing):
thing[j] = [int(sample[0] * scalingfactor), int(sample[1] * scalingfactor)]
I thought I could do the following, but the results are not the same:
np.multiply(thing, scalingfactor)
Is there are more efficient way to normalize a matrix?
Use vectorized elementwise multiplication and then change dtype (that does the floor-ing) -
(thing*scalingfactor).astype(int) # for thing as array type
Or use np.floor on the scaled version -
np.floor(thing*scalingfactor)
Using the posted code from the question : np.multiply(thing, scalingfactor) would work too, just needs the additional floor-ing step, as suggested earlier.
I have a batch of N sequences of integers of length L which is embedded into a N*L*d tensor. This sequence is auto-encoded by my network architecture. So, I have:
from theano import tensor as T
X = T.imatrix('X') # N*L elements in [0,C]
EMB = T.tensor('Embedding') # N*L*d
... # some code goes here :-)
PY = T.tensor('PY') # N*L*C probability of the predicted class in [0,C]
cost = -T.log(PY[X])
as far as I could get, the indexing is in the first dimension of the tensor, so I had to use a theano.scan. Is there a way to index the tensor directly?
Sounds like you want a 3 dimensional version of theano.tensor.nnet.categorical_crossentropy?
If so, then I think you could simply flatten the matrix of true class label indexes into a vector and the 3D tensor of predicted class probabilities into a matrix and then use the built in function.
cost = T.nnet.categorical_crossentropy(
Y.reshape((Y.shape[0] * Y.shape[1], X.shape[2])),
X.flatten())
The order of entries in Y may need to be adjusted first (e.g. via a dimshuffle) to make sure the entries in the matrix and vector being compared correspond to each other.
Here we assume, as the question suggests, that the sequences are not padded -- they are all exactly L elements in length. If the sequences are actually padded then you may need to do something much more complicated to avoid computing cost elements inside the padding regions.