So I started learning ML and I need to code in Python. I follow a tutorial on back propagation. So, in it, I am presented with the problem of turning an algorithm to support matrix multiplication so it would run faster. The following code is from a function that updates the biases and weights. I needed to change it so instead of running each input-output pair at a single time, I would run the entire inputs and outputs matrices. The mini-batch is a list with 10 elements. Each element in the list is two tuples, one tuple is the input, a 784x1 sized matrix. The other tuple is the output, a 10x1 sized matrix. I tried to group the inputs to a 784x10x1 array and then convert it to 784x10 array. I did it in two ways as shown in the following code:
# for clearance - batch[0] is the first tuple in the element in mini_batch, which is, as recalled, the input array.
# mini_batch[0][0] is the input array of the first element in mini_batch, which is a 784x1 array as I mentioned earlier
inputs3 = np.array([batch[0] for batch in mini_batch]).reshape(len(mini_batch[0][0]), len(mini_batch))
inputs2 = np.array([batch[0].ravel() for batch in mini_batch]).transpose()
both inputs3 and inputs2 are 784x10 arrays but for some reason, they are not equal. I don't understand why, so I would really appreciate if someone could explain to me why there's a difference.
>>> A = np.array([[1,2,3],[4,5,6]])
>>> A.reshape(3,2)
array([[1, 2],
[3, 4],
[5, 6]])
>>> A.transpose()
array([[1, 4],
[2, 5],
[3, 6]])
From this short example you can see that A.transpose() != A.reshape(3,2).
Imagine a blank matrix with dimensions 3x2. A.reshape(3,2) will read values form A(a 2x3 matrix) left to right starting from the top row and storing them in the blank matrix. Making these matrices have different values.
Related
Problem:
I have an array that represents products, lets say 3 for example
prod = [1,2,3]
then I have a correlation matrix for those products (just a number that represents something between two products, lets call c_ij for simplicity), in this case a 3x3 matrix
corr = [[c_11,c_12,c_13],
[c_21,c_22,c_23],
[c_31,c_32,c_33]]
The problem is that a need to shuffle the prod array, then I need to shuffle the corr matrix in a way that corr[i,j] still represent the correlation between prod[i] and prod[j]
My solution:
I know I can use a integer array as index to shuffle multiple array in the same way, like this:
order = [0,1,2]
new_order = np.random.permutation(order) # [2,0,1] for example
shuf_prod = prod[new_order]
Looking in the web I find that to make this work in a matrix I need to transform the order array in a matrix like
new_order = [2,0,1]
new_index = [ [[2,2,2],[0,0,0],[1,1,1]],
[[2,0,1],[2,0,1],[2,0,1]] ]
new_corr = corr[tuple(new_index)]
# this output what I want that is:
# [[c_33,c_31,c_32],
# [c_13,c_11,c_12],
# [c_23,c_21,c_22]]
Question:
The entire solution of shuffling look chunky and not efficient, this is a performance critical application so there is a faster way to do this? (I don't really care for simplicity of code, just performance)
If this is a good way of doing this, how I can create the new_index matrix from new_order array?
EDIT: Michael Szczesny solved the problem
new_corr = corr[new_order].T[new_order].T
you can use the indices directly as subscripts to the matrix as long as you provide the right shape for the second axis:
import numpy as np
mat = np.array([[3,4,5],
[4,8,9],
[5,9,7]])
order = np.array([2,0,1])
mat[order,order[:,None]]
array([[7, 5, 9],
[5, 3, 4],
[9, 4, 8]])
As I was going through pytorch documentation I came across a term layout = torch.strided in many of the functions. Can anyone help me in understanding where is it used and how. The description says it's the the desired layout of returned Tensor. What does layout mean and how many types of layout are there ?
torch.rand(*sizes, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False)
strides is number of steps (or jumps) that is needed to go from one element to next element, in a given dimension. In computer memory, the data is stored linearly in a contiguous block of memory. What we view is just a (re)presentation.
Let's take an example tensor for understanding this:
# a 2D tensor
In [62]: tensor = torch.arange(1, 16).reshape(3, 5)
In [63]: tensor
Out[63]:
tensor([[ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10],
[11, 12, 13, 14, 15]])
With this tensor in place, the strides are:
# get the strides
In [64]: tensor.stride()
Out[64]: (5, 1)
What this resultant tuple (5, 1) says is:
to traverse along the 0th dimension/axis (Y-axis), let's say we want to jump from 1 to 6, we should take 5 steps (or jumps)
to traverse along the 1st dimension/axis (X-axis), let's say we want to jump from 7 to 8, we should take 1 step (or jump)
The order (or index) of 5 & 1 in the tuple represents the dimension/axis. You can also pass the dimension, for which you want the stride, as an argument:
# get stride for axis 0
In [65]: tensor.stride(0)
Out[65]: 5
# get stride for axis 1
In [66]: tensor.stride(1)
Out[66]: 1
With that understanding, we might have to ask why is this extra parameter needed when we create the tensors? The answer to that is for efficiency reasons. (How can we store/read/access the elements in the (sparse) tensor most efficiently?).
With sparse tensors (a tensor where most of the elements are just zeroes), so we don't want to store these values. we only store the non-zero values and their indices. With a desired shape, the rest of the values can then be filled with zeroes, yielding the desired sparse tensor.
For further reading on this, the following articles might be of help:
numpy.ndarray.strides
torch.layout
torch.sparse
P.S: I guess there's a typo in the torch.layout documentation which says
Strides are a list of integers ...
The composite data type returned by tensor.stride() is a tuple, not a list.
For quick understanding, layout=torch.strided corresponds to dense tensors while layout=torch.sparse_coo corresponds to sparse tensors.
From another perspective, we can understand it together with torch.tensor.view.
A tensor can be viewed indicates it is contiguous. If we change the view of a tensor, the strides will change accordingly, but the data will keep the same. More specifically, view returns a new tensor with the same data but different shape, and strides is compatible with the view to indicate how to access the data in the memory.
For example
In [1]: import torch
In [2]: a = torch.arange(15)
In [3]: a.data_ptr()
Out[3]: 94270437164688
In [4]: a.stride()
Out[4]: (1,)
In [5]: a = a.view(3, 5)
In [6]: a.data_ptr() # share the same data pointer
Out[6]: 94270437164688
In [7]: a.stride() # the stride changes as the view changes
Out[7]: (5, 1)
In addition, the idea of torch.strided is basically the same as strides in numpy.
View this question for more detailed understanding.
How to understand numpy strides for layman?
As per the official pytorch documentation here,
A torch.layout is an object that represents the memory layout of a
torch.Tensor. Currently, we support torch.strided (dense Tensors) and
have experimental support for torch.sparse_coo (sparse COO Tensors).
torch.strided represents dense Tensors and is the memory layout that
is most commonly used. Each strided tensor has an associated
torch.Storage, which holds its data. These tensors provide
multi-dimensional, strided view of a storage. Strides are a list of
integers: the k-th stride represents the jump in the memory necessary
to go from one element to the next one in the k-th dimension of the
Tensor. This concept makes it possible to perform many tensor
operations efficiently.
Example:
>>> x = torch.Tensor([[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]])
>>> x.stride()
(5, 1)
>>> x.t().stride()
(1, 5)
layout means the way memory organize the element in that tensor,I think currently there are 2 types of layout to store tensor
one is torch.strided and another is torch.sparse_coo
strided means the element is arranged one by one in a very dense way, think about a strided troops, squares,so each soldier actually has neighbours.
while for sparse_coo I think should be deal with sparse matrix, the exact storage structure I am not sure but I guess it just stores non-zero elements' indices and values
It need to separate for those two types because for sparse matrix no need to arrange element one by one in a dense form, because it will take maybe one hundred steps for the non-zero element get to its next non-zero elements
I have a dataframe containing a column vector with around 500 000 rows of array vector. What I'm trying to do is unloading the content of this column into a 2 dimension array but I don't know the fastest way to do it.
This is the format of the array I'm trying to obtain ([1, 2], [3, 4] and [5, 6] are array contained in my dataframe):
array([[1, 2],
[3, 4],
[5, 6]])
I tried to_numpy, as_matrix, and .values but it gives me a 1D array which is not what I'm looking for:
array([array([1, 2]),
array([3, 4]),
array([5, 6])])
The only methods which gave me the result I want are np.asarray() and np.array() but they take too much time in my case.
What I want is the same array I obtain with using numpy array methods (vector1,2 and 8) but faster if possible because it takes too much time when we have lot of data.
Thank you for your help !
edit : Here is my function which does the following: it takes in parameter a dataframe which which contains two columns : id and vectors which is a serie of array objects.
id vectors
1 array([1,2,3], dtype='float32')
2 array([3,4,5], dtype='float32')
3 array([6,7,8], dtype='float32')
[11530 rows x 2 columns]
What i want to do with this function is unloading the content of column id in a list which is fast and easy and the content of column vectors into an array. So i want a 2 dimensional array of array vectors.
def filter_df(df, request):
start = time.time()
filtered_df = df
ids = filtered_df['id'].tolist()
filtered_df_vectors = filtered_df['vectors'].tolist()
vectors9 = np.array(filtered_df['vectors'].tolist())
vectors1 = np.asarray(filtered_df_vectors)
vectors2 = np.array([f for f in filtered_df_vectors],dtype=np.float32)
vectors3 = filtered_df['vectors'].as_matrix()
vectors4 = filtered_df['vectors'].to_numpy()
vectors5 = filtered_df['vectors'].values
vectors6 = filtered_df.iloc[:,-1].values
vectors8 = np.array(filtered_df['vectors'].values.tolist())
vectors9 = np.array(filtered_df['vectors'].tolist())
filter_duration= time.time()-start
logger.info(f"duration: {filter_duration}s")
return ids,vectors2,filter_duration
I can't copy paste the exact output it returns me for the resulted arrays because it will be unreadable for you so i will just show the two type of array i obtain with the multiple methods i tested.
For vectors 1, 2, 8 and 9 where i use numpy methods, i obtain this format which is the one i'm looking for but it takes two much time (around 0,7 second which is too slow for my case). I wont copy paste the exact array i obtain because it will be unreadable for you. Know just that [1,2,3] represent Here is what i obtain :
array([[1,2,3],
[4,5,6],
[7,8,9]], dtype=float32)
ndim : 2
dtype('float32')
shape : (11530, 300)
size : 3459000
For vectors 3, 4, 5 and 6 where i use no numpy methods like pandas to_numpy or as_matrix are fast (~0.05 sec) but returns me with the same entry an array of this form:
array([array([1,2,3], dtype=float32),
array([4,5,6], dtype=float32),
array([7,8,9], dtype=float32)], dtype=object)
ndim : 1
dtype('O')
shape : (11530,)
size : 11530
I don't understand why it doesn't give me the same array as numpy methods gives me.
I sometimes work with 1D arrays:
A = np.array([1, 2, 3, 4])
or 2D arrays (mono or stereo signals read with scipy.io.wavfile):
A = np.array([[1, 2], [3, 4], [5, 6], [7,8]])
Without having to distinguish these 2 cases with an if A.ndim == 2:..., is there a simple one-line solution to multiply this array A by an 1D array B = np.linspace(0., 1., 4)?
If A is 1D, then it's just A * B, and if A is 2D, what I mean is multiply each line of A by each element of B.
Note: this question arises naturally when working with both mono and stereo sounds read with scipy.io.wavfile.
Approach #1
We can use einsum to cover generic ndarrays -
np.einsum('i...,i->i...',A,B)
The trick that works here is the ellipsis that broadcasts for the trailing dimensions after the first axis as they are in A and keeps them in the output, while keeping the first axes for the two inputs aligned, which is the intended multiplication here. With A as 1D array, there's no broadcasting and that essentially reduces to : np.einsum('i,i->i',A,B) under the hoods.
Schematically put :
A : i x ....
B : i
out : i x ....
Hence, covers for A with any number of dimensions.
More info on the use of ellipsis from the docs :
To enable and control broadcasting, use an ellipsis. Default
NumPy-style broadcasting is done by adding an ellipsis to the left of
each term, like np.einsum('...ii->...i', a). To take the trace along
the first and last axes, you can do np.einsum('i...i', a), or to do a
matrix-matrix product with the left-most indices instead of rightmost,
you can do np.einsum('ij...,jk...->ik...', a, b).
Approach #2
Using the fact that we are trying to align the first axis of A with the only axis of 1D array B, we can simply transpose A, multiply with B and finally transpose back -
(A.T*B).T
I am trying to solve a binary classification problem with the sequential model from Keras
and have to meet a given Balanced Error Rate (BER)
So I thought it would be a good idea to use the BER instead of accuracy as a metric.
My custom metric implementation for BER looks like this:
def balanced_error_rate(y_true, y_pred):
labels = theano.shared(np.asmatrix([[0, 1]], dtype='int8'))
label_matrix = K.repeat_elements(labels, K.shape(y_true)[0], axis=1)
true_matrix = K.repeat_elements(y_true, K.shape(labels)[0], axis=1)
pred_matrix = K.repeat_elements(K.round(y_pred), K.shape(labels)[0], axis=1)
class_lens = K.sum(K.equal(label_matrix, true_matrix), axis=1)
return K.sum(K.sum(class_lens - K.sum(K.equal(label_matrix, K.not_equal(true_matrix,pred_matrix)), axis=1), axis=0)/class_lens, axis=0)/2
The idea is to create a matrix from the available labels and compare it to the input data (then sum the ones) to get the number of elements of this label....
My problem is that:
> K.shape(y_true)
Shape.0
> Typeinfo:
> type(y_true)
<class 'theano.tensor.var.TensorVariable'>
> type(K.shape(y_true))
<class 'theano.tensor.var.TensorVariable'>
...and I can't find out why.
I am now looking for:
A way to get the array dimensions / an explanation why shape acts like it does / the reason why y_true seems to have 0 dimensions
or
A method to create a tensor matrix with a given with/height by repeating a given row/column vector.
or
A smarter solution to calculate the BER using tensor functions.
A way to get the array dimensions / an explanation why shape acts like it does / the reason why y_true seems to have 0 dimensions
The deal with print and abstraction libraries like Theano is that you usually do not get the values but a represenation of the value. So if you do
print(foo.shape)
You won't get the actual shape but a representation of the operation that is done at runtime. Since this is all computed on an external device the computation is not run immediately but only after creating a function with appropriate inputs (or calling foo.shape.eval()).
Another way to print the value is to use theano.printing.Print when using the value, e.g.:
shape = theano.printing.Print('shape of foo')(foo.shape)
# use shape (not foo.shape!)
A method to create a tensor matrix with a given with/height by repeating a given row/column vector.
See theano.tensor.repeat for that. Example in numpy (usage is quite similar):
>>> x
array([[1, 2, 3]])
>>> x.repeat(3, axis=0)
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])