I would like to multyply the following matrices (using numpy) in the most efficient way.
This is the code for the matrixes:
a = np.array([[1, 5], [2, 6], [3, 7], [4, 8]])
m = np.array([[1, 0, 0, 1], [1, 0, 1, 0], [0, 1, 0, 1], [0, 1, 1, 1]])
These are the matrixes visualized better:
a:
[[1 5]
[2 6]
[3 7]
[4 8]]
m:
[[1 0 0 1]
[1 0 1 0]
[0 1 0 1]
[0 1 1 1]]
I want to multiply a by (the first column of matrix m), like this
a m[:,0] x0
[[1 5] [[1] [[1 5]
[2 6] * [1] = [2 6]
[3 7] [0] [0 0]
[4 8]] [0]] [0 0]
And then I want to multiply a by (the second column of matrix m), like this
a * m[:,1] = x1
And then 3rd and 4th column
a * m[:,2] = x2
a * m[:,3] = x3
And finally, I want to put the resulting matrices x0,x1,x2,x3 all in one matrix.
X = [x0, x1, x2, x3]
The size X in this example is 4 x 8.
The final result in this example is:
X =
[[[1 5 0 0 0 0 1 5]
[2 6 0 0 2 6 0 0]
[0 0 3 7 0 0 3 7]
[0 0 4 8 4 8 4 8]]
I would like to know how to do this with build-in functions of numpy, and using generators, instead of using 2 for loops, if it is possible.
This is just an example. In reality the matrixes have large dimensions and it is important that the multiplications are done as fast as possible.
Thank you
You may achieve it with broadcast and reshape
arr_out = (a[:,None] * m[...,None]).reshape(4,8)
Out[176]:
array([[1, 5, 0, 0, 0, 0, 1, 5],
[2, 6, 0, 0, 2, 6, 0, 0],
[0, 0, 3, 7, 0, 0, 3, 7],
[0, 0, 4, 8, 4, 8, 4, 8]])
You could transpose and expand the dimensions of m, to get the wanted result:
m.T[...,None] * a
array([[[1, 5],
[2, 6],
[0, 0],
[0, 0]],
[[0, 0],
[0, 0],
[3, 7],
[4, 8]],
...
If you want to stack the arrays horizontally to end up with a 2D array, use np.hstack:
np.hstack(m.T[...,None] * a)
array([[1, 5, 0, 0, 0, 0, 1, 5],
[2, 6, 0, 0, 2, 6, 0, 0],
[0, 0, 3, 7, 0, 0, 3, 7],
[0, 0, 4, 8, 4, 8, 4, 8]])
Or reshaping as:
(a[:,None] * m[...,None]).reshape(m.shape[0], -1)
This is the answer that I was looking for. Thank you Yatu and hpaulj.
X = m.T[...,None] * a
for i in range(4):
reshaped = np.hstack(X[i,:,:])
reshaped_simpler = np.hstack(X)
print (reshaped_simpler)
I got the rest of the answer from the following link:
reshape numpy 3D array to 2D
I rearranged the for loop because I got a Warning regarding the generators going to be depricated in future versions of Numpy.
Related
I have the following diagonal matrix
a = array([[1, 0, 0, 0],
[0, 2, 0, 0],
[0, 0, 3, 0],
[0, 0, 0, 4]])
And the desired out come is the following
array([[1, 3, 4, 5],
[3, 2, 5, 6],
[4, 5, 3, 7],
[5, 6, 7, 4]])
Each element is the sum of the corresponding diagonals.
Thanks a lot
Try:
>>> np.diag(a) + np.diag(a)[:, None] - a
array([[1, 3, 4, 5],
[3, 2, 5, 6],
[4, 5, 3, 7],
[5, 6, 7, 4]])
Addendum
What if a is a DataFrame?
Then: np.diag(a) + np.diag(a)[:, None] - a is also a DataFrame (with same index and columns as a).
What if a is a numpy array, but I want a DataFrame result?
Then use: pd.DataFrame(...) instead.
You can use:
# get diagonal
diag = np.diag(a)
# outer sum
out = diag+diag[:,None]
# or
# out = np.outer(diag, diag)
# reset diagonal
np.fill_diagonal(out, diag)
print(out)
output:
[[1 3 4 5]
[3 2 5 6]
[4 5 3 7]
[5 6 7 4]]
I'm new to the tensorflow world. With tensorflow.reverse_sequence we need to pass sequential_length but what are its exact requirements?
I played with the values provided in the docs. I cant grasp the concept. I'm curious about its properties and exact usage.
This is my code:
import tensorflow as tf
from tensorflow import keras
#seq_lengths = [7, 2, 3, 5]
seq_lengths = [0, 0, 0, 0]
input = [[1, 2, 3, 4, 5, 0, 0, 0], [1, 2, 0, 0, 0, 0, 0, 0],
[1, 2, 3, 4, 0, 0, 0, 0], [1, 2, 3, 4, 5, 6, 7, 8]]
output = tf.reverse_sequence(input, seq_lengths, seq_axis=1, batch_axis=0)
print(output)
Even if the answer is already present in the comment, i'm answering here for the community.
seq_lengths is a 1-D with values in int32, int64, value should not exceed the length of the input array.
To explain to you how it works, here is the example.
seq_lengths = [2,2,2,2]
input = [[1, 2, 3, 4, 5, 0, 0, 0], [1, 2, 0, 0, 0, 0, 0, 0],
[1, 2, 3, 4, 0, 0, 0, 0], [1, 2, 3, 4, 5, 6, 7, 8]]
output = tf.reverse_sequence(input, seq_lengths, seq_axis=1, batch_axis=0)
Output:
tf.Tensor(
[[2 1 3 4 5 0 0 0]
[2 1 0 0 0 0 0 0]
[2 1 3 4 0 0 0 0]
[2 1 3 4 5 6 7 8]], shape=(4, 8), dtype=int32)
As you can see, I have given seq_lengths of 2 for all the input array, which means only first 2 elements of each array will be reversed, rest will remain in the original order of sequence.
You can try giving any number in seq_lengths (max 8 in the above case) and try yourself.
After some vectorized calculations, I get a sparse block matrix with all my results stacked in blocks of same size.
>>> A = [[1, 1],
... [1, 1]]
>>> B = [[2, 2],
... [2, 2]]
>>> C = [[3, 3],
... [3, 3]]
>>> results = scipy.sparse.block_diag(A, B, C)
>>> print(results.toarray())
[[1 1 0 0 0 0]
[1 1 0 0 0 0]
[0 0 2 2 0 0]
[0 0 2 2 0 0]
[0 0 0 0 3 3]
[0 0 0 0 3 3]]
How can I get back these arrays A,B,C in an efficient way, if necessery by providing their shape (2,2)?
In [177]: >>> A = [[1, 1],
...: ... [1, 1]]
...: >>> B = [[2, 2],
...: ... [2, 2]]
...: >>> C = [[3, 3],
...: ... [3, 3]]
...: >>> results = sparse.block_diag([A, B, C])
...:
In [178]: results
Out[178]:
<6x6 sparse matrix of type '<class 'numpy.int64'>'
with 12 stored elements in COOrdinate format>
block_diag does not preserve the inputs; rather it creates coo format matrix, representing the whole matrix, not the pieces.
In [194]: results.data
Out[194]: array([1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3], dtype=int64)
In [195]: results.row
Out[195]: array([0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5], dtype=int32)
In [196]: results.col
Out[196]: array([0, 1, 0, 1, 2, 3, 2, 3, 4, 5, 4, 5], dtype=int32)
In [179]: results.A
Out[179]:
array([[1, 1, 0, 0, 0, 0],
[1, 1, 0, 0, 0, 0],
[0, 0, 2, 2, 0, 0],
[0, 0, 2, 2, 0, 0],
[0, 0, 0, 0, 3, 3],
[0, 0, 0, 0, 3, 3]], dtype=int64)
block_diag pass the arrays to sparse.bmat. That in turn makes a coo matrix from each, and then merges the coo attributes into 3 arrays, which are inputs to the global sparse matrix.
There is another sparse format bsr that may preserve the blocks (until conversion to csr for calculation), but I'll have to experiment to see that's the case.
Let's make a bsr from that results coo:
In [186]: bresults = sparse.bsr_matrix(results)
In [187]: bresults
Out[187]:
<6x6 sparse matrix of type '<class 'numpy.int64'>'
with 12 stored elements (blocksize = 2x2) in Block Sparse Row format>
In [188]: bresults.blocksize
Out[188]: (2, 2)
In [189]: bresults.data
Out[189]:
array([[[1, 1],
[1, 1]],
[[2, 2],
[2, 2]],
[[3, 3],
[3, 3]]], dtype=int64)
So it deduces that there are blocks, just as you desired.
In [191]: bresults.indices
Out[191]: array([0, 1, 2], dtype=int32)
In [192]: bresults.indptr
Out[192]: array([0, 1, 2, 3], dtype=int32)
So it's a csr like storage, but with the data arranged in blocks.
It may be possible to construct this from your A,B,C without the block_diag intermediary, but I'd have to look at the docs more.
that's a funny little problem.
I don't think there is a function that solves this in one line, but there's a way to do it programmatically.
Check out what res.data prints out, I use it here.
This works when shapes are all the same.
from scipy.sparse import block_diag
a = [[1, 2, 4],
[3, 4, 4]]
b = [[2, 2, 1],
[2, 2, 1]]
c = [[3, 3, 6],
[3, 3, 6]]
res = block_diag((a, b, c))
def goBack(res, shape):
s = shape[0]*shape[1]
num = int(len(res.data)/s)
for i in range (num):
mat = res.data[i*s:(i+1)*s].reshape(shape)
print(mat)
goBack(res, [2,3])
Output:
[[1 2 4]
[3 4 4]]
[[2 2 1]
[2 2 1]]
[[3 3 6]
[3 3 6]]
Edit:
Okay, this does not work when any of the elements of the provided matrices is zero, as then it would not be counted in res.data.
Also, forget it, the link provided by cleb should help you.
Say that I have a matrix:
X = [
[0, 0, 0, 0, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0],
[2, 0, 0, 3, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 4, 0, 0, 0]
]
but it is stored in a sparse format:
# [row, column, value]
X_sparse = [
[1, 2, 1],
[3, 0, 2],
[3, 3, 3]
[6, 1, 4],
]
I also have a vector
b = [1,2,3,4,5]
I would like to pass X_sparse to TensorFlow as data and multiply it with b (i.e. np.dot(X, b)). What is the correct way to do this in TensorFlow? Additionally, this is a toy example, while in real-life X has many rows and columns, so I'd rather not want to work with it converted to non-sparse format.
I don't see an easy way to do so without using a dense tensor representation (tf.sparse_tensor_to_dense()). Tensorflow is however built for operations on huge matrices, so in many cases it shouldn't be a problem...
A quick example:
import tensorflow as tf
import numpy as np
X_sparse_np = np.array([
[1, 2, 1],
[3, 0, 2],
[3, 3, 3],
[6, 1, 4],
])
b_np = np.array([1,2,3,4,5])
b = tf.constant(b_np)
# Converting your input data into a SparseTensor:
indices = X_sparse_np[:, :2].reshape((-1, 2))
values = X_sparse_np[:, 2]
dense_shape = [7, 5]
X_sparse = tf.SparseTensorValue(indices, values, dense_shape)
# Performing your operation on its dense representation:
res = tf.sparse_tensor_to_dense(X_sparse) * b
# opt. Re-obtaining the sparse representation of your result:
sparse_res = tf.SparseTensor(indices, tf.gather_nd(res, indices), dense_shape)
with tf.Session() as sess:
print(sess.run(res))
# [[ 0 0 0 0 0]
# [ 0 0 3 0 0]
# [ 0 0 0 0 0]
# [ 2 0 0 12 0]
# [ 0 0 0 0 0]
# [ 0 0 0 0 0]
# [ 0 8 0 0 0]]
print(sess.run(sparse_res))
# SparseTensorValue(
# indices=array([[1, 2],
# [3, 0],
# [3, 3],
# [6, 1]]),
# values=array([ 3, 2, 12, 8]),
# dense_shape=array([7, 5]))
Probably there was no better option back then, there is a function for this now - doing the matrix multiplication after converting to dense makes the whole thing kind of pointless...
import tensorflow as tf
import numpy as np
A_vals = np.array([
[1, 2, 1],
[3, 0, 2],
[3, 3, 3],
[6, 1, 4],
])
A_sparse = tf.SparseTensor(A_vals[:,:2], tf.cast(A_vals[:,2], tf.float32), [7, 5])
b = tf.constant([1,2,3,4,5], tf.float32)
# sparse-dense matrix multiplication
res = tf.sparse.sparse_dense_matmul(A_sparse, b[:,None])
I created a matrix using:
Matrix = [[0 for x in range(5)] for z in range(5)]
I am trying to extract the elements above the diagnal and store it in an array.
For example:
[0, 0, 0, 1, 1]
[1, 0, 0, 0, 0]
[1, 1, 0, 0, 1]
[0, 1, 1, 0, 0]
[0, 1, 0, 1, 0]
U=[0,0,1,1,0,0,0,0,1,0]
A=[1,4,9]
[0,1,2]
[2,3,6]
U=[4,9,2]
You can just use List Comprehensions.
from random import randrange
Matrix = [[randrange(10) for x in range(5)] for z in range(5)]
>>>Matrix
[[6, 3, 7, 9, 3], [8, 6, 4, 0, 4], [0, 0, 1, 3, 2], [7, 7, 2, 3, 7], [3, 3, 5, 6, 3]]
[Matrix[i][j] for i in range(0,5) for j in range(i+1,5)]
[3, 7, 9, 3, 4, 0, 4, 3, 2, 7]
So here is a Solution, i changed your matrix to generate any random numbers so that you can see better which numbers are taken into account. TRIU = Triangle Upper is the function that takes a Matrix in your given format and takes the Upper Triangle, which is above the diagonal.
#import numpy as np
from random import randrange
Matrix = [[randrange(10) for x in range(5)] for z in range(5)]
def triu(matrix):
length = len(matrix[0])
U = list()
diagLine = 0
for row in Matrix:
length -= 1
colCounter = 0
for col in row:
if colCounter > diagLine:
U.append(col)
colCounter += 1
diagLine += 1
return U
#print np.matrix(Matrix)
print triu(Matrix)
Result:
[[0 0 2 4 0]
[6 4 8 9 0]
[6 2 2 3 0]
[2 9 6 5 5]
[1 5 8 9 2]]
[0, 2, 4, 0, 8, 9, 0, 3, 0, 5]
[Finished in 0.2s]