I have a loop for that generate a new vector (100,) in each iteration. So the code loop likes
for i in range (10):
for j in range (4):
#Create a new vector (100,)
#Concatenate 4 vector together to make (400,) #400=4*10
#Append the concatenation vectors (400,) in vertical to make (10,400) array
My expected is that generates a matrix size of (10,400) that concatenate vectors in these loops
Currently, my solution is
matrix_= np.empty([10,400])
for i in range (10):
vector_horz=[]
for j in range (4):
#Create a new vector (100,)
vector_rnd=#Random make a vector/list with size of (100,1)
#Concatenate 4 vector together to make (400,) #400=4*10
vector_horz.append(vector_rnd)
#Append the concatenation vectors (400,) in vertical to make (10,400) array
matrix_(:,i)=vector_horz
However, it said that my size of matrix and vector_horz cannot assign. Could you give me another solution?
Option 1
(Recommended) First generate your data and create an array at the end:
data = []
for i in range(10):
for j in range(4):
temp_list = ... # random vector of 100 elements
data.extend(temp_list)
arr = np.reshape(data, (10, 400))
Option 2
Alternatively, initialise an empty array with np.empty and assign one slice at a time:
arr = np.empty((10, 400))
for i in range(10):
for j in range(4):
temp_list = ...
arr[i, j * 100 : (j + 1) * 100] = temp_list
Related
Is there an efficient way to represent a sparse matrix as ijv (3 arrays : row, column, value) form.
Using nested loop seems very naive and slow for large matrices.
The code comes from here:
# Python program for Sparse Matrix Representation
# using arrays
# assume a sparse matrix of order 4*5
# let assume another matrix compactMatrix
# now store the value,row,column of arr1 in sparse matrix compactMatrix
sparseMatrix = [[0,0,3,0,4],[0,0,5,7,0],[0,0,0,0,0],[0,2,6,0,0]]
# initialize size as 0
size = 0
for i in range(4):
for j in range(5):
if (sparseMatrix[i][j] != 0):
size += 1
# number of columns in compactMatrix(size) should
# be equal to number of non-zero elements in sparseMatrix
rows, cols = (3, size)
compactMatrix = [[0 for i in range(cols)] for j in range(rows)]
k = 0
for i in range(4):
for j in range(5):
if (sparseMatrix[i][j] != 0):
compactMatrix[0][k] = i
compactMatrix[1][k] = j
compactMatrix[2][k] = sparseMatrix[i][j]
k += 1
for i in compactMatrix:
print(i)
# This code is contributed by MRINALWALIA
I am going to print sparse matrix to file in ijv form and read it in C++.
scipy.sparse.coo_matrix just give me:
print(coo_matrix([[0,0,3,0,4],[0,0,5,7,0],[0,0,0,0,0],[0,2,6,0,0]]))
(0, 2) 3
(0, 4) 4
(1, 2) 5
(1, 3) 7
(3, 1) 2
(3, 2) 6
with np.where() I can get the index of nonzero elements, but how about the v array?
Probably do you know a more efficient method (I am not going to use swig, ... to wrap the code)?
Edit
size=np.count_nonzero(sparseMatrix)
rows, cols = np.where(sparseMatrix)
compactMatrix = np.zeros((3, size))
for i in range(size):
compactMatrix[0][i] = rows[i]
compactMatrix[1][i] = cols[i]
compactMatrix[2][i] = sparseMatrix[rows[i]][cols[i]]
print(compactMatrix)
That's what I thought finally.
Suppose i have an array shaped as a:
import numpy as np
n = 10
d = 5
a = np.zeros(shape = np.repeat(n,d))
And that I want to obtain the values corresponding to indexes (0,...,:,...,0) for the : along dimensions, resulting in a (n,d)-shaped array b, with b[i,j] = a[0,...,0,i,0,...,0] where the i is in the jth dimension.
How can i extractb from a ?
Get the flattened indices and just index for a vectorized solution -
n = len(a)
d = a.ndim
idxs = np.multiply.outer(n**np.arange(d), np.arange(n))
out = a.flat[idxs]
Easiest is to do a for loop:
# get the first slice of `a` along given dimension `j`
def get_slice(a,j):
idx = [0]*len(a.shape)
idx[j] = slice(None)
return a[tuple(idx)]
out = np.stack([get_slice(a,j) for j in range(len(a.shape))])
And out.shape is (10,5)
I meet a problem to convert a python matrix of torch.tensor to a torch.tensor
For example, M is an (n,m) matrix, with each element M[i][j] is a torch.tensor with same size (p, q, r, ...). How to convert python list of list M to a torch.tensor with size (n,m,p,q,r,...)
e.g.
M = []
for i in range(5):
row = []
for j in range(10):
row.append(torch.rand(3,4))
M.append(row)
How to convert above M to a torch.tensor with size (5,10,3,4).
Try torch.stack() to stack a list of tensors on the first dimension.
import torch
M = []
for i in range(5):
row = []
for j in range(10):
row.append(torch.rand(3,4))
row = torch.stack(row)
M.append(row)
M = torch.stack(M)
print(M.size())
# torch.Size([5, 10, 3, 4])
Try this.
ref = np.arange(3*4*5).reshape(3,4,5) # numpy array
values = [ref.copy()+i for i in range(6)] # List of numpy arrays
b = torch.from_numpy(np.array(values)) # torch-array from List of numpy arrays
References
Converting NumPy Array to Torch Tensor
The general solution to this question is being worked on in this github issue, but I was wondering if there are workarounds using tf.gather (or something else) to achieve array indexing using a multi-index. One solution I came up with was to broadcast multiply each index in the multi-idx with the cumulative product of the tensor shape, which produces indices suitable for indexing the flattened tensor:
import tensorflow as tf
import numpy as np
def __cumprod(l):
# Get the length and make a copy
ll = len(l)
l = [v for v in l]
# Reverse cumulative product
for i in range(ll-1):
l[ll-i-2] *= l[ll-i-1]
return l
def ravel_multi_index(tensor, multi_idx):
"""
Returns a tensor suitable for use as the index
on a gather operation on argument tensor.
"""
if not isinstance(tensor, (tf.Variable, tf.Tensor)):
raise TypeError('tensor should be a tf.Variable')
if not isinstance(multi_idx, list):
multi_idx = [multi_idx]
# Shape of the tensor in ints
shape = [i.value for i in tensor.get_shape()]
if len(shape) != len(multi_idx):
raise ValueError("Tensor rank is different "
"from the multi_idx length.")
# Work out the shape of each tensor in the multi_idx
idx_shape = [tuple(j.value for j in i.get_shape()) for i in multi_idx]
# Ensure that each multi_idx tensor is length 1
assert all(len(i) == 1 for i in idx_shape)
# Create a list of reshaped indices. New shape will be
# [1, 1, dim[0], 1] for the 3rd index in multi_idx
# for example.
reshaped_idx = [tf.reshape(idx, [1 if i !=j else dim[0]
for j in range(len(shape))])
for i, (idx, dim)
in enumerate(zip(multi_idx, idx_shape))]
# Figure out the base indices for each dimension
base = __cumprod(shape)
# Now multiply base indices by each reshaped index
# to produce the flat index
return (sum(b*s for b, s in zip(base[1:], reshaped_idx[:-1]))
+ reshaped_idx[-1])
# Shape and slice starts and sizes
shape = (Z, Y, X) = 4, 5, 6
Z0, Y0, X0 = 1, 1, 1
ZS, YS, XS = 3, 3, 4
# Numpy matrix and index
M = np.random.random(size=shape)
idx = [
np.arange(Z0, Z0+ZS).reshape(ZS,1,1),
np.arange(Y0, Y0+YS).reshape(1,YS,1),
np.arange(X0, X0+XS).reshape(1,1,XS),
]
# Tensorflow matrix and indices
TM = tf.Variable(M)
TF_flat_idx = ravel_multi_index(TM, [
tf.range(Z0, Z0+ZS),
tf.range(Y0, Y0+YS),
tf.range(X0, X0+XS)])
TF_data = tf.gather(tf.reshape(TM,[-1]), TF_flat_idx)
with tf.Session() as S:
S.run(tf.initialize_all_variables())
# Obtain data via flat indexing
data = S.run(TF_data)
# Check that it agrees with data obtained
# by numpy smart indexing
assert np.all(data == M[idx])
However, this only works on tensors of rank 3 due to this (current) limitation limiting broadcasts to tensors of rank 3.
At the moment I can only think of doing a chained gather, transpose, gather, transpose, gather, but this is unlikely to be efficient. e.g.
shape = (8, 9, 10)
A = tf.random_normal(shape)
data = tf.gather(tf.transpose(tf.gather(A, [1, 3]), [1,0,2]), ...)
Any ideas?
It sounds like you want gather_nd.
i need to manipulate an numpy array:
My Array has the followng format:
x = [1280][720][4]
The array stores image data in the third dimension:
x[0][0] = [Red,Green,Blue,Alpha]
Now i need to manipulate my array to the following form:
x = [1280][720]
x[0][0] = Red + Green + Blue / 3
My current code is extremly slow and i want to use the numpy array manipulation to speed it up:
for a in range(0,719):
for b in range(0,1279):
newx[a][b] = x[a][b][0]+x[a][b][1]+x[a][b][2]
x = newx
Also, if possible i need the code to work for variable array sizes.
Thansk Alot
Use the numpy.mean function:
import numpy as np
n = 1280
m = 720
# Generate a n * m * 4 matrix with random values
x = np.round(np.random.rand(n, m, 4)*10)
# Calculate the mean value over the first 3 values along the 2nd axix (starting from 0)
xnew = np.mean(x[:, :, 0:3], axis=2)
x[:, :, 0:3] gives you the first 3 values in the 3rd dimension, see: numpy indexing
axis=2 specifies, along which axis of the matrix the mean value is calculated.
Slice the alpha channel out of the array, and then sum the array along the RGB axis and divide by 3:
x = x[:,:,:-1]
x_sum = x.sum(axis=2)
x_div = x_sum / float(3)