I have a multidimensional numpy array of dtype object, which was filled with other arrays.
As an example, here is a code reproducing that behavior:
arr = np.empty((3,4,2,1), dtype=object)
for i in range(arr.shape[0]):
for j in range(arr.shape[1]):
for k in range(arr.shape[2]):
for l in range(arr.shape[3]):
arr[i, j, k, l] = np.random.random(10)
Since all the inside arrays have the same size, I would like in this example to "incorporate" the last level into the array and make it an array of size (3,4,2,1,10).
I cannot really change the above code, so what I am looking for is a clean way (few lines, possibly without for loops) to generate this new array once created.
Thank you.
If I understood well your problem you could use random.random_sample() which should give the same result:
arr = np.random.random_sample((3, 4, 2, 1, 10))
After edit the solution is arr = np.array(arr.tolist())
Just by adding a new for loop :
arr = np.empty((3,4,2,1,10), dtype=object)
for i in range(arr.shape[0]):
for j in range(arr.shape[1]):
for k in range(arr.shape[2]):
for l in range(arr.shape[3]):
for m in range(arr.shape[4]):
arr[i, j, k, l, m] = np.random.randint(10)
However, you can one line this code with an optimized numpy function, every random function from numpy has a size parameter to build a array with random number with a particular shape :
arr = np.random.random((3,4,2,1,10))
EDIT :
You can flatten the array, replace every single number by a 1D array of length 10 and then reshape it to your desired shape :
import numpy as np
arr = np.empty((3,4,2,1), dtype=object)
for i in range(arr.shape[0]):
for j in range(arr.shape[1]):
for k in range(arr.shape[2]):
for l in range(arr.shape[3]):
arr[i, j, k, l] = np.random.randint(10)
flat_arr = arr.flatten()
for i in range(len(flat_arr)):
flat_arr[i] = np.random.randint(0, high=10, size=(10))
res_arr = flat_arr.reshape((3,4,2,1))
Related
I am working on a project for cryptography course working with Charm jhuisi link .
I have two numpy matrix: V(2,3) belonging to ZR and M(3x2) belonging to G1. I want to bring V to G1, so I can exponentiate M^V. To perform this operation, in Charm I cannot simply use M**V, but I have to do it element by element.
from charm.toolbox.pairinggroup import PairingGroup,ZR,G1,G2,GT,pair
import numpy as np
M = np.array([[group.random(G1) for i in range(2)] for j in range(3)])
V_t = np.transpose(np.array([[group.random(ZR) for i in range(2)] for j in range(3)]))
matrix = np.array([[M[i][j] ** V[j][i] for j in range(3)] for i in range(2)]
but it returns me an error "IndexError: index 2 is out of bounds for axis 0 with size 2"
Can someone who has used Charm before help me, please?
Your code is a confused mix of python lists and numpy arrays. Lets first do the calculation with lists, and paying particular attention to keeping indices right.
Make 2 lists:
In [358]: M = [[1,2,3],[4,5,6]]
In [359]: V = [[1,2],[3,4],[5,6]]
Starting with an empty list, fill it with a new list:
In [360]: res = []
In [361]: for i in range(2):
...: res1 = []
...: for j in range(3):
...: res1.append(M[i][j]**V[j][i])
...: res.append(res1)
...:
In [362]: res
Out[362]: [[1, 8, 243], [16, 625, 46656]]
Note how i range is 2, j range is 3, matching the lengths of the lists.
The same calculation using numpy arrays:
In [363]: np.array(M)**np.array(V).T
Out[363]:
array([[ 1, 8, 243],
[ 16, 625, 46656]])
np.array(M) is (2,3) shape; np.array(V) is (3,2). To perform the elementwise power, V has to be transposed to (2,3).
The nested loop can be written as a comprehension - again with the same care over indices:
In [364]: [[M[i][j]**V[j][i] for j in range(3)] for i in range(2)]
Out[364]: [[1, 8, 243], [16, 625, 46656]]
V_t
What is V_t? I see
In [365]: from random import random
In [366]: V = np.transpose(np.array([[random() for i in range(2)] for j in rang
...: e(3)]))
In [367]: V
Out[367]:
array([[0.8748556 , 0.10373381, 0.23399403],
[0.95388354, 0.24060715, 0.38468676]])
In [368]: V.shape
Out[368]: (2, 3)
Have you done some undocumented transpose to produce a (3,2)? If so then you need to use V_t[i][j]. Are your problems just the result of a sloppy use of the transpose?
Aren't your indices just the wrong way around?
import numpy as np
from numpy.random import random
M = np.array([[random() for i in range(2)] for j in range(3)])
V = np.transpose(np.array([[random() for i in range(2)] for j in range(3)]))
matrix = [[M[i][j] ** V[j][i] for j in range(2)] for i in range(3)]
Edit
Here's a wild idea. Try:
import numpy as np
from god_knows_what import random
M = np.array([[random() for i in range(2)] for j in range(3)], dtype=object)
V = np.transpose(np.array([[random() for i in range(2)] for j in range(3)], dtype=object))
matrix = np.array([[M[i][j] ** V[j][i] for j in range(2)] for i in range(3)], dtype=object)
If the last line fails, try
matrix = np.array([[M[i][j] ** V[j][i] for j in range(3)] for i in range(2)], dtype=object)
import numpy as np
import scipy.sparse
x = np.random.randint(0, 1000, (1000, 100))
# prob better way to do this
d = np.random.random((1000,1000))
d[d < 0.99] = 0
y = scipy.sparse.csr_matrix(d)
What I would like to do is to create a new matrix z containing the values of y at the indices in x.
ie [0, 0] of z should contain the y[0, x[0, 0]]
[0, 1] of z should contain the y[0, x[0, 1]]
%time for i in range(1000): x[i, y[i]].todense()
~247ms
%time for i in range(1000): np.take(x[i].todense(), y[i])
~150ms
both of the above work, but I am looking for a faster method- this is currently the bottleneck on my code.
Please assume that representing the whole scipy.sparse matrix as dense isn't feasible.
edit:
%time z = np.vstack([q.todense()[0, p] for q, p in zip(x, y)])
is ~110ms
The answer seems to be to use an appropriately shaped broadcasting index, as outlined here: How to generate multi-dimensional 2D numpy index using a sub-index for one dimension
(answer deserves more upvotes)!
%time res = y[np.arange(0, 1000).reshape((-1, 1)), x].todense()
The code below allows me to add a vector to each row of a given matrix using Numpy:
import numpy as np
m = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
v = np.array([1, 1, 0])
print("Original vector:")
print(v)
print("Original matrix:")
print(m)
result = np.empty_like(m)
for i in range(4):
result[i, :] = m[i, :] + v
print("\nAfter adding the vector v to each row of the matrix m:")
print(result)
How do I perform a similar addition operation, but going column by column?
I have tried the following:
import numpy as np
array1 = np.array([[5,5,3],[2,2,3]])
print(array1)
addition = np.array([[1],[1]])
print(addition)
for i in range(3):
array1[:,i] = array1[:,i] + addition
print(array1)
However, I get the following broadcasting error:
ValueError: could not broadcast input array from shape (2,2) into shape (2)
Just match the number of dimensions, numpy will broadcast the arrays as needed. In the first example, it should be:
result = m + v.reshape((1, -1))
In the second example, the addition is already 2D so it will be just:
array1 + addition
You can alternatively, add a dimension via Numpy None syntax and then do the addition:
array1 += addition[:,None]
I meet a problem to convert a python matrix of torch.tensor to a torch.tensor
For example, M is an (n,m) matrix, with each element M[i][j] is a torch.tensor with same size (p, q, r, ...). How to convert python list of list M to a torch.tensor with size (n,m,p,q,r,...)
e.g.
M = []
for i in range(5):
row = []
for j in range(10):
row.append(torch.rand(3,4))
M.append(row)
How to convert above M to a torch.tensor with size (5,10,3,4).
Try torch.stack() to stack a list of tensors on the first dimension.
import torch
M = []
for i in range(5):
row = []
for j in range(10):
row.append(torch.rand(3,4))
row = torch.stack(row)
M.append(row)
M = torch.stack(M)
print(M.size())
# torch.Size([5, 10, 3, 4])
Try this.
ref = np.arange(3*4*5).reshape(3,4,5) # numpy array
values = [ref.copy()+i for i in range(6)] # List of numpy arrays
b = torch.from_numpy(np.array(values)) # torch-array from List of numpy arrays
References
Converting NumPy Array to Torch Tensor
Is there an easy way to index a numpy multidimensional array along the last dimension, using an array of indices? For example, take an array a of shape (10, 10, 20). Let's assume I have an array of indices b, of shape (10, 10) so that the result would be c[i, j] = a[i, j, b[i, j]].
I've tried the following example:
a = np.ones((10, 10, 20))
b = np.tile(np.arange(10) + 10, (10, 1))
c = a[b]
However, this doesn't work because it then tries to index like a[b[i, j], b[i, j]], which is not the same as a[i, j, b[i, j]]. And so on. Is there an easy way to do this without resorting to a loop?
There are several ways to do this. Let's first generate some test data:
In [1]: a = np.random.rand(10, 10, 20)
In [2]: b = np.random.randint(20, size=(10,10)) # random integers in range 0..19
One way to solve the question would be to create two index vectors, where one is a row vector and the other a column vector of 0..9 using meshgrid:
In [3]: i1, i0 = np.meshgrid(range(10), range(10), sparse=True)
In [4]: c = a[i0, i1, b]
This works because i0, i1 and b will all be broadcasted to 10x10 matrices. Quick test for correctness:
In [5]: all(c[i, j] == a[i, j, b[i, j]] for i in range(10) for j in range(10))
Out[5]: True
Another way would be to use choose and rollaxis:
# choose needs a sequence of length 20, so move last axis to front
In [22]: aa = np.rollaxis(a, -1)
In [23]: c = np.choose(b, aa)
In [24]: all(c[i, j] == a[i, j, b[i, j]] for i in range(10) for j in range(10))
Out[24]: True