I am working on a project for cryptography course working with Charm jhuisi link .
I have two numpy matrix: V(2,3) belonging to ZR and M(3x2) belonging to G1. I want to bring V to G1, so I can exponentiate M^V. To perform this operation, in Charm I cannot simply use M**V, but I have to do it element by element.
from charm.toolbox.pairinggroup import PairingGroup,ZR,G1,G2,GT,pair
import numpy as np
M = np.array([[group.random(G1) for i in range(2)] for j in range(3)])
V_t = np.transpose(np.array([[group.random(ZR) for i in range(2)] for j in range(3)]))
matrix = np.array([[M[i][j] ** V[j][i] for j in range(3)] for i in range(2)]
but it returns me an error "IndexError: index 2 is out of bounds for axis 0 with size 2"
Can someone who has used Charm before help me, please?
Your code is a confused mix of python lists and numpy arrays. Lets first do the calculation with lists, and paying particular attention to keeping indices right.
Make 2 lists:
In [358]: M = [[1,2,3],[4,5,6]]
In [359]: V = [[1,2],[3,4],[5,6]]
Starting with an empty list, fill it with a new list:
In [360]: res = []
In [361]: for i in range(2):
...: res1 = []
...: for j in range(3):
...: res1.append(M[i][j]**V[j][i])
...: res.append(res1)
...:
In [362]: res
Out[362]: [[1, 8, 243], [16, 625, 46656]]
Note how i range is 2, j range is 3, matching the lengths of the lists.
The same calculation using numpy arrays:
In [363]: np.array(M)**np.array(V).T
Out[363]:
array([[ 1, 8, 243],
[ 16, 625, 46656]])
np.array(M) is (2,3) shape; np.array(V) is (3,2). To perform the elementwise power, V has to be transposed to (2,3).
The nested loop can be written as a comprehension - again with the same care over indices:
In [364]: [[M[i][j]**V[j][i] for j in range(3)] for i in range(2)]
Out[364]: [[1, 8, 243], [16, 625, 46656]]
V_t
What is V_t? I see
In [365]: from random import random
In [366]: V = np.transpose(np.array([[random() for i in range(2)] for j in rang
...: e(3)]))
In [367]: V
Out[367]:
array([[0.8748556 , 0.10373381, 0.23399403],
[0.95388354, 0.24060715, 0.38468676]])
In [368]: V.shape
Out[368]: (2, 3)
Have you done some undocumented transpose to produce a (3,2)? If so then you need to use V_t[i][j]. Are your problems just the result of a sloppy use of the transpose?
Aren't your indices just the wrong way around?
import numpy as np
from numpy.random import random
M = np.array([[random() for i in range(2)] for j in range(3)])
V = np.transpose(np.array([[random() for i in range(2)] for j in range(3)]))
matrix = [[M[i][j] ** V[j][i] for j in range(2)] for i in range(3)]
Edit
Here's a wild idea. Try:
import numpy as np
from god_knows_what import random
M = np.array([[random() for i in range(2)] for j in range(3)], dtype=object)
V = np.transpose(np.array([[random() for i in range(2)] for j in range(3)], dtype=object))
matrix = np.array([[M[i][j] ** V[j][i] for j in range(2)] for i in range(3)], dtype=object)
If the last line fails, try
matrix = np.array([[M[i][j] ** V[j][i] for j in range(3)] for i in range(2)], dtype=object)
Related
I would like to do this particular computation: with a square ndarray A in 4 dimension of size (N, )*4, I would like to compute the 2 dimension array B such that
for n in range(N):
for m in range(N):
B[n, m] = sum(A[i, j, n-i, m-j] for i in range(n) for j in range(m))
Is it possible to vectorize this computation with numpy?
It somehow looks like a kind of convolution, but on one array only
It's hard to visualize the whole array action
N=4; A = np.arange(N**4).reshape(N,N,N,N)
this tests the same:
for n in range(N):
for m in range(N):
I = np.arange(n)[:,None]; J = np.arange(m)
B[n,m] = A[I,J,n-I,m-J].sum()
It's harder to "vectorize" the n,m since the indexed portion of A changes with n and m. At the last iteration:
In [248]: n,m
Out[248]: (3, 3)
In [249]: A[I,J,n-I,m-J]
Out[249]:
array([[ 15, 30, 45],
[ 75, 90, 105],
[135, 150, 165]])
while if either n or m is 0, it's an "empty" array with sum of 0.
I have a multidimensional numpy array of dtype object, which was filled with other arrays.
As an example, here is a code reproducing that behavior:
arr = np.empty((3,4,2,1), dtype=object)
for i in range(arr.shape[0]):
for j in range(arr.shape[1]):
for k in range(arr.shape[2]):
for l in range(arr.shape[3]):
arr[i, j, k, l] = np.random.random(10)
Since all the inside arrays have the same size, I would like in this example to "incorporate" the last level into the array and make it an array of size (3,4,2,1,10).
I cannot really change the above code, so what I am looking for is a clean way (few lines, possibly without for loops) to generate this new array once created.
Thank you.
If I understood well your problem you could use random.random_sample() which should give the same result:
arr = np.random.random_sample((3, 4, 2, 1, 10))
After edit the solution is arr = np.array(arr.tolist())
Just by adding a new for loop :
arr = np.empty((3,4,2,1,10), dtype=object)
for i in range(arr.shape[0]):
for j in range(arr.shape[1]):
for k in range(arr.shape[2]):
for l in range(arr.shape[3]):
for m in range(arr.shape[4]):
arr[i, j, k, l, m] = np.random.randint(10)
However, you can one line this code with an optimized numpy function, every random function from numpy has a size parameter to build a array with random number with a particular shape :
arr = np.random.random((3,4,2,1,10))
EDIT :
You can flatten the array, replace every single number by a 1D array of length 10 and then reshape it to your desired shape :
import numpy as np
arr = np.empty((3,4,2,1), dtype=object)
for i in range(arr.shape[0]):
for j in range(arr.shape[1]):
for k in range(arr.shape[2]):
for l in range(arr.shape[3]):
arr[i, j, k, l] = np.random.randint(10)
flat_arr = arr.flatten()
for i in range(len(flat_arr)):
flat_arr[i] = np.random.randint(0, high=10, size=(10))
res_arr = flat_arr.reshape((3,4,2,1))
import numpy as np
import scipy.sparse
x = np.random.randint(0, 1000, (1000, 100))
# prob better way to do this
d = np.random.random((1000,1000))
d[d < 0.99] = 0
y = scipy.sparse.csr_matrix(d)
What I would like to do is to create a new matrix z containing the values of y at the indices in x.
ie [0, 0] of z should contain the y[0, x[0, 0]]
[0, 1] of z should contain the y[0, x[0, 1]]
%time for i in range(1000): x[i, y[i]].todense()
~247ms
%time for i in range(1000): np.take(x[i].todense(), y[i])
~150ms
both of the above work, but I am looking for a faster method- this is currently the bottleneck on my code.
Please assume that representing the whole scipy.sparse matrix as dense isn't feasible.
edit:
%time z = np.vstack([q.todense()[0, p] for q, p in zip(x, y)])
is ~110ms
The answer seems to be to use an appropriately shaped broadcasting index, as outlined here: How to generate multi-dimensional 2D numpy index using a sub-index for one dimension
(answer deserves more upvotes)!
%time res = y[np.arange(0, 1000).reshape((-1, 1)), x].todense()
In R, when I execute the code below:
> X=matrix(1,2,3)
> c=c(1,2,3)
> X*c
R gives out the following output:
[,1] [,2] [,3]
[1,] 1 3 2
[2,] 2 1 3
But when I do the below on Python:
>>> import numpy as np
>>> X=np.array([[1,1,1],[1,1,1]])
>>> c=np.array([1,2,3])
>>> X*c
the Python code above gives the following output:
array([[1, 2, 3],
[1, 2, 3]])
Is there any way that I can make the Python to come up with the identical output as R? I think I somehow have to tell Python that I want the numpy to multiply each element of the matrix X by each element of the vector c along the column, instead of along the row, but I am not sure how to go about this.
In [18]: np.reshape([1,2,3]*2,(2,3),order='F')
Out[18]:
array([[1, 3, 2],
[2, 1, 3]])
This starts with a list multiply, which is replication:
In [19]: [1,2,3]*2
Out[19]: [1, 2, 3, 1, 2, 3]
The rest uses numpy to reshape it into a (2,3) array, but with consecutive values going down, 'F' order.
Not knowning R, and in particular the c(1,2,3) expression, I can't say that's what's going on in R.
===
You talk about rows with columns, but I don't see how that works in your example. That said, we can easily perform outer like products
===
This reproduces your R_Product (at least in a few test cases):
In [138]: def foo(X,c):
...: X1 = X.ravel()
...: Y = np.resize(c,X1.shape)*X1
...: return Y.reshape(X.shape, order='F')
...:
In [139]: foo(np.ones((2,3)),np.arange(1,4))
Out[139]:
array([[1., 3., 2.],
[2., 1., 3.]])
In [140]: foo(np.arange(6).reshape(2,3),np.arange(1,4))
Out[140]:
array([[ 0, 6, 8],
[ 2, 3, 15]])
I'm using the resize function to replicate c to match the total number of elements of X. And order F to stack them in the desired column order. The default for numpy is order C.
In numpy replicating an array to match another is not common, at least not in this sense. Replicating by row or column, as in broadcasting is common. And of course reshaping.
I am the OP.
I was looking for a quick and easy solution, but I guess there is no straightforward functionality in Python that allows us to do this. So, I had to make a function that multiplies a matrix with a vector in the same manner that R does:
def R_product(X,c):
"""
Computes the regular R product
(not same as the matrix product) between
a 2D Numpy Array X, and a numpy vector c.
Args:
X: 2D Numpy Array
c: A Numpy vector
Returns: the output of X*c in R.
(This is different than X/*/c in R)
"""
X_nrow = X.shape[0]
X_ncol = X.shape[1]
X_dummy = np.zeros(shape=((X_nrow * X_ncol),1))
nrow = X_dummy.shape[0]
nc = nrow // len(c)
Y = np.zeros(shape=(nrow,1))
for j in range(X_ncol):
for u in range(X_nrow):
X_element = X[u,j]
if u == X_nrow - 1:
idx = X_nrow * (j+1) - 1
else:
idx = X_nrow * j + (u+1) - 1
X_dummy[idx,0] = X_element
for i in range(nc):
for j in range(len(c)):
Y[(i*len(c)+j):(i*len(c)+j+1),:] = (X_dummy[(i*len(c)+j):(i*len(c)+j+1),:]) * c[j]
for z in range(nrow-nc*len(c)):
Y[(nc*len(c)+z):(nc*len(c)+z+1),:] = (X_dummy[(nc*len(c)+z):(nc*len(c)+z+1),:]) * c[z]
return Y.reshape(X_ncol, X_nrow).transpose() # the answer I am looking for
Should work.
Is there an easy way to index a numpy multidimensional array along the last dimension, using an array of indices? For example, take an array a of shape (10, 10, 20). Let's assume I have an array of indices b, of shape (10, 10) so that the result would be c[i, j] = a[i, j, b[i, j]].
I've tried the following example:
a = np.ones((10, 10, 20))
b = np.tile(np.arange(10) + 10, (10, 1))
c = a[b]
However, this doesn't work because it then tries to index like a[b[i, j], b[i, j]], which is not the same as a[i, j, b[i, j]]. And so on. Is there an easy way to do this without resorting to a loop?
There are several ways to do this. Let's first generate some test data:
In [1]: a = np.random.rand(10, 10, 20)
In [2]: b = np.random.randint(20, size=(10,10)) # random integers in range 0..19
One way to solve the question would be to create two index vectors, where one is a row vector and the other a column vector of 0..9 using meshgrid:
In [3]: i1, i0 = np.meshgrid(range(10), range(10), sparse=True)
In [4]: c = a[i0, i1, b]
This works because i0, i1 and b will all be broadcasted to 10x10 matrices. Quick test for correctness:
In [5]: all(c[i, j] == a[i, j, b[i, j]] for i in range(10) for j in range(10))
Out[5]: True
Another way would be to use choose and rollaxis:
# choose needs a sequence of length 20, so move last axis to front
In [22]: aa = np.rollaxis(a, -1)
In [23]: c = np.choose(b, aa)
In [24]: all(c[i, j] == a[i, j, b[i, j]] for i in range(10) for j in range(10))
Out[24]: True