I want to perform a specific operation. Namely, from a matrix:
A = np.array([[1,2],
[3,4]])
To the following
B = np.array([[1, 0, 0, 2, 0, 0],
[0, 1, 0, 0, 2, 0],
[0, 0, 1, 0, 0, 2],
[3, 0, 0, 4, 0, 0],
[0, 3, 0, 0, 4, 0],
[0, 0, 3, 0, 0, 4]])
Or in words: multiply every entry by the identity matrix and keep the same order.
Now I have accomplished this by using numpy, using the following code. Here N and M are the dimensions of the starting matrix, and the dimension of the identity matrix.
l_slice = 3
n_slice = 2
A = np.reshape(np.arange(1, 1+N ** 2), (N, N))
B = np.array([i * np.eye(M) for i in A.flatten()])
C = B.reshape(N, N, M, M).reshape(N, N * M, M).transpose([0, 2, 1]).reshape((N * M, N * M))
where C has my desired properties.
But now I want do this modification in Keras/Tensorflow, where the matrix A is the outcome of one of my layers.
However, I am not sure yet if I will be able to properly create matrix B. Especially when batches are involved, I think I will somehow mess up the dimensions of my problem.
Can anyone with more Keras/Tensorflow experience comment on this 'reshape' and how he/she sees this happening within Keras/Tensorflow?
Here is a way to do that with TensorFlow:
import tensorflow as tf
data = tf.placeholder(tf.float32, [None, None])
n = tf.placeholder(tf.int32, [])
eye = tf.eye(n)
mult = data[:, tf.newaxis, :, tf.newaxis] * eye[tf.newaxis, :, tf.newaxis, :]
result = tf.reshape(mult, n * tf.shape(data))
with tf.Session() as sess:
a = sess.run(result, feed_dict={data: [[1, 2], [3, 4]], n: 3})
print(a)
Output:
[[1. 0. 0. 2. 0. 0.]
[0. 1. 0. 0. 2. 0.]
[0. 0. 1. 0. 0. 2.]
[3. 0. 0. 4. 0. 0.]
[0. 3. 0. 0. 4. 0.]
[0. 0. 3. 0. 0. 4.]]
By the way, you can do basically the same in NumPy, which should be faster than your current solution:
import numpy as np
data = np.array([[1, 2], [3, 4]])
n = 3
eye = np.eye(n)
mult = data[:, np.newaxis, :, np.newaxis] * eye[np.newaxis, :, np.newaxis, :]
result = np.reshape(mult, (n * data.shape[0], n * data.shape[1]))
print(result)
# The output is the same as above
EDIT:
I'll try to give some intuition about why/how this works, sorry if it's too long. It is not that hard but I think it's sort of tricky to explain. Maybe it is easier to see how the following multiplication works
import numpy as np
data = np.array([[1, 2], [3, 4]])
n = 3
eye = np.eye(n)
mult1 = data[:, :, np.newaxis, np.newaxis] * eye[np.newaxis, np.newaxis, :, :]
Now, mult1 is a sort of "matrix of matrices". If I give two indices, I will get the diagonal matrix for the corresponding element in the original one:
print(mult1[0, 0])
# [[1. 0. 0.]
# [0. 1. 0.]
# [0. 0. 1.]]
So you could say this matrix could be visualize like this:
| 1 0 0 | | 2 0 0 |
| 0 1 0 | | 0 2 0 |
| 0 0 1 | | 0 0 2 |
| 3 0 0 | | 4 0 0 |
| 0 3 0 | | 0 4 0 |
| 0 0 3 | | 0 0 4 |
However this is deceiving, because if you try to reshape this to the final shape the result is not the right one:
print(np.reshape(mult1, (n * data.shape[0], n * data.shape[1])))
# [[1. 0. 0. 0. 1. 0.]
# [0. 0. 1. 2. 0. 0.]
# [0. 2. 0. 0. 0. 2.]
# [3. 0. 0. 0. 3. 0.]
# [0. 0. 3. 4. 0. 0.]
# [0. 4. 0. 0. 0. 4.]]
The reason is that reshaping (conceptually) "flattens" the array first and then gives the new shape. But the flattened array in this case is not what you need:
print(mult1.ravel())
# [1. 0. 0. 0. 1. 0. 0. 0. 1. 2. 0. 0. 0. 2. 0. ...
You see, it first traverses the first submatrix, then the second, etc. What you want though is for it to traverse first the first row of the first submatrix, then the first row of the second submatrix, then second row of first submatrix, etc. So basically you want something like:
Take the first two submatrices (the ones with 1 and 2)
Take all the first rows ([1, 0, 0] and [2, 0, 0]).
Take the first of these ([1, 0, 0])
Take each of its elements (1, 0 and 0).
And then continue for the rest. So if you think about it, we traversing first the axis 0 (row of "matrix of matrices"), then 2 (rows of each submatrix), then 1 (column of "matrix of matrices") and finally 3 (columns of submatrices). So we can just reorder the axis to do that:
mult2 = mult1.transpose((0, 2, 1, 3))
print(np.reshape(mult2, (n * data.shape[0], n * data.shape[1])))
# [[1. 0. 0. 2. 0. 0.]
# [0. 1. 0. 0. 2. 0.]
# [0. 0. 1. 0. 0. 2.]
# [3. 0. 0. 4. 0. 0.]
# [0. 3. 0. 0. 4. 0.]
# [0. 0. 3. 0. 0. 4.]]
And it works! So in the solution I posted, to avoid the tranposing, I just make the multiplication so the order of the axes is exactly that:
mult = data[
:, # Matrix-of-matrices rows
np.newaxis, # Submatrix rows
:, # Matrix-of-matrices columns
np.newaxis # Submatrix columns
] * eye[
np.newaxis, # Matrix-of-matrices rows
:, # Submatrix rows
np.newaxis, # Matrix-of-matrices columns
: # Submatrix columns
]
I hope that makes it slightly clearer. To be honest, in this case in particular I could came up with the solution quickly because I had to solve a similar problem not too long ago, and I guess you end up building an intuition of these things.
Another way to achieve the same effect in numpy is to use the following:
A = np.array([[1,2],
[3,4]])
B = np.repeat(np.repeat(A, 3, axis=0), 3, axis=1) * np.tile(np.eye(3), (2,2))
Then, to replicate it in tensorflow, we can use tf.tile, but there is no tf.repeat, however someone has provided this function on tensorflow tracker.
def tf_repeat(tensor, repeats):
"""
Args:
input: A Tensor. 1-D or higher.
repeats: A list. Number of repeat for each dimension, length must be the same as the number of dimensions in input
Returns:
A Tensor. Has the same type as input. Has the shape of tensor.shape * repeats
"""
with tf.variable_scope("repeat"):
expanded_tensor = tf.expand_dims(tensor, -1)
multiples = [1] + list(repeats)
tiled_tensor = tf.tile(expanded_tensor, multiples=multiples)
repeated_tesnor = tf.reshape(tiled_tensor, tf.shape(tensor) * repeats)
return repeated_tesnor
and thus the tensorflow implementation will look like the following. Here I also consider that the first dimension represents batches, and thus we do not operate on it.
N = 2
M = 3
nbatch = 2
Ain = np.reshape(np.arange(1, 1 + N*N*nbatch), (nbatch, N, N))
A = tf.placeholder(tf.float32, shape=(nbatch, N, N))
B = tf.tile(tf.eye(M), [N, N]) * tf_repeat(A, [1, M, M])
with tf.Session() as sess:
print(sess.run(C, feed_dict={A: Ain}))
and the result:
[[[1. 0. 0. 2. 0. 0.]
[0. 1. 0. 0. 2. 0.]
[0. 0. 1. 0. 0. 2.]
[3. 0. 0. 4. 0. 0.]
[0. 3. 0. 0. 4. 0.]
[0. 0. 3. 0. 0. 4.]]
[[5. 0. 0. 6. 0. 0.]
[0. 5. 0. 0. 6. 0.]
[0. 0. 5. 0. 0. 6.]
[7. 0. 0. 8. 0. 0.]
[0. 7. 0. 0. 8. 0.]
[0. 0. 7. 0. 0. 8.]]]
Related
I need help with making this so I can convert a string into a number in a list, I have done this but if I wanted to do it this way I would have to wright a dictionary with 100 definitions which I do not want to do. The code is just to show what I found all ready. As you can see this would take 100 definitions if I were to do it this way.
x1 = [0,0,0,0,0,0,0,0,0,0]
x2 = [0,0,0,0,0,0,0,0,0,0]
x3 = [0,0,0,0,0,0,0,0,0,0]
x4 = [0,0,0,0,0,0,0,0,0,0]
x5 = [0,0,0,0,0,0,0,0,0,0]
x6 = [0,0,0,0,0,0,0,0,0,0]
x7 = [0,0,0,0,0,0,0,0,0,0]
x8 = [0,0,0,0,0,0,0,0,0,0]
x9 = [0,0,0,0,0,0,0,0,0,0]
x10 = [0,0,0,0,0,0,0,0,0,0]
my_dict_grid = {
'x2[3]' : x2[3]
}
x = 'x2[3]'
print(my_dict_grid[x])
If you have multiple arrays you are managing all at once, create a multi-dimensional array:
x = [
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0]
]
In that case, you can just index by row then column:
x[2][3]
Based on your comment, you want to randomly change values in the array. In that case, the approach above is not at all what you want. You want to pick two random numbers, and index to them in x to change them:
import random
for _ in range(5):
updated = False
while not updated:
i = random.randrange(10)
j = random.randrange(10)
if x[i][j] == 0:
x[i][j] = 1
updated = True
Original answer to the initial question:
(this is here more as an interesting thing, not as a viable approach)
Okay. Assuming that you have to do it the way you have described, you can generate a dictionary with all of the string keys:
my_dict_grid = {
f"x{i + 1}[{j}]": arr[j]
for i, arr in enumerate([x1, x2, x3, x4, x5, x6, x7, x8, x9, x10])
for j in range(10)
}
However, I have to stress that this is not a good idea.
3 different ways to solve this with oneliners, depending of the output you want:
my_list = [[ 0 for _ in range(10)] for _ in range(10)]
my_dict = {"x"+str(i+1):[ 0 for _ in range(10)] for i in range(10)}
my_dict2 = {"x"+str((i+1)%10)+"["+str(int((i+1)/10))+"]": 0 for i in range(100)}
print(my_list) #[[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0,...
print(my_dict) #{'x10': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'x9': [0,...
print(my_dict2)#{'x4[3]': 0, 'x1[9]': 0, 'x6[6]': 0, 'x2[8]': 0,...
A more mathematical, but very practical way to do this with numpy:
import numpy as np
grid_shape = [10, 10] # define 10 x 10 grid
num_ones = 5
cells = np.zeros(grid_shape[0]*grid_shape[1]) # define 10*10 = 100 cells as flat array
cells[0:num_ones] = 1 # Set the first 5 entries to 1
np.random.shuffle(cells) # Shuffle the entries, such that the 1's are at random position
grid = cells.reshape(grid_shape) # shape the grid into the desired shape
Running the code above and will e.g. result in grid=
[[0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]
[0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]
Note that, by changing grid_shape you can resize your grid, and by changing num_ones, you will adapt the number of ones in your grid. Also, it is guaranteed that there will always be num_ones ones in your grid (given that num_ones is smaller or equal the number of elements in the grid).
I'm using numpy's ndarray, and I'm wondering is there a way that allows me to insert multiple elements to different locations all at once?
For example, I have an image, and I want to pad the image with 0s. This is what I currently have:
def zero_padding(self):
padded = self.copy()
padded.img = np.insert(self.img, 0, 0, axis = 0)
padded.img = np.insert(padded.img, padded.img.shape[0], 0, axis = 0)
padded.img = np.insert(padded.img, 0, 0, axis = 1)
padded.img = np.insert(padded.img, padded.img.shape[1], 0, axis = 1)
return padded
where padded is an instance of the image.
Sure, you can use the fancy indexing techinque of NumPy as follows:
import numpy as np
if __name__=='__main__':
A = np.zeros((5, 5))
A[[1, 2], [0, 3]] = 1
print(A)
Output:
[[0. 0. 0. 0. 0.]
[1. 0. 0. 0. 0.]
[0. 0. 0. 1. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]]
Cheers
If i want to do an outer product of 2 vectors to create a 2d matrix, each element a product of the two respective elements in the original vectors:
b = np.arange(5).reshape((1, 5))
a = np.arange(5).reshape((5, 1))
a * b
array([[ 0, 0, 0, 0, 0],
[ 0, 1, 2, 3, 4],
[ 0, 2, 4, 6, 8],
[ 0, 3, 6, 9, 12],
[ 0, 4, 8, 12, 16]])
I want the same for 3 (or for n) vectors.
An equivalent non numpy answer:
a = np.arange(5)
b = np.arange(5)
c = np.arange(5)
res = np.zeros((a.shape[0], b.shape[0], c.shape[0]))
for ia in range(len(a)):
for ib in range(len(b)):
for ic in range(len(c)):
res[ia, ib, ic] = a[ia] * b[ib] * c[ic]
print(res)
out:
[[[ 0. 0. 0. 0. 0.]
[ 0. 0. 0. 0. 0.]
[ 0. 0. 0. 0. 0.]
[ 0. 0. 0. 0. 0.]
[ 0. 0. 0. 0. 0.]]
[[ 0. 0. 0. 0. 0.]
[ 0. 1. 2. 3. 4.]
[ 0. 2. 4. 6. 8.]
[ 0. 3. 6. 9. 12.]
[ 0. 4. 8. 12. 16.]]
[[ 0. 0. 0. 0. 0.]
[ 0. 2. 4. 6. 8.]
[ 0. 4. 8. 12. 16.]
[ 0. 6. 12. 18. 24.]
[ 0. 8. 16. 24. 32.]]
[[ 0. 0. 0. 0. 0.]
[ 0. 3. 6. 9. 12.]
[ 0. 6. 12. 18. 24.]
[ 0. 9. 18. 27. 36.]
[ 0. 12. 24. 36. 48.]]
[[ 0. 0. 0. 0. 0.]
[ 0. 4. 8. 12. 16.]
[ 0. 8. 16. 24. 32.]
[ 0. 12. 24. 36. 48.]
[ 0. 16. 32. 48. 64.]]]
How to do this with numpy [no for loops]?
Also, how to do this for a general function, not necessarily *?
NumPy provides you with np.outer() for computing the outer product.
This is a less powerful version of more versatile approaches:
ufunc.outer()
np.tensordot()
np.einsum()
np.einsum() is the only one capable of handling more than two input arrays:
import numpy as np
def prod(items, start=1):
for item in items:
start = start * item
return start
a = np.arange(5)
b = np.arange(5)
c = np.arange(5)
r0 = np.zeros((a.shape[0], b.shape[0], c.shape[0]))
for ia in range(len(a)):
for ib in range(len(b)):
for ic in range(len(c)):
r0[ia, ib, ic] = a[ia] * b[ib] * c[ic]
r1 = prod([a[:, None, None], b[None, :, None], c[None, None, :]])
# same as: r1 = a[:, None, None] * b[None, :, None] * c[None, None, :]
# same as: r1 = a.reshape(-1, 1, 1) * b.reshape(1, -1, 1) * c.reshape(1, 1, -1)
print(np.all(r0 == r2))
# True
r2 = np.einsum('i,j,k->ijk', a, b, c)
print(np.all(r0 == r2))
# True
# as per #hpaulj suggestion
r3 = prod(np.ix_(a, b, c))
print(np.all(r0 == r3))
# True
Of course, the broadcasting approach (which is the same that you used with the array.reshape() version of your code, except that it uses a slightly different syntax for providing the correct shape), can be automatized by explicitly building the slicing (or equivalently the array.reshape() parameters).
In [166]: a = np.arange(2)
...: b = np.arange(3)
...: c = np.arange(4)
As shown in comments and answer:
In [167]: R = np.einsum('i,j,k',a,b,c)
We can also np.ix_ construct arrays that broadcast against each other. This is often used to construct block indexing arrays, but works here as well:
In [168]: A,B,C = np.ix_(a,b,c)
In [169]: A,B,C
Out[169]:
(array([[[0]],
[[1]]]),
array([[[0],
[1],
[2]]]),
array([[[0, 1, 2, 3]]]))
In [170]: R1 = A*B*C
testing:
In [171]: np.allclose(R,R1)
Out[171]: True
That broadcasted product can be done in one line with:
In [172]: np.prod(np.array(np.ix_(a,b,c),object)).shape
Out[172]: (2, 3, 4)
Without that explicit object dtype casting I get a future warning about creating an ragged array.
np.meshgrid(a,b,c, sparse=True, indexing='ij') is an alternative to ix_.
While these ix_ etc expressions are nice, you should become thoroughly comfortable using:
a[:, None, None] * b[None, :, None] * c[None, None, :]
This kind of dimension expansion gives you the most power and flexibility.
The simplest approach (from this answer) is to use:
functools.reduce(np.multiply.outer, (a, b, c))
This works for any number of dimensions, and unlike np.prod(np.ix_(...)) it does not result in numpy deprecation warnings about introducing jagged arrays.
I am new in numpy, and I am having troubles with simple managment of numpy arrays.
I am doing a task in which it said that loops has to be avoid as much as possible, and I need to edit the values of an array through another array of indexes.
indexes # [3, 16]
y # [0. 1. 1. 1. 0. 1. 0. 0. 0. 0. 1. 1. 1. 0. 1. 0. 0. 0. 1. 1.]
y[indexes] = 2 # [0. 1. 1. 2. 0. 1. 0. 0. 0. 0. 1. 1. 1. 0. 1. 0. 2. 0. 1. 1.]
But I don't need change the value simply by 2. I need make a conditional change. This what I have got, but I would need something like
y[indexes] = 0 if y[indexes] == 1 else 0
>>> [0. 1. 1. 0. 0. 1. 0. 0. 0. 0. 1. 1. 1. 0. 1. 0. 1. 0. 1. 1.]
And the line above should be the results.
This is the loop way answer, but I need a numpy way if exists:
for index in indexes:
y[index] = 1 if y[index] == 0 else 0
Thanks in advance.
I don't know if I understood your question. But I hope this helps you.
tip 01
import numpy as np
indexes = [1, 5, 7] # index list
y = np.array([9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]) #array example
y[indexes][2] #3rd(0,1,>>2<<) item of y array (1,5,>>7<<).
In this case it is y[7] equal 16.
tip 02
This can also be useful.
y = np.array([0,1,1,0,3,0,1,0,1,0])
y
array([0, 1, 1, 0, 3, 0, 1, 0, 1, 0])
np.where(y != 1, y, 0)
y
array([0, 0, 0, 0, 3, 0, 0, 0, 0, 0])
In python I would like to build a matrix from four 2d numpy arrays
m = np.eye(3, 3)
c = np.random.rand(2, 3)
cT = c.T
z = np.zeros([min(np.shape(c)), min(np.shape(c))])
and the new matrix shape is defined as:
[[m, cT],
[c, z]]
or like this (with numerical data):
1. 0. 0. 0.0109 0.5339
0. 1. 0. 0.4991 0.9854
0. 0. 1. 0.5942 0.7565
0.0109 0.4991 0.5942 0. 0.
0.5339 0.9854 0.7565 0. 0.
I would like to ask you what would be the easiest was and also the quickest (CPU-wise) in python using numpy
The most straightforward way is to copy each piece of data across to the appropriate slice
>>> m = np.eye(3, 3)
>>> c = np.random.rand(2, 3)
>>> cT = c.T
>>> z = np.empty([min(np.shape(c)), min(np.shape(c))])
>>> X = np.eye(5, 5)
>>> X[:3, :3] = m
>>> X[:3, -2:] = c.T
>>> X[-2:, :3] = c
>>> X[-2:, -2:] = z
>>> X
array([[ 1. , 0. , 0. , 0.98834141, 0.69806125],
[ 0. , 1. , 0. , 0.97342311, 0.97368278],
[ 0. , 0. , 1. , 0.28701318, 0.08705423],
[ 0.98834141, 0.97342311, 0.28701318, 0. , 0. ],
[ 0.69806125, 0.97368278, 0.08705423, 0. , 0. ]])
>>>
Combining vstack and hstack can do this:
from numpy import ones, hstack, vstack
a, b, c, d = ones((3,3)), 2*ones((3,2)), 3*ones((2,3)), 4*ones((2,2))
x = hstack(( vstack((a, c)), vstack((b, d)) ))
[[ 1. 1. 1. 2. 2.]
[ 1. 1. 1. 2. 2.]
[ 1. 1. 1. 2. 2.]
[ 3. 3. 3. 4. 4.]
[ 3. 3. 3. 4. 4.]]