Develop Reference

Modify order of column in a 2D numpy array

I'm working with a 2D numpy array like this:
array = np.asarray([[1,1,1,1,1],
[2,2,2,2,2],
[3,3,3,3,3],
[4,4,4,4,4],
[5,5,5,5,5]])
I also have a 1D containing indexes to change the order of the columns of my first array.
So with this index array:
column_order = np.asarray([3,2,1,0,4])
I should have someting like this:
[[4 4 4 4 4]
[3 3 3 3 3]
[2 2 2 2 2]
[1 1 1 1 1]
[5 5 5 5 5]]
This is my code:
array = np.asarray([[1,1,1,1,1],
[2,2,2,2,2],
[3,3,3,3,3],
[4,4,4,4,4],
[5,5,5,5,5]])
column_order = np.asarray([3,2,1,0,4])
new_array = np.copy(array)
for idx , i in enumerate(column_order):
new_array[i] = array[idx]
print(new_array)
The problem is that it is a bit slow on large size array.
I would like to know if there is most efficient way to do this ?

How to solve: ValueError: operands could not be broadcast together with shapes (4,) (4,6)

I have to sum 2 arrays with broadcasting. This is the first:
a = [0 1 2 3]
And this is the second:
A = [[ 0 1 2 3 4 5]
[ 6 7 8 9 10 11]
[12 13 14 15 16 17]
[18 19 20 21 22 23]]
This is the code I had tried until now:
a = np.array(a)
A = np.array(A)
G = a + A
print(G)
But when I run, it throws this error: ValueError: operands could not be broadcast together with shapes (4,) (4,6)
How to solve it?

Arrays need to have compatible shapes and same number of dimensions when performing a mathematical operation. That is, you can't add two arrays of shape (4,) and (4, 6), but you can add arrays of shape (4, 1) and (4, 6).
You can add that extra dimension as follows:
a = np.array(a)
a = np.expand_dims(a, axis=-1) # Add an extra dimension in the last axis.
A = np.array(A)
G = a + A
Upon doing this and broadcasting, a will practically become
[[0 0 0 0 0 0]
[1 1 1 1 1 1]
[2 2 2 2 2 2]
[3 3 3 3 3 3]]
for the purpose of addition (the actual value of a won't change, a will still be [[0] [1] [2] [3]]; the above is the array A will be added to).

How to overwrite 2-D numpy multi times symmetrically with given index?

I'm trying to change values in matrix a with given index matrix d and matrix e.
And the matrix should always be symmetrical.
What I come up with is to overwrite the primal matrix with given index, and try to make it symmetrical, then go for another overwrite, until all the given index matrix have been gone through. It's not efficient.
But I'm stuck with how make it symmetrical.
For example:
a = np.ones([4,4],dtype=np.object) #the primal matrix
d = np.array([[1],
[2],
[0],
[0]]) #the first index matrix
a[np.arange(a.shape[0])[:,None],d] =2 #the element change to 2 with the indexes shown in d matrix
Now the result is:
a = np.array([[1 2 1 1]
[1 1 2 1]
[2 1 1 1]
[2 1 1 1]])
After making it symmetrical (if a[ i ][ j ] was selected in d matrix, a[ j ][ i ] should also be changed to 2, how to do this part).
The expected output should be :
a = np.array([[1 2 2 2]
[2 1 2 1]
[2 2 1 1]
[2 1 1 1]])
Then, for another overwrite again:
e = np.array([[0],[2],[1],[1]])
a[np.arange(a.shape[0])[:,None],e] =3
Now the result is:
a = np.array([[3 2 2 2]
[2 1 3 1]
[2 3 1 1]
[2 3 1 1]])
Make it symmetrical, (I don't know how to do this part) the final output should be : (overwrite the values if they were given 2 or 1 before)
a = np.array([[3 2 2 2]
[2 1 3 3]
[2 3 1 1]
[2 3 1 1]])
What should I do to get symmetrical matrix?
And, is there anyway to change the primal matrix a directly to get the final result? In a more efficient way?
Thanks in advance !!

You can simply switch the first and second indices and apply the change, the result would be symmetrical:
a[np.arange(a.shape[0])[:,None], d] = 2
a[d, np.arange(a.shape[0])[:,None]] = 2
output:
[[1 2 2 2]
[2 1 2 1]
[2 2 1 1]
[2 1 1 1]]
Same with any number of other changes:
a[np.arange(a.shape[0])[:,None], e] = 3
a[e, np.arange(a.shape[0])[:,None]] = 3
output:
[[3 2 2 2]
[2 1 3 3]
[2 3 1 1]
[2 3 1 1]]

Is there a way to permute a subset of a matrix?

I'm working on a way to find the lowest 1-Norm of a given Matrix using a permutation of its rows. The problem is that the permutation can't be fully random. There are 4 subsets of rows in the Matrix having a special parameter. I want to permute just the rows having this one parameter and keeping those on the same spot.
Ex. The first column defines the type of row.
A = [
1, val_11, val_12, ... #1. Row
2, val_21, val_22, ... #2. Row
2, val_31, val_32, ... #3. Row
2, val_41, val_42, ... #4. Row
1, val_51, val_52, ... #5. Row
]
So in this example I want to permute the 1. and 5. Row AND permute the 2., 3. and 4. Row keeping the Types like [1;2;2;2;1] in place.

You just have to carefully define your permutations. Fancy indexing will then do the job :
Example :
from numpy.random import randint
M0 = randint(10,size=(5,5))
after=[4,2,3,1,0]
M0 = M[after]
print(M0)
print(M)
[[4 9 3 0 0]
[3 1 7 6 0]
[6 6 5 0 9]
[0 4 7 1 3]
[0 0 1 0 6]]
[[0 0 1 0 6]
[6 6 5 0 9]
[0 4 7 1 3]
[3 1 7 6 0]
[4 9 3 0 0]]

How to reshape a vector to TensorFlow's filters?

I want to transfer some weights trained by another network to TensorFlow, the weights are stored in a single vector like this:
[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]
By using numpy, I can reshape it to two 3 by 3 filters like this:
1 2 3 9 10 11
3 4 5 12 13 14
6 7 8 15 16 17
Thus, the shape of my filters are (1,2,3,3). However, in TensorFlow, the shape of filters are (3,3,2,1):
tf_weights = tf.Variable(tf.random_normal([3,3,2,1]))
After reshaping the tf_weights to the expected shape, the weight becomes a mess and I can't get the expected convolution result.
To be specific, when the shape of an image or filter is [number,channel,size,size], I wrote a convolution function and it gives the correct answer,but it's too slow:
def convol(images,weights,biases,stride):
"""
Args:
images:input images or features, 4-D tensor
weights:weights, 4-D tensor
biases:biases, 1-D tensor
stride:stride, a float number
Returns:
conv_feature: convolved feature map
"""
image_num = images.shape[0] #the number of input images or feature maps
channel = images.shape[1] #channels of an image,images's shape should be like [n,c,h,w]
weight_num = weights.shape[0] #number of weights, weights' shape should be like [n,c,size,size]
ksize = weights.shape[2]
h = images.shape[2]
w = images.shape[3]
out_h = (h+np.floor(ksize/2)*2-ksize)/2+1
out_w = out_h
conv_features = np.zeros([image_num,weight_num,out_h,out_w])
for i in range(image_num):
image = images[i,...,...,...]
for j in range(weight_num):
sum_convol_feature = np.zeros([out_h,out_w])
for c in range(channel):
#extract a single channel image
channel_image = image[c,...,...]
#pad the image
padded_image = im_pad(channel_image,ksize/2)
#transform this image to a vector
im_col = im2col(padded_image,ksize,stride)
weight = weights[j,c,...,...]
weight_col = np.reshape(weight,[-1])
mul = np.dot(im_col,weight_col)
convol_feature = np.reshape(mul,[out_h,out_w])
sum_convol_feature = sum_convol_feature + convol_feature
conv_features[i,j,...,...] = sum_convol_feature + biases[j]
return conv_features
Instead, by using tensorflow's conv2d like this:
img = np.zeros([1,3,224,224])
img = img - 1
img = np.rollaxis(img, 1, 4)
weight_array = googleNet.layers[1].weights
weight_array = np.reshape(weight_array,[64,3,7,7])
biases_array = googleNet.layers[1].biases
tf_weight = tf.Variable(weight_array)
tf_img = tf.Variable(img)
tf_img = tf.cast(tf_img,tf.float32)
tf_biases = tf.Variable(biases_array)
conv_feature = tf.nn.bias_add(tf.nn.conv2d(tf_img,tf_weight,strides=[1,2,2,1],padding='SAME'),tf_biases)
sess = tf.Session()
sess.run(tf.initialize_all_variables())
feautre = sess.run(conv_feature)
The feature map I got is wrong.

Don't use np.reshape. It might mess up the order of your values.
Use np.rollaxis instead:
>>> a = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18])
>>> a = a.reshape((1,2,3,3))
>>> a
array([[[[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9]],
[[10, 11, 12],
[13, 14, 15],
[16, 17, 18]]]])
>>> b = np.rollaxis(a, 1, 4)
>>> b.shape
(1, 3, 3, 2)
>>> b = np.rollaxis(b, 0, 4)
>>> b.shape
(3, 3, 2, 1)
Note that the order of the two axes with size 3 haven't changed. If I were to label them, the two rollaxis operations caused the shapes to change as (1, 2, 31, 32) -> (1, 31, 32, 2) -> (31, 32, 2, 1). Your final array looks like:
>>> b
array([[[[ 1],
[10]],
[[ 2],
[11]],
[[ 3],
[12]]],
[[[ 4],
[13]],
[[ 5],
[14]],
[[ 6],
[15]]],
[[[ 7],
[16]],
[[ 8],
[17]],
[[ 9],
[18]]]])

Sample Tensor Manipulations
I dont know if this might be of help. Consider the Reshape ,Gather, Dynamic_partition and Split operations and adapt this to your needs.
In what comes below is the illustration of these operations that can be adapted to use in your situation. I copied this from my git repo. I will believe if you run this examples in ipython you can figure out what you really want and get even better insight.
Reshape ,Gather, Dynamic_partition and Split
Gather Operation ( tf.gather( ) )
Generate an array and test the gather operation. Note this approach for fast prototyping:
We generate an array in Numpy and test the operations of tensor flow on it.
Use: Gather slices from params according to indices.
indices must be an integer tensor of any dimension (usually 0-D or 1-D). This is best illustrated by an example:
array = np.array([[1,2,3],[4,9,6],[2,3,4],[7,8,0]])
array.shape
(4, 3)
In [27]:
gather_output0 = tf.gather(array,1)
gather_output01 = tf.gather(array,2)
gather_output02 = tf.gather(array,3)
gather_output11 = tf.gather(array,[1,2])
gather_output12 = tf.gather(array,[1,3])
gather_output13 = tf.gather(array,[3,2])
gather_output = tf.gather(array,[1,0,2])
gather_output1 = tf.gather(array,[1,1,2])
gather_output2 = tf.gather(array,[1,2,1])
In [28]:
with tf.Session() as sess:
print (gather_output0.eval());print("\n")
print (gather_output01.eval());print("\n")
print (gather_output02.eval());print("\n")
print (gather_output11.eval());print("\n")
print (gather_output12.eval());print("\n")
print (gather_output13.eval());print("\n")
print (gather_output.eval());print("\n")
print (gather_output1.eval());print("\n")
print (gather_output2.eval());print("\n")
#print (gather_output2.eval());print("\n")
[4 9 6]
[2 3 4]
[7 8 0]
[[4 9 6]
[2 3 4]]
[[4 9 6]
[7 8 0]]
[[7 8 0]
[2 3 4]]
[[4 9 6]
[1 2 3]
[2 3 4]]
[[4 9 6]
[4 9 6]
[2 3 4]]
[[4 9 6]
[2 3 4]
[4 9 6]]
And looking at this simple example:
Initialise simple array
test gather operation
In [11]:
array_simple = np.array([1,2,3])
In [15]:
print "shape of simple array is: ", array_simple.shape
shape of simple array is: (3,)
In [57]:
gather1 = tf.gather(array1,[0])
gather01 = tf.gather(array1,[1])
gather02 = tf.gather(array1,[2])
gather2 = tf.gather(array1,[1,2])
gather3 = tf.gather(array1,[0,1])
with tf.Session() as sess:
print (gather1.eval());print("\n")
print (gather01.eval());print("\n")
print (gather02.eval());print("\n")
print (gather2.eval());print("\n")
print (gather3.eval());print("\n")
[1]
[2]
[3]
[2 3]
[1 2]
tf.reshape( )
Note:
* Use the same array that was initiated
* Do reshape using tf.reshape( )
In [64]:
array.shape # Confirm array shape
Out[64]:
(4, 3)
In [74]:
print ("This is the array\n" ,array) # see the output and compare with the initial array,
This is the array
[[1 2 3]
[4 9 6]
[2 3 4]
[7 8 0]]
In [84]:
reshape_ops= tf.reshape(array,[-1,4]) # Note the parameters in reshpe
reshape_ops1= tf.reshape(array,[-1,3]) # Note the parameters in reshpe
reshape_ops2= tf.reshape(array,[-1,6]) # Note the parameters in reshpe
reshape_ops_back1= tf.reshape(array,[6,-1]) # Note the parameters in reshpe
reshape_ops_back2= tf.reshape(array,[3,-1]) # Note the parameters in reshpe
reshape_ops_back3= tf.reshape(array,[4,-1]) # Note the parameters in reshpe
In [86]:
with tf.Session() as sess:
print(reshape_ops.eval());print("\n")
print(reshape_ops1.eval());print("\n")
print(reshape_ops2.eval());print("\n")
print ("Output when we reverse the parameters:");print("\n")
print(reshape_ops_back1.eval());print("\n")
print(reshape_ops_back2.eval());print("\n")
print(reshape_ops_back3.eval());print("\n")
[[1 2 3 4]
[9 6 2 3]
[4 7 8 0]]
[[1 2 3]
[4 9 6]
[2 3 4]
[7 8 0]]
[[1 2 3 4 9 6]
[2 3 4 7 8 0]]
Output when we reverse the parameters:
[[1 2]
[3 4]
[9 6]
[2 3]
[4 7]
[8 0]]
[[1 2 3 4]
[9 6 2 3]
[4 7 8 0]]
[[1 2 3]
[4 9 6]
[2 3 4]
[7 8 0]]
Note: The input size and output size must be the same. ---otherwise it gives error. Simple way to check this out is to make sure the input can be paritioned into the the reshape parameters by doing simple multiplications.
Dynamic_cell_partitions
This is declared as :
tf.dynamic_partition (array, partitions, num_partitions, name=None)
Note:
* we decalare number_partitions --- number of partitions
* Use our array initialised earlier
* We declare the partition as [0 1 0 1] . This signifies the partitions we want 0's fall to one partition and 1 the other partitions given that we have two num_partitions=2.
* The output is a list
In [96]:
print ("This is the array\n" ,array) # This is output array
This is the array
[[1 2 3]
[4 9 6]
[2 3 4]
[7 8 0]]
We show how to make two and three partitions below
In [123]:
num_partitions = 2
num_partitions1 = 3
partitions = [0, 0, 1, 1]
partitions1 = [0 ,1 ,1, 2 ]
In [119]:
dynamic_ops =tf.dynamic_partition(array, partitions, num_partitions, name=None) # 2 partitions
dynamic_ops1 =tf.dynamic_partition(array, partitions1, num_partitions1, name=None) # 3 partitions
In [125]:
with tf.Session() as sess:
run = sess.run(dynamic_ops)
run1 = sess.run(dynamic_ops1)
print("Output for 2 partitions: ")
print (run[0]);print("\n")
print(run[1]) ;print("\n")# Compare result with initial array. Out is list
print("Output for three partitions: ")
print (run1[0]);print("\n")
print (run1[1]);print("\n")
print (run1[2]);print("\n")
Output for 2 partitions:
[[1 2 3]
[4 9 6]]
[[2 3 4]
[7 8 0]]
Output for three partitions:
[[1 2 3]]
[[4 9 6]
[2 3 4]]
[[7 8 0]]
tf.split( )
Make sure you use an up to date tensorflow version. Otherwise in older versions, this implemetation will give error
This is specified in the documentation as below:
tf.split(value, num_or_size_splits, axis=0, num=None, name='split').
It splits a tensor into subtensors. This is best illustrated by an example:
* we define (5,30) aray in numpy
* we split the array along axis 1
* We specify the number of splits as 1-Dimen Tensor along axis 1. So we have 3 splits.
Specify an array
Create a (5 by 30) numpy array. The syntax using numpy is shown below
In [2]:
ArrayBeforeSplitting = np.arange(150).reshape(5,30)
print ("Array shape without split operation is : " ,ArrayBeforeSplitting.shape)
('Array shape without split operation is : ', (5, 30))
specify number of splits
In [3]:
split_1D = tf.Variable([8,13,9])
print("specify number of partions using 1-Dimen Variable:" , tf.shape(split_1D))
('specify number of partions using 1-Dimen Variable:', <tf.Tensor 'Shape:0' shape=(1,) dtype=int32>)
Use tf.split
Make 3 splits aong y axis so that we have (5,8) ,(5,13),(5,9) splits. The axis 1 add up to give 30-- we can see axis 1 has 30 elements so the partition along that axis should add up to 30 otherwise it gives error.
In [6]:
split1,split2,split3 = tf.split(ArrayBeforeSplitting,split_1D,1)
# we have 3 splits along axis 1 specified spcifically
# by the split_1D . That is split axis 1D (with 30 elements) into partions with 8 ,13, and 9 elements while the x axis
#remains constant
In [7]:
#INitialise global variables. because split_ID is a variable and needs to be initialised before being
#used in a computational graph
init_op = tf.global_variables_initializer()
In [16]:
with tf.Session() as sess:
sess.run(init_op) # run variable initialisation.
result=split1.eval();print("\n")
print(result)
print("the shape of the first split operation is : ",result.shape)
result2=split2.eval();print("\n")
print(result2)
print("the shape of the second split operation is : ",result2.shape)
result3=split3.eval();print("\n")
print(result3)
print("the shape of the third split operation is : ",result3.shape)
[[ 0 1 2 3 4 5 6 7]
[ 30 31 32 33 34 35 36 37]
[ 60 61 62 63 64 65 66 67]
[ 90 91 92 93 94 95 96 97]
[120 121 122 123 124 125 126 127]]
('the shape of the first split operation is : ', (5, 8))
[[ 8 9 10 11 12 13 14 15 16 17 18 19 20]
[ 38 39 40 41 42 43 44 45 46 47 48 49 50]
[ 68 69 70 71 72 73 74 75 76 77 78 79 80]
[ 98 99 100 101 102 103 104 105 106 107 108 109 110]
[128 129 130 131 132 133 134 135 136 137 138 139 140]]
('the shape of the second split operation is : ', (5, 13))
Hope this helps!