Related
I have two NumPy arrays that I would like to multiply with each other across every row. To illustrate what I mean I have put the code below:
import numpy as np
a = np.array([
[1,2],
[3,4],
[5,6],
[7,8]])
b = np.array([
[1,2],
[4,4],
[5,5],
[7,10]])
final_product=[]
for i in range(0,b.shape[0]):
product=a[i,:]*b
final_product.append(product)
Rather than using loops and lists, is there more direct, faster and elegant way of doing the above row-wise multiplication in NumPy?
By using proper reshaping and repetition you can achieve what you are looking for, here is a simple implementation:
a.reshape(4,1,2) * ([b]*4)
If the length is dynamic you can do this:
a.reshape(a.shape[0],1,a.shape[1]) * ([b]*a.shape[0])
Note : Make sure a.shape[1] and b.shape[1] remains equal, while a.shape[0] and b.shape[0] can differ.
This type of problems can be handled by np.einsum(see Doc & this post) for more understanding. It is one of the most efficient ways in this regard:
np.einsum("ij, kj->ikj", a, b)
Try:
n = b.shape[0]
print(np.multiply(np.repeat(a, n, axis=0).reshape((a.shape[0], n, -1)), b))
Prints:
[[[ 1 4]
[ 4 8]
[ 5 10]
[ 7 20]]
[[ 3 8]
[12 16]
[15 20]
[21 40]]
[[ 5 12]
[20 24]
[25 30]
[35 60]]
[[ 7 16]
[28 32]
[35 40]
[49 80]]]
I had written a python program to sort a two-dimensional array using the second column and if elements in the second column are the same sort by the first column. Though I solved the problem with my rudimentary python knowledge.
I think it can be improved. Can anyone help optimizing it?
Please also suggest if using other data types for sorting will be good option?
#created a two dimensional array
two_dim_array=[[2, 5], [9, 1], [4, 8], [10, 0], [50, 32], [33, 31],[1, 5], [12, 5], [22, 5], [32, 5], [9, 5],[3, 31], [91, 32] ]
#get the length of the array
n_ship=len(two_dim_array)
#sorting two dimensional array by using second column
sort_by_second_column=sorted(two_dim_array, key=lambda x: x[1], reverse=False)
#declared new variable for storing second soeted array
new_final_data=[]
#parameter used to slice the two dimensional column
first_slice=0
#tmp=[]
index=[0]
for m in range(1, n_ship):
#print('m is: '+str(m)+'final_data[m-1][1] is: '+str(final_data[m-1][1])+'final_data[m][1] is: '+str(final_data[m][1]))
#subtracting second column elements to detect changes and saved to array
if(abs(sort_by_second_column[m-1][1]-sort_by_second_column[m][1])!=0):
index.append(m)
# print(index)
l=1
# used the above generated index to slice the data
for z in range(len(index)):
tmp=[]
if(l==1):
first_slice=0
last=index[z+1]
mid_start=index[z]
# print('l is start'+ 'first is '+str(first_slice)+'last is'+str(last))
v=sort_by_second_column[:last]
elif l==len(index):
first_slice=index[z]
# print('l is last'+str(1)+ 'first is '+str(first_slice)+'last is'+str(last))
v=sort_by_second_column[first_slice:]
else:
first_slice=index[z]
last=index[z+1]
#print('l is middle'+str(1)+ 'first is '+str(first_slice)+'last is'+str(last))
v=sort_by_second_column[first_slice:last]
tmp.extend(v)
tmp=sorted(tmp, key=lambda x: x[0], reverse=False)
#print(tmp)
new_final_data.extend(tmp)
# print(new_final_data)
l+=1
for l in range(n_ship):
print(str(new_final_data[l][0])+' '+str(new_final_data[l][1]))
''' Input
2 5
9 1
4 8
10 0
50 32
33 31
1 5
12 5
22 5
32 5
9 5
3 31
91 32
Output
10 0
9 1
1 5
2 5
9 5
12 5
22 5
32 5
4 8
3 31
33 31
50 32
91 32'''
You should read the documentation on sorted(), as this is exactly what you need to use:
https://docs.python.org/3/library/functions.html#sorted
newarray=sorted(two_dim_array, key=lambda x:(x[1],x[0]))
Outputs:
[10, 0]
[9, 1]
[1, 5]
[2, 5]
[9, 5]
[12, 5]
[22, 5]
[32, 5]
[4, 8]
[3, 31]
[33, 31]
[50, 32]
[91, 32]
I have the following Numpy array of shape (4, 4, 3):
a = [[[ 0 1 2]
[ 3 4 5]
[ 6 7 8]
[ 9 10 11]]
[[12 13 14]
[15 16 17]
[18 19 20]
[21 22 23]]
[[24 25 26]
[27 28 29]
[30 31 32]
[33 34 35]]
[[36 37 38]
[39 40 41]
[42 43 44]
[45 46 47]]]
I am looking for an elegant solution to re-arrange the elements in that array to get the following 3D array of shape (3, 4, 4):
a_new = [[[ 0 3 6 9]
[12 15 18 21]
[24 27 30 33]
[36 39 42 45]]
[[ 1 4 7 10]
[13 16 19 22]
[25 28 31 34]
[37 40 43 46]]
[[ 2 5 8 11]
[14 17 20 23]
[26 29 32 35]
[38 41 44 47]]]
Use np.transpose -
a.transpose(2,0,1)
Or use np.rollaxis -
np.rollaxis(a,2,0) # Or np.rollaxis(a,-1,0)
In case somebody asks the same question for pure Python:
mylist = [[[1,2,3], [4,5,6]], [[7,8,9], [10, 11, 12]]]
flat = sum(sum(mylist, []), [])
groups = 3
print [flat[r::groups] for r in range(groups)]
[[1, 4, 7, 10], [2, 5, 8, 11], [3, 6, 9, 12]]
The fastest way I can think of is to use numpy's swapaxes function in combination with the transpose function.
anew=np.swapaxes(a,0,1).T
I'm trying to create a list of lists, such that each inner list has 8 elements, in a python one-liner.
So far I have the following:
locations = [[alphabet.index(j) for j in test]]
That maps to one big list inside of a list:
[[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]]
But I want to split it up to be multiple inner lists, each 8 elements:
[[1,2,3,4,5,6,7,8],[9,10,11,12,13,14,15,16]]
Any idea how I can acheive this?
Use list slicing with range() to get the starting indexes:
In [3]: test = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
In [4]: [test[i:i+8] for i in range(0, len(test), 8)]
Out[4]: [[1, 2, 3, 4, 5, 6, 7, 8], [9, 10, 11, 12, 13, 14, 15, 16]]
As a function:
In [7]: def slicing(list_, elem_):
...: return [list_[i:i+elem_] for i in range(0, len(list_), elem_)]
In [8]: slicing(test, 8)
Out[8]: [[1, 2, 3, 4, 5, 6, 7, 8], [9, 10, 11, 12, 13, 14, 15, 16]]
Another solution could be to use NumPy
import numpy as np
data = [x for x in xrange(0, 64)]
data_split = np.array_split(np.asarray(data), 8)
Output:
for a in data_split:
print a
[0 1 2 3 4 5 6 7]
[ 8 9 10 11 12 13 14 15]
[16 17 18 19 20 21 22 23]
[24 25 26 27 28 29 30 31]
[32 33 34 35 36 37 38 39]
[40 41 42 43 44 45 46 47]
[48 49 50 51 52 53 54 55]
[56 57 58 59 60 61 62 63]
I am trying to gather slices of a tensor in terms of the last dimension for partial connection between layers. Because the output tensor's shape is [batch_size, h, w, depth], I want to select slices based on the last dimension, such as
# L is intermediate tensor
partL = L[:, :, :, [0,2,3,8]]
However, tf.gather(L, [0, 2,3,8]) seems to only work for the first dimension (right?) Can anyone tell me how to do it?
As of TensorFlow 1.3 tf.gather has an axis parameter, so the various workarounds here are no longer necessary.
https://www.tensorflow.org/versions/r1.3/api_docs/python/tf/gather
https://github.com/tensorflow/tensorflow/issues/11223
There's a tracking bug to support this use-case here: https://github.com/tensorflow/tensorflow/issues/206
For now you can:
transpose your matrix so that dimension to gather is first (transpose is expensive)
reshape your tensor into 1d (reshape is cheap) and turn your gather column indices into a list of individual element indices at linear indexing, then reshape back
use gather_nd. Will still need to turn your column indices into list of individual element indices.
With gather_nd you can now do this as follows:
cat_idx = tf.concat([tf.range(0, tf.shape(x)[0]), indices_for_dim1], axis=0)
result = tf.gather_nd(matrix, cat_idx)
Also, as reported by user Nova in a thread referenced by #Yaroslav Bulatov's:
x = tf.constant([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
idx = tf.constant([1, 0, 2])
idx_flattened = tf.range(0, x.shape[0]) * x.shape[1] + idx
y = tf.gather(tf.reshape(x, [-1]), # flatten input
idx_flattened) # use flattened indices
with tf.Session(''):
print y.eval() # [2 4 9]
The gist is flatten the tensor and use strided 1D addressing with tf.gather(...).
Yet another solution using tf.unstack(...), tf.gather(...) and tf.stack(..)
Code:
import tensorflow as tf
import numpy as np
shape = [2, 2, 2, 10]
L = np.arange(np.prod(shape))
L = np.reshape(L, shape)
indices = [0, 2, 3, 8]
axis = -1 # last dimension
def gather_axis(params, indices, axis=0):
return tf.stack(tf.unstack(tf.gather(tf.unstack(params, axis=axis), indices)), axis=axis)
print(L)
with tf.Session() as sess:
partL = sess.run(gather_axis(L, indices, axis))
print(partL)
Result:
L =
[[[[ 0 1 2 3 4 5 6 7 8 9]
[10 11 12 13 14 15 16 17 18 19]]
[[20 21 22 23 24 25 26 27 28 29]
[30 31 32 33 34 35 36 37 38 39]]]
[[[40 41 42 43 44 45 46 47 48 49]
[50 51 52 53 54 55 56 57 58 59]]
[[60 61 62 63 64 65 66 67 68 69]
[70 71 72 73 74 75 76 77 78 79]]]]
partL =
[[[[ 0 2 3 8]
[10 12 13 18]]
[[20 22 23 28]
[30 32 33 38]]]
[[[40 42 43 48]
[50 52 53 58]]
[[60 62 63 68]
[70 72 73 78]]]]
A correct version of #Andrei's answer would read
cat_idx = tf.stack([tf.range(0, tf.shape(x)[0]), indices_for_dim1], axis=1)
result = tf.gather_nd(matrix, cat_idx)
You can try this way, for instance(in most cases in NLP at the least),
The parameter is of shape [batch_size, depth] and the indices are [i, j, k, n, m] of which the length is batch_size. Then gather_nd can be helpful.
parameters = tf.constant([
[11, 12, 13],
[21, 22, 23],
[31, 32, 33],
[41, 42, 43]])
targets = tf.constant([2, 1, 0, 1])
batch_nums = tf.range(0, limit=parameters.get_shape().as_list()[0])
indices = tf.stack((batch_nums, targets), axis=1) # the axis is the dimension number
items = tf.gather_nd(parameters, indices)
# which is what we want: [13, 22, 31, 42]
This snippet first find the fist dimension through the batch_num and then fetch the item along that dimension by the target number.
Tensor doesn't have attribute shape, but get_shape() method. Below is runnable by Python 2.7
import tensorflow as tf
import numpy as np
x = tf.constant([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
idx = tf.constant([1, 0, 2])
idx_flattened = tf.range(0, x.get_shape()[0]) * x.get_shape()[1] + idx
y = tf.gather(tf.reshape(x, [-1]), # flatten input
idx_flattened) # use flattened indices
with tf.Session(''):
print y.eval() # [2 4 9]
Implementing 2. from #Yaroslav Bulatov's:
#Your indices
indices = [0, 2, 3, 8]
#Remember for final reshaping
n_indices = tf.shape(indices)[0]
flattened_L = tf.reshape(L, [-1])
#Walk strided over the flattened array
offset = tf.expand_dims(tf.range(0, tf.reduce_prod(tf.shape(L)), tf.shape(L)[-1]), 1)
flattened_indices = tf.reshape(tf.reshape(indices, [-1])+offset, [-1])
selected_rows = tf.gather(flattened_L, flattened_indices)
#Final reshape
partL = tf.reshape(selected_rows, tf.concat(0, [tf.shape(L)[:-1], [n_indices]]))
Credit to How to select rows from a 3-D Tensor in TensorFlow?