Develop Reference

Remove rows from tensor by matching

I'm trying to do some operation like if there is tensor in pytorch
a = torch.tensor([[1,0]
,[0,1]
,[2,0]
,[3,2]])
b = torch.tensor([[0,1]
,[2,0]])
I want to remove the rows [0,1], [2,0] which are the rows of b from a.
Is there any way to do this?
# result
a = torch.tensor([[1,0]
,[3,2]])

You could do it if the tensor shapes were broadcastable.
For a tensor a of shape (?, d) and a tensor b of shape (d,), you could write something like:
cmp = a.eq(b).all(dim=1).logical_not(), i.e. compare each d-dimensional row of a with b and give me the indices where the comparison is False.
From these you can then easily your new tensor like so:
a = a[cmp]
I doubt you'll find an elegant way of doing this when b itself contains a batch dimension; your best bet would be to write a for loop.
Full example:
>>> xs = torch.tensor([[1,0], [0,1], [2,0], [3,2]])
>>> ys = torch.tensor([[0,1],[2,0]])
>>> for y in ys:
... xs = xs[xs.eq(y).all(dim=1).logical_not()]
>>> xs
tensor([[1, 0],
[3, 2]])

You can do something like this exploiting broadcasting:
import torch
a = torch.tensor([[1, 0], [0, 1], [2, 0], [3, 2]])
b = torch.tensor([[0, 1], [2, 0]])
indices = ((a == b[:, None]).sum(axis = 2) != a.shape[1]).all(axis = 0)
print(indices)
print(a[indices])
indices =
tensor([ True, False, False, True])
a[indices] =
tensor([[1, 0],
[3, 2]])
Works for all tensors a and b of shapes m x n and p x n respectively i.e, the number of columns (a.shape[1]) must be same and you can compare among any no. of rows.

Moving some matrix b(kxk) over the matrix a(nxm) with given stride, multiplying the corresponding elements, and adding them into the new matrix

So here i have this problem.
Given 2D numpy arrays 'a' and 'b' of sizes m×n and k×k
respectively (k <= n, k <= m), 2 integers 'stride' and 'padding' and
'f' function. You need to
first pad 'a' matrix with 0s on each side,
then move 'b' over 'a' with stride 'stride', then multiply their elements by the corresponding 'b' elements,
add the resulting k * k numbers
apply the 'f' function to the result
and place them in the new matrix.
a = np.array([[1, 1, 2],
[0, 1, 3],
[1, 3, 0],
[4, 5, 2]])
b = np.array([[1, 0],
[0, 1]])
stride = 1
padding = 0
f = lambda x: x**2
print(conv(a, b, stride, padding, f))
>>[[4, 16],
[9, 1],
[36, 25]]
I don't understand how I should handle it in case if the stride is too large, for example if I set stride=2 in the example above, what will the program do? Will it take at first the [[1,1], [0,1]] then skip to the [[0,1], [1,3]], or somehow differently?
And what functions or method will be useful in this example, I already know how to pad matrices with 0s, but is there something else that could be useful?

def padding(a, padd):
matrix = np.zeros((len(a)+2*padd,len(a[0])+2*padd))
for i in range(len(a)):
for j in range(len(a[0])):
matrix[i+padd,j+padd] = a[i,j]
return matrix
def conv(a, b, stride, padd, f):
output = np.zeros((len(a)-(len(b)-1),len(b)))
c = padding(a,padd)
matrices = []
for i in range(len(output)):
column = stride - 1
for j in range(len(output[0])):
output[i,j] = np.sum(a[i:i+len(b),j+column:j+column+len(b)] * b)
return f(output)
a = np.array([[1, 1, 2],
[0, 1, 3],
[1, 3, 0],
[4, 5, 2]])
b = np.array([[1, 0],
[0, 1]])
stride = 1
pad = 0
f = lambda x: x**2
print(conv(a, b, stride, pad, f))
Hello from 1991 )

How to remove non-symmetric pairs in a numpy array?

Given a numpy Nx2 numpy array data of ints (we can assume that data has no duplicate rows), I need to keep only the rows whose elements satisfy the relationship
(data[i,0] == data[j,1]) & (data[i,1] == data[j,0])
For instance with
import numpy as np
data = np.array([[1, 2],
[2, 1],
[7, 3],
[6, 6],
[5, 6]])
I should return
array([[1, 2], # because 2,1 is present
[2, 1], # because 1,2 is present
[6, 6]]) # because 6,6 is present
One verbose way to do this is
def filter_symmetric_pairs(data):
result = np.empty((0,2))
for i in range(len(data)):
for j in range(len(data)):
if (data[i,0] == data[j,1]) & (data[i,1] == data[j,0]):
result = np.vstack([result, data[i,:]])
return result
and I came up with a more concise:
def filter_symmetric_pairs(data):
return data[[row.tolist() in data[:,::-1].tolist() for row in data]]
Can somebody suggest a better numpy idiom?

Here are a couple of different methods you may use to do that. The first one is the "obvious" quadratic solution, which is simple but may give you trouble if you have a big input array. The second one should work as long as you don't have a huge range of numbers in the input, and it has the advantage of working with a linear amount of memory.
import numpy as np
# Input data
data = np.array([[1, 2],
[2, 1],
[7, 3],
[6, 6],
[5, 6]])
# Method 1 (quadratic memory)
d0, d1 = data[:, 0, np.newaxis], data[:, 1]
# Compare all values in first column to all values in second column
c = d0 == d1
# Find where comparison matches both ways
c &= c.T
# Get matching elements
res = data[c.any(0)]
print(res)
# [[1 2]
# [2 1]
# [6 6]]
# Method 2 (linear memory)
# Convert pairs into single values
# (assumes positive values, otherwise shift first)
n = data.max() + 1
v = data[:, 0] + (n * data[:, 1])
# Symmetric values
v2 = (n * data[:, 0]) + data[:, 1]
# Find where symmetric is present
m = np.isin(v2, v)
res = data[m]
print(res)
# [[1 2]
# [2 1]
# [6 6]]

You can sort the arrays preserving the row contents using argsort for both the original and reversed arrays, then just check which rows are equal and use that as a mask for slice data.
import numpy as np
data = np.array([[1, 2],
[2, 1],
[7, 3],
[6, 6],
[5, 6]])
data_r = data[:,::-1]
sorter = data.argsort(axis=0)[:,0]
sorter_r = data_r.argsort(axis=0)[:,0]
mask = (data.take(sorter, axis=0) == data_r.take(sorter_r, axis=0)).all(axis=1)
data[mask]
# returns:
array([[1, 2],
[2, 1],
[6, 6]])

Another solution dawned on me, which sees data as the edge list of a directed graph and filters only bidirected edges (my problem is thus equivalent to detecting mutual edges in a graph):
def filter_symmetric_pairs(data):
rank = max(data.flatten() + 1)
adj = np.zeros((rank, rank))
adj[data[:,0], data[:,1]] = 1 # treat the coordinates as edges of directed graph, compute adjaciency matrix
bidirected_edges = (adj == adj.T) & (adj == 1) # impose symmetry and a nonzero value
return np.vstack(np.nonzero(bidirected_edges)).T # list indices of components satisfying the above constraint

using tf.where() to select 3d tensor by 2d conditions & replacing elements in a 2d indices with keys and values

There are 2 questions in the title. I am confused by both questions because tensorflow is such a static programming language (I really want to go back to either pytorch or chainer).
I give 2 examples. please answer me in tensorflow codes or providing the relevant function links.
1) tf.where()
data0 = tf.zeros([2, 3, 4], dtype = tf.float32)
data1 = tf.ones([2, 3, 4], dtype = tf.float32)
cond = tf.constant([[0, 1, 1], [1, 0, 0]])
# cond.shape == (2, 3)
# tf.where() works for 1d condition with 2d data,
# but not for 2d indices with 3d tensor
# currently, what I am doing is:
# cond = tf.stack([cond] * 4, 2)
data = tf.where(cond > 0, data1, data0)
# data should be [[0., 1., 1.], [1., 0., 0.]]
(I don't know how to broadcast cond to 3d tensor)
2) change element in 2d tensor
# all dtype == tf.int64
t2d = tf.Variable([[0, 1, 2], [3, 4, 5]])
k, v = tf.constant([[0, 2], [1, 0]]), tf.constant([-2, -3])
# TODO: change values at positions k to v
# I cannot do [t2d.copy()[i] = j for i, j in k, v]
t3d == [[[0, 1, -2], [3, 4, 5]],
[[0, 1, 2], [-3, 4, 5]]]
Thank you so much in advance. XD

This are two quite different questions, and they should probably have been posted as such, but anyway.
1)
Yes, you need to manually broadcast all the inputs to [tf.where](https://www.tensorflow.org/api_docs/python/tf/where] if they are different. For what is worth, there is an (old) open issue about it, but so far implicit broadcasting it has not been implemented. You can use tf.stack like you suggest, although tf.tile would probably be more obvious (and may save memory, although I'm not sure how it is implemented really):
cond = tf.tile(tf.expand_dims(cond, -1), (1, 1, 4))
Or simply with tf.broadcast_to:
cond = tf.broadcast_to(tf.expand_dims(cond, -1), tf.shape(data1))
2)
This is one way to do that:
import tensorflow as tf
t2d = tf.constant([[0, 1, 2], [3, 4, 5]])
k, v = tf.constant([[0, 2], [1, 0]]), tf.constant([-2, -3])
# Tile t2d
n = tf.shape(k)[0]
t2d_tile = tf.tile(tf.expand_dims(t2d, 0), (n, 1, 1))
# Add aditional coordinate to index
idx = tf.concat([tf.expand_dims(tf.range(n), 1), k], axis=1)
# Make updates tensor
s = tf.shape(t2d_tile)
t2d_upd = tf.scatter_nd(idx, v, s)
# Make updates mask
upd_mask = tf.scatter_nd(idx, tf.ones_like(v, dtype=tf.bool), s)
# Make final tensor
t3d = tf.where(upd_mask, t2d_upd, t2d_tile)
# Test
with tf.Session() as sess:
print(sess.run(t3d))
Output:
[[[ 0 1 -2]
[ 3 4 5]]
[[ 0 1 2]
[-3 4 5]]]

Slice numpy array of dimension N with array of dimension N-1 [duplicate]

What is the most elegant way to access an n dimensional array with an (n-1) dimensional array along a given dimension as in the dummy example
a = np.random.random_sample((3,4,4))
b = np.random.random_sample((3,4,4))
idx = np.argmax(a, axis=0)
How can I access now with idx a to get the maxima in a as if I had used a.max(axis=0)? or how to retrieve the values specified by idx in b?
I thought about using np.meshgrid but I think it is an overkill. Note that the dimension axis can be any usefull axis (0,1,2) and is not known in advance. Is there an elegant way to do this?

Make use of advanced-indexing -
m,n = a.shape[1:]
I,J = np.ogrid[:m,:n]
a_max_values = a[idx, I, J]
b_max_values = b[idx, I, J]
For the general case:
def argmax_to_max(arr, argmax, axis):
"""argmax_to_max(arr, arr.argmax(axis), axis) == arr.max(axis)"""
new_shape = list(arr.shape)
del new_shape[axis]
grid = np.ogrid[tuple(map(slice, new_shape))]
grid.insert(axis, argmax)
return arr[tuple(grid)]
Quite a bit more awkward than such a natural operation should be, unfortunately.
For indexing a n dim array with a (n-1) dim array, we could simplify it a bit to give us the grid of indices for all axes, like so -
def all_idx(idx, axis):
grid = np.ogrid[tuple(map(slice, idx.shape))]
grid.insert(axis, idx)
return tuple(grid)
Hence, use it to index into input arrays -
axis = 0
a_max_values = a[all_idx(idx, axis=axis)]
b_max_values = b[all_idx(idx, axis=axis)]

using indexing in numpy https://docs.scipy.org/doc/numpy-1.10.1/reference/arrays.indexing.html#advanced-indexing
a = np.array([[1, 2], [3, 4], [5, 6]])
a
> a: array([[1, 2],
[3, 4],
[5, 6]])
idx = a.argmax(axis=1)
idx
> array([1, 0, 0], dtype=int64)
since you want all rows but only columns with idx indexes you can use [0, 1, 2] or np.arange(a.shape[0]) for the row indexes
rows = np.arange(a.shape[0])
a[rows, idx]
>array([3, 2, 1])
which is the same as a.max(axis=1)
a.max(axis=1)
>array([3, 2, 1])
if you have 3 dimensions you add the indexes of the 3rd dimension as well:
index2 = np.arange(a.shape[2])
a[rows, idx, index2]

I suggest the following:
a = np.array([[1, 3], [2, -2], [1, -1]])
a
>array([[ 1, 3],
[ 2, -2],
[ 1, -1]])
idx = a.argmax(axis=1)
idx
> array([1, 0, 0], dtype=int64)
np.take_along_axis(a, idx[:, None], axis=1).squeeze()
>array([3, 2, 1])
a.max(axis=1)
>array([3, 2, 1])