Given a network with two or more placeholders of varying dimensionality e.g.
x1 = tf.placeholder(tf.int32, [None, seq_len])
x2 = tf.placeholder(tf.int32, [None, seq_len])
xn = tf.placeholder(tf.int32, [None, None, seq_len]
The first dimension in each placeholder corresponds to the minibatch size. seq_len is the length of the inputs. The second dimension is like a list of inputs that I need to process together with x1 and x2 for each index in the minibatch. How can I group these tensors to operate on them by batch index?
For example
x1 = [[1, 2, 3], [4, 5, 6]]
x2 = [[7, 8, 9], [8, 7, 6]]
xn = [[[1, 5, 2], [7, 2, 8], [3, 2, 5]], [[8, 9, 8]]]
I need to keep x1[0] i.e. [1, 2, 3], x2[0] i.e. [7, 8, 9], and xn[0] i.e. [[1, 5, 2], [7, 2, 8], [3, 2, 5]] together, because I need to perform matrix operations between x1[i] and each element in xn[i] for all i.
Notice that the dimensionality of xn is jagged.
Still not sure if I understand your question. If I understand correctly, your challenge comes from the jagged nature of the dimensionality of xn. I have the below way to "unrolling" along batch index. The result is an array with a size of batch_size; each element in the array is a Tensor. Of course you can perform other operations for all these individual tensors before evaluating them.
I have to use tf.scan to perform the operation for each element of xn[i] because its first dimension is dynamic. There might exist better solutions though.
x1 = np.array([[1, 2, 3]])
xn = np.array([[[1, 5, 2], [7, 2, 8], [3, 2, 5]]])
batch_size = x1.shape[0]
result = []
for batch_idx in range(batch_size):
x1_i = x1[batch_idx]
xn_i = xn[batch_idx]
result.append(tf.scan(fn=lambda a, x: x * x1_i, elems=xn_i, initializer=x1_i))
with tf.Session() as sess:
print sess.run([result[0]])
# result, this is x1[0] multiply each element in xn[0] for all i (element-wise).
# free free to plug in your own matrix operations in the `fn` arg of `tf.scan`.
[array([[ 1, 10, 6],
[ 7, 4, 24],
[ 3, 4, 15]])]
Related
I have a 3D array that I like to repeat 4 times.
Achieved via a mixture of Numpy and Python methods:
>>> z = np.arange(9).reshape(3,3)
>>> z
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
>>> z2 = []
>>> for i in range(4):
z2.append(z)
>>> z2
[array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]]), array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]]), array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]]), array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])]
>>> z2 = np.array(z2)
>>> z2
array([[[0, 1, 2],
[3, 4, 5],
[6, 7, 8]],
[[0, 1, 2],
[3, 4, 5],
[6, 7, 8]],
[[0, 1, 2],
[3, 4, 5],
[6, 7, 8]],
[[0, 1, 2],
[3, 4, 5],
[6, 7, 8]]])
Achieved via Pure NumPy:
>>> z2 = np.repeat(z[np.newaxis,...], 4, axis=0)
>>> z2
array([[[0, 1, 2],
[3, 4, 5],
[6, 7, 8]],
[[0, 1, 2],
[3, 4, 5],
[6, 7, 8]],
[[0, 1, 2],
[3, 4, 5],
[6, 7, 8]],
[[0, 1, 2],
[3, 4, 5],
[6, 7, 8]]])
Are the elements created by numpy.repeat() views of the original numpy.array() or unique elements?
If the latter, is there an equivalent NumPy functions that can create views of the original array the same way as numpy.repeat()?
I think such an ability can help reduce the buffer space of z2 in the event size of z is large and when there are many repeats of z involved.
A follow-up on one of #FrankYellin answer:
>>> z = np.arange(9).reshape(3,3)
>>> z
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
>>> z2 = np.repeat(z[np.newaxis,...], 1_000_000_000, axis=0)
>>> z2.nbytes
72000000000
>>> y2 = np.broadcast_to(z, (1_000_000_000, 3, 3))
>>> y2.nbytes
72000000000
The nbytes from using np.broadcast_to() is the same as np.repeat(). This is surprising given that the former returns a readonly view on the original z array with the given shape. Having said this, I did notice that np.broadcast_to() created the y2 array instantaneously, while the creation of z2 via np.repeat() took abt 40 seconds to complete. Hence,np.broadcast_to() yielded significantly faster performance.
If you want a writable version, it is doable, but it's really ugly.
If you want a read-only version, np.broadcast_to(z, (4, 3, 3)) should be all you need.
Now the ugly writable version. Be careful. You can corrupt memory if you mess the arguments up.
> z.shape
(3, 3)
> z.strides
(24, 8)
from numpy.lib.stride_tricks import as_strided
z2 = as_strided(z, shape=(4, 3, 3), strides=(0, 24, 8))
and you end up with:
>>> z2[1, 1, 1]
4
>>> z2[1, 1, 1] = 100
>>> z2[2, 1, 1]
100
>>>
You are using strides to say that I want to create a second array overlayed on top of the first array. You set its new shape, and you prefix 0 to the previous stride, indicating that the first dimension has no effect on the data you want.
Make sure you understand strides.
numpy.repeat creates a new array and not a view (you can check it by looking the __array_interface__ field). In fact, it is not possible to create a view on the original array in the general case since Numpy views does not support such pattern. A views is basically just an object containing a pointer to a raw memory buffer, some strides, a shape and a type. While it is possible to repeat one item N times with a 0 stride, it is not possible to repeat 2 items N times (without adding a new dimension to the output array). Thus, no there is no way to build a function like numpy.repeat having the same array output shape to repeat items of the last axis. If adding a new dimension is Ok, then you can build an array with a new dimension and a stride set to 0. Repeating the last dimension is possible though. The answer of #FrankYellin gives a good example. Note that reshaping/ravel the resulting array cause a mandatory copy. Supporting such advanced views would make the Numpy code more complex or/and less efficient for a feature that is only used rarely by users.
I have an arbitrarily nested ragged tensor x I need to perform masking on. Something like:
x = tf.ragged.constant([
[[12, 9], [5]],
[[10], [6, 8], [42]],
])
The easiest way for me to mask will be by index of an element along the 1st axis. Is there a way to get a ragged arange with the same row lengths/splits like:
x = tf.ragged.constant([
[[0, 1], [2]],
[[0], [1, 2], [3]],
])
Try this code:
import tensorflow as tf
x = tf.ragged.constant([
[[12, 9], [5]],
[[10], [6, 8], [42]],
])
starts = tf.gather(x.nested_row_splits[1], x.nested_row_splits[0])[1:-1]
starts = tf.cast(starts, tf.int32)
len = tf.shape(x.flat_values)[0]
starts = tf.scatter_nd(starts[:,tf.newaxis], starts, [len])
starts = tf.scan(lambda a, x: a + x, starts)
output = tf.range(len) - starts
x = tf.RaggedTensor.from_nested_row_splits(output, x.nested_row_splits)
print(x)
I managed to solve this by merging the n-d tensor up to rank2, calculating a ragged range over that, and then essentially reshaping it from the original nested row lengths:
x = tf.ragged.constant([
[[12, 9], [5]],
[[10], [6, 8], [42]],
])
x_2d = x.merge_dims(inner_axis=-1, outer_axis=1)
arange_2d = tf.ragged.range(x_2d.row_lengths())
arange_nd = tf.RaggedTensor.from_nested_row_lengths(
arange_2d.flat_values,
x.nested_row_lengths(),
)
>>> arange_nd
<tf.RaggedTensor [[[0, 1], [2]], [[0], [1, 2], [3]]]>
See this issue for an alternative solution from one of the Tensorflow maintainers.
Say I have one matrix and one vector as follows:
import torch
x = torch.tensor([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
y = torch.tensor([0, 2, 1])
is there a way to slice it x[y] so the result is:
res = [1, 6, 8]
So basically I take the first element of y and take the element in x that corresponds to the first row and the elements' column.
You can specify the corresponding row index as:
import torch
x = torch.tensor([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
y = torch.tensor([0, 2, 1])
x[range(x.shape[0]), y]
tensor([1, 6, 8])
Advanced indexing in pytorch works just as NumPy's, i.e the indexing arrays are broadcast together across the axes. So you could do as in FBruzzesi's answer.
Though similarly to np.take_along_axis, in pytorch you also have torch.gather, to take values along a specific axis:
x.gather(1, y.view(-1,1)).view(-1)
# tensor([1, 6, 8])
I have a really big numpy array(145000 rows * 550 cols). And I wanted to create rolling slices within subarrays. I tried to implement it with a function. The function lagged_vals behaves as expected but np.lib.stride_tricks does not behave the way I want it to -
def lagged_vals(series,l):
# Garbage implementation but still right
return np.concatenate([[x[i:i+l] for i in range(x.shape[0]) if i+l <= x.shape[0]] for x in series]
,axis = 0)
# Sample 2D numpy array
something = np.array([[1,2,2,3],[2,2,3,3]])
lagged_vals(something,2) # Works as expected
# array([[1, 2],
# [2, 2],
# [2, 3],
# [2, 2],
# [2, 3],
# [3, 3]])
np.lib.stride_tricks.as_strided(something,
(something.shape[0]*something.shape[1],2),
(8,8))
# array([[1, 2],
# [2, 2],
# [2, 3],
# [3, 2], <--- across subarray stride, which I do not want
# [2, 2],
# [2, 3],
# [3, 3])
How do I remove that particular row in the np.lib.stride_tricks implementation? And how can I scale this cross array stride removal for a big numpy array ?
Sure, that's possible with np.lib.stride_tricks.as_strided. Here's one way -
from numpy.lib.stride_tricks import as_strided
L = 2 # window length
shp = a.shape
strd = a.strides
out_shp = shp[0],shp[1]-L+1,L
out_strd = strd + (strd[1],)
out = as_strided(a, out_shp, out_strd).reshape(-1,L)
Sample input, output -
In [177]: a
Out[177]:
array([[0, 1, 2, 3],
[4, 5, 6, 7]])
In [178]: out
Out[178]:
array([[0, 1],
[1, 2],
[2, 3],
[4, 5],
[5, 6],
[6, 7]])
Note that the last step of reshaping forces it to make a copy there. But that's can't be avoided if we need the final output to be a 2D. If we are okay with a 3D output, skip that reshape and thus achieve a view, as shown with the sample case -
In [181]: np.shares_memory(a, out)
Out[181]: False
In [182]: as_strided(a, out_shp, out_strd)
Out[182]:
array([[[0, 1],
[1, 2],
[2, 3]],
[[4, 5],
[5, 6],
[6, 7]]])
In [183]: np.shares_memory(a, as_strided(a, out_shp, out_strd) )
Out[183]: True
I have a tensor defined as follows:
temp_var = tf.Variable(initial_value=np.asarray([[1, 2, 3],[4, 5, 6],[7, 8, 9],[10, 11, 12]]))
I also have an array of indexes of rows to be fetched from tensor:
idx = tf.constant([0, 2])
Now I want to take a subset of temp_var at those indexes i.e. idx
I know that to take a single index or a slice, we can do something like
temp_var[single_row_index, :]
or
temp_var[start:end, :]
But how to fetch rows indicated by idx array?
Something like temp_var[idx, :] ?
The tf.gather() op does exactly what you need: it selects rows from a matrix (or in general (N-1)-dimensional slices from an N-dimensional tensor). Here's how it would work in your case:
temp_var = tf.Variable([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]))
idx = tf.constant([0, 2])
rows = tf.gather(temp_var, idx)
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
print(sess.run(rows)) # ==> [[1, 2, 3], [7, 8, 9]]