I am preparing the input tensor for the tensorflow RNN.
Currently I am doing the following
rnn_format = list()
for each in range(batch_size):
rnn_format.append(tf.slice(input2Dpadded,[each,0],[max_steps,10]))
lstm_input = tf.stack(rnn_format)
Would it be possible to do this at once, without loop, with some tensorflow function?
As suggested by Peter Hawkins, you can use gather_nd with the appropriate indices to get there.
Your uniform cropping on the inner dimension can simply be done before the call to gather_nd.
Example:
import tensorflow as tf
import numpy as np
sess = tf.InteractiveSession()
# integer image simply because it is more readable to me
im0 = np.random.randint(10, size=(20,20))
im = tf.constant(im0)
max_steps = 3
batch_size = 10
# create the appropriate indices here
indices = (np.arange(max_steps) +
np.arange(batch_size)[:,np.newaxis])[...,np.newaxis]
# crop then call gather_nd
res = tf.gather_nd(im[:,:10], indices).eval()
# check that the resulting tensors are equal to what you had previously
for each in range(batch_size):
assert(np.all(tf.slice(im, [each,0],[max_steps,10]).eval() == res[each]))
EDIT
If your slices indices are in a tensor, you simply replace numpy's operations with tensorflow's operations when creating indices:
# indices stored in a 1D array
my_indices = tf.constant([1, 8, 3, 0, 0])
indices = (np.arange(max_steps) +
my_indices[:,tf.newaxis])[...,tf.newaxis]
Further remarks:
indices is created by taking advantage of broadcasting during the addition: arrays are virtually tiled so that their dimensions match. Broadcasting is supported by numpy and by tensorflow in a similar fashion.
Ellipsis ... is part of the standard numpy slicing notation, it basically fills all remaining dimensions left by the other slicing indices. So [..., newaxis] is basically equivalent to expand_dims(ยท, -1).
Try tf.split or tf.split_v. See here:
https://www.tensorflow.org/api_docs/python/tf/split
Does that help?
Related
I have the following TensorFlow tensors.
tensor1 = tf.constant(np.random.randint(0,255, (2,512,512,1)), dtype='int32') #All elements in range [0,255]
tensor2 = tf.constant(np.random.randint(0,255, (2,512,512,1)), dtype='int32') #All elements in range [0,255]
tensor3 = tf.keras.backend.flatten(tensor1)
tensor4 = tf.keras.backend.flatten(tensor2)
tensor5 = tf.constant(np.random.randint(0,255, (255,255)), dtype='int32') #All elements in range [0,255]
I wish to use the values stored in tensor 3 and tensor 4 to make a tuple and query the element at position given by the tuple in tensor 5. For example, let's say 0th element in tensor 3, that is tensor3[0]=5 and tensor4[0]=99.
So the tuple becomes (5,99). I wish to look up the value of element (5,99) in tensor 5. I wish to do it for all elements in Tensor3 and Tensor4 in a batch processing manner. That is I do not want to loop over all values in the range of (len(Tensor3)). I did the following to achieve this.
tensor6 = tensor5[tensor3[0],tensor4[0]]
But tensor6 has the shape (255,255) where as I was hoping to get a tensor of shape (len(tensor3),len(tensor3)). I wanted to evaluate tensor5 at all possible locations in len(tensor3). That is at (0,0),...(1000,1000),....(2000,2000),.... I am using TensorFlow version 1.12.0. How can I achieve this?
I have managed to get something working in Tensorflow v 1.12, but do let me know if it is the expected code:
import tensorflow as tf
print(tf.__version__)
import numpy as np
tensor1 = tf.constant(np.random.randint(0,255, (2,512,512,1)), dtype='int32') #All elements in range [0,255]
tensor2 = tf.constant(np.random.randint(0,255, (2,512,512,1)), dtype='int32') #All elements in range [0,255]
tensor3 = tf.keras.backend.flatten(tensor1)
tensor4 = tf.keras.backend.flatten(tensor2)
tensor5 = tf.constant(np.random.randint(0,255, (255,255)), dtype='int32') #All elements in range [0,255]
elems = (tensor3, tensor4)
a = tf.map_fn(lambda x: tensor5[x[0], x[1]], elems, dtype=tf.int32)
print(tf.Session().run(a))
Based on the comment below I'd like to add an explanation for the map_fn used in the code. Since for loops are not supported without eager_execution, map_fn is (sort of) equivalent to for loops.
A map_fn has the following parameters: operation_performed, input_arguments, optional_dtype. What happens under the hood is that a for loop is run along the length of the values in input_arguments (which must contain an iterable object) and then for each value obtained operation_performed is performed. For further clarification please refer docs.
The names given to the arguments of the function is my way of interpreting them, as I'd like understand it, and is not given in the official docs. :)
I am trying bit manipulation APIs in TensorFlow 1.15.
I've got a Tensor of bool (or of int8 with 0 or 1 values), and I want to densely pack the True or False values into bits of int32 Tensor.
For example (assuming target is int8), given Tensor of shape 16: [0,0,0,0, 0,0,0,1, 0,0,0,1, 0,0,1,0], I want to get a Tensor of shape 2: [0x01, 0x12]
How can I do that efficiently?
Does TensorFlow 2.x provide better operations for this?
The original Tensor is a result of comparison operator. Is there a way to make comparison return bit-packed Tensor directly?
P.S. right now I am reshaping the original Tensor into a 32xN, then doing bit left shift by a constant 32x1 tensor: [31..0], then sum along the first dimension.
If you are working with integers (1's and 0's) in an array.
You could try this common approach.
import tensorflow as tf # TensorFlow 2.1.0
import numpy as np
def convert_bits(storage, bit_array, bit_size):
for n_bit in range(bit_size):
storage.assign_add(tf.reverse(bit_array, axis = [-1])[n_bit]*(2**n_bit))
data = [0,1,0,0,0,0,0,1,0,0,0,0,1,0,1,0]
tf_data = tf.convert_to_tensor(data)
bit_size = 8 ## int8
np_array = np.array([])
for index in range(int(len(data)/bit_size)):
temp_storage = tf.Variable(initial_value=0)
convert_bits(temp_storage, tf_data[int(index*bit_size):int((index+1)*bit_size)], bit_size)
np_array = np.append(np_array, temp_storage.numpy())
print(np_array)
tf_array = tf.convert_to_tensor(np_array, dtype = tf.int32)
print(tf_array) # tf.Tensor([65 10], shape=(2,), dtype=int32)
You could modify and remove the tf.reverse depending on the endianness of the bits.
Also, as much as possible if you can use TensorFlow operations and data types in your code, you should use it.
So the task is to optimise a Neural Network with a PSO. The PSO needs a one-dimensional list of all the weights and biases, like so [0.1 0.244 ... 0.214]. The NN needs an array of arrays with different dimensions, like so [[x,y], [m,n], ...(all the hidden layer matrices)... ,[p,q]] X and y are the dimensions for the input layer, then all the hidden layers and finally p and q - the dimensions of the output layer.
I can easily flatten the array to pass it to the PSO, but I need a method that takes the modified array and reshapes it back into the same array of arrays with the same dimensions as the starting one from the NN.
The dimensions depend on the amount of neurons in a layer, we have that information from the start.
I have tried to keep track of the shapes array and create an indices array to know when to stop but it doesn't seem to work. I am trying something with slicing now but no cigar yet. A modification to the NN is also possible but how to create it so it takes a predefined list of weights? There might be a very nice and efficient way to do it but I just haven't thought of it yet... Any suggestions?
Example:
a = np.array([1,2,3])
b = np.array([7,8,9,10])
c = np.array([12,13,14,15,16])
b.reshape(2,2)
arr = []
arr.append(a)
arr.append(b)
arr.append(c)
This is a very simple example of what the list of weights is as the NN works with it - a list of multi-dimensional array. Arr can be converted into a numpy array of objects if necessary with np.asarray(arr).
Flattening is easy, here is how I do it (there might be a better that doesn't need a loop, if you know, I'd be thankful if you shared).
Flattening:
new_arr = np.array([])
for i in range(len(arr)):
new_arr = np.append(arr, arr[i].flatten())
My question is how to take new_arr and put it back together to look like arr and is there a beautiful and fast way to do it.
You can save the shape in a variable (it's just a tuple). Try something like:
...
old_shape = arr.shape
# ... do flattening here
new_arr.reshape(old_shape)
new_arr = np.array([])
shapes=[]
for i in range(len(arr)):
new_arr = np.append(new_arr, arr[i].flatten())
shapes.append(arr[i].shape)
#do whatever
restoredArray =[]
offset=0
for i in range(len(shapes)):
s = shapes[i]
n = np.prod(s)
restoredArray.append(new_arr[offset:(offset+n)].reshape(s))
offset+=n
I have a tensor probs with probs.shape = (max_time, num_batches, num_labels).
And I have a tensor targets with targets.shape = (max_seq_len, num_batches) where the values are label indices, i.e. for the third dimension in probs.
Now I want to get a tensor probs_y with probs.shape = (max_time, num_batches, max_seq_len) where the third dimension is the index in targets. Basically
probs_y[:,i,:] = probs[:,i,targets[:,i]]
for all 0 <= i < num_batches.
How can I achieve this?
A similar problem with solution was posted here.
The solution there, if I understand correctly, would be:
probs_y = probs[:,T.arange(targets.shape[1])[None,:],targets]
But that doesn't seem to work. I get:
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices.
Also, isn't the creation of the temporal T.arange a bit costly? Esp when I try to workaround by really making it a full dense integer array. There should be a better way.
Maybe theano.map? But as far as I understand, that doesn't parallelize the code, so this is also not a solution.
This works for me:
import theano
import theano.tensor as T
max_time, num_batches, num_labels = 3, 4, 6
max_seq_len = 5
probs_ = np.arange(max_time * num_batches * num_labels).reshape(
max_time, num_batches, num_labels)
targets_ = np.arange(num_batches * max_seq_len).reshape(max_seq_len,
num_batches) % (num_batches - 1) # mix stuff up
probs, targets = map(theano.shared, (probs_, targets_))
print probs_
print targets_
probs_y = probs[:, T.arange(targets.shape[1])[:, np.newaxis], targets.T]
print probs_y.eval()
Above used a transposed version of your indices. Your exact proposition also works
probs_y2 = probs[:, T.arange(targets.shape[1])[np.newaxis, :], targets]
print probs_y2.eval()
print (probs_y2.dimshuffle(0, 2, 1) - probs_y).eval()
So maybe your problem is somewhere else.
As for speed, I am at a loss as to what could be faster than this. map, which is a specialization of scan almost certainly is not. I do not know to what extent the arange is actually built rather than simply iterated over.
This question has been asked before, but the solution only works for 1D/2D arrays, and I need a more general answer.
How do you create a repeating array without replicating the data? This strikes me as something of general use, as it would help to vectorize python operations without the memory hit.
More specifically, I have a (y,x) array, which I want to tile multiple times to create a (z,y,x) array. I can do this with numpy.tile(array, (nz,1,1)), but I run out of memory. My specific case has x=1500, y=2000, z=700.
One simple trick is to use np.broadcast_arrays to broadcast your (x, y) against a z-long vector in the first dimension:
import numpy as np
M = np.arange(1500*2000).reshape(1500, 2000)
z = np.zeros(700)
# broadcasting over the first dimension
_, M_broadcast = np.broadcast_arrays(z[:, None, None], M[None, ...])
print M_broadcast.shape, M_broadcast.flags.owndata
# (700, 1500, 2000), False
To generalize the stride_tricks method given for a 1D array in this answer, you just need to include the shape and stride length for each dimension of your output array:
M_strided = np.lib.stride_tricks.as_strided(
M, # input array
(700, M.shape[0], M.shape[1]), # output dimensions
(0, M.strides[0], M.strides[1]) # stride length in bytes
)