Develop Reference

boolean_mask or sparse dot product in tensorflow

tl;dr what is the most efficient way to dynamically choose some entries of a tensor.
I am trying to implement syntactic GCN in Tensorflow. Basically, I need to have a different weight matrix for every label (lets ignore biases for this question) and choose at each run the relevant entries to use, those would be chosen by a sparse matrix (for each entry there is at most one label in one direction and mostly no edge so not even that).
More concretely, when I have a sparse matrix of labeled edges (zero-one), is it better to use it in a mask, a sparse-dense tensor multiplication or maybe just use normal multiplication (I guess not the latter, but for simplicty use it in the example)
example:
units = 6 # output size
x = ops.convert_to_tensor(inputs[0], dtype=self.dtype)
labeled_edges = ops.convert_to_tensor(inputs[1], dtype=self.dtype)
edges_shape = labeled_edges.get_shape().as_list()
labeled_edges = expand_dims(labeled_edges, -2)
labeled_edges = tile(
labeled_edges, [1] * (len(edges_shape) - 1) + [units, 1])
graph_kernel = math_ops.multiply(self.kernel, labeled_edges) # here is the question basically
outputs = standard_ops.tensordot(x, graph_kernel, [[1], [0]])
outputs = math_ops.reduce_sum(outputs, [-1])

To answer your tl;dr question, you can try using either of the following:
tf.nn.embedding_lookup : typical usage is tf.nn.embedding_lookup(params, ids). It returns a Tensor, which 0-axis entries are a subset of Tensor params. The indices of kept entries are defined by Tensor ids.
tf.nn.embedding_lookup_sparse : is the same as tf.nn.embedding_lookup but takes ids as a SparseTensor.

Get started to launch google-cloud-ml with my own dataset

I succeeded at executing all the steps of the online tutorial for google cloud ml.
But since the dataset used in this tutorial is already a TFRecord, I didn't understand well in what way transform say my numpy dataset to a TFRecord one.
Then, I tried to create my TFRecord using this a little bit modified code compared to the official convert_to_records.py. What I understand is that we can only convert primitive variables to TFRecord and that is why the trick to convert a list of float to bytes is used.
Then I have to somewhere convert back my string to a list of floats. Thus, I tried to perform this task with either the line 97 or the line 98 in my modified script model.py.
Unfortunately, none of these attempts is working. I always get the following error message :
ValueError: rank of shape must be at least 2 not: 1
This is because the shape of my variable features is (batch_size,) and not (batch_size, IMAGE_PIXELS). But I don't understand why.
Am I trying to launch google-cloud-ml the wrong way or are there some more parameters to tweak ?

The error indicates a rank 2 (matrix) is expected but the value is actually rank 1 (a vector). I suspect this is because np.tostring() returns a single string rather than a list of strings.
I think that is somewhat tangential as I don't think your float-to-string and string-to-float conversions are consistent. You convert float-to-string using numpy's builtin tostring() method. That returns the byte representation of the data: i.e.
import numpy as np
x = np.array([1.0, 2.0])
print x.tostring()
Returns
�?#
And not
['1.0', '2.0']
The latter is what tf.string_to_number expects.
You could make the float-to-string and string-to-float conversions consistent but I think a better solution is to just represent the data as floats. For example:
def _int64_feature(value):
return tf.train.Feature(int64_list=tf.train.Int64List(value=value))
def _float_feature(value):
return tf.train.Feature(float_list=tf.train.FloatList(value=value))
e = tf.train.Example(features=tf.train.Features(feature={
'labels': _int64_feature([10]),
'features': _float_feature([100.0, 200, ....])}))
feature_map = {
'labels': tf.FixedLenFeature(
shape=[1], dtype=tf.int64, default_value=[-1]),
'features': tf.FixedLenFeature(
shape=[NUM_PIXELS], dtype=tf.float32),
}
result = tf.parse_example([e.SerializeToString()], features=feature_map)
A Feature proto allows float32's to be stored inside float_list. You only need to convert floats to bytes if you are using float64. Your data is float32 so that's unnecessary.

It might help to analyze both the output of read_data_sets.py and the output of parse_example operation in your model.py
What read_data_sets produces
read_data_sets, as you point out, creates numpy arrays for each image. They have shape [28, 28, 1] for height x width x channels (the images are monochrome) and in your original call to read_data_sets, you were specifying that you wanted the image data as uint8 arrays. When you call tostring on the uint8 numpy array, the shape information is discarded and since each uint8 is a single byte, you end up with a byte string of length 784 with one entry for each pixel in the original 28x28x1 numpy array in row-major order. This is then stored as a bytes_list in the resulting tf.train.Example.
To recap, each entry in the feature map under the features key has a list of bytes with exactly one entry. That entry is a string of length 784 where each 'character' in the string is a value between 0-255 representing the monochrome pixel value for a point in the original 28x28 image. The following is a sample instance of tf.train.Example as printed by Python:
features {
feature {
key: "features"
value {
bytes_list {
value: "\000\000\257..."
}
}
}
feature {
key: "labels"
value {
int64_list {
value: 10
}
}
}
}
What parse_example expects and returns
tf.parse_example accepts a vector of tf.string objects as input. These objects are serialized tf.train.Example objects. In your code, util.read_examples produces exactly that.
The other argument to tf.parse_example is the schema to the examples. As mentioned before, the features entry in your Example is a tf.string as defined above. For reference, your code has:
def parse_examples(examples):
feature_map = {
'labels': tf.FixedLenFeature(
shape=[], dtype=tf.int64, default_value=[-1]),
'features': tf.FixedLenFeature(
shape=[], dtype=tf.string),
}
return tf.parse_example(examples, features=feature_map)
The interest thing, related to the error message you received, is the shape parameter. That shape parameter specifies the shape of a single instance, in this case, by specifying that shape=[] you are saying that each image is a rank-0 string, which is to say, a plain-old string (i.e., not a vector, not a matrix, etc.). This requires that the bytes_list have exactly one element. That's exactly what you are storing in each features field of your tf.train.Example.
Even though the shape property refers to the shape of a single instance, the output of tf.parse_example for the features field will be the whole batch of examples. This can be a bit confusing. So while each individual example has a single string (shape=[]), the batch is a vector of strings (shape=[batch_size]).
Using the image
Having the image data in a string is not very useful; we need to convert it back to numerical data. The TensorFlow op to do this is tf.decode_raw (Jeremy Lewi explained why tf.string_to_number won't work here):
image_bytes = tf.decode_raw(parsed['features'], out_type=tf.uint8)
image_data = tf.cast(image_bytes, tf.float32)
(be sure to set out_type=tf.uint8 since that was the data type that was output in read_data_sets). Typically, you're going to want to cast the result to a tf.float32. Sometimes it's even useful to reshape the tensor to recover the original shapes, e.g.,
# New shape is [batch_size, height, width, channels]. We use
# -1 as the first dimension in case batches are variable size.
image_data = tf.reshape(image_data, [-1, 28, 28, 1])
(NB: you probably don't need that in your code).
Alternatively, you could store the data as tf.float32 by calling read_data_sets with dtype=tf.float32 (the default). Then you can construct your tf.train.Example as explained by Jeremy Lewi, who also gave the code to parse such examples. However, the shapes will be different in that case. The shape of each instance (as indicated by the shape in FixedLenFeature) is now IMAGE_PIXELS, and the shape of the features entry in the output of tf.parsed_example is [batch_size, IMAGE_PIXELS].
The tradeoff between uint8 and float32, of course, is that the data on disk will be approximately four times as large for the latter, but you avoid the extra cast needed for the former. In the case of MNIST where there isn't much data, the added clarity of directly dealing with float data is probably worth the extra space.

TensorFlow: how can I sum a list of tf.Variables?

I've got a 3D array that is of tf.Variable type. tf.reduce_sum only works on individual tensors. I've tried doing:
tf.reduce_sum([tf.reduce_sum(mat) for mat in var_3Dlist])
...but tf.reduce_sum() expects a tensor and not a list. Can I convert it to a tensor somehow, or is there another, better way to do it? I haven't found anything in the API.

The tf.reduce_sum() op works on 3-D tensors and variables (and in general any rank or tensor or variable). However, if you have a list of 2-D tensors (or variables), you should use the tf.add_n() op to add together the values in the list:
var_3Dlist = ... # List of 3-D variables.
sum_list = [tf.reduce_sum(mat) for mat in var_3Dlist]
sum = tf.add_n(sum_list)

TypeError from theano While using 3D numpy array

I am trying something similar to code below
datax=theano.shared(value=rng.rand(5,500,45))
x=T.dmatrix('x')
i=T.lscalar('i')
W=theano.shared(value=rng.rand(90,45,500))
Hb=theano.shared(value=np.zeros(90))
w_v_bias=T.dot(W,x).sum(axis=2).sum(axis=1)+Hb
z=theano.function([i],w_v_bias,givens={x:datax[i*5:(i+1)*5]})
z(0)
Theano is giving me a TypeError with msg:
Cannot convert Type TensorType(float64, 3D) (of Variable Subtensor{int64:int64:}.0) into Type TensorType(float64, matrix). You can try to manually convert Subtensor{int64:int64:}.0 into a TensorType(float64, matrix)
What I am doing wrong here?
Edit
As mentioned by daniel changing x to dtensor3 will result in another error.
ValueError: Input dimension mis-match. (input[0].shape[1] = 5, input[1].shape[1] = 90)
Apply node that caused the error: Elemwise{add,no_inplace}(Sum{axis=[1], acc_dtype=float64}.0, DimShuffle{x,0}.0)
Another way is to modify my train function but then I won't be able to do batch learning.
z=theano.function([x],w_v_bias)
z(datax[0])
I am trying to implement RBM with integer values for visible units.

The problem is that datax is a 3D tensor and datax[index*5:(index+1)*5] is also a 3D tensor but you're trying to assign that to x which is a 2D tensor (i.e. a matrix).
Changing
x = T.dmatrix('x')
to
x = T.dtensor3('x')
solves this problem but creates a new one because the dimensions of W and x don't match up to perform the dot product. It's unclear what the desired outcome is.

Solved it after few hit and trials.
What I needed was to change
x=T.dmatrix('x')
w_v_bias=T.dot(W,x).sum(axis=2).sum(axis=1)+Hb
to
x=T.dtensor3('x')
w_v_bias=T.dot(x,W).sum(axis=3).sum(axis=1)+Hb
Now it produces (5,90) array after adding Hb elementwise to each of the five vectors of dot product.

represent an index inside a list as x,y in python

I have a list which contains 1000 integers. The 1000 integers represent 20X50 elements of dimensional array which I read from a file into the list.
I need to walk through the list with an indicator in order to find close elements to each other. I want that my indicator will be represented not only by a simple index i, but as a two indices x,y so I can know where is my indicator along the list.
I tried to reshape the list like that:
data = np.array( l )
shape = ( 20, 50 )
data.reshape( shape )
but I don't know how to access the data array.
Update: Is there any way to find the indices of x, y for an integers that are smaller than NUM(let's say NUM=12)

According to documentation of numpy.reshape , it returns a new array object with the new shape specified by the parameters (given that, with the new shape, the amount of elements in the array remain unchanged) , without changing the shape of the original object, so when you are calling the data.reshape() function you should also assign it back to data for it to reflect in data.
Example -
data = data.reshape( shape ) # where shape = (20,50)
Also, another way to change the shape, is to directly assign the new shape to the data.shape property.
Example -
shape = (20,50)
data.shape = shape # where shape is the new shape