I am currently struggeling with the Problem of combining static features with a sequence of input data within a batch.
I have two channels of input data, one is processed via a convolutional neural network (eg. vgg-16 or comparable) and outputs a feature map.
My second input channel contains a list (with variable length) of input data.
Each single entry of that list and the calculated feature map should be fed into a classifier.
I know that I can use a TimeDistributed Wrapper to process sequences of data, but that only partially solves my problem:
The calculation of the feature map in the first input channel is costly and should only be performed once per batch
As the list in the second channel has a variable length, I cannot make use of a repeat Layer to to duplicate the feature map properly, additionally I run into memory problems as I cannot hold several hundreds (or thousand) copies of the feature map in gpu memory
What is the best way to properly combine static data (ones per batch) with a sequence of data?
Related
I have a lot of csv file containing approximately 1000 rows and 2 columns where the data looks like this:
21260.35679 0.008732499
21282.111 0.008729349
21303.86521 0.008721652
21325.61943 0.008708224
These two are the features where the output will be a device name. Each csv file is data from a specific device of different times and there are also many devices. What I am trying to do is train the data and then classify the device name using CNN. If there is any incoming data outside of the trained observation, it should be classified as anomaly.
I am trying to convert those values to image matrix so that I can use CNN to train this data. But I what I am concerned about is, the second columns contains value less than 1 or and close to zero and the value is also float. If I convert it to integer it becomes zero and if all the values becomes zero then it doesn't make any sense.
How to solve this? And is it even possible to use CNN on these datasets?
From your description, your problem seems to be a sequence classification.
You have many temporal sequences. Each sequence has the same quantity of 2D elements and is associated to a device. Given a sequence as input, you want to predict the corresponding device.
This kind of temporal dependencies are better captured by RNNs. I would suggest giving a look at LSTM .
Suppose we have a batch of images as 4D tensor of shape (batches, height, width, channels). And suppose we have some images transforming functions f0(img...), f1(img...) ..., each function processes just one image (not batch) represented as 3D tensor. Also output tensors from this functions may be different in shape than input tensors, even some of functions may produce more than one tensor and functions may have some extra arguments besides image.
Also suppose that we have non-eager execution mode, meaning that we build a graph to .pb file first and later graph is executed mostly on GPU for efficiency.
As it is common for TF, first batches dimension size might be unknown, which is signified by value None. Real size of this dimension is only known at graph evaluation time on input data (user may try to feed batches of varying sizes).
We're given a sequence of required image transformations specified as the list of functions with extra arguments. The task is to somehow process input batch of images through these functions. More than that it is needed to do in most efficient way, tf.py_functions are not allowed, meaning that if all transforming functions are implemented and run on GPU-side only then it is not allowed for intermediate results to go back and forth from gpu to cpu as inputs to py_functions.
The main difficulty is due to batches count being not known (equal to None). Hence we can't use python loop to processs each image through function. Of cause we can fix some maximum possible batch size and create a loop up to this maximum size, and for loop iterations when there is no input we can conditionally not process this input and pass empty tensor forth. Another problem that on each stage input and output list length may be different, e.g. when image transofmation splits input image into 3-4 unequal in shape tensors or does multi-crop with different windows.
I think the right approach would be to use tf.while_loop somehow, but I haven't understood how it works, and how to apply this while to case when each stage has different length of possible tensors of different shape.
There is tf.map_fn which is perfect for the case when all inputs have same shape and outputs too. Then we can pass down the transofmation path single 4D tensor. But this is not the case, inputs can be different in shapes and outputs too.
Maybe there is anything like python's list but in TF side? Meaning list to keep tensors of different shape, but TF-only, which doesn't leave GPU like python list does. If there exists such list and there is analogy of tf.map_fn then we can use this mapping to process list of tensors of different shape, by applying one function. It partially and mostly would solve my task, at least help me for sure.
I have multiple datasets, each with a different number of images (and different image dimensions) in it. In the training loop I want to load a batch of images randomly from among all the datasets but so that each batch only contains images from a single dataset. For example, I have datasets A, B, C, D and each has images 01.jpg, 02.jpg, … n.jpg (where n depends on the dataset), and let’s say the batch size is 3. In the first loaded batch, for example, I may get images [B/02.jpg, B/06.jpg, B/12.jpg], in the next batch [D/01.jpg, D/05.jpg, D/12.jpg], etc.
So far I have considered the following:
Use a different DataLoader for each dataset, e.g. dataloaderA, dataloaderB, etc., and then in each training loop randomly select one of the dataloaders and get a batch from it. However, this will require a for loop and for large number of datasets it would be very slow since it can’t be split among workers to do it in parallel.
Use a single DataLoader with all of the images from all datasets together but with a custom collate_fn which will create a batch using only images from the same dataset. (I’m not sure how exactly to go about this.)
I have looked at the ConcatDataset class but from its source code it looks like if I use it and try getting a new batch the images in it will be mixed up from among different datasets which I don’t want.
What would be the best way to do this? Thanks!
You can use ConcatDataset, and provide a batch_sampler to DataLoader.
concat_dataset = ConcatDataset((dataset1, dataset2))
ConcatDataset.comulative_sizes will give you the boundaries between each dataset you have:
ds_indices = concat_dataset.cumulative_sizes
Now, you can use ds_indices to create a batch sampler. See the source for BatchSampler for reference. Your batch sampler just has to return a list with N random indices that will respect the ds_indices boundaries. This will guarantee that your batches will have elements from the same dataset.
I have a working input pipeline that uses multiple threads to read data from tfrecords files, decodes and preprocesses the data, and serves them to the graph. I used this example as a starting point. My problem is that I need the input batch again in the loss layer, to compute a reconstruction error of the input. If I now just add the dequeuing operation multiple times to my graph, i.e. once at the input layer and once at the loss layer, I obviously get two times a different batch since tensorflow evaluates the operation twice. What is the easiest way to overcome this?
Probably you can try adding identity operation after dequeuing and connect other nodes to that identity operation.
I want to use SURF feature for Neural Network from Opencv.
I have successfully extracted features and created a nn.
But, the point where I am stocked is for every images I get varying length of feature vectors and each of 128 values.
So, my question is how should I feed these values into my 128 nodes of input layers for a single image. I know how to assign the output values for each set of input sets
Thank, you