I can't seem to get Theano to reshape my tensors as want it to. The reshaping in the code bellow is supposed to keep keep_dims dimensions and flatten all remaining ones into a single array.
The code fails with IndexError: index out of bounds on the reshape line if I run it with a test value. Otherwise, the function seems to compile, but fails upon first real input with ValueError: total size of new array must be unchanged.
When I tried using just numpy for an equivalent code, it worked normally. Is there anything I am doing wrong? Or is there any easy way to see the resulting dimensions that are used for the reshaping (ipdb does not help since everything is a Theano variable)?
import theano
import theano.tensor as T
import numpy as np
theano.config.compute_test_value = 'warn'
theano.config.optimizer = 'None'
class Layer(object):
def __init__(self, name):
self.name = name
self.inputs = []
self.outputs = []
def get_init_weights(self, shape):
rows, cols = shape
w_init = np.reshape(np.asarray([rnd.uniform(-0.05, 0.05)
for _ in xrange(rows * cols)]),
newshape=(rows, cols))
return w_init
class Embedding(Layer):
def __init__(self, name, dict_size, width, init='uniform_005'):
super(Embedding, self).__init__(name)
self.width = width
self.dict_size = dict_size
e_init = self.get_init_weights((dict_size, width))
self.e = theano.shared(value=e_init, name=self.name)
def connect(self, inputs):
output = self.e[inputs]
self.inputs.append(inputs)
self.outputs.append(output)
return output
class Flatten(Layer):
def __init__(self, name, keep_dims=1):
super(Flatten, self).__init__(name)
self.params = []
self.keep_dims = keep_dims
def connect(self, inputs):
keep_dims = self.keep_dims
# this line fails
output = inputs.reshape(inputs.shape[0:keep_dims] +
(T.prod(inputs.shape[keep_dims:]),),
ndim=(keep_dims + 1))
return output
if __name__ == '__main__':
x = T.itensor3('x') # batch embedding * embedding size * number of different embeddings
x.tag.test_value = np.random.randint(0, 50, (5, 20, 3)).astype('int32')
emb_layer = Embedding('e', dict_size=50, width=10)
y = emb_layer.connect(x)
flat_layer = Flatten('f')
y = flat_layer.connect(y)
func = theano.function([x], y, allow_input_downcast=True)
The problem relates to how you're combining the two components of the new shape. The reshape command requires an lvector for the new shape.
Since you're using the test values mechanism you can debug this problem by simply printing test value bits and pieces. For example, I used
print inputs.shape.tag.test_value
print inputs.shape[0:keep_dims].tag.test_value
print inputs.shape[keep_dims:].tag.test_value
print T.prod(inputs.shape[keep_dims:]).tag.test_value
print (inputs.shape[0:keep_dims] + (T.prod(inputs.shape[keep_dims:]),)).tag.test_value
print T.concatenate([inputs.shape[0:keep_dims], [T.prod(inputs.shape[keep_dims:])]]).tag.test_value
This shows a fix to the problem: using T.concatenate to combine the keep_dims and the product of the remaining dims.
Related
Hypothesis complains vehemently that this was slow:
#composite
def f_and_g_and_padding(draw, in_channels = channel_ints, out_channels = channel_ints, fs = shapes_2d, fill=None, elements=well_behaved_floats):
shape_f = draw(basic_shape)
padding = draw(shapes_2d)
fs = draw(fs)
in_channels = draw(in_channels)
out_channels = draw(out_channels)
batch_size = draw(shape_ints)
shape_f = (batch_size, in_channels, fs[0], fs[1])
f = draw(stnp.arrays(dt_numpy, shape_f, elements=elements, fill=fill))
h_in = f.shape[2] + padding[0] * 2
w_in = f.shape[3] + padding[1] * 2
shape_g = (out_channels, in_channels, h_in, w_in)
g = draw(stnp.arrays(dt_numpy, shape_g, elements=elements, fill=fill))
return (f, g, padding)
I have tried to find out why, but failed. See: How to use pytest, hypothesis and line_profiler / kernprof together?.
So, my question remains: Why?
Here are the other strategies used:
well_behaved_floats = stnp.from_dtype(dtype=dt_numpy, allow_infinity=False, allow_nan=False)
small_floats = stnp.from_dtype(dtype=dt_numpy, min_value=-10000, max_value=10000, allow_infinity=False, allow_nan=False)
floats_0_1 = stnp.from_dtype(dtype=dt_numpy, min_value=-1, max_value=1, allow_infinity=False, allow_nan=False)
small_ints = stnp.from_dtype(dtype=numpy.dtype("i4"), allow_infinity=False, allow_nan=False, min_value=-10, max_value=10)
small_positive_ints = stnp.from_dtype(dtype=numpy.dtype("i4"), allow_infinity=False, allow_nan=False, min_value=0, max_value=10)
one_or_greater = st.integers(min_value=1)
shape_ints = st.integers(min_value=1, max_value=4)
channel_ints = st.integers(min_value=1, max_value=10)
basic_shape = stnp.array_shapes(min_dims=4, max_dims=4, min_side=1, max_side=10)
ones = st.integers(min_value=1, max_value=1)
shapes_2d = stnp.array_shapes(min_dims=2, max_dims=2, min_side=1, max_side=4)
Used like this:
#given(f_and_g_and_padding(elements=ones))
def test_padding(f_g_padding: Tuple[numpy.ndarray, numpy.ndarray, Tuple[int, int]]):
f, g, padding = f_g_padding
run_test(Tensor(f), Tensor(g), padding=padding)
There's no filtering, just plain simple drawing and numpy arrays.
fwiw here's the hypothesis config:
hypothesis.settings.register_profile("default",
derandomize=True,
deadline=None,
print_blob=True,
report_multiple_bugs=False,
suppress_health_check=[HealthCheck.too_slow])
I'd expect that your basic_shapes strategy is the culprit; with a minimum of four dimensions you're already into n^4 elements in the average side length and that's going to be slow. Consider reducing the max_side for this strategy; if that's unacceptable you might need to generate shapes with Hypothesis but elements with numpy.random.
I'd also recommend against passing allow_infinity=False, allow_nan=False to strategies for integers, or for bounded floats - in either case non-finite numbers are already ruled out, so while they don't do anything it's a hit to readability.
I'm struggling in creating a data generator in PyTorch to extract 2D images from many 3D cubes saved in .dat format
There is a total of 200 3D cubes each having a 128*128*128 shape. Now I want to extract 2D images from all of these cubes along length and breadth.
For example, a is a cube having size 128*128*128
So I want to extract all 2D images along length i.e., [:, i, :] which will get me 128 2D images along the length, and similarly i want to extract along width i.e., [:, :, i], which will give me 128 2D images along the width. So therefore i get a total of 256 2D images from 1 3D cube, and i want to repeat this whole process for all 200 cubes, there by giving me 51200 2D images.
So far I've tried a very basic implementation which is working fine but is taking approximately 10 minutes to run. I want you guys to help me create a more optimal implementation keeping in mind time and space complexity. Right now my current approach has a time complexity of O(n2), can we dec it further to reduce the time complexity
I'm providing below the current implementation
from os.path import join as pjoin
import torch
import numpy as np
import os
from tqdm import tqdm
from torch.utils import data
class DataGenerator(data.Dataset):
def __init__(self, is_transform=True, augmentations=None):
self.is_transform = is_transform
self.augmentations = augmentations
self.dim = (128, 128, 128)
seismicSections = [] #Input
faultSections = [] #Ground Truth
for fileName in tqdm(os.listdir(pjoin('train', 'seis')), total = len(os.listdir(pjoin('train', 'seis')))):
unrolledVolSeismic = np.fromfile(pjoin('train', 'seis', fileName), dtype = np.single) #dat file contains unrolled cube, we need to reshape it
reshapedVolSeismic = np.transpose(unrolledVolSeismic.reshape(self.dim)) #need to transpose the axis to get height axis at axis = 0, while length (axis = 1), and width(axis = 2)
unrolledVolFault = np.fromfile(pjoin('train', 'fault', fileName),dtype=np.single)
reshapedVolFault = np.transpose(unrolledVolFault.reshape(self.dim))
for idx in range(reshapedVolSeismic.shape[2]):
seismicSections.append(reshapedVolSeismic[:, :, idx])
faultSections.append(reshapedVolFault[:, :, idx])
for idx in range(reshapedVolSeismic.shape[1]):
seismicSections.append(reshapedVolSeismic[:, idx, :])
faultSections.append(reshapedVolFault[:, idx, :])
self.seismicSections = seismicSections
self.faultSections = faultSections
def __len__(self):
return len(self.seismicSections)
def __getitem__(self, index):
X = self.seismicSections[index]
Y = self.faultSections[index]
return X, Y
Please Help!!!
why not storing only the 3D data in mem, and let the __getitem__ method "slice" it on the fly?
class CachedVolumeDataset(Dataset):
def __init__(self, ...):
super(...)
self._volumes_x = # a list of 200 128x128x128 volumes
self._volumes_y = # a list of 200 128x128x128 volumes
def __len__(self):
return len(self._volumes_x) * (128 + 128)
def __getitem__(self, index):
# extract volume index from general index:
vidx = index // (128 + 128)
# extract slice index
sidx = index % (128 + 128)
if sidx < 128:
# first dim
x = self._volumes_x[vidx][:, :, sidx]
y = self._volumes_y[vidx][:, :, sidx]
else:
sidx -= 128
# second dim
x = self._volumes_x[vidx][:, sidx, :]
y = self._volumes_y[vidx][:, sidx, :]
return torch.squeeze(x), torch.squeeze(y)
I've rewritten a bit of what was done here in an attempt to not have to use recursion so as to produce the images. While I can get what appears to be the correct string of random functions, I am unable to get the correct output arrays so as to build the image.
You'll notice I've put the xVar function first in the random functions because it will operate on an empty string and give me back values. This is similar to what the original code does except that (by recursion) uses the value 0 to pick out one of three functions that will operate on empty strings. I am thinking that the results are passed back in so that functions such as np.sin will work.
I think the issue might lie in my usage of the identity decorator func(*testlist), perhaps I'm using it incorrectly.
import numpy as np, random
from PIL import Image
width, height = 256,256
xArray = np.linspace(0.0, 1.0, width).reshape((1, width, 1))
yArray = np.linspace(0.0, 1.0, height).reshape((height, 1, 1))
def xVar(): return xArray
def yVar(): return yArray
def safeDivide(a, b): return np.divide(a, np.maximum(b, 0.001))
def add(x,y):
added = np.add(x, y)
return added
def Color():
randColorarray = np.array([random.random(), random.random(), random.random()]).reshape((1, 1, 3))
return randColorarray
# def circle(x,y):
# circles = (x- 100) ** 2 + (y - 100) ** 2
# return circles
functions = (Color, xVar, yVar, np.sin, np.multiply, safeDivide)
depth = 5
def functionArray(depth = 0):
FunctList = []
FunctList.append(xVar)
for x in range(depth):
func = random.choice(functions)
FunctList.append(func)
return FunctList
def ImageBuilder():
FunctionList = functionArray(depth)
testlist = []
for func in FunctionList:
values = func(*testlist)
return values
vals = ImageBuilder()
repetitions = (int(xArray / vals.shape[0]), int(yArray / vals.shape[1]), int(3 / vals.shape[2]))
img = np.tile(vals, repetitions)
# Convert to 8-bit, send to PIL and save
img8Bit = np.uint8(np.rint(img.clip(0.0, 1.0) * 255.0))
Image.fromarray(img8Bit).save('Images/' + '.png', "PNG")
Depending on which random function is chosen, I'll either get
values = func(*testlist)
ValueError: invalid number of arguments
or
TypeError: safeDivide() missing 2 required positional arguments: 'a' and 'b'
Note however that the linked program does not get a safe divide error and both a and b are not being explicitly passed in (as is the same with np.multiply).
Thanks for any help.
I have a simple sum pooling implemented in keras tensorflow, using AveragePooling2D*N*N, so it creates a sum of the elements in pool with some shape, same padding so the shape won't change:
import numpy as np
import seaborn as sns
import matplotlib.pylab as plt
import tensorflow as tf
from tensorflow.keras.backend import square
#generating the example matrix
def getMatrixByDefinitions(definitions, width, height):
matrix = np.zeros((width, height))
for definition in definitions:
x_cor = definition[1]
y_cor = definition[0]
value = definition[2]
matrix.itemset((x_cor, y_cor), value)
return matrix
generated = getMatrixByDefinitions(width=32, height=32, definitions =[[7,16,1]])
def avg_pool(pool):
return tf.keras.layers.AveragePooling2D(pool_size=(pool,pool), strides=(1, 1), padding='same')
def summer(pool, tensor):
return avg_pool(pool)(tensor)*pool*pool
def numpyToTensor(numpy_data):
numpy_as_array = np.asarray(numpy_data)
tensor_data = numpy_as_array.reshape(1, numpy_data.shape[1], numpy_data.shape[1], 1)
return tensor_data
data = numpyToTensor(generated)
pooled_data = summer(11, data)
def printMatrixesToHeatMap(matrixes, title):
# f = pyplot.figure() # width and height in inches
matrix_count = len(matrixes)
width_ratios = [4] * matrix_count + [0.2]
mergedMatrixes = matrixes[0][0]
for matrix in matrixes:
mergedMatrixes = np.concatenate((mergedMatrixes, matrix[0]), axis=0)
vmin = np.min(mergedMatrixes)
vmax = np.max(mergedMatrixes)
fig, axs = plt.subplots(ncols=matrix_count + 1, gridspec_kw=dict(width_ratios=width_ratios))
fig.set_figheight(20)
fig.set_figwidth(20 * matrix_count + 5)
axis_id = 0
for matrix in matrixes:
sns.heatmap(matrix[0], annot=True, cbar=False, ax=axs[axis_id], vmin=vmin, vmax=vmax)
axs[axis_id].set_title(matrix[1])
axis_id = axis_id + 1
#fig.colorbar(axs[1].collections[0], cax=axs[matrix_count])
fig.savefig(title+".pdf", bbox_inches='tight')
def tensorToNumpy(tensor):
width = tensor.get_shape()[1]
height = tensor.get_shape()[2]
output = tf.reshape(tensor, [width, height])
#output = output.eval(session=tf.compat.v1.Session())
output = output.numpy()
return np.array(output)
printMatrixesToHeatMap([[tensorToNumpy(pooled_data), "Pooled data"]],
"name")
After testing it on very simple 2D array I have found out it does not do what I expect (original and pooled data):
You can see that the single one sum-pooled (according to average pooling) ended up with sum greater than real sum, which is 1, near the borders. (in this case max can be used, but the real data are more complex and we need sum) This would mean that average near borders is count not from padded data but the original. Or is this misunderstanding of padding from my side? I need to have ones on indices where 1.1, 1.2, 1.4 is. Why is this and how can I solve such problem?
Note that I do not want to manually set the correct sum, so I am looking for a way to achieve this in keras pooling itself.
It seems to be a problem with the "SAME" padding algorithm. Unfortunately,there is no way of specifying an explicit padding to the avg_pool2d op. It is possible to manually pad the input with tf.pad though. Here is a really naive approach to padding that will work with odd shaped pooling filters and strides size of 1 :
generated = getMatrixByDefinitions(width=32, height=32, definitions =[[7,16,1]])
gen_nhwc = tf.constant(generated[np.newaxis,:,:,np.newaxis])
pool = 11
paddings = [[0,0],[pool//2,pool//2],[pool//2,pool//2],[0,0]]
gen_pad = tf.pad(gen_nhwc, paddings, "CONSTANT")
res = tf.nn.avg_pool2d(gen_pad, (pool,pool), (1,1),"VALID")*pool*pool
result = np.squeeze(res.numpy())
printMatrixesToHeatMap([[generated, "input"],[result, "output"]], "name")
Results in images :
Edit : I created an issue on Github regarding the problem.
My question is if there was an issue in changing def step(self,x) function since the original was faulty.
I attempted to change def step(self,x) to x.any. It resulted in a prediction error where all predictions were 1 I attempted to implement an OR Perceptron neural network from a book by following the codes given. However, I received an error The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
This is the code:
from nn import Perceptron
import numpy as np
X = np.array([[0,0],[0,1],[1,0],[1,1]])
print(X[1])
y = np.array([0],[1],[1],[0])
print("[INFO] training perceptron...")
p = Perceptron(X.shape[1],alpha = 0.1)
p.fit(X,y,epochs=20)
print("[INFO] testing perceptron...")
for (x,target) in zip(X,y):
pred=p.predict(X)
print("[INFO] data={}, ground-truth={}, pred={}". format(x, target[0], pred))
The package that I imported was:
import numpy as np
class Perceptron:
def __init__(self, N, alpha = 0.1):
self.W = np.random.randn(N+1)/np.sqrt(N)
self.alpha = alpha
def step(self,x):
if x>0:
return 1
else:
return 0
def fit(self, X, y, epochs = 10):
X = np.c_[X,np.ones((X.shape[0]))]
for epoch in np.arange(0, epochs):
for (x,target) in zip(X,y):
p = self.step(np.dot(x, self.W))
if p!= target:
error = p-target
self.W += -self.alpha * error * x
def predict(self,X,addBias=True):
X = np.atleast_2d(X)
if addBias:
X=np.c_[X, np.ones((X.shape[0]))]
return self.step(np.dot(X,self.W))
My apologies if its a silly question as I spent the whole day thinking about it to no avail.
Thanks in advance!
The error that you are facing is because step() is coded to evaluate 1 element of the array at a time but when you pass an array to it in the predict function it has to do something like this:
[0.266,1.272,-1.282,0.889] > 1
The interpreter doesn't know which value to evaluate since it's an array and hence gives the error. Using any or all would check for 'any' or 'all' value in the array and give you 0 or 1 correspondingly, which is why you get an array of 1s when you write x.any().
Another thing that bothered me about the code you imported was that the forward pass is done in a loop, which is not very efficient or pythonic. A vectorized implementation is way better. I have changed the step function and fit function in that imported code to be vectorized and it runs fine for me.
import numpy as np
class Perceptron:
def __init__(self, N, alpha = 0.1):
self.W = np.random.randn(N+1)/np.sqrt(N)
self.alpha = alpha
def step(self,x):
return 1. * (x > 0)
def fit(self, X, y, epochs = 10):
X = np.c_[X,np.ones((X.shape[0]))]
for epoch in np.arange(0, epochs):
Z = np.dot(X, self.W)
p = self.step(Z)
if np.any(p != y):
error = (p-y)
self.W += -self.alpha * np.dot(X.T,error)
def predict(self,X,addBias=True):
X = np.atleast_2d(X)
if addBias:
X=np.c_[X, np.ones((X.shape[0]))]
return self.step(np.dot(X,self.W))
Now the step function is returning a binary array where the value is 1 when the input is greater than 0 else 0. For example if you had an array say:
X= [0.266,1.272,-1.282,0.889]
would be converted to:
[1,1,0,1]
I also changed the fit function so that it does everything vectorized.
One other thing that I did to my code was this :
Instead of
y = np.array([0],[1],[1],[0])
I did
y = np.array([0,1,1,0])
to get it working. I hope this helps. Be sure to ask anything if you don't understand.