Sorry if I messed up the title, I didn't know how to phrase this. Anyways, I have a tensor of a set of values, but I want to make sure that every element in the tensor has a range from 0 - 255, (or 0 - 1 works too). However, I don't want to make all the values add up to 1 or 255 like softmax, I just want to down scale the values.
Is there any way to do this?
Thanks!
You are trying to normalize the data. A classic normalization formula is this one:
normalize_value = (value − min_value) / (max_value − min_value)
The implementation on tensorflow will look like this:
tensor = tf.div(
tf.subtract(
tensor,
tf.reduce_min(tensor)
),
tf.subtract(
tf.reduce_max(tensor),
tf.reduce_min(tensor)
)
)
All the values of the tensor will be betweetn 0 and 1.
IMPORTANT: make sure the tensor has float/double values, or the output tensor will have just zeros and ones. If you have a integer tensor call this first:
tensor = tf.to_float(tensor)
Update: as of tensorflow 2, tf.to_float() is deprecated and instead, tf.cast() should be used:
tensor = tf.cast(tensor, dtype=tf.float32) # or any other tf.dtype, that is precise enough
According to the feature scaling in Wikipedia you can also try the Scaling to unit length:
It can be implemented using this segment of code:
In [3]: a = tf.constant([2.0, 4.0, 6.0, 1.0, 0])
In [4]: b = a / tf.norm(a)
In [5]: b.eval()
Out[5]: array([ 0.26490647, 0.52981293, 0.79471946, 0.13245323, 0. ], dtype=float32)
sigmoid(tensor) * 255 should do it.
Let the input be
X = tf.constant([[0.65,0.61, 0.59, 0.62, 0.6 ],[0.25,0.31, 0.89, 0.52, 0.6 ]])
We can define a scaling function
def rescale(X, a=0, b=1):
repeat = X.shape[1]
xmin = tf.repeat(tf.reshape(tf.math.reduce_min(X, axis=1), shape=[-1,1]), repeats=repeat, axis=1)
xmax = tf.repeat(tf.reshape(tf.math.reduce_max(X, axis=1), shape=[-1,1]), repeats=repeat, axis=1)
X = (X - xmin) / (xmax-xmin)
return X * (b - a) + a
This outputs X in range [0,1]
>>rescale(X)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
array([[1. , 0.333334 , 0. , 0.5000005 , 0.16666749],
[0. , 0.09375001, 1. , 0.42187497, 0.54687506]],
dtype=float32)>
To scale in range [0, 255]
>> rescale(X, 0, 255)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
array([[255. , 85.00017 , 0. , 127.50012 , 42.50021 ],
[ 0. , 23.906252, 255. , 107.57812 , 139.45314 ]],
dtype=float32)>
In some contexts, you need to normalize each image separately - for example adversarial datasets where each image has noise. The following normalizes each image according to its own min and max, assuming the inputs have typical size Batch x YDim x XDim x Channels:
cast_input = tf.cast(inputs,dtype=tf.float32) # e.g. MNIST is integer
input_min = tf.reduce_min(cast_input,axis=[1,2]) # result B x C
input_max = tf.reduce_max(cast_input,axis=[1,2])
ex_min = tf.expand_dims(input_min,axis=1) # put back inner dimensions
ex_max = tf.expand_dims(input_max,axis=1)
ex_min = tf.expand_dims(ex_min,axis=1) # one at a time - better way?
ex_max = tf.expand_dims(ex_max,axis=1) # Now Bx1x1xC
input_range = tf.subtract(ex_max, ex_min)
floored = tf.subtract(cast_input,ex_min) # broadcast
scale_input = tf.divide(floored,input_range)
I would like to expand the dimensions in one short like you can in Numpy, but tf.expand_dims seems to only accept one dimension at a a time - open to suggestions here. Thanks!
If you want the maximum value to be the effective upper bound of the 0-1 range and there's a meaningful zero then using this:
import tensorflow as tf
tensor = tf.constant([0, 1, 5, 10])
tensor = tf.divide(tensor, tf.reduce_max(tensor))
tf.print(tensor)
would result in:
[0 0.1 0.5 1]
Related
I'm trying to populate an array in python more efficiently. I have a 5x3 matrix A that I am transforming into a 3x3 matrix (Z) by calculating z11, z12, ..., z33 independently. The code below works, but it's clunky and I'm hoping to automate this into a loop so that it will take an A matrix of any size (n x m) and transform it into a Z matrix of size (m x m). If someone could help me out I would greatly appreciate it!
import numpy as np
A = np.array([[1,0,0],
[0,1,0],
[0,1,1],
[0,0,-1],
[0,0,1]])
A1=A[:,0]
A2=A[:,1]
A3=A[:,2]
C = np.array([-2,-2, -9,-6,-4])
X = np.array([-4,-4,-8])
z11 = (sum(A1*A1))*(C[0]/X[0])
z12 = (sum(A1*A2))*(C[0]/X[1])
z13 = (sum(A1*A3))*(C[0]/X[2])
z21 = (sum(A2*A1))*(C[1]/X[0])
z22 = (sum(A2*A2))*(C[1]/X[1])
z23 = (sum(A2*A3))*(C[1]/X[2])
z31 = (sum(A3*A1))*(C[2]/X[0])
z32 = (sum(A3*A2))*(C[2]/X[1])
z33 = (sum(A3*A3))*(C[2]/X[2])
Z = np.array([[z11,z12,z13],
[z21,z22,z23],
[z31,z32,z33]])
We can use the broadcasting to achieve the same. First let's increase A by one dimension using A[:, None] and then multiply it with A. Since shape of A[:, None] is (3, 1, 5) and shape of A is (3, 5), numpy first repeats(intuitively) the array corresponding to dimension where both array don't match and then does the multiplication. This way each column of A gets multiplied with every other column(to makes sure that columns are multiplied, I have used transpose) Then we can take sum along the last axis and multiply with C[:, None] to achieve the desired output.
Use:
m = A.shape[1]
B = A[:, None].T * A.T
Z = np.sum(B, axis = -1).astype(float)*C[:m, None]/X
Output:
>>> Z
array([[0.5 , 0. , 0. ],
[0. , 1. , 0.25 ],
[0. , 2.25 , 3.375]])
I have to run the snippet shown below about 200000 times in a row and the snippet needs about 0.12585 seconds for 1000 iterations. Datapoints has a shape of (3, 2704, 64)
output = []
maxium = 0
for datapoint in datapoints:
tmp = []
for data in datapoint:
maxium = max(data)
if maxium == 0:
tmp.append(data)
else:
tmp.append(data / maxium)
output.append(tmp)
I have tried to rewrite it using map() but this gives me an average of 0.23237 seconds per iteration. This is probably due to the multiple max(y) and list() calls.
np.asarray(list(map(lambda datapoint: list(map(lambda data: data / max(data) if max(data) > 0 else y, datapoint)), datapoints)))
Is there a possibility to optimize the code again to improve performance?
Well here's a short answer:
def bar(datapoints):
m = np.amax(datapoints, axis=2)
m[m == 0] = 1
return datapoints / m[:,:,np.newaxis]
Here's an explanation of how you might have got there (it's how I did get there!):
Let's start off with some example data:
>>> x = np.array([[[1, 2, 3, 4], [11, -12, 13, -14]], [[26, 27, 28, 29], [0, 0, 0, 0]]])
Now check what you get on your original function:
def foo(datapoints):
output = []
maxium = 0
for datapoint in datapoints:
tmp = []
for data in datapoint:
maxium = max(data)
if maxium == 0:
tmp.append(data)
else:
tmp.append(data / maxium)
output.append(tmp)
return numpy.array(output)
The result is:
>>> foo(x)
array([[[ 0.25 , 0.5 , 0.75 , 1. ],
[ 0.84615385, -0.92307692, 1. , -1.07692308]],
[[ 0.89655172, 0.93103448, 0.96551724, 1. ],
[ 0. , 0. , 0. , 0. ]]])
Now let's try out amax:
>>> np.amax(x, axis=0)
array([[26, 27, 28, 29],
[11, 0, 13, 0]])
>>> np.amax(x, axis=2)
array([[ 4, 13],
[29, 0]])
Ah ha, looks like axis=2 is what we're after. Now we want to divide the original array by this, but only in the places where the max is non-zero. How do only divide in some places? The answer is: we divide everywhere, but in some places we divide by 1 so it has no effect. So let's replace zeros with ones:
>>> m = np.amax(x, axis=2)
>>> m[m == 0] = 1
>>> m
array([[ 4, 13],
[29, 1]])
Finally, let's divide by this, broadcasting back over axis 2 which we took the maximum over earlier:
>>> x / m[:,:,np.newaxis]
array([[[ 0.25 , 0.5 , 0.75 , 1. ],
[ 0.84615385, -0.92307692, 1. , -1.07692308]],
[[ 0.89655172, 0.93103448, 0.96551724, 1. ],
[ 0. , 0. , 0. , 0. ]]])
Putting that all together you get bar() at the top.
Try something like this:
maximum = datapoints.max(axis=2, keepdims=True)
output = np.where(maximum==0, datapoints, datapoints/maximum)
You would see a warning invalid value encounter in true_divide but it should work as expected.
Update as #ArthurTacca pointed out:
output = datapoints/np.where(maximum==0, 1, maximum)
will eliminate the warning.
Yes you can definitely speed this up w/ vectorized numpy operations. Here's how I would do it, if I understand what you're trying to do correctly:
import numpy as np
# I use a randomly initialized array here, replace this with your input
arr = np.random.random(size=(3, 2704, 64))
# Find max for 3rd dimension, returns array w/ shape (3, 2704)
max_arr = np.max(arr, axis=2)
# Set up divisor, returns array w/ shape (3, 2704)
divisor = np.where(max_arr == 0, 1, max_arr)
# Use expand_dims to add third dimension, returns array w/ shape (3, 2704, 1)
divisor = np.expand_dims(divisor, axis=2)
# Perform division, shape is (3, 2704, 64)
ans = np.divide(arr, divisor)
From your code, I gather that you intend to scale your data by the max of your 3rd axis, but in the event of there being 0, forego scaling instead. You seem to also want your output to have the same shape as your input, which explains the way you structured output and tmp. That's why I left the code snippet to end w/ output in a numpy array, but if you need it in its original form regardless, its a simple loop to re-arrange your data:
output = []
for i in ans:
tmp = []
for j in i:
tmp.append(list(j))
output.append(tmp)
For future reference, furnish your questions with more detail. It will make it easier for people to participate, and you'll increase the chance of getting your questions answered quickly!
I have a series of 1D time series that, through a series of convolutional layers, end up in the form of:
(batch_size, time_series_length, num_filters)
I would like to manually upsample the tensors by inserting alternating zeros (much like a tranposed convolution), such that the new dimensionality becomes
(batch_size, 2*time_series_length, num_filters)
in order to be able to include an additional step before a convolutional layer. It is simple to do this in numpy, for example, with np.insert, but how does one do it with tensors?
I have looked at a few similar posts such as this, but I don't understand how to do this with multiple dimensions while preserving the other dimensions. Any thoughts?
I was working on a similar problem with images. I wanted to go from batch, height, width, in_channels to batch, 2*height, 2*width, in_channels. Like you said this is very much like a transposed convolution so I ended up using tf.nn.conv2d_transpose with strides=2 and filters=tf.ones([1, 1, 1, 1]):
upsampled_images = tf.nn.conv2d_transpose(images, tf.ones([1, 1, 1, 1]), output_shape, strides=2, padding='VALID')
This worked perfectly so I think the same will be true for 1d by just using tf.nn.conv1d_transpose with filters=tf.ones([1, 1, 1]).
I know this question is old and you probably figured out a way since, but I was looking for the answer for long myself, so it will probably help others.
EDIT
As pointed out by #A Roebel, this answer works only for single-channel images.
Here is an extension to the multi-channel case, with a complete example:
import tensorflow as tf
image = tf.random.normal(shape=[1, 2, 2, 2])
def enlarge_one_channel_images(images):
batch_size, height, width, n_channels = tf.shape(image) # might not work in graph mode
output_shape = [batch_size, 2*height, 2*width, 1]
upsampled_images = tf.nn.conv2d_transpose(images, tf.ones([1, 1, 1, 1]), output_shape, strides=2, padding='VALID')
return upsampled_images
image_reshaped = tf.transpose(image, [3, 0, 1, 2])[..., None]
batch_size, height, width, n_channels = tf.shape(image) # might not work in graph mode
expected_output_shape = [batch_size, 2*height, 2*width, 1]
image_reshaped_enlarged = tf.map_fn(
enlarge_one_channel_images,
image_reshaped,
fn_output_signature=tf.TensorSpec(expected_output_shape)
)
image_enlarged = tf.transpose(image_reshaped_enlarged[..., 0], [1, 2, 3, 0])
As also pointed out by #A Roebel in his answer this might not be the most efficient solution however.
I have not run the tests myself, but I agree that the additional convolution with the identity filter will surely slow things down, although I am not sure exactly what the expected acceleration when using tf.function can be.
The short answer is: use tf.scatter_nd
The tricky part is constructing the indices for this operation.
The following code example shows how you can do this for Tensors with arbitrarily many dimensions.
import itertools
import numpy as np
import tensorflow as tf
def pad_strided(x, strides, name=None):
# Preparatory steps and sanity checks.
input_shape = x.shape.as_list()
# Because life gets easier, we let the consumer specify a striding value for EACH dimension
assert len(strides) == len(input_shape), "Rank of strides and x.shape must be the same"
output_shape = [s_in * s for s_in, s in zip(input_shape, strides)]
"""
Calculate the striding indices for EACH dimension.
"""
index_ranges = [list(range(0, s_out, s)) for s_out, s in zip(output_shape, strides)]
"""
Expand the indices per dimension. The resulting array has shape [n_elements, n_dims].
n_elements is the number of values in the input tensor x. So the product of the input
shape. n_dims is the number of input (and output) dimensions.
"""
indices_flat = np.array(list(itertools.product(*index_ranges)))
"""
Reshape the flat index array to have the same dimensions as the input plus an additional
dimension. If the input had [s0, s1, ..., sn], then indices will have
[s0, s1, ..., sn, n_dims]. I.e. the rank will be 1 higher than that of the input tensor.
"""
indices = np.reshape(indices_flat, input_shape + [-1])
""" Now we simply call the TensorFlow operator """
with tf.variable_scope(name, default_name="pad_strided"):
t_indices = tf.constant(indices, dtype=tf.int32, name="indices")
t_output_shape = tf.constant(output_shape, name="output_shape")
return tf.scatter_nd(t_indices, x, t_output_shape)
session = tf.Session()
batch_size = 1
time_series_length = 6
num_filters = 3
t_in = tf.random.uniform((batch_size, time_series_length, num_filters))
# Specify a stride 2 for the time_series dimension
t_out = pad_strided(t_in, strides=[1, 2, 1])
original, strided = session.run([t_in, t_out])
print(f"Input Tensor:\n{original[:,:,:]}")
print(f"Output Tensor:\n{strided[:,:,:]}")
The output would then be for instance
Input Tensor:
[[[0.0678339 0.07883668 0.49193358]
[0.5029118 0.8639555 0.74302936]
[0.995087 0.6315181 0.11990702]
[0.95606446 0.29059124 0.12656784]
[0.8278991 0.8518325 0.4033165 ]
[0.78434443 0.7894305 0.6251142 ]]]
Output Tensor:
[[[0.0678339 0.07883668 0.49193358]
[0. 0. 0. ]
[0.5029118 0.8639555 0.74302936]
[0. 0. 0. ]
[0.995087 0.6315181 0.11990702]
[0. 0. 0. ]
[0.95606446 0.29059124 0.12656784]
[0. 0. 0. ]
[0.8278991 0.8518325 0.4033165 ]
[0. 0. 0. ]
[0.78434443 0.7894305 0.6251142 ]
[0. 0. 0. ]]]
I just had the same problem and found a problem in the solution shared by zaccharie-ramzi. The given solutions does not work with signals with more than a singe channel. I suggest here a fix for the solution with conXd_transpose together with a more efficient solution by means of reshaping and padding.
If you store the code below in a script named ./upsample_with_padding.py
you can reproduce the following experiments. The script starts with tensor
sig = tf.ones((60,10000,args.n_channels))
that is supposed to be upsampled by a factor upfac by means of inserting 0s in time direction for all channels. Default upfac is 4, default number of channels is 2.
You can run it with argument check to see the shapes and check that the results obtained with the padding solution and the solution using the corrected implementation of the answer with transposed convolution are equivalent.
> ./upsample_with_padding.py --check
upsig_conv (60, 40000, 2)
upsig_pad (60, 40000, 2)
diff: tf.Tensor(0.0, shape=(), dtype=float32)
Comparing the computational speed wee can see that the use of padding is much more efficient
> ./upsample_with_padding.py
timeit conv: 9.84551206199103
timeit pad : 1.459020125999814
This is expected because the convXd_transpose operation will perform padding as well but then has to convolve with a identity filter.
Here the script
#! /usr/bin/env python3
import os
# silence verbose TF feedback
if 'TF_CPP_MIN_LOG_LEVEL' not in os.environ:
os.environ['TF_CPP_MIN_LOG_LEVEL'] = "2"
from argparse import ArgumentParser
import tensorflow as tf
import timeit
def up_pad(sig, upfac):
upsigp = tf.expand_dims(sig, axis=2)
upsigp = tf.pad(upsigp, ((0, 0), (0, 0), (0, upfac-1), (0, 0)))
return tf.reshape(upsigp, shape=(sig.shape[0], sig.shape[1]*upfac, sig.shape[2]))
def up_conv(sig, upfac):
upsigc = tf.expand_dims(sig, axis=-1)
filter = tf.ones([1, 1, 1, 1])
return tf.nn.conv2d_transpose(upsigc, filters=filter, strides=(upfac,1), padding="VALID", data_format="NHWC",
output_shape=(sig.shape[0], sig.shape[1]*upfac, sig.shape[2], 1))[:,:,:,0]
parser=ArgumentParser()
parser.add_argument("--check", action="store_true")
parser.add_argument("--upfac", default=4, type=int)
parser.add_argument("--n_channels", default=2, type=int)
args=parser.parse_args()
sig = tf.ones((60,10000,args.n_channels))
if args.check:
upsig_conv = up_conv(sig, upfac=args.upfac)
upsig_pad = up_pad(sig, upfac=args.upfac)
print(f"upsig_conv {upsig_conv.shape}")
print(f"upsig_pad {upsig_pad.shape}")
print("diff:", tf.reduce_max(tf.abs(upsig_conv - upsig_pad)))
else:
print("timeit conv:",timeit.timeit(f'up_conv(sig, upfac={args.upfac})', globals=globals(), number=3000))
print("timeit pad :",timeit.timeit(f'up_pad(sig, upfac={args.upfac})', globals=globals(), number=3000))
Here is a solution which inserts factor - 1 zeros in between time samples for a tensor of shape (batch_size, time_series_length, num_channels):
def upsample(x, factor):
# x has shape (batch_size, time_series_length, num_channels)
L = tf.shape(x)[1] # time series length
## repeat each sample `factor` times
x = tf.repeat(x, tf.repeat(factor, L), axis=1)
## create a mask in order to replace the inserted samples by zeroes
mask = tf.reshape(tf.repeat([ tf.concat([[factor], tf.zeros(factor-1)], 0) ], L, axis=0), [-1])
# mask looks like [1, 0, 0, 0, 1, 0, 0, 0, 1, ...] (here factor = 4)
## multiply by mask
x = x * mask[tf.newaxis, :, tf.newaxis] # mask is reshaped to broadcast multiplication along axis 1
## low-pass filtering:
# from scipy.signal import firwin2
# filters = tf.convert_to_tensor(firwin2(32*factor, [0.0, 0.95/factor, 1.0/factor, 1.0], [1.0, 1.0, 0.0, 0.0], window="blackman"), tf.float32)[:,tf.newaxis, tf.newaxis]
# x = tf.nn.conv1d(x, filters, 1, 'SAME')
return x
Just like the question says, I'm trying to remove all zeros vectors (i.e [0, 0, 0, 0]) from a tensor.
Given:
array([[ 0. , 0. , 0. , 0. ],
[ 0.19999981, 0.5 , 0. , 0. ],
[ 0.4000001 , 0.29999995, 0.10000002, 0. ],
...,
[-0.5999999 , 0. , -0.0999999 , -0.20000005],
[-0.29999971, -0.4000001 , -0.30000019, -0.5 ],
[ 0. , 0. , 0. , 0. ]], dtype=float32)
I had tried the following code (inspired by this SO):
x = tf.placeholder(tf.float32, shape=(10000, 4))
zeros_vector = tf.zeros(shape=(1, 4), dtype=tf.float32)
bool_mask = tf.not_equal(x, zero_vector)
omit_zeros = tf.boolean_mask(x, bool_mask)
But bool_mask seem also to be of shape (10000, 4), like it was comparing every element in the x tensor to zero, and not rows.
I thought about using tf.reduce_sum where an entire row is zero, but that will omit also rows like [1, -1, 0, 0] and I don't want that.
Ideas?
One possible way would be to sum over the absolute values of the row, in this way it will not omit rows like [1, -1, 0, 0] and then compare it with a zero vector. You can do something like this:
intermediate_tensor = reduce_sum(tf.abs(x), 1)
zero_vector = tf.zeros(shape=(1,1), dtype=tf.float32)
bool_mask = tf.not_equal(intermediate_tensor, zero_vector)
omit_zeros = tf.boolean_mask(x, bool_mask)
I tried solution by Rudresh Panchal and it doesn't work for me. Maybe due versions change.
I found tipo in the first row: reduce_sum(tf.abs(x), 1) -> tf.reduce_sum(tf.abs(x), 1).
Also, bool_mask has rank 2 instead of rank 1, which is required:
tensor: N-D tensor.
mask: K-D boolean tensor, K <= N and K must be known statically. In other words, the shape of bool_mask must be for example [6] not [1,6]. tf.squeeze works well to reduce dimension.
Corrected code which works for me:
intermediate_tensor = tf.reduce_sum(tf.abs(x), 1)
zero_vector = tf.zeros(shape=(1,1), dtype=tf.float32)
bool_mask = tf.squeeze(tf.not_equal(intermediate_tensor, zero_vector))
omit_zeros = tf.boolean_mask(x, bool_mask)
Just cast the tensor to tf.bool and use it as a boolean mask:
boolean_mask = tf.cast(x, dtype=tf.bool)
no_zeros = tf.boolean_mask(x, boolean_mask, axis=0)
I am implementing linear regression in Python, and I think I am doing something wrong while converting matrix to numpy array, but cannot seem to figure it out.
Any help will be appreciated.
I am loading data from a csv file that has 100 columns. y is the last column. I am not using col 1 and 2 for regression.
communities=np.genfromtxt("communities.csv", delimiter = ",", dtype=float)
xdata = communities[1:,2:99]
x = np.array([np.concatenate((v,[1]))for v in xdata])
y = communities[1:,99]
Function definition
def standRegress(xArr, yArr):
xMat = mat(xArr); yMat = mat(yArr).T
xTx = xMat.T*xMat
if linalg.det(xTx)==0.0:
print"singular matrix"
return
ws = xTx.I*(xMat.T*yMat)
return ws
calling the function
w = standRegress(x,y)
xMat = mat(x) #shape(1994L,98L)
yMat = mat(y) #shape (1L, 1994L)
yhat = xMat*w #shape (1994L, 1L)
Next I am trying to calculate RMSE and this is where I am having problem
yMatT = yMat.T #shape(1994L, 1L)
err = yhat - yMatT #shape(1994L, 1L)
error = np.array(err)
total_error = np.dot(error,error)
rmse = np.sqrt(total_error/len(p))
I get an error while I am doing the dot product and thus not able to calculate rmse. I will appreciate if someone can help me find my mistake.
Error:
---> 11 np.dot(error,error)
12 #test = (error)**2
13 #test.sum()/len(y)
ValueError: matrices are not aligned
I'm not quite sure what the last dot is supposed to do. But you can't multiple error with itself this way. dot does a matrix multiplication, thus the dimensions have to align.
See, e.g., the following example:
import numpy as np
A = np.ones((3, 4))
B = np.ones((3, 4))
print np.dot(A, B)
This yields the error ValueError: matrices are not aligned.
What is possible, however, is:
print np.dot(A.T, B)
Output:
[[ 3. 3. 3. 3.]
[ 3. 3. 3. 3.]
[ 3. 3. 3. 3.]
[ 3. 3. 3. 3.]]
In your example error is just a column vector - but stored as a 2D array:
A = np.ones((3, 1))
B = np.ones((3, 1))
print np.dot(A, B)
Same error.
So you can either transpose one argument - as shown above - or extract one column as a 1D array:
print np.dot(A[:, 0], B[:, 0])
Output:
3.0