I've got a numpy_array of size (3275412, 50, 22) which represents my data reshaped for LSTM purposes and I have got a target vector of shape (3275412,).
I want to balance my data so that there is approximately the same number of data with target 0 and 1.
The way I prepared the data makes that I can not do this balancing operation before reshaping.
Firstly, I wanted to apply make_imbalance function (see this link for details) but I can't apply it on a 2-D array (got an error).
My question is : what's the most efficient way to do it for a 3D array ?
My thoughts: I thought about firstly "flatten" my 3-D array to a 2-D array by "concatenating" the second and third dimension (but don't know how so please tell me ??) then apply make_imbalance and then reshape the result to a 3-D array (again, don't know how to do). It seems a little bit tricky however...
So any help would be appreciated, either for an other imbalancing method or for help about reshaping 3D->2D or vice-versa
You can use np.reshape with -1 for unknown dimension size.
data2d = data3d.reshape(data3d.shape[0], -1)
will give you a 2d array of shape (n_samples, n_features)
with the second and the third dimensions merged.
data2d_new, y_new = make_imbalance(data2d, y)
After make_imbalance call, you will get a 2d array with a shape (n_samples_new, n_features), where the number of rows is "unknown" but you know your other two 'feature' dimensions of the original 3d array, so
data3d_new = data2d.reshape(-1, data3d.shape[1], data3d.shape[2])
will give you back the balanced 3d dataset.
Related
My goal is to to turn a row vector into a column vector and vice versa. The documentation for numpy.ndarray.transpose says:
For a 1-D array, this has no effect. (To change between column and row vectors, first cast the 1-D array into a matrix object.)
However, when I try this:
my_array = np.array([1,2,3])
my_array_T = np.transpose(np.matrix(myArray))
I do get the wanted result, albeit in matrix form (matrix([[66],[640],[44]])), but I also get this warning:
PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.
my_array_T = np.transpose(np.matrix(my_array))
How can I properly transpose an ndarray then?
A 1D array is itself once transposed, contrary to Matlab where a 1D array doesn't exist and is at least 2D.
What you want is to reshape it:
my_array.reshape(-1, 1)
Or:
my_array.reshape(1, -1)
Depending on what kind of vector you want (column or row vector).
The -1 is a broadcast-like, using all possible elements, and the 1 creates the second required dimension.
If your array is my_array and you want to convert it to a column vector you can do:
my_array.reshape(-1, 1)
For a row vector you can use
my_array.reshape(1, -1)
Both of these can also be transposed and that would work as expected.
IIUC, use reshape
my_array.reshape(my_array.size, -1)
I'm trying to write a function that performs Convolution, and I'm getting a little challenged trying to create the output volume using numpy. Specifically, I have an input image that is represented as an array of dimensions (150,150,3). Now, I want to convolve over this image with a set of kernels num_kernels, which are arrays of dimension (4,4,3), and I want these kernels to move over the image with a stride of 2. My thought process has been:
(1) I'll create an output array which is comprised of taking (4,4,3) size chunks out of the input array and stretching these out into rows, and ultimately making a large matrix of these.
(2) Then, I'll create a parameter array composed of all of my (4,4,3) kernels stretched out into rows, which will also make a large matrix.
(3) Then I can dot product these matrices together and reshape the output matrix into the proper dimensions.
My rough psuedo-code start to number (1) is as follows.
def Convolution(input, filter_size, num_filters, stride):
X = input
output_Volume = np.zeros(#dimensions)
weights = np.zeros(#dimensions)
#get weights from other function
for width in range(0,150,2):
for height in range(0,150,2):
row = X(#indexes here to take out chunk).flatten
output_Volume.append(row) #something of this sort
return #dot product output volume and weights
If someone could provide a specific code example of how to implement this (most helpful would be answers to (1) and (2)) in Python (I'm using numpy), it would be much appreciated. Thank you!
Hello I am a newbie with the tensorflow and currently, I am working with colour Images and it's PCAS.
I have extracted PCAS in a form of "Red","Green" and "Blue" and also computed the weights which are associated with "Red","Green" and "Blue" components.
After doing the all the above stuff I want to combine all three 2D matrices into the single 3D matrix.
For a tensorflow it would be a 3D tensor.
def multi(h0,ppca,mu,i,scope=None):
with tf.variable_scope(scope or"multi"):
return tf.matmul(ppca[:,:,0],h0[i,:,:,0]) + tf.reshape(mu[:,0],[4096,1]) , tf.matmul(ppca[:,:,1],h0[i,:,:,1]) + tf.reshape(mu[:,1],[4096,1]) ,tf.matmul(ppca[:,:,2],h0[i,:,:,2]) + tf.reshape(mu[:,2],[4096,1])
So from the above function, I will get all three different 2D tensors and want to combine those 2D tensors to single 3D tensor which has dimensions [4096,1,3]
How can I do that?
any help is highly appreciated.
You need to concat them like this:
three_d_image = tf.concat(0, [[r], [g], [b]])
This tells tensorflow to concat them along the x dimension and treat each tensor as a matrix.
Doing the same without the additional brackets around the r,g,b tensors will try to concat them to one large 2D matrix
A clean, easy way to do it is using the tf.stack operation (tf.pack in older versions of tensorflow), it concatenats all tensors along a new dimension. If you want your new dimension to be after all previous, you need to set the axis argument to the number of dimensions of your tensors.
three_d_image = tf.stack([r,g,b], axis=2)
one of the solutions is that you can add one more empty dimension to your 2Ds so you will have 3 matrices of 3D dimension [4096,1,1] then you can concat these 3 matrices by axis 2 tf.concat(2,matrices) gives you [4096,1,3]
the second solution can be concat of axis 1, tf.concat(1,matrices) then reshape it to 3D
I have two images , image 1 of dimension (32,43,3) and image2 of dimension (67,86,3) . How can i store this in a numpy array , Whenever i try to append the array
image=cv2.imread(image1,0)
image=cv2.resize(image,(32,43))
x_train=np.array(image.flatten())
x_train=x_train.reshape(-1,3,32,43)
X_train =np.append(X_train,x_train) #X_train is my array
image=cv2.imread(image2,0)
image=cv2.resize(image,(67,86))
x_train=np.array(image.flatten())
x_train=x_train.reshape(-1,3,67,86)
X_train =np.append(X_train,x_train)
Value Error: total size of new array must be unchanged.
i want the X_train in shape (-1,depth,height,width).So that i can feed it into my neural network. Is there any way to store images of different dimension in array and feed into neural network ?
Don't use np.append. If you must join arrays, start with np.concatenate. It'll force you to pay more attention to the compatibility of dimensions.
You can't join 2 arrays with shapes (32,43,3) (67,86,3) to make a larger array of some compatible shape. The only dimension they share is the last.
These reshapes don't make sense either: (-1,3,32,43), (-1,3,67,86).
It works, but it also messes up the 'image'. You aren't just adding a 4th dimension. It looks like you want to do some axis swapping or transpose as well. Practice with some small arrays so you can see what's happening, e.g. (2,4,3).
What final shape do you expect for Xtrain?
You can put these two images in a object dtype array, which is basically the same as the list [image1, image2]. But I doubt if your neuralnet can do anything practical with that.
If you reshaped the (32,43,3) array to (16,86,3) you could concatenate that with (67,86,3) on axis=0 to produce a (83,86,3) array. If you needed the 3 to be first, I'd use np.transpose(..., (2,0,1)).
Conversely reshape (67,86,3) to (2*67,43,3).
Passing the (32,43,3) to (32,86,3) is another option.
Joining them on a new 4th dimension, requires that the number of 'rows' match as well as the number of 'columns'.
I m working on audio stuff with python and I have the data into a 2d array with left and right channels in two different dimensions
so my array shape looks like this (exemple) :
(360448, 2)
I'm trying to find a way to reshape the array so the data fits into a 3d array where each cell represent each sample in this way : [left_sample,right_sample,0] so the final array will have shape (X,X,3)
is there a way to reshape the array( without iterating and assigning the values) ?