I'm trying to use some sklearn estimators for classifications on the coefficients of some fast fourier transform (technically Discrete Fourier Transform). I obtain a numpy array X_c as output of np.fft.fft(X) and I want to transform it into a real numpy array X_r, with each (complex) column of the original X_c transformed into two (real/float) columns in X_r, i.e the shape goes from (r, c) to (r, 2c). So I use .view(np.float64). and it works at first.
The problem is that if I first decide to keep only some coefficients of the original complex array with X_c2 = X_c[:, range(3)] and then to do the same thing as before instead of having the number of columns doubled, I obtain the number of ranks doubled (the imaginary part of each element is put in a new row below the original).
I really don't understand why this happens.
To make myself clearer, here is a toy example:
import numpy as np
# I create a complex array
X_c = np.arange(8, dtype = np.complex128).reshape(2, 4)
print(X_c.shape) # -> (2, 4)
# I use .view to transform it into something real and it works
# the way I want it.
X_r = X_c.view(np.float64)
print(X_r.shape) # -> (2, 8)
# Now I subset the array.
indices_coef = range(3)
X_c2 = X_c[:, indices_coef]
print(X_c2.shape) # -> (2, 3)
X_r2 = X_c2.view(np.float64)
# In the next line I obtain (4, 3), when I was expecting (2, 6)...
print(X_r2.shape) # -> (4, 3)
Does anyone see a reason for this difference of behavior?
I get a warning:
In [5]: X_c2 = X_c[:,range(3)]
In [6]: X_c2
Out[6]:
array([[ 0.+0.j, 1.+0.j, 2.+0.j],
[ 4.+0.j, 5.+0.j, 6.+0.j]])
In [7]: X_c2.view(np.float64)
/usr/local/bin/ipython3:1: DeprecationWarning: Changing the shape of non-C contiguous array by
descriptor assignment is deprecated. To maintain
the Fortran contiguity of a multidimensional Fortran
array, use 'a.T.view(...).T' instead
#!/usr/bin/python3
Out[7]:
array([[ 0., 1., 2.],
[ 0., 0., 0.],
[ 4., 5., 6.],
[ 0., 0., 0.]])
In [12]: X_c2.strides
Out[12]: (16, 32)
In [13]: X_c2.flags
Out[13]:
C_CONTIGUOUS : False
F_CONTIGUOUS : True
So this copy (or is a view?) is Fortran order. The recommended X_c2.T.view(float).T produces the same 4x3 array without the warning.
As your first view shows, a complex array has the same data layout as twice the number of floats.
I've seen funny shape behavior when trying to view a structured array. I'm wondering the complex dtype is behaving much like a dtype('f8,f8') array.
If I change your X_c2 so it is a copy, I get the expected behavior
In [19]: X_c3 = X_c[:,range(3)].copy()
In [20]: X_c3.flags
Out[20]:
C_CONTIGUOUS : True
F_CONTIGUOUS : False
OWNDATA : True
WRITEABLE : True
ALIGNED : True
UPDATEIFCOPY : False
In [21]: X_c3.strides
Out[21]: (48, 16)
In [22]: X_c3.view(float)
Out[22]:
array([[ 0., 0., 1., 0., 2., 0.],
[ 4., 0., 5., 0., 6., 0.]])
That's reassuring. But I'm puzzled as to why the [:, range(3)] indexing creates a F order view. That should be advance indexing.
And indeed, a true slice does not allow this view
In [28]: X_c[:,:3].view(np.float64)
---------------------------------------------------------------------------
ValueError: new type not compatible with array.
So the range indexing has created some sort of hybrid object.
Related
I'd like to assign multiple values to a tensor, but it seems that it's not supported at least in the way that is possible using numpy.
a = np.zeros((4, 4))
v = np.array([0, 2, 3, 1])
r = np.arange(4)
a[r, v] = 1
>>> a
array([[1., 0., 0., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.],
[0., 1., 0., 0.]])
The above works, but the tensorflow equivalent doesn't:
import tensorflow as tf
a = tf.zeros((4, 4))
v = tf.Variable([0, 2, 3, 1])
r = tf.range(4)
a[r, v].assign(1)
TypeError: Only integers, slices, ellipsis, tf.newaxis and scalar tensors are valid indices, got <tf.Tensor: shape=(4,), dtype=int32, numpy=array([0, 1, 2, 3])>
How could this be achieved? Are loops the only option? In my case the resulting array is indeed only slices of an identity matrix rearranged, so maybe that could be taken advantage of somehow.
Your example, which is updating a zero tensor at some indices to a certain value is most of time achieved through tf.scatter_nd :
idx = tf.stack([r,v],axis=-1)
tf.scatter_nd(idx, updates=tf.ones(4), shape=(4,4))
For more complex cases, you can look at the following functions:
tf.tensor_scatter_nd_add: Adds sparse updates to an existing tensor according to indices.
tf.tensor_scatter_nd_sub: Subtracts sparse updates from an existing tensor according to indices.
tf.tensor_scatter_nd_max: to copy element-wise maximum values from one tensor to another.
tf.tensor_scatter_nd_min: to copy element-wise minimum values from one tensor to another.
tf.tensor_scatter_nd_update: Scatter updates into an existing tensor according to indices.
You can read more in the guide: Introduction to tensor slicing
How can the tuple with shape (0,2) be passed?
a = np.empty((0,2))
a
>>> array([], shape=(0, 2), dtype=float64)
Also what is the difference between tuple and list while passing the shape as a parameter in np.empty()?
arr = np.empty((2,2))
arr
>>> array([[ 0., 0.],
[ 0., 0.]])
arr1 = np.empty([2,2])
arr1
>>> array([[ 0., 0.],
[ 0., 0.]])
How both tuple and list gives same output?
Passing a tuple with 0, such as (0, 2) as shape parameter is allowed, but the resulting array is empty since it contains 0*2 = 0 elements. Written out in words, it's "zero rows with 2 elements in each row" where "2 elements in each row" is not of much consequence since there are no rows.
Such arrays arise when slicing goes wrong: for example,
b = np.ones((2, 2))
a = b[2:, :]
makes a an array of shape (0, 2) as there are no values of the first index that fall in the given slice. There is no reason to create such arrays intentionally.
There is no difference between np.empty([2, 2]) and np.empty((2, 2)), the array creation method, as many other NumPy methods, accepts a list instead of a tuple. It's still recommended to use tuples for shape parameters, because arr.shape is always a tuple.
I am new in python coming from matlab. Now when i want to save a vector in matlab to a preallocated matrix i do this (matlab code)
a = zeros(5, 2)
b = zeros(5, 1)
# save elements of b in the first column of a
a(:, 1) = b
Now i am using numpy in python. I do not really know how to describe this problem. What am i doing here is essentially this
a = np.zeros([5, 2])
b = np.ones([5, 1])
a[:, 0] = np.reshape(b, a[:, 0].shape)
because the following solution is not working:
a[:, 0] = b # Not working
Can anyone point out other ways of doing it, more closely to the matlab style?
Simple way would be -
a[:,[0]] = b
Sample run -
In [217]: a = np.zeros([5, 2])
...: b = np.ones([5, 1])
...:
In [218]: a[:,[0]] = b
In [219]: a
Out[219]:
array([[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 1., 0.],
[ 1., 0.]])
Basically with this slicing of using a scalar a[:,0], number of dimensions are reduced (the dimension along which the scalar is used is removed) for assignment. When we specify a list of index/indices like a[:,[0]], the dimensions are preserved, i.e. kept as 2D and that allows us to assign b, which is also 2D. Let's test that out -
In [225]: a[:,0].shape
Out[225]: (5,) # 1D array
In [226]: a[:,[0]].shape
Out[226]: (5, 1) # 2D array
In [227]: b.shape
Out[227]: (5, 1) # 2D array
For reference, here's a link to the slicing scheme. Quoting the relevant part from it -
An integer, i, returns the same values as i:i+1 except the
dimensionality of the returned object is reduced by 1.
In particular, a selection tuple with the p-th element an integer (and all other
entries :) returns the corresponding sub-array with dimension N - 1.
I try to store a list of different shaped arrays as a dtype=object array using np.save (I'm aware I could just pickle the list but I'm really curious how to do this).
If I do this:
import numpy as np
np.save('test.npy', [np.zeros((2, 2)), np.zeros((3,3))])
it works.
But this:
np.save('test.npy', [np.zeros((2, 2)), np.zeros((2,3))])
Gives me an error:
ValueError: could not broadcast input array from shape (2,2) into shape (2)
I guess np.save converts the list into an array first, so I tried:
x=np.array([np.zeros((2, 2)), np.zeros((3,3))])
y=np.array([np.zeros((2, 2)), np.zeros((2,3))])
Which has the same effect (first one works, second one doesn't.
The resulting x behaves as expected:
>>> x.shape
(2,)
>>> x.dtype
dtype('O')
>>> x[0].shape
(2, 2)
>>> x[0].dtype
dtype('float64')
I also tried to force the 'object' dtype:
np.array([np.zeros((2, 2)), np.zeros((2,3))], dtype=object)
Without success. It seems numpy tries to broadcast the array with equal first dimension into the new array and realizes too late that their shape is different. Oddly it seems to have worked at one point - so I'm really curious what the difference is, and how to do this properly.
EDIT:
I figured out the case it worked before: The only difference seems to be that the numpy arrays in the list have another data type.
It works with dtype('<f8'), but it doesn't with dtype('float64'), I'm not even sure what the difference is.
EDIT 2:
I found a very non-pythonic way to solve my issue, I add it here, maybe it helps to understand what I wanted to do:
array_list=np.array([np.zeros((2, 2)), np.zeros((2,3))])
save_array = np.empty((len(array_list),), dtype=object)
for idx, arr in enumerate(array_list):
save_array[idx] = arr
np.save('test.npy', save_array)
One of the first things that np.save does is
arr = np.asanyarray(arr)
So yes it is trying to turn your list into an array.
Constructing an object array from arbitrary sized arrays or lists is tricky. np.array(...) tries to create as high a dimensional array as it can, even attempting to concatenate the inputs if possible. The surest way is to do what you did - make the empty array and fill it.
A slightly more compact way of constructing the object array:
In [21]: alist = [np.zeros((2, 2)), np.zeros((2,3))]
In [22]: arr = np.empty(len(alist), dtype=object)
In [23]: arr[:] = alist
In [24]: arr
Out[24]:
array([array([[ 0., 0.],
[ 0., 0.]]),
array([[ 0., 0., 0.],
[ 0., 0., 0.]])], dtype=object)
Here are 3 scenarios:
Arrays that match in shape, combine into a 3d array:
In [27]: np.array([np.zeros((2, 2)), np.zeros((2,2))])
Out[27]:
array([[[ 0., 0.],
[ 0., 0.]],
[[ 0., 0.],
[ 0., 0.]]])
In [28]: _.shape
Out[28]: (2, 2, 2)
Arrays that don't match on the first dimension - create object array
In [29]: np.array([np.zeros((2, 2)), np.zeros((3,2))])
Out[29]:
array([array([[ 0., 0.],
[ 0., 0.]]),
array([[ 0., 0.],
[ 0., 0.],
[ 0., 0.]])], dtype=object)
In [30]: _.shape
Out[30]: (2,)
And awkward intermediate case (which may even be described as a bug). The first dimensions match, but the second ones don't):
In [31]: np.array([np.zeros((2, 2)), np.zeros((2,3))])
...
ValueError: could not broadcast input array from shape (2,2) into shape (2)
[ 0., 0.]])], dtype=object)
It's as though it initialized a (2,2,2) array, and then found that the (2,3) wouldn't fit. And the current logic doesn't allow it to backup and create the object array as it did in the previous scenario.
If you wanted to put the two (2,2) arrays in object array you'd have to use the create and fill logic.
Say I have two objects X, Y of shape k,1,n and k,m,n. I know that numpy will automatically extend/repeat X along the first dimension when I do operations such as X + Y. Does this magic work for all mathematical operations that are supported/included in numpy?
For example, can I do scipy.special.binom(X,Y) and get the expected result? I have tried some of the special functions, and I don't receive an error. Does not receiving an error allow me to conclude that the broadcasting was done correctly?
numpy does apply broadcasting for all operators, eg. *+-/ etc. It also applies it where possible to ufunc functions. That's part of the ufunc definition.
scipy.special.binom is, according to its docs a ufunc. It is compiled so I can't look at the code to verify this, but I can do a few simple tests:
In [379]: special.binom([1,2,3],[[1],[2]])
Out[379]:
array([[ 1., 2., 3.],
[ 0., 1., 3.]])
In [380]: special.binom([1,2,3,4],[[1],[2]])
Out[380]:
array([[ 1., 2., 3., 4.],
[ 0., 1., 3., 6.]])
In [385]: special.binom(np.arange(6).reshape(3,2,1),np.arange(6).reshape(3,1,2)).shape
Out[385]: (3, 2, 2)
The (2,3) and (2,4) output dimensions match the broadcasted inputs. That consistent with broadcasting.
np.dot is an example of a numpy function where broadcasting does not apply.