Normally, when we know where should we insert the newaxis, we can do a[:, np.newaxis,...]. Is there any good way to insert the newaxis at certain axis?
Here is how I do it now. I think there must be some much better ways than this:
def addNewAxisAt(x, axis):
_s = list(x.shape)
_s.insert(axis, 1)
return x.reshape(tuple(_s))
def addNewAxisAt2(x, axis):
ind = [slice(None)]*x.ndim
ind.insert(axis, np.newaxis)
return x[ind]
That singleton dimension (dim length = 1) could be added as a shape criteria to the original array shape with np.insert and thus directly change its shape, like so -
x.shape = np.insert(x.shape,axis,1)
Well, we might as well extend this to invite more than one new axes with a bit of np.diff and np.cumsum trick, like so -
insert_idx = (np.diff(np.append(0,axis))-1).cumsum()+1
x.shape = np.insert(x.shape,insert_idx,1)
Sample runs -
In [151]: def addNewAxisAt(x, axis):
...: insert_idx = (np.diff(np.append(0,axis))-1).cumsum()+1
...: x.shape = np.insert(x.shape,insert_idx,1)
...:
In [152]: A = np.random.rand(4,5)
In [153]: addNewAxisAt(A, axis=1)
In [154]: A.shape
Out[154]: (4, 1, 5)
In [155]: A = np.random.rand(5,6,8,9,4,2)
In [156]: addNewAxisAt(A, axis=5)
In [157]: A.shape
Out[157]: (5, 6, 8, 9, 4, 1, 2)
In [158]: A = np.random.rand(5,6,8,9,4,2,6,7)
In [159]: addNewAxisAt(A, axis=(1,3,4,6))
In [160]: A.shape
Out[160]: (5, 1, 6, 1, 1, 8, 1, 9, 4, 2, 6, 7)
np.insert does
slobj = [slice(None)]*ndim
...
slobj[axis] = slice(None, index)
...
new[slobj] = arr[slobj2]
Like you it constructs a list of slices, and modifies one or more elements.
apply_along_axis constructs an array, and converts it to indexing tuple
outarr[tuple(i.tolist())] = res
Other numpy functions work this way as well.
My suggestion is to make initial list large enough to hold the None. Then I don't need to use insert:
In [1076]: x=np.ones((3,2,4),int)
In [1077]: ind=[slice(None)]*(x.ndim+1)
In [1078]: ind[2]=None
In [1080]: x[ind].shape
Out[1080]: (3, 2, 1, 4)
In [1081]: x[tuple(ind)].shape # sometimes converting a list to tuple is wise
Out[1081]: (3, 2, 1, 4)
Turns out there is a np.expand_dims
In [1090]: np.expand_dims(x,2).shape
Out[1090]: (3, 2, 1, 4)
It uses reshape like you do, but creates the new shape with tuple concatenation.
def expand_dims(a, axis):
a = asarray(a)
shape = a.shape
if axis < 0:
axis = axis + len(shape) + 1
return a.reshape(shape[:axis] + (1,) + shape[axis:])
Timings don't tell me much about which is better. They are the 2 µs range, where simply wrapping the code in a function makes a difference.
Related
I have an empty list: x = [].
I have a numpy array, y, of shape: (180, 161). I can't necessarily define x to be an np.empty of a particular shape, because I won't know the shape of y ahead of time.
I want to append y to x so that x will have a .shape of (1, 180, 161).
Then if I append more, I want it to be (n, 180, 161)
I tried .append and .stack, but I've had a variety of errors:
TypeError: only size-1 arrays can be converted to Python scalars
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 3 dimension(s) and the array at index 1 has 2 dimension(s)
And so on. It seems that this should be simple, but it's strangely difficult.
Assuming all items in x have the same shape, you can first construct a list and then construct the NumPy array from the list.
There, you have two options:
np.array() which is faster but not flexible
np.stack() which is slower but allows you to choose over which axis should the stack happen (it is roughly equivalent to np.array().transpose(...).copy()
The code would look like:
import numpy as np
n = 100
x = [np.random.randint(0, 10, (10, 20)) for _ in range(n)]
# same as: y = np.stack(x, 0)
y = np.array(x)
print(y.shape)
# (100, 10, 20)
Of course this line:
x = [np.random.randint(0, 10, (10, 20)) for _ in range(n)]
can be replaced with:
x = []
for _ in range(n):
x.append(np.random.randint(0, 10, (10, 20)))
You could also use np.append(), e.g.:
def stacker(arrs):
result = arrs[0][None, ...]
for arr in arrs[1:]:
result = np.append(result, arr[None, ...], 0)
return result
but with horrific performances:
n = 1000
shape = (100, 100)
x = [np.random.randint(0, n, shape) for _ in range(n)]
%timeit np.array(x)
# 10 loops, best of 3: 21.1 ms per loop
%timeit np.stack(x)
# 10 loops, best of 3: 21.6 ms per loop
%timeit stacker(x)
# 1 loop, best of 3: 11 s per loop
and, as you can see, performance-wise, the list-based method is way faster.
You can reshape y to be (1, *y.shape).
Then for appending an array you can say:
y_1 = np.vstack((y, new_arr))
where y_1.shape produces a (2, *y.shape) numpy array.
To save memory you can say y = np.vstack((y, new_arr))
You might have to reshape your array to (1, *y.shape) however.
This is a very basic example:
import numpy as np
a = np.ones((1,2,3))
b = np.ones((1,2,3))
np.vstack((a,b)).shape # (2,2,3)
Let me know if this helps!
If you keep x as a list then if you just want to maintain the shape by appending, it is possible:
>>> import numpy as np
>>> x = []
>>> y = np.arange(12).reshape(3,4)
>>> x.append(y)
>>> np.shape(x)
(1, 3, 4)
>>> x.append(y)
>>> np.shape(x)
(2, 3, 4)
>>> for i in range(10):
... x.append(y)
>>> np.shape(x)
(12, 3, 4)
But considering you are dealing with np.arrays it may not be convenient for you to keep x as list, so you may try this:
>>> x = np.array(x)
>>> x.shape
(12, 3, 4)
>>> y[None,...].shape
(1, 3, 4)
>>> np.append(x, y[None,...],axis=0).shape
(13, 3, 4)
Word of caution:
As pointed out by #hpaulj :
np.append should be avoided, as it is extremely slow, probably only faster than:
x = np.array([*x, y])
The correct usage would be:
x = np.concatenate([x, y[None,...]], axis=0)
Either way, concatenating or appending is generally a speed bump in numpy. So unless you absolutely need to create an array this way, you should work with lists. Also most functions applied to np.arrays work on lists as well. Note, functions applied to arrays, not methods of an np.array object. For example:
>>> x = list((1, 2, 3, 4))
>>> np.shape(x)
(4,)
>>> x.shape
Traceback (most recent call last):
File "<ipython-input-100-9f2b259887ef>", line 1, in <module>
x.shape
AttributeError: 'list' object has no attribute 'shape'
So I would suggest appending to list, and then after you have done appending all the arrays, convert the list to np.array if you require.
Suppose I have a 5x10x3 array, which I interpret as 5 'sub-arrays', each consisting of 10 rows and 3 columns. I also have a seperate 1D array of length 5, which I call b.
I am trying to insert a new column into each sub-array, where the column inserted into the ith (i=0,1,2,3,4) sub-array is a 10x1 vector where each element is equal to b[i].
For example:
import numpy as np
np.random.seed(777)
A = np.random.rand(5,10,3)
b = np.array([2,4,6,8,10])
A[0] should look like:
A[1] should look like:
And similarly for the other 'sub-arrays'.
(Notice b[0]=2 and b[1]=4)
What about this?
# Make an array B with the same dimensions than A
B = np.tile(b, (1, 10, 1)).transpose(2, 1, 0) # shape: (5, 10, 1)
# Concatenate both
np.concatenate([A, B], axis=-1) # shape: (5, 10, 4)
One method would be np.pad:
np.pad(A, ((0,0),(0,0),(0,1)), 'constant', constant_values=[[[],[]],[[],[]],[[],b[:, None,None]]])
# array([[[9.36513084e-01, 5.33199169e-01, 1.66763960e-02, 2.00000000e+00],
# [9.79060284e-02, 2.17614285e-02, 4.72452812e-01, 2.00000000e+00],
# etc.
Or (more typing but probably faster):
i,j,k = A.shape
res = np.empty((i,j,k+1), np.result_type(A, b))
res[...,:-1] = A
res[...,-1] = b[:, None]
Or dstack after broadcast_to:
np.dstack([A,np.broadcast_to(b[:,None],A.shape[:2])]
I have a question regarding the conversion between (N,) dimension arrays and (N,1) dimension arrays. For example, y is (2,) dimension.
A=np.array([[1,2],[3,4]])
x=np.array([1,2])
y=np.dot(A,x)
y.shape
Out[6]: (2,)
But the following will show y2 to be (2,1) dimension.
x2=x[:,np.newaxis]
y2=np.dot(A,x2)
y2.shape
Out[14]: (2, 1)
What would be the most efficient way of converting y2 back to y without copying?
Thanks,
Tom
reshape works for this
a = np.arange(3) # a.shape = (3,)
b = a.reshape((3,1)) # b.shape = (3,1)
b2 = a.reshape((-1,1)) # b2.shape = (3,1)
c = b.reshape((3,)) # c.shape = (3,)
c2 = b.reshape((-1,)) # c2.shape = (3,)
note also that reshape doesn't copy the data unless it needs to for the new shape (which it doesn't need to do here):
a.__array_interface__['data'] # (22356720, False)
b.__array_interface__['data'] # (22356720, False)
c.__array_interface__['data'] # (22356720, False)
Use numpy.squeeze:
>>> x = np.array([[[0], [1], [2]]])
>>> x.shape
(1, 3, 1)
>>> np.squeeze(x).shape
(3,)
>>> np.squeeze(x, axis=(2,)).shape
(1, 3)
Slice along the dimension you want, as in the example below. To go in the reverse direction, you can use None as the slice for any dimension that should be treated as a singleton dimension, but which is needed to make shapes work.
In [786]: yy = np.asarray([[11],[7]])
In [787]: yy
Out[787]:
array([[11],
[7]])
In [788]: yy.shape
Out[788]: (2, 1)
In [789]: yy[:,0]
Out[789]: array([11, 7])
In [790]: yy[:,0].shape
Out[790]: (2,)
In [791]: y1 = yy[:,0]
In [792]: y1.shape
Out[792]: (2,)
In [793]: y1[:,None]
Out[793]:
array([[11],
[7]])
In [794]: y1[:,None].shape
Out[794]: (2, 1)
Alternatively, you can use reshape:
In [795]: yy.reshape((2,))
Out[795]: array([11, 7])
the opposite translation can be made by:
np.atleast_2d(y).T
Another option in your toolbox could be ravel:
>>> y2.shape
(2, 1)
>>> y_ = y2.ravel()
>>> y_.shape
(2,)
Again, a copy is made only if needed, but this is not the case:
>>> y2.__array_interface__["data"]
(2700295136768, False)
>>> y_.__array_interface__["data"]
(2700295136768, False)
For further details, you can take a look at this answer.
In Python, I have a list and a numpy array.
I would like to multiply the array by the list in such a way that I get an array where the 3rd dimension represents the input array multiplied by each element of the list. Therefore:
in_list = [2,4,6]
in_array = np.random.rand(5,5)
result = ...
np.shape(result) ---> (3,5,5)
where (0,:,:) is the input array multiplied by the first element of the list (2);
(1,:,:) is the input array multiplied by the second element of the list (4), etc.
I have a feeling this question will be answered by broadcasting, but I'm not sure how to go around doing this.
You want np.multiply.outer. The outer method is defined for any NumPy "ufunc", including multiplication. Here's a demonstration:
In [1]: import numpy as np
In [2]: in_list = [2, 4, 6]
In [3]: in_array = np.random.rand(5, 5)
In [4]: result = np.multiply.outer(in_list, in_array)
In [5]: result.shape
Out[5]: (3, 5, 5)
In [6]: (result[1, :, :] == in_list[1] * in_array).all()
Out[6]: True
As you suggest, broadcasting gives an alternative solution: if you convert in_list to a 1d NumPy array of length 3, you can then reshape to an array of shape (3, 1, 1), and then a multiplication with in_array will broadcast appropriately:
In [9]: result2 = np.array(in_list)[:, None, None] * in_array
In [10]: result2.shape
Out[10]: (3, 5, 5)
In [11]: (result2[1, :, :] == in_list[1] * in_array).all()
Out[11]: True
I have an M-dimensional np.ndarray, where M <= N. Beyond this condition, the array may have any shape. I want to convert this array to N-dimensional, with dimensions 0 through M kept the same and dimensions M through N set to 1.
I can almost accomplish this behavior by copying the array using np.array and supplying the the ndmin argument. However, this places extra axis to the 'first' rather than 'last' positions:
>>> a3d = np.zeros((2,3,4))
>>> a5d = np.array(a3d, ndmin = 5)
>>> a5d.shape
(1, 1, 2, 3, 4) #actual shape
(2, 3, 4, 1, 1) #desired shape
Is there a way to specify where the added dimensions should go? Is there an alternate function I can use here which can result in my desired output?
Obviously in the example above I could manipulate the array after the fact to put axes in the order I want them, but since the orignal array could have had anywhere from 0 to 5 dimensions (and I want to keep original dimensions in the original order), I can't think of a way to do that without a tedious series of checks on the original shape.
I'd use .reshape ...
>>> a3d = a3d.reshape(a3d.shape + (1, 1))
>>> a3d.shape
(2, 3, 4, 1, 1)
If you want to pad up to a certain dimensionality:
>>> a3d = np.zeros((2,3,4))
>>> ndim = 5
>>> padded_shape = (a3d.shape + (1,)*ndim)[:ndim]
>>> new_a3d = a3d.reshape(padded_shape)
>>> new_a3d.shape
(2, 3, 4, 1, 1)
Just set
a5d = np.array(a3d)
a5d.shape = a3d.shape + (1, 1)
print a5d.shape
(2, 3, 4, 1, 1)
since the arrays are of the same physical size