I have a 3d numpy array of following form:
array([[[ 1., 5., 4.],
[ 1., 5., 4.],
[ 1., 2., 4.]],
[[ 3., 6., 4.],
[ 6., 6., 4.],
[ 6., 6., 4.]]])
Is there a efficient way to convert it to a 2d array of form:
array([[1, 1, 1, 5, 5, 2, 4, 4, 4],
[3, 6, 6, 6, 6, 6, 4, 4, 4]])
Thanks a lot!
In [54]: arr = np.array([[[ 1., 5., 4.],
[ 1., 5., 4.],
[ 1., 2., 4.]],
[[ 3., 6., 4.],
[ 6., 6., 4.],
[ 6., 6., 4.]]])
In [61]: arr.reshape((arr.shape[0], -1), order='F')
Out[61]:
array([[ 1., 1., 1., 5., 5., 2., 4., 4., 4.],
[ 3., 6., 6., 6., 6., 6., 4., 4., 4.]])
The array arr has shape (2, 3, 3). We wish to keep the first axis of length 2, and flatten the two axes of length 3.
If we call arr.reshape(h, w) then NumPy will attempt to reshape arr to shape (h, w). If we call arr.reshape(h, -1) then NumPy will replace the -1 with whatever integer is needed for the reshape to make sense -- in this case, arr.size/h.
Hence,
In [63]: arr.reshape((arr.shape[0], -1))
Out[63]:
array([[ 1., 5., 4., 1., 5., 4., 1., 2., 4.],
[ 3., 6., 4., 6., 6., 4., 6., 6., 4.]])
This is almost what we want, but notice that the values in each subarray, such as
[[ 1., 5., 4.],
[ 1., 5., 4.],
[ 1., 2., 4.]]
are being traversed by marching from left to right before going down to the next row.
We want to march down the rows before going on to the next column.
To achieve that, use order='F'.
Usually the elements in a NumPy array are visited in C-order -- where the last index moves fastest. If we visit the elements in F-order then the first index moves fastest. Since in a 2D array of shape (h, w), the first axis is associated with the rows and the last axis the columns, traversing the array in F-order marches down each row before moving on to the next column.
Related
I'd like to return a 2D numpy.array with multiple rolls of a given 1D numpy.array.
>>> multiroll(np.arange(10), [-1, 0, 1, 2])
array([[1., 0., 9., 8.],
[2., 1., 0., 9.],
[3., 2., 1., 0.],
[4., 3., 2., 1.],
[5., 4., 3., 2.],
[6., 5., 4., 3.],
[7., 6., 5., 4.],
[8., 7., 6., 5.],
[9., 8., 7., 6.],
[0., 9., 8., 7.]])
Is there some combination of numpy.roll, numpy.tile, numpy.repeat, or other functions that does this?
Here's what I've tried
def multiroll(array, rolls):
"""Create multiple rolls of 1D vector"""
m = len(array)
n = len(rolls)
shape = (m, n)
a = np.empty(shape)
for i, roll in enumerate(rolls):
a[:,i] = np.roll(array, roll)
return a
I'd expected there's a more "Numpythonic" way of doing this that doesn't use the loop.
Approach #1 : For elegance
Here's one way with broadcasting -
In [44]: a
Out[44]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
In [45]: rolls
Out[45]: array([-1, 0, 1, 2])
In [46]: a[(np.arange(len(a))[:,None]-rolls) % len(a)]
Out[46]:
array([[1, 0, 9, 8],
[2, 1, 0, 9],
[3, 2, 1, 0],
[4, 3, 2, 1],
[5, 4, 3, 2],
[6, 5, 4, 3],
[7, 6, 5, 4],
[8, 7, 6, 5],
[9, 8, 7, 6],
[0, 9, 8, 7]])
Approach #2 : For memory/perf-efficiency
Idea mostly borrowed from - this post.
We can leverage np.lib.stride_tricks.as_strided based scikit-image's view_as_windows to get sliding windows. More info on use of as_strided based view_as_windows.
from skimage.util.shape import view_as_windows
def multiroll_stridedview(a, r):
r = np.asarray(r)
# Concatenate with sliced to cover all rolls
a_ext = np.concatenate((a,a[:-1]))
# Get sliding windows; use advanced-indexing to select appropriate ones
n = len(a)
return view_as_windows(a_ext,n)[:,(n-r)%n]
Approach #3 : For mathematical beauty (and efficiency ?)
Using a fft kernel in the frequency domain you can process a whole matrix at once. This method only work with integer
A = np.array([[1., 1., 1., 1.],
[2., 2., 2., 2.],
[3., 3., 3., 3.],
[4., 4., 4., 4.],
[5., 5., 5., 5.],
[6., 6., 6., 6.],
[7., 7., 7., 7.],
[8., 8., 8., 8.],
[9., 9., 9., 9.],
[0., 0., 0., 0.]]).transpose()
m,n = A.shape
#shift vector
s=[-1,0,1,2]
#transformation kernel (shift theorem)
fftkernel = np.exp(-2*1j*np.pi/n*np.outer(v,np.arange(0,n)))
#Apply the shift
res=np.round(np.fft.ifft(np.fft.fft(A,axis = 1) * fftkernel ,axis = 1)).real.transpose()
We get:
array([[1., 0., 9., 8.],
[2., 1., 0., 9.],
[3., 2., 1., 0.],
[4., 3., 2., 1.],
[5., 4., 3., 2.],
[6., 5., 4., 3.],
[7., 6., 5., 4.],
[8., 7., 6., 5.],
[9., 8., 7., 6.],
[0., 9., 8., 7.]])
You can get more information about how this code work here
For left circular shift you can use:
fftkernel = np.exp(2*1j*np.pi/n*np.outer(v,np.arange(0,n)))
Without minus sign.
I have some array A, and the corresponding elements of the array bins contain each row's bin assignment. I want to construct an array S, such that
S[0, :] = (A[(bins == 0), :]).sum(axis=0)
This is rather easy to do with np.stack and list comprehensions, but it seems overly complicated and not terribly readable. Is there a more general way to sum (or even apply some general function to) slices of arrays with bin assignments? scipy.stats.binned_statistic is along the right lines, but requires that bin assignments and values to compute the functions on are the same shape (since I am using slices, this is not the case).
For example, if
A = np.array([[1., 2., 3., 4.],
[2., 3., 4., 5.],
[9., 8., 7., 6.],
[8., 7., 6., 5.]])
and
bins = np.array([0, 1, 0, 2])
then it should result in
S = np.array([[10., 10., 10., 10.],
[2., 3., 4., 5. ],
[8., 7., 6., 5. ]])
Here's an approach with matrix-multiplication using np.dot -
(bins == np.arange(bins.max()+1)[:,None]).dot(A)
Sample run -
In [40]: A = np.array([[1., 2., 3., 4.],
...: [2., 3., 4., 5.],
...: [9., 8., 7., 6.],
...: [8., 7., 6., 5.]])
In [41]: bins = np.array([0, 1, 0, 2])
In [42]: (bins == np.arange(bins.max()+1)[:,None]).dot(A)
Out[42]:
array([[ 10., 10., 10., 10.],
[ 2., 3., 4., 5.],
[ 8., 7., 6., 5.]])
Performance boost
A more efficient way to create the mask (bins == np.arange(bins.max()+1)[:,None]), would be like so -
mask = np.zeros((bins.max()+1, len(bins)), dtype=bool)
mask[bins, np.arange(len(bins))] = 1
You can use np.add.reduceat:
import numpy as np
# index to sort the bins
sort_index = bins.argsort()
# indices where the array needs to be split at
indices = np.concatenate(([0], np.where(np.diff(bins[sort_index]))[0] + 1))
# sum values where the bins are the same
np.add.reduceat(A[sort_index], indices, axis=0)
# array([[ 10., 10., 10., 10.],
# [ 2., 3., 4., 5.],
# [ 8., 7., 6., 5.]])
I am attempting to add two arrays.
np.zeros((6,9,20)) + np.array([1,2,3,4,5,6,7,8,9])
I want to get something out that is like
array([[[ 1., 1., 1., ..., 1., 1., 1.],
[ 2., 2., 2., ..., 2., 2., 2.],
[ 3., 3., 3., ..., 3., 3., 3.],
...,
[ 7., 7., 7., ..., 7., 7., 7.],
[ 8., 8., 8., ..., 8., 8., 8.],
[ 9., 9., 9., ..., 9., 9., 9.]],
[[ 1., 1., 1., ..., 1., 1., 1.],
[ 2., 2., 2., ..., 2., 2., 2.],
[ 3., 3., 3., ..., 3., 3., 3.],
...,
[ 7., 7., 7., ..., 7., 7., 7.],
[ 8., 8., 8., ..., 8., 8., 8.],
[ 9., 9., 9., ..., 9., 9., 9.]],
So adding entries to each of the matrices at the corresponding column. I know I can code it in a loop of some sort, but I am trying to use a more elegant / faster solution.
You can bring broadcasting into play after extending the dimensions of the second array with None or np.newaxis, like so -
np.zeros((6,9,20))+np.array([1,2,3,4,5,6,7,8,9])[None,:,None]
If I understand you correctly, the best thing to use is NumPy's Broadcasting. You can get what you want with the following:
np.zeros((6,9,20))+np.array([1,2,3,4,5,6,7,8,9]).reshape((1,9,1))
I prefer using the reshape method to using slice notation for the indices the way Divakar shows, because I've done a fair bit of work manipulating shapes as variables, and it's a bit easier to pass around tuples in variables than slices. You can also do things like this:
array1.reshape(array2.shape)
By the way, if you're really looking for something as simple as an array that runs from 0 to N-1 along an axis, check out mgrid. You can get your above output with just
np.mgrid[0:6,1:10,0:20][1]
You could use tile (but you would also need swapaxes to get the correct shape).
A = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9])
B = np.tile(A, (6, 20, 1))
C = np.swapaxes(B, 1, 2)
In order to do calculations, I have a set of arrays: "sub" array (as you can see below), and I want to reshape it in an array as given by "test" array:
import numpy as np
sub = np.array([[[[ 1., 1.],
[ 1., 1.]],
[[ 2., 2.],
[ 2., 2.]],
[[ 3., 3.],
[ 3., 3.]],
[[ 4., 4.],
[ 4., 4.]]],
[[[ 5., 5.],
[ 5., 5.]],
[[ 6., 6.],
[ 6., 6.]],
[[ 7., 7.],
[ 7., 7.]],
[[ 8., 8.],
[ 8., 8.]]]])
test=np.array([[[ 1., 1., 2., 2.],
[ 1., 1., 2., 2.],
[ 3., 3., 4., 4.],
[ 3., 3., 4., 4.]],
[[ 5., 5., 6., 6.],
[ 5., 5., 6., 6.],
[ 7., 7., 8., 8.],
[ 7., 7., 8., 8.]]])
I have found on a post a part of code which seems to work for my case, but I have some errors...
k,l,m,n,p =2,2,2,2,2
conc = np.array([np.ones([p,m,n],dtype=int)*i for i in range(k*l)])
test_reshape=np.vstack([np.hstack(sub[i:i+l]) for i in range(0,k*l*p,l)])
Here's an alternative way to swap, slice and stack your array into shape:
>>> t = sub.swapaxes(1, 3).T.swapaxes(1, 3)
>>> x = np.c_[t[::2, 0], t[1::2, 0]]
>>> y = np.c_[t[::2, 1], t[1::2, 1]]
>>> np.array((np.r_[x[0], x[1]], np.r_[y[0], y[1]]))
array([[[ 1., 1., 2., 2.],
[ 1., 1., 2., 2.],
[ 3., 3., 4., 4.],
[ 3., 3., 4., 4.]],
[[ 5., 5., 6., 6.],
[ 5., 5., 6., 6.],
[ 7., 7., 8., 8.],
[ 7., 7., 8., 8.]]])
Edit: Or instead, squeeze, slice and stack:
>>> x = np.c_[sub[:1][:,::2], sub[:1][:,1::2]].squeeze()
>>> y = np.c_[sub[1:][:,::2], sub[1:][:,1::2]].squeeze()
>>> np.array((np.r_[x[0], x[1]], np.r_[y[0], y[1]]))
# the required array
Perhaps there exists a pure numpy solution, but I'm not aware of it and it'll use quite a few tricks with strides. The solution below is thus not as efficient, because it uses python's for loops (making it less quick), but it 'll get your result in a general way, so without it depending on the size of your actual 4D array.
np.vstack( (sub[vol,2*sheet:2*sheet+2].reshape((4,-1)).T for vol in range(2) for sheet in range(2))).reshape((2,4,-1)
import numpy as np
sub = np.array(...)
test = np.array([np.hstack((np.vstack(( s[0],s[1] )),
np.vstack(( s[2],s[3] )))) for s in sub])
print test
In the OP's example the shape of sub is (2,4,2,2), but that the code above would work as is for an array of shape (n,4,m,m). For different shapes of the type (n,k,m,m) the code above can be adapted to different requirements.
Eventually I would like to add that when you look at the code you literally see what the code is achieving, and this may be compensating other defects of the code in terms of efficiency (i.e., copying vs reshaping).
A better solution (i.e, not mine ;-) and some aftertoughts
I have found this answer from unutbu (that contains a link to a more general solution) that the OP can easily (?) adapt to her/his needs. Due to the complex reshaping that is
involved data is however copied, hence the OP may want to measure the different performances of the two approaches, taking into account the incidence of the "reshaping" on the total run time of her/his program (i.e., imho shaving 0.3s on a runtime of 2' wouldn't be worth the effort)
Examplar interactive session
In the following, the data and the procedures are literally lifted from
the above mentioned answer from unutbu, with the last two statements added by me to show the addresses of the data buffers of the three ndarrays, x, y and z.
In [1]: import numpy as np
In [2]: x = np.arange(16).reshape((4,2,2))
In [3]: y = x.reshape(2,2,2,2).swapaxes(1,2).reshape(4,-1)
In [4]: x
Out[4]:
array([[[ 0, 1],
[ 2, 3]],
[[ 4, 5],
[ 6, 7]],
[[ 8, 9],
[10, 11]],
[[12, 13],
[14, 15]]])
In [5]: y
Out[5]:
array([[ 0, 1, 4, 5],
[ 2, 3, 6, 7],
[ 8, 9, 12, 13],
[10, 11, 14, 15]])
In [6]: z = x.T
In [7]: [a.__array_interface__['data'][0] for a in (x, y, z)]
Out[7]: [46375856, 45578800, 46375856]
In [8]:
This can be done using a reshape/swapaxes trick:
In [92]: sub.reshape(2,2,2,2,2).swapaxes(2,3).reshape(test.shape)
Out[92]:
array([[[ 1., 1., 2., 2.],
[ 1., 1., 2., 2.],
[ 3., 3., 4., 4.],
[ 3., 3., 4., 4.]],
[[ 5., 5., 6., 6.],
[ 5., 5., 6., 6.],
[ 7., 7., 8., 8.],
[ 7., 7., 8., 8.]]])
In [94]: np.allclose(sub.reshape(2,2,2,2,2).swapaxes(2,3).reshape(test.shape), test)
Out[94]: True
I confess I do not know how to generate this kind of solution without some guessing. But it appears that when you want to rearrange "blocks" in an array, there is a way to do it by reshaping to a higher dimension, swapping some axes, then reshaping to the desired shape. Given that sub.shape is (2, 4, 2, 2) reshaping to a higher dimension must mean (2, 2, 2, 2, 2). So you only have to test for a solution of the form
sub.reshape(2,2,2,2,2).swapaxes(i,j).reshape(test.shape)
and that is easy to do:
for i,j in IT.combinations(range(5), 2):
if np.allclose(sub.reshape(2,2,2,2,2).swapaxes(i,j).reshape(test.shape), test):
print(i,j)
reveals the right axes to swap:
(2, 3)
In python we do some thing like this for example:
n = 30
A = numpy.zeros(shape=(n,n))
for i in range(0, n):
for j in range(0, n):
A[i, j] = i+j
#i+j just example of assignment
To manage a 2-dim array. It's so simple. just use nest loop to walk around rows and columns.
But my friend told me why it's so complicated. Could you give me the another way to manage it?
He told me in Mathematica have some way more easier to manage n-dim array (I'm not sure. I've never use Mathematica)
Can you give me the alternative way to manage value assignment on n-dim matrix/array(in Numpy) or list(ordinary one in Python)?
You are looking for numpy.fromfunction:
>>> numpy.fromfunction(lambda x, y: x + y, (5, 5))
array([[ 0., 1., 2., 3., 4.],
[ 1., 2., 3., 4., 5.],
[ 2., 3., 4., 5., 6.],
[ 3., 4., 5., 6., 7.],
[ 4., 5., 6., 7., 8.]])
You can simplify slightly using operator:
>>> from operator import add
>>> numpy.fromfunction(add, (5, 5))
array([[ 0., 1., 2., 3., 4.],
[ 1., 2., 3., 4., 5.],
[ 2., 3., 4., 5., 6.],
[ 3., 4., 5., 6., 7.],
[ 4., 5., 6., 7., 8.]])
You can use the mathematical rules for matrixes and vectors:
n = 30
w = numpy.arange(n).reshape(1,-1)
A = w+w.T