I have a simulation model that integrates a set of variables whose states are represented by numpy arrays of an arbitrary number of dimensions. After the simulation, I now have a list of arrays whose elements represent the variable state at a particular point in time.
In order to output the simulation results I want to split these arrays into multiple 1D arrays where the elements correspond to the same component of the state variable through time. Here is an example of a 2D state variable over a number of time steps.
import numpy as np
# Arbitrary state that is constant
arr = np.arange(9).reshape((3, 3))
# State variable through 3 time steps
state = [arr.copy() for _ in range(3)]
# Stack the arrays up to 3d. Axis could be rolled here if it makes it easier.
stacked = np.stack(state)
The output I need to get is:
[np.array([0, 0, 0]), np.array([1, 1, 1]), np.array([2, 2, 2]), ...]
I've tried doing np.split(stacked, sum(stacked.shape[:-1]), axis=...) (tried everything for axis=) but get the following error: ValueError: array split does not result in an equal division. Is there a way to do this using np.split or maybe np.nditer that will work for the general case?
I guess this would be equivalent to doing:
I, J, K = stacked.shape
result = []
for i in range(I):
for j in range(J):
result.append(stacked[i, j, :])
Which is also the ordering I'm hoping to get. Easy enough, however I'm hoping there is something in numpy that I can take advantage of for this that will be more general.
If I reshape it to a 9x3 array, then a simple list() will turn it into a list of 3 element arrays:
In [190]: stacked.reshape(-1,3)
Out[190]:
array([[0, 0, 0],
[1, 1, 1],
[2, 2, 2],
[3, 3, 3],
[4, 4, 4],
[5, 5, 5],
[6, 6, 6],
[7, 7, 7],
[8, 8, 8]])
In [191]: list(stacked.reshape(-1,3))
Out[191]:
[array([0, 0, 0]),
array([1, 1, 1]),
array([2, 2, 2]),
array([3, 3, 3]),
array([4, 4, 4]),
array([5, 5, 5]),
array([6, 6, 6]),
array([7, 7, 7]),
array([8, 8, 8])]
np.split(stacked.reshape(-1,3),9) produces a list of 1x3 arrays.
np.split only works on one axis, but you want to split on the 1st 2 - hence the need for a reshape or ravel.
And forget about nditer. That's a stepping stone to reworking code in cython. It does not help with ordinary iteration - except that when used in ndindex it can streamline your i,j double loop:
In [196]: [stacked[idx] for idx in np.ndindex(stacked.shape[:2])]
Out[196]:
[array([0, 0, 0]),
array([1, 1, 1]),
array([2, 2, 2]),
array([3, 3, 3]),
array([4, 4, 4]),
array([5, 5, 5]),
array([6, 6, 6]),
array([7, 7, 7]),
array([8, 8, 8])]
======================
With the different state, just stack on a different axis
In [302]: state
Out[302]:
[array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]]), array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]]), array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])]
In [303]: np.stack(state,axis=2).reshape(-1,3)
Out[303]:
array([[0, 0, 0],
[1, 1, 1],
[2, 2, 2],
[3, 3, 3],
[4, 4, 4],
[5, 5, 5],
[6, 6, 6],
[7, 7, 7],
[8, 8, 8]])
stack is rather like np.array, except it gives more control over where the dimension is added. But do look at it's code.
You could use np.split on a flattened version and cut into appropriate number of parts as 1D lists, like so -
np.split(stacked.ravel(),np.prod(stacked.shape[:2]))
Sample run -
In [406]: stacked
Out[406]:
array([[[0, 0, 0],
[1, 1, 1]],
[[2, 2, 2],
[3, 3, 3]],
[[4, 4, 4],
[5, 5, 5]],
[[6, 6, 6],
[7, 7, 7]]])
In [407]: np.split(stacked.ravel(),np.prod(stacked.shape[:2]))
Out[407]:
[array([0, 0, 0]),
array([1, 1, 1]),
array([2, 2, 2]),
array([3, 3, 3]),
array([4, 4, 4]),
array([5, 5, 5]),
array([6, 6, 6]),
array([7, 7, 7])]
Related
Say I have some time-series data in the form of a simple array.
X1 = np.array[(1, 2, 3, 4]
The Hankel matrix can be obtained by using scipy.linalg.hankel, which would look something like this:
hankel(X1)
array([[1, 2, 3, 4],
[2, 3, 4, 0],
[3, 4, 0, 0],
[4, 0, 0, 0]])
Now assume I had a larger array in the form of
X2 = np.array([1, 2, 3, 4, 5, 6, 7])
What I want to do is fill in the zeros in this matrix with the numbers that are next in the index (specific to each row). Taking the same Hankel matrix earlier by using the first four values in the array X2, I'd like to see the following output:
hankel(X2[:4])
array([[1, 2, 3, 4],
[2, 3, 4, 5],
[3, 4, 5, 6],
[4, 5, 6, 7]])
How would I do this? I'd ideally like to use this for larger data.
Appreciate any tips or pointers given. Thanks!
If you have a matrix with the appropriate index values into your dataset, you can use integer array indexing directly into your dataset.
To create the index matrix, you can simply use the upper-left quadrant of a double-sized Hankel array. There are likely simpler ways to create the index matrix, but this does the trick.
>>> X = np.array([9, 8, 7, 6, 5, 4, 3])
>>> N = 4 # the size of the "window"
>>> indices = scipy.linalg.hankel(np.arange(N*2))[:N, :N]
>>> indices
array([[0, 1, 2, 3],
[1, 2, 3, 4],
[2, 3, 4, 5],
[3, 4, 5, 6]])
>>> X[indices]
array([[9, 8, 7, 6],
[8, 7, 6, 5],
[7, 6, 5, 4],
[6, 5, 4, 3]])
I am attempting to compute the distance matrix for an ndarray that I have converted from pandas. I tried to convert the pandas df currently in this format:
move_df =
movement
0 [4, 3, 6, 2]
1 [5, 2, 3, 6, 2]
2 [4, 7, 2, 3, 6, 1]
3 [4, 4, 4, 3]
... ...
33410 [2, 6, 3, 1, 8]
[33410 x 1 columns]
to a numpy ndarray by using the following:
1) m = move_df.to_numpy()
2) m = pd.DataFrame(move_df.tolist()).values
3) m = [move_df.tolist() for i in move_df.columns]
Each of these conversions resulted in a numpy array in this format:
[[list([4, 3, 6, 2])]
[list([5, 2, 3, 6, 2])]
[list([4, 7, 2, 3, 6, 1])]
[list([4, 4, 4, 3])]
...
[list([2, 6, 3, 1, 8])]]
So when I try to run dtaidistance matrix, I get the following error:
d_m = dtw.distance_matrix(m)
TypeError: unsupported operand type(s) for -: 'list' and 'list'
But when I create a list of lists by copying and pasting several of the numpy arrays created with any of the methods mentioned above, the code works. But this is not feasible in the long run since the arrays are over 30k rows. Is there something I am doing wrong in the conversion from pandas df to numpy array? I used
print(type(m))
and it outputs that it is a numpy array and I already know that I cannot subtract a list from a list, hence the error.
EDIT:
For move_df.head(10).to_dict()
{'movement': {0: [4, 3, 6, 2],
1: [5, 2, 3, 6, 2],
2: [4, 7, 2, 3, 6, 1],
3: [4, 4, 4, 3],
4: [3, 6, 2, 3, 3],
5: [6, 2, 1],
6: [1, 1, 1, 1],
7: [7, 2, 3, 1, 1],
8: [7, 2, 3, 2, 1],
9: [6, 2, 3, 1]}}
(one of the dtaidistance authors here)
The dtaidistance package expects one of three formats:
A 2D numpy array (where all sequences have the same length by definition)
A Python list of 1D numpy.array or array.array.
A Python list of Python lists
In your case you could do:
series = move_df['movement'].to_list()
dtw.distance_matrix(series)
which works then on a list of lists.
To use the fast C implementation an array is required (either Numpy or std lib array). If you want to keep different lengths you can do
series = move_df['movement'].apply(lambda a: np.array(a, dtype=np.double)).to_list()
dtw.distance_matrix_fast(series)
Note that it might make sense to do the apply operation inplace on your move_df datastructure such that you only have to do it once and not keep track of two nearly identical datastructures. After you do this, the to_list call is sufficient. Thus:
move_df['movement'] = move_df['movement'].apply(lambda a: np.array(a, dtype=np.double))
series = move_df['movement'].to_list()
dtw.distance_matrix_fast(series)
If you want to use a 2D numpy matrix, you would need to truncate or pad all series to be the same length as is explained in other answers (for dtw padding is more common to not lose information).
ps. This assumes you want to do univariate DTW, the ndim subpackage for multivariate time series expects a different datastructure.
Assuming you want to form an array with the lists of length 4:
m = df['movement'].str.len().eq(4)
a = np.array(df.loc[m, 'movement'].to_list())
output:
array([[4, 3, 6, 2],
[4, 4, 4, 3],
[1, 1, 1, 1],
[6, 2, 3, 1]])
used input:
df = pd.DataFrame({'movement': [[4, 3, 6, 2],
[5, 2, 3, 6, 2],
[4, 7, 2, 3, 6, 1],
[4, 4, 4, 3],
[3, 6, 2, 3, 3],
[6, 2, 1],
[1, 1, 1, 1],
[7, 2, 3, 1, 1],
[7, 2, 3, 2, 1],
[6, 2, 3, 1]]})
A dataframe created with:
In [112]: df = pd.DataFrame({'movement': {0: [4, 3, 6, 2],
...: 1: [5, 2, 3, 6, 2],
...: 2: [4, 7, 2, 3, 6, 1],
...: 3: [4, 4, 4, 3],
...: 4: [3, 6, 2, 3, 3],
...: 5: [6, 2, 1],
...: 6: [1, 1, 1, 1],
...: 7: [7, 2, 3, 1, 1],
...: 8: [7, 2, 3, 2, 1],
...: 9: [6, 2, 3, 1]}})
has an object dtype column that contains lists. The array derived from that column is object dtype:
In [121]: arr = df['movement'].to_numpy()
In [122]: arr
Out[122]:
array([list([4, 3, 6, 2]), list([5, 2, 3, 6, 2]),
list([4, 7, 2, 3, 6, 1]), list([4, 4, 4, 3]),
list([3, 6, 2, 3, 3]), list([6, 2, 1]), list([1, 1, 1, 1]),
list([7, 2, 3, 1, 1]), list([7, 2, 3, 2, 1]), list([6, 2, 3, 1])],
dtype=object)
By selecting the column I get a 1d array, not the 2d you get. Otherwise it's the same
This cannot be converted into a 2d numeric dtype array. For most purposes we can think of this as a list of lists.
In [123]: arr.tolist()
Out[123]:
[[4, 3, 6, 2],
[5, 2, 3, 6, 2],
[4, 7, 2, 3, 6, 1],
[4, 4, 4, 3],
[3, 6, 2, 3, 3],
[6, 2, 1],
[1, 1, 1, 1],
[7, 2, 3, 1, 1],
[7, 2, 3, 2, 1],
[6, 2, 3, 1]]
If the lists were all the same length, or if we pick a subset, it is possible to construct a 2d array:
In [125]: arr[[0,3,6,9]]
Out[125]:
array([list([4, 3, 6, 2]), list([4, 4, 4, 3]), list([1, 1, 1, 1]),
list([6, 2, 3, 1])], dtype=object)
In [126]:
In [126]: np.stack(arr[[0,3,6,9]])
Out[126]:
array([[4, 3, 6, 2],
[4, 4, 4, 3],
[1, 1, 1, 1],
[6, 2, 3, 1]])
Padding and slicing could also be used to force the lists to matching lengths - but that could mean losing information.
But without knowing what dtw.distance_matrix expects (looks like it wants a 2d numeric array), or what these lists represent, I can't go further.
The fundamental point is that your dataframe contains lists that vary in length.
I'm a MatLab user who recently converted to python. I am running a for loop that cuts a longer signal into individual trials, normalizes them to 100% trial and then would like to have the trials listed horizontally in a single variable. My code is
RHipFE=np.empty([101, 1])
newlength = 101
for i in range(0,len(R0X)-1,2):
iHipFE=redataf.RHipFE[R0X[i]:R0X[i+1]]
x=np.arange(0,len(iHipFE),1)
new_x = np.linspace(x.min(), x.max(), newlength)
iHipFEn = interpolate.interp1d(x, iHipFE)(new_x)
RHipFE=np.concatenate((RHipFE,iHipFEn),axis=1)
When I run this, I get the error "ValueError: all the input arrays must have same number of dimensions". Which I assume is because RHipFE is (101,1) while iHipFEn is (101,). Is the best solution to make iHipFEn (101,1)? If so, how does one do this in the above for loop?
Generally it's faster to collect arrays in a list, and use some form of concatenate once. List append is faster than concatenate:
In [51]: alist = []
In [52]: for i in range(3):
...: alist.append(np.arange(i,i+5))
...:
In [53]: alist
Out[53]: [array([0, 1, 2, 3, 4]), array([1, 2, 3, 4, 5]), array([2, 3, 4, 5, 6])]
Various ways of joining
In [54]: np.vstack(alist)
Out[54]:
array([[0, 1, 2, 3, 4],
[1, 2, 3, 4, 5],
[2, 3, 4, 5, 6]])
In [55]: np.column_stack(alist)
Out[55]:
array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4],
[3, 4, 5],
[4, 5, 6]])
In [56]: np.stack(alist, axis=1)
Out[56]:
array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4],
[3, 4, 5],
[4, 5, 6]])
In [57]: np.array(alist)
Out[57]:
array([[0, 1, 2, 3, 4],
[1, 2, 3, 4, 5],
[2, 3, 4, 5, 6]])
Internally, vstack, column_stack, stack expand the dimension of the components, and concatenate on the appropriate axis:
In [58]: np.concatenate([l[:,None] for l in alist],axis=1)
Out[58]:
array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4],
[3, 4, 5],
[4, 5, 6]])
I'd like to obtain a 1D array of indexes from a 3D matrix.
For instance given x = np.random.randint(10, size=(10,3,3)), I'd like to do something like np.argmax(x, axis=(1,2)) just like you can do with np.max, that is, obtain a 1D array of length 10 containing the indexes (0 to 8) of the maximums of each submatrix of size (3,3).
I have not found anything helpful so far and I want to avoid looping on the first dimension (and use np.argmax(x)) as it is quite big.
Cheers!
Reshape to merge those last two axes and then use np.argmax -
idx = x.reshape(x.shape[0],-1).argmax(-1)
out = np.unravel_index(idx, x.shape[-2:])
Sample run -
In [263]: x = np.random.randint(10, size=(4,3,3))
In [264]: x
Out[264]:
array([[[0, 9, 2],
[7, 7, 8],
[2, 5, 9]],
[[1, 7, 2],
[8, 9, 0],
[2, 8, 3]],
[[7, 5, 0],
[7, 1, 6],
[5, 1, 1]],
[[0, 7, 3],
[5, 4, 1],
[9, 8, 9]]])
In [265]: idx = x.reshape(x.shape[0],-1).argmax(-1)
In [266]: np.unravel_index(idx, x.shape[-2:])
Out[266]: (array([0, 1, 0, 2]), array([1, 1, 0, 0]))
If you meant getting the merged index, then its simpler -
x.reshape(x.shape[0],-1).argmax(1)
Sample run -
In [283]: x
Out[283]:
array([[[2, 3, 7],
[8, 1, 0],
[3, 6, 9]],
[[8, 0, 5],
[2, 2, 9],
[9, 0, 9]],
[[1, 9, 2],
[5, 0, 3],
[7, 2, 1]],
[[1, 6, 5],
[2, 3, 7],
[7, 4, 6]]])
In [284]: x.reshape(x.shape[0],-1).argmax(1)
Out[284]: array([8, 5, 1, 5])
I want to extract the second and the 3rd to the fifth columns of the NumPy array, how would I go about it?
A = array([[0, 1, 2, 3, 4, 5, 6], [4, 5, 6, 7, 4, 5, 6]])
A[:, [1, 4:6]]
This obviously doesn't work.
Assuming I've understood you -- it's usually a good idea to explicitly specify the output you want, because it's not obvious -- you could use numpy.r_:
In [27]: A
Out[27]:
array([[0, 1, 2, 3, 4, 5, 6],
[4, 5, 6, 7, 4, 5, 6]])
In [28]: A[:, [1,3,4,5]]
Out[28]:
array([[1, 3, 4, 5],
[5, 7, 4, 5]])
In [29]: A[:, r_[1, 3:6]]
Out[29]:
array([[1, 3, 4, 5],
[5, 7, 4, 5]])
In [37]: A[1:, r_[1, 3:6]]
Out[37]: array([[5, 7, 4, 5]])
which you can then flatten or reshape as you like. r_ is basically a convenience function to generate the right indices, e.g.
In [30]: r_[1, 3:6]
Out[30]: array([1, 3, 4, 5])
Perhaps you are looking for this?
In [10]: A[1:, [1]+range(3,6)]
Out[10]: array([[5, 7, 4, 5]])
Note this gives you the second, fourth, fifth and six columns of all rows but the first.
The second element is A[:,1]. Elements 3-5 (I'm assuming you want inclusive) are A[:,2:5]. You won't be able to extract them with a single call. To get them as an array, you could do
import numpy as np
A = np.array([[0, 1, 2, 3, 4, 5, 6], [4, 5, 6, 7, 4, 5, 6]])
my_cols = np.hstack((A[:,1][...,np.newaxis], A[:,2:5]))
The np.newaxis stuff is just to make A[:,1] a 2D array, consistent with A[:,2:5].
Hope this helps.