Alright, here the given data;
There are three numpy arrays of the shapes:
(i, 4, 2), (i, 4, 3), (i, 4, 2)
the i is shared among them but is variable.
The dtype is float32 for everything.
The goal is to interweave them in a particular order. Let's look at the data at index 0 for these arrays:
[[-208. -16.]
[-192. -16.]
[-192. 0.]
[-208. 0.]]
[[ 1. 1. 1.]
[ 1. 1. 1.]
[ 1. 1. 1.]
[ 1. 1. 1.]]
[[ 0.49609375 0.984375 ]
[ 0.25390625 0.984375 ]
[ 0.25390625 0.015625 ]
[ 0.49609375 0.015625 ]]
In this case, the concatened target array would look something like this:
[-208, -16, 1, 1, 1, 0.496, 0.984, -192, -16, 1, 1, 1, ...]
And then continue on with index 1.
I don't know how to achieve this, as the concatenate function just keeps telling me that the shapes don't match. The shape of the target array does not matter much, just that the memoryview of it must be in the given order for upload to a gpu shader.
Edit: I could achieve this with a few python for loops, but the performance impact would be a problem in this program.
Use np.dstack and flatten with np.ravel() -
np.dstack((a,b,c)).ravel()
Now, np.dstack is basically stacking along the third axis. So, alternatively we can use np.concatenate too along that axis, like so -
np.concatenate((a,b,c),axis=2).ravel()
Sample run -
1) Setup Input arrays :
In [613]: np.random.seed(1234)
...: n = 3
...: m = 2
...: a = np.random.randint(0,9,(n,m,2))
...: b = np.random.randint(11,99,(n,m,2))
...: c = np.random.randint(101,999,(n,m,2))
...:
2) Check input values :
In [614]: a
Out[614]:
array([[[3, 6],
[5, 4]],
[[8, 1],
[7, 6]],
[[8, 0],
[5, 0]]])
In [615]: b
Out[615]:
array([[[84, 58],
[61, 87]],
[[48, 45],
[49, 78]],
[[22, 11],
[86, 91]]])
In [616]: c
Out[616]:
array([[[104, 359],
[376, 560]],
[[472, 720],
[566, 115]],
[[344, 556],
[929, 591]]])
3) Output :
In [617]: np.dstack((a,b,c)).ravel()
Out[617]:
array([ 3, 6, 84, 58, 104, 359, 5, 4, 61, 87, 376, 560, 8,
1, 48, 45, 472, 720, 7, 6, 49, 78, 566, 115, 8, 0,
22, 11, 344, 556, 5, 0, 86, 91, 929, 591])
What I would do is:
np.hstack([a, b, c]).flatten()
assuming a, b, c are the three arrays
Related
I have accelerometer data (x,y,z) which is being updated every 50ms. I need to store 80 values of the data into a 3D numpy array (1, 80, 3). For example:
[[[x,y,z] (at 0ms)
[x,y,z] (at 50ms)
...
[x,y,z]]] (at 4000ms)
After getting the first 80 values, I need to update the array with upcoming values, for example:
[[[x,y,z] (at 50ms)
[x,y,z] (at 100ms)
...
[x,y,z]]] (at 4050ms)
I'm sure there is a way to update the array without needing to manually write 80 variables to store the data into, but I can't think of it. Would really appreciate some help here.
It sounds like you want your array to always be 80 long, so what I would suggest is roll the array and then update the last value.
import numpy as np
data = np.arange(80*3).reshape(80, 3)
data
>>> array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
...,
[231, 232, 233],
[234, 235, 236],
[237, 238, 239]])
data = np.roll(data, -1, axis=0)
data
>>> array([[ 3, 4, 5], # this is second row (index 1) in above array
[ 6, 7, 8], # third row
[ 9, 10, 11], # etc.
...,
[234, 235, 236],
[237, 238, 239],
[ 0, 1, 2]]) # the first row has been rolled to the last position
# now update last position with new data
data[-1] = [x, y, z] # new xyz data
data
>>> data
>>> array([[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
...,
[234, 235, 236],
[237, 238, 239],
[ 76, 76, 76]]) # new data updates in correct position in array
You can use vstack (initializing the array for the first iteration):
data=[x,y,x] # first iteration
data=np.vstack([data,[x,y,z]]) # for the rest
print(data) # you would have a Nx3 array
For the update every N seconds it is easier if you use a FIFO or a ring buffer:
https://pypi.org/project/numpy_ringbuffer/
If I have a 1d array a and want to map a function f over each element, I could do
>>> import numpy as np
>>> a = np.arange(5)
>>> def f(x):
... return 3*x + x**2 #whatever
>>> np.fromiter(map(f,a),float)
array([ 0., 4., 10., 18., 28.])
I'd like to do something analogous with more complex arrays. One example calculation is this: compose paired 3x3 arrays with matrix multiplication
>>> a = np.arange(5*2*3**2).reshape(5,2,3,3)
>>> def f(x):
... return np.matmul(x[0],x[1])
# is there a smarter way?
>>> np.array([f(x) for x in a])
array([[[ 42, 45, 48],
[ 150, 162, 174],
[ 258, 279, 300]],
[[ 1716, 1773, 1830],
[ 1986, 2052, 2118],
[ 2256, 2331, 2406]],
[[ 5334, 5445, 5556],
[ 5766, 5886, 6006],
[ 6198, 6327, 6456]],
[[10896, 11061, 11226],
[11490, 11664, 11838],
[12084, 12267, 12450]],
[[18402, 18621, 18840],
[19158, 19386, 19614],
[19914, 20151, 20388]]])
Another example calculation would be transform every vector in an array of vectors by matrix multiplication
>>> a = np.arange(3*5).reshape(5,3)
>>> def f(x):
... M = np.arange(3*3).reshape(3,3)
... return np.dot(M,x)
>>> np.array([f(x) for x in a])
array([[ 5, 14, 23],
[ 14, 50, 86],
[ 23, 86, 149],
[ 32, 122, 212],
[ 41, 158, 275]])
Is there a nice way to do such computations with an np.fromiter like approach? What is the most pythonic way to do these operations with numpy? Is there an approach which handles every example problem here as np.specialnumpything(map(f,a))?
This is just as easily implemented with broadcasting. Namely:
a = np.arange(5)
a*3 + a**2
array([ 0, 4, 10, 18, 28])
a = np.arange(5*2*3**2).reshape(5,2,3,3)
a[:, 0] # a[:, 1]
array([[[ 42, 45, 48],
[ 150, 162, 174],
[ 258, 279, 300]],
[[ 1716, 1773, 1830],
[ 1986, 2052, 2118],
[ 2256, 2331, 2406]],
[[ 5334, 5445, 5556],
[ 5766, 5886, 6006],
[ 6198, 6327, 6456]],
[[10896, 11061, 11226],
[11490, 11664, 11838],
[12084, 12267, 12450]],
[[18402, 18621, 18840],
[19158, 19386, 19614],
[19914, 20151, 20388]]])
a = np.arange(3*5).reshape(5,3)
M = np.arange(3*3).reshape(3,3)
M.dot(a.T).T
array([[ 5, 14, 23],
[ 14, 50, 86],
[ 23, 86, 149],
[ 32, 122, 212],
[ 41, 158, 275]])
np.einsum('kj, ij -> ik', M, a)
array([[ 5, 14, 23],
[ 14, 50, 86],
[ 23, 86, 149],
[ 32, 122, 212],
[ 41, 158, 275]])
I would use the builtin numpy.nditer, which could be what you're looking for:
https://docs.scipy.org/doc/numpy/reference/arrays.nditer.html
from the examples:
>>> a = np.arange(6).reshape(2,3)
>>> a
>>> array([[0, 1, 2],
[3, 4, 5]])
>>> with np.nditer(a, op_flags=['readwrite']) as it:
... for x in it:
... x[...] = 2 * x
...
>>> a
array([[ 0, 2, 4],
[ 6, 8, 10]])
I want to create a bumpy array from two different bumpy arrays. For example:
Say I have 2 arrays a and b.
a = np.array([1,3,4])
b = np.array([[1,5,51,52],[2,6,61,62],[3,7,71,72],[4,8,81,82],[5,9,91,92]])
I want it to loop through each indices in array a and find it in array b and then save the row of b into c. Like below:
c = np.array([[1,5,51,52],
[3,7,71,72],
[4,8,81,82]])
I have tried doing:
c=np.zeros(shape=(len(b),4))
for i in b:
c[i]=a[b[i][:]]
but get this error "arrays used as indices must be of integer (or boolean) type"
Approach #1
If a is sorted, we can use np.searchsorted, like so -
idx = np.searchsorted(a,b[:,0])
idx[idx==a.size] = 0
out = b[a[idx] == b[:,0]]
Sample run -
In [160]: a
Out[160]: array([1, 3, 4])
In [161]: b
Out[161]:
array([[ 1, 5, 51, 52],
[ 2, 6, 61, 62],
[ 3, 7, 71, 72],
[ 4, 8, 81, 82],
[ 5, 9, 91, 92]])
In [162]: out
Out[162]:
array([[ 1, 5, 51, 52],
[ 3, 7, 71, 72],
[ 4, 8, 81, 82]])
If a is not sorted, we need to use sorter argument with searchsorted.
Approach #2
We can also use np.in1d -
b[np.in1d(b[:,0],a)]
I am trying to put many numpy files to get one big numpy file, I tried to follow those two links Append multiple numpy files to one big numpy file in python and Python append multiple files in given order to one big file this is what I did:
import matplotlib.pyplot as plt
import numpy as np
import glob
import os, sys
fpath ="/home/user/Desktop/OutFileTraces.npy"
npyfilespath ="/home/user/Desktop/test"
os.chdir(npyfilespath)
with open(fpath,'wb') as f_handle:
for npfile in glob.glob("*.npy"):
# Find the path of the file
filepath = os.path.join(npyfilespath, npfile)
print filepath
# Load file
dataArray= np.load(filepath)
print dataArray
np.save(f_handle,dataArray)
dataArray= np.load(fpath)
print dataArray
An example of the result that I have:
/home/user/Desktop/Trace=96
[[ 0.01518007 0.01499514 0.01479736 ..., -0.00392216 -0.0039761
-0.00402747]]
[[-0.00824758 -0.0081808 -0.00811402 ..., -0.0077236 -0.00765425
-0.00762086]]
/home/user/Desktop/Trace=97
[[ 0.00614908 0.00581004 0.00549154 ..., -0.00814741 -0.00813457
-0.00809347]]
[[-0.00824758 -0.0081808 -0.00811402 ..., -0.0077236 -0.00765425
-0.00762086]]
/home/user/Desktop/Trace=98
[[-0.00291786 -0.00309509 -0.00329287 ..., -0.00809861 -0.00797789
-0.00784175]]
[[-0.00824758 -0.0081808 -0.00811402 ..., -0.0077236 -0.00765425
-0.00762086]]
/home/user/Desktop/Trace=99
[[-0.00379887 -0.00410453 -0.00438963 ..., -0.03497837 -0.0353842
-0.03575151]]
[[-0.00824758 -0.0081808 -0.00811402 ..., -0.0077236 -0.00765425
-0.00762086]
this line represents the first trace:
[[-0.00824758 -0.0081808 -0.00811402 ..., -0.0077236 -0.00765425
-0.00762086]]
It is repeated all the time.
I asked the second question two days ago, at first I think that I had the best answer, but after trying to model to print and lot the final file 'OutFileTraces.npy' I found that my code:
1/ doesn't print numpy files from folder 'test' with respecting their order(trace0,trace1, trace2,...)
2/ saves only the last trace in the file, I mean by that when print or plot the OutFileTraces.npy, I found just one trace , it is the first one.
So I need to correct my code because really I am blocked. I would be very grateful if you could help me.
Thanks in advance.
Glob produces unordered lists. You need to sort explicitly with an extra line as the sorting procedure is in-place and does not return the list.
npfiles = glob.glob("*.npy")
npfiles.sort()
for npfile in npfiles:
...
NumPy files contain a single array. If you want to store several arrays in a single file you may have a look at .npz files with np.savez https://docs.scipy.org/doc/numpy/reference/generated/numpy.savez.html#numpy.savez I have not seen this in use widely, so you may wish seriously to consider alternatives.
If your arrays are all of the same shape and store related data, you can make a larger array. Say that the current shape is (N_1, N_2) and that you have N_0 such arrays. A loop with
all_arrays = []
for npfile in npfiles:
all_arrays.append(np.load(os.path.join(npyfilespath, npfile)))
all_arrays = np.array(all_arrays)
np.save(f_handle, all_array)
will produce a file with a single array of shape (N_0, N_1, N_2)
If you need per-name access to the arrays, HDF5 files are a good match. See http://www.h5py.org/ (a full intro is too much for a SO reply, see the quick start guide http://docs.h5py.org/en/latest/quick.html)
As discussed in
loading arrays saved using numpy.save in append mode
it is possible to save multiple times to an open file, and it possible to load multiple times. That's not documented, and probably not preferred, but it works. savez archive is the preferred method for saving multiple arrays.
Here's a toy example:
In [777]: with open('multisave.npy','wb') as f:
...: arr = np.arange(10)
...: np.save(f, arr)
...: arr = np.arange(20)
...: np.save(f, arr)
...: arr = np.ones((3,4))
...: np.save(f, arr)
...:
In [778]: ll multisave.npy
-rw-rw-r-- 1 paul 456 Feb 13 08:38 multisave.npy
In [779]: with open('multisave.npy','rb') as f:
...: arr = np.load(f)
...: print(arr)
...: print(np.load(f))
...: print(np.load(f))
...:
[0 1 2 3 4 5 6 7 8 9]
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
[[ 1. 1. 1. 1.]
[ 1. 1. 1. 1.]
[ 1. 1. 1. 1.]]
Here's a simple example of saving a list of arrays of the same shape
In [780]: traces = [np.arange(10),np.arange(10,20),np.arange(100,110)]
In [781]: traces
Out[781]:
[array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),
array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]),
array([100, 101, 102, 103, 104, 105, 106, 107, 108, 109])]
In [782]: arr = np.array(traces)
In [783]: arr
Out[783]:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109]])
In [785]: np.save('mult1.npy', arr)
In [786]: data = np.load('mult1.npy')
In [787]: data
Out[787]:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109]])
In [788]: list(data)
Out[788]:
[array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),
array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]),
array([100, 101, 102, 103, 104, 105, 106, 107, 108, 109])]
temp1 = tempObj[0][0]
temp2 = tempObj[0][1]
if len(tempObj) > 1:
for i in range(1, len(tempObj)):
temp1 = np.vstack((temp1, tempObj[i][0]))
temp2 = np.vstack((temp2, tempObj[i][1]))
The code is in Python. In the above code, I am trying to separate the numpy 3-d array (tempObj) into two 2-d arrays (temp1 & temp2) so that the respective columns are merged.
Is there a better or simpler way to do this?
use tempObj[:, 0, :] and tempObj[:, 1, :]. here is an example:
import numpy as np
tempObj = np.array([
[
[1,5], [2,7], [3,8], [4,4],
[6, 5], [4, 7], [13, 8], [9, 4]
],
[
[21,25], [22,72], [32,82], [34,43],
[64, 54], [44, 74], [1443, 48], [94, 44]
],
[
[211, 215], [212, 712], [312, 812], [314, 413],
[614, 514], [414, 714], [11443, 148], [194, 414]
]
])
# Your code:
############################################
temp1 = tempObj[0][0]
temp2 = tempObj[0][1]
if len(tempObj) > 1:
for i in range(1, len(tempObj)):
temp1 = np.vstack((temp1, tempObj[i][0]))
temp2 = np.vstack((temp2, tempObj[i][1]))
print(temp1)
print(temp2)
# my suggestion
############################################
print(tempObj[:, 0, :])
print(tempObj[:, 1, :])
You need to make sure that the type of tempObj (or any part of it) is numpy.ndarray and not list. You can do it by typing print(type(tempObj)). The following example is what I suspect that is your code that gives an error of TypeError: list indices must be integers, not tuple:
tempObj = [
np.array([
[1,5], [2,7], [3,8], [4,4],
[6, 5], [4, 7], [13, 8], [9, 4]
]),
np.array([
[21,25], [22,72], [32,82], [34,43],
[64, 54], [44, 74], [1443, 48], [94, 44]
]),
np.array([
[211, 215], [212, 712], [312, 812], [314, 413],
[614, 514], [414, 714], [11443, 148], [194, 414]
])
]