I am working on an real-time application. For this I need to store around 20 arrays per second. Each arrays consists of n Points with their respective x and y coordinate (z may follow as well in the future).
What I did come up with is some kind of a Ring Buffer, which takes the length of the total arrays (it's frames of a video btw.) and the number of the points with their coordinate (this doesn't change within one execution, but is variable for executions following).
My Buffer inits with an numpy array filled with zeros: np.zeros((lengthOfSlices,numberOfTrackedPoints))
However this seems to be problematic, because I write the whole Points for a Slice into the array at once, not after another. That means I can't broadcast the array as the shape is not correct.
Is there a numPythonic way to initialize the array with zeros and store vectorwise afterwards?
Below you can find what I have now:
class Buffer():
def __init__(self, lengthOfSlices, numberOfTrackedPoints):
self.data = np.zeros((lengthOfSlices,numberOfTrackedPoints))
self.index = 0
def extend(self, x):
'adds array x to ring buffer'
x_index = (self.index + np.arange(x.size)) % self.data.size
self.data[x_index] = x
self.index = x_index[-1] + 1
def get(self):
'returns the first-in-first-out data in the ring buffer'
idx = (self.index + np.arange(self.data.size)) % self.data.size
return self.data[idx]
You need to reshape the array based on the lenght of the frame.
Simple example:
>>> import numpy as np
>>> A = np.zeros(100)
>>> B = np.reshape(A, (10,10))
>>> B[0]
array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
So that's probably something like self.data = np.reshape(self.data, (lengthOfAFrame, 20))
EDIT:
Apparently reshaping is not your (only?) problem, you might check collections.deque for a python implementation of a circular buffer (source and example)
Related
I'm working with Python3 and I would like to load datas from several CSV files.
Each CSV (one measurement) has 3 columns (3 different physical quantities). I want to load each quantity on 3 separate variables. For one CSV file this is quite simple, I used :
TIME,CH1,CH2 = loadtxt(file_path,usecols=(3,4,5),delimiter=',',skiprows=2,unpack=True)
and it worked fine. Now I would like to extend this procedure so I can load several CSV files. Each array would be 2D, each column representing one CSV file. Instead of having several CSV with three variables, I will have 3 2D arrays, which is much more convenient for data analysis.
I thought I could try something like this :
TIME = matrix(zeros((20480,len(file_path)))) # 20480 length of each column
CH1 = matrix(zeros((20480,len(file_path)))) # len(file_path) number of CSV files
CH2 = matrix(zeros((20480,len(file_path))))
for k in range(0,len(file_path)): # reading each CSV file
TIME[:,k],CH1[:,k],CH2[:,k] = loadtxt(file_path[k],usecols=(3,4,5),delimiter=',',skiprows=2,unpack=True)
But it's telling me :
ValueError: could not broadcast input array from shape (20480) into shape (20480,1)
In the end I would like variables looking like this :
TIME = matrix([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
...,
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]])
Each column is from one different CSV file.
I think this is a quite usual problem, but I don't really get how arrays works in Python. I get this idea from Matlab which is quite straightforward but here I don't know why indexing arrays with TIME[:][:] doesn't work.
Have you any idea how I could do this ?
Thanks.
Use np.array, not np.matrix
I can't emphasize this enough. np.matrix exists only for legacy reasons. See this answer for an explanation of the difference. np.matrix requires 2 dimensions, while np.array permits a single dimension when indexing. This seems to be the source of your error.
Here's a minimal example exhibiting the behaviour you are seeing:
A = np.array([[1, 2, 3], [4, 5, 6]])
B = np.matrix(A)
print(A[:, 0].shape) # (2,)
print(B[:, 0].shape) # (2, 1)
Therefore, define your resultant arrays as np.array objects:
m = 20480
n = len(file_path)
shape = (m, n)
TIME = np.zeros(shape)
CH1 = np.zeros(shape)
CH2 = np.zeros(shape)
I have a 3-d array G with a size that changes in a loop. In Matlab, I first create an empty array G = [];
then I create the first element of G from another existing array D with size 256x256, it is simple to do that in matlab as follows
G(:,:,1) = D(:,:)
How can I do the same thing in Python?
Consider preallocating.
In python's numpy you can preallocate like this:
G = np.zeros([depth, height, width])
then you can slice G in a way similar to matlab, and substitue matrices in it. If you still want to have an array of changing size, you can create a list with your 2D arrays and then convert it to a np.array, like so:
G = list()
for i in range(N):
G.append(D)
G = np.array(G)
an empty 3d array should look something like this
n = 256
threeD = [[[0 for k in xrange(n)] for j in xrange(n)] for i in xrange(n)]
or if you want just one 256x256 2d array in the larger 3d array (which is what i think you are trying in Matlab)
threeD = [[[0 for k in xrange(n)] for j in xrange(n)]]
where n is the size of each "dimension".
it will give you an array full of 0, or you can replace 0 with None if that is more desirable as an empty array
also, it isn't really an "array" in python, it's a "list"
You can use
G[:,:,1]=D[:,:]
Example:
>>> G=np.zeros((2,2,2))
>>> D=np.ones((2,2))
>>> G[:,:,1]=D[:,:]
>>> G
array([[[ 0., 1.],
[ 0., 1.]],
[[ 0., 1.],
[ 0., 1.]]])
I have a cube of size N * N * N, say N=8. Each dimension of the cube is discretised to 1, so that I have labelled points (0,0,0), (0,0,1)..(N,N,N). At each labelled points, I would like to assign a random value, and thus produce an array which stores value at each vertex. For example val[0,0,0]=1, val[0,0,1]=1.2 val[0,1,0]=1.3, ...
How do I write a python code to acheive this?
Did you mean this:
import numpy as np
n = 5
val = np.empty((n, n, n)) # Create an 3d array full of 0's
val[0,0,0] = 11
val[0,0,1] = 33
print(val[0, 0])
array([ 11., 33., 0., 0., 0.])
You could simply generate lists of lists. While not in any way efficient, it would allow you to access your cube like val[0][0][0].
arr = [[[] for _ in range(8)] for _ in range(8)]
arr[0][0].append(1)
For large matrices, look into using numpy. This is the problem that it's designed to solve
numpy has a beautiful function which generate multidimensional grid. It is easy to work with it when number of dimension is low and is known in advance, but what to do when number of dimension is only known at time of execution or simply big and it takes too long to type. I guess I am looking for something like
import numpy as np
x = np.meshgrid(y)
where y is an array of arrays of evaluation points, for example
y = [array([-3., 0., 3.]) array([-3., 0., 3.]) array([-3., 0., 3.])]
Suggestions?
Use the *-operator (i.e. the unpacking operator):
x = np.meshgrid(*y)
See https://docs.python.org/2/tutorial/controlflow.html#unpacking-argument-lists
Having looked over the man pages for numpy's eye and identity, I'd assumed that identity was a special case of eye, since it has fewer options (e.g. eye can fill shifted diagonals, identity cannot), but could plausibly run more quickly. However, this isn't the case on either small or large arrays:
>>> np.identity(3)
array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]])
>>> np.eye(3)
array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]])
>>> timeit.timeit("import numpy; numpy.identity(3)", number = 10000)
0.05699801445007324
>>> timeit.timeit("import numpy; numpy.eye(3)", number = 10000)
0.03787708282470703
>>> timeit.timeit("import numpy", number = 10000)
0.00960087776184082
>>> timeit.timeit("import numpy; numpy.identity(1000)", number = 10000)
11.379066944122314
>>> timeit.timeit("import numpy; numpy.eye(1000)", number = 10000)
11.247124910354614
What, then, is the advantage of using identity over eye?
identity just calls eye so there is no difference in how the arrays are constructed. Here's the code for identity:
def identity(n, dtype=None):
from numpy import eye
return eye(n, dtype=dtype)
As you say, the main difference is that with eye the diagonal can may be offset, whereas identity only fills the main diagonal.
Since the identity matrix is such a common construct in mathematics, it seems the main advantage of using identity is for its name alone.
To see the difference in an example, run the below codes:
import numpy as np
#Creates an array of 4 x 4 with the main diagonal of 1
arr1 = np.eye(4)
print(arr1)
print("\n")
#or you can change the diagonal position
arr2 = np.eye(4, k=1) # or try with another number like k= -2
print(arr2)
print("\n")
#but you can't change the diagonal in identity
arr3 = np.identity(4)
print(arr3)
np.identity returns a square matrix (special case of a 2D-array) which is an identity matrix with the main diagonal (i.e. 'k=0') as 1's and the other values as 0's. you can't change the diagonal k here.
np.eye returns a 2D-array, which fills the diagonal, i.e. 'k' which can be set, with 1's and rest with 0's.
So, the main advantage depends on the requirement. If you want an identity matrix, you can go for identity right away, or can call the np.eye leaving the rest to defaults.
But, if you need a 1's and 0's matrix of a particular shape/size or have a control over the diagonal you can go for eye method.
Just like how a matrix is a special case of an array, np.identity is a special case of np.eye.
Additional references:
Eye and Identity - HackerRank