I'm working with Python3 and I would like to load datas from several CSV files.
Each CSV (one measurement) has 3 columns (3 different physical quantities). I want to load each quantity on 3 separate variables. For one CSV file this is quite simple, I used :
TIME,CH1,CH2 = loadtxt(file_path,usecols=(3,4,5),delimiter=',',skiprows=2,unpack=True)
and it worked fine. Now I would like to extend this procedure so I can load several CSV files. Each array would be 2D, each column representing one CSV file. Instead of having several CSV with three variables, I will have 3 2D arrays, which is much more convenient for data analysis.
I thought I could try something like this :
TIME = matrix(zeros((20480,len(file_path)))) # 20480 length of each column
CH1 = matrix(zeros((20480,len(file_path)))) # len(file_path) number of CSV files
CH2 = matrix(zeros((20480,len(file_path))))
for k in range(0,len(file_path)): # reading each CSV file
TIME[:,k],CH1[:,k],CH2[:,k] = loadtxt(file_path[k],usecols=(3,4,5),delimiter=',',skiprows=2,unpack=True)
But it's telling me :
ValueError: could not broadcast input array from shape (20480) into shape (20480,1)
In the end I would like variables looking like this :
TIME = matrix([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
...,
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]])
Each column is from one different CSV file.
I think this is a quite usual problem, but I don't really get how arrays works in Python. I get this idea from Matlab which is quite straightforward but here I don't know why indexing arrays with TIME[:][:] doesn't work.
Have you any idea how I could do this ?
Thanks.
Use np.array, not np.matrix
I can't emphasize this enough. np.matrix exists only for legacy reasons. See this answer for an explanation of the difference. np.matrix requires 2 dimensions, while np.array permits a single dimension when indexing. This seems to be the source of your error.
Here's a minimal example exhibiting the behaviour you are seeing:
A = np.array([[1, 2, 3], [4, 5, 6]])
B = np.matrix(A)
print(A[:, 0].shape) # (2,)
print(B[:, 0].shape) # (2, 1)
Therefore, define your resultant arrays as np.array objects:
m = 20480
n = len(file_path)
shape = (m, n)
TIME = np.zeros(shape)
CH1 = np.zeros(shape)
CH2 = np.zeros(shape)
Related
I'd like to assign multiple values to a tensor, but it seems that it's not supported at least in the way that is possible using numpy.
a = np.zeros((4, 4))
v = np.array([0, 2, 3, 1])
r = np.arange(4)
a[r, v] = 1
>>> a
array([[1., 0., 0., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.],
[0., 1., 0., 0.]])
The above works, but the tensorflow equivalent doesn't:
import tensorflow as tf
a = tf.zeros((4, 4))
v = tf.Variable([0, 2, 3, 1])
r = tf.range(4)
a[r, v].assign(1)
TypeError: Only integers, slices, ellipsis, tf.newaxis and scalar tensors are valid indices, got <tf.Tensor: shape=(4,), dtype=int32, numpy=array([0, 1, 2, 3])>
How could this be achieved? Are loops the only option? In my case the resulting array is indeed only slices of an identity matrix rearranged, so maybe that could be taken advantage of somehow.
Your example, which is updating a zero tensor at some indices to a certain value is most of time achieved through tf.scatter_nd :
idx = tf.stack([r,v],axis=-1)
tf.scatter_nd(idx, updates=tf.ones(4), shape=(4,4))
For more complex cases, you can look at the following functions:
tf.tensor_scatter_nd_add: Adds sparse updates to an existing tensor according to indices.
tf.tensor_scatter_nd_sub: Subtracts sparse updates from an existing tensor according to indices.
tf.tensor_scatter_nd_max: to copy element-wise maximum values from one tensor to another.
tf.tensor_scatter_nd_min: to copy element-wise minimum values from one tensor to another.
tf.tensor_scatter_nd_update: Scatter updates into an existing tensor according to indices.
You can read more in the guide: Introduction to tensor slicing
This question already has answers here:
What's the best way to create a "3D identity matrix" in Numpy?
(3 answers)
Closed 2 years ago.
The standard usage of the np.diag(a) function when given a 1D array a is to create a 2D array with the diagonal entries being the elements of a. In my case, a is a 2D array with size n x m. My goal is to generate an n x n x m array in a manner similar to the np.diag() function, where each n x n slice is a matrix of zeros with the m'th row of a in the diagonal. What is the best way of doing this? Clearly it can be done with the np.diag() function and a for loop, but I am wondering whether a vectorized version of this exists with numpy.
One way to accomplish this is to use the function np.broadcast_to, which broadcasts a given array to a new shape. I had trouble broadcasting the m dimension to the end of the array, but broadcasting it as the first dimension and then transposing along the first and last dimensions also seemed to work just fine.
Please see the code snippet below:
# Specify dimensions
n = 4
m = 3
# Create diagonal matrix
D = np.eye(n)
# Broadcast diagonal and transpose
B = np.transpose(np.broadcast_to(D, (m,) + D.shape), (2, 1, 0))
# Verify shape
print(B.shape)
--> (4, 4, 3)
# Verify correct slice
print(B[:, :, 0])
--> array([[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.]])
Hope this helps!
I try to store a list of different shaped arrays as a dtype=object array using np.save (I'm aware I could just pickle the list but I'm really curious how to do this).
If I do this:
import numpy as np
np.save('test.npy', [np.zeros((2, 2)), np.zeros((3,3))])
it works.
But this:
np.save('test.npy', [np.zeros((2, 2)), np.zeros((2,3))])
Gives me an error:
ValueError: could not broadcast input array from shape (2,2) into shape (2)
I guess np.save converts the list into an array first, so I tried:
x=np.array([np.zeros((2, 2)), np.zeros((3,3))])
y=np.array([np.zeros((2, 2)), np.zeros((2,3))])
Which has the same effect (first one works, second one doesn't.
The resulting x behaves as expected:
>>> x.shape
(2,)
>>> x.dtype
dtype('O')
>>> x[0].shape
(2, 2)
>>> x[0].dtype
dtype('float64')
I also tried to force the 'object' dtype:
np.array([np.zeros((2, 2)), np.zeros((2,3))], dtype=object)
Without success. It seems numpy tries to broadcast the array with equal first dimension into the new array and realizes too late that their shape is different. Oddly it seems to have worked at one point - so I'm really curious what the difference is, and how to do this properly.
EDIT:
I figured out the case it worked before: The only difference seems to be that the numpy arrays in the list have another data type.
It works with dtype('<f8'), but it doesn't with dtype('float64'), I'm not even sure what the difference is.
EDIT 2:
I found a very non-pythonic way to solve my issue, I add it here, maybe it helps to understand what I wanted to do:
array_list=np.array([np.zeros((2, 2)), np.zeros((2,3))])
save_array = np.empty((len(array_list),), dtype=object)
for idx, arr in enumerate(array_list):
save_array[idx] = arr
np.save('test.npy', save_array)
One of the first things that np.save does is
arr = np.asanyarray(arr)
So yes it is trying to turn your list into an array.
Constructing an object array from arbitrary sized arrays or lists is tricky. np.array(...) tries to create as high a dimensional array as it can, even attempting to concatenate the inputs if possible. The surest way is to do what you did - make the empty array and fill it.
A slightly more compact way of constructing the object array:
In [21]: alist = [np.zeros((2, 2)), np.zeros((2,3))]
In [22]: arr = np.empty(len(alist), dtype=object)
In [23]: arr[:] = alist
In [24]: arr
Out[24]:
array([array([[ 0., 0.],
[ 0., 0.]]),
array([[ 0., 0., 0.],
[ 0., 0., 0.]])], dtype=object)
Here are 3 scenarios:
Arrays that match in shape, combine into a 3d array:
In [27]: np.array([np.zeros((2, 2)), np.zeros((2,2))])
Out[27]:
array([[[ 0., 0.],
[ 0., 0.]],
[[ 0., 0.],
[ 0., 0.]]])
In [28]: _.shape
Out[28]: (2, 2, 2)
Arrays that don't match on the first dimension - create object array
In [29]: np.array([np.zeros((2, 2)), np.zeros((3,2))])
Out[29]:
array([array([[ 0., 0.],
[ 0., 0.]]),
array([[ 0., 0.],
[ 0., 0.],
[ 0., 0.]])], dtype=object)
In [30]: _.shape
Out[30]: (2,)
And awkward intermediate case (which may even be described as a bug). The first dimensions match, but the second ones don't):
In [31]: np.array([np.zeros((2, 2)), np.zeros((2,3))])
...
ValueError: could not broadcast input array from shape (2,2) into shape (2)
[ 0., 0.]])], dtype=object)
It's as though it initialized a (2,2,2) array, and then found that the (2,3) wouldn't fit. And the current logic doesn't allow it to backup and create the object array as it did in the previous scenario.
If you wanted to put the two (2,2) arrays in object array you'd have to use the create and fill logic.
Having looked over the man pages for numpy's eye and identity, I'd assumed that identity was a special case of eye, since it has fewer options (e.g. eye can fill shifted diagonals, identity cannot), but could plausibly run more quickly. However, this isn't the case on either small or large arrays:
>>> np.identity(3)
array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]])
>>> np.eye(3)
array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]])
>>> timeit.timeit("import numpy; numpy.identity(3)", number = 10000)
0.05699801445007324
>>> timeit.timeit("import numpy; numpy.eye(3)", number = 10000)
0.03787708282470703
>>> timeit.timeit("import numpy", number = 10000)
0.00960087776184082
>>> timeit.timeit("import numpy; numpy.identity(1000)", number = 10000)
11.379066944122314
>>> timeit.timeit("import numpy; numpy.eye(1000)", number = 10000)
11.247124910354614
What, then, is the advantage of using identity over eye?
identity just calls eye so there is no difference in how the arrays are constructed. Here's the code for identity:
def identity(n, dtype=None):
from numpy import eye
return eye(n, dtype=dtype)
As you say, the main difference is that with eye the diagonal can may be offset, whereas identity only fills the main diagonal.
Since the identity matrix is such a common construct in mathematics, it seems the main advantage of using identity is for its name alone.
To see the difference in an example, run the below codes:
import numpy as np
#Creates an array of 4 x 4 with the main diagonal of 1
arr1 = np.eye(4)
print(arr1)
print("\n")
#or you can change the diagonal position
arr2 = np.eye(4, k=1) # or try with another number like k= -2
print(arr2)
print("\n")
#but you can't change the diagonal in identity
arr3 = np.identity(4)
print(arr3)
np.identity returns a square matrix (special case of a 2D-array) which is an identity matrix with the main diagonal (i.e. 'k=0') as 1's and the other values as 0's. you can't change the diagonal k here.
np.eye returns a 2D-array, which fills the diagonal, i.e. 'k' which can be set, with 1's and rest with 0's.
So, the main advantage depends on the requirement. If you want an identity matrix, you can go for identity right away, or can call the np.eye leaving the rest to defaults.
But, if you need a 1's and 0's matrix of a particular shape/size or have a control over the diagonal you can go for eye method.
Just like how a matrix is a special case of an array, np.identity is a special case of np.eye.
Additional references:
Eye and Identity - HackerRank
I have a folder with many images (ordered by their creation time) that I can read into numpy float32 arrays. I want to write these arrays down to the filesystem in a single file in two different formats that a C programm (I can not modify) will access.
The first format is easy:
The values for the arrays one after another from left to right from top to bottom for every array. (The arrays come one after another that way). That I can do with np.tofile trivially.
The second format is more complicated:
For every pixel-coordinate (x, y) I want to write the corresponding pixels of all images one after another sequentially into the file. I tried to stack the arrays and then trnaspose the result. But when I write that down to the filesystem using np.tofile, the file contains the same arrangement of data as with the first format.
How can I tell numpy to rearrange the data?
For the second format, you could use column_stack followed by ravel
In [8]: img1 = np.arange(5, dtype='float32')
In [9]: img2 = np.arange(5, dtype='float32')
In [10]: np.column_stack((img1,img2)).ravel()
Out[10]: array([ 0., 0., 1., 1., 2., 2., 3., 3., 4., 4.], dtype=float32)