In order to make say it simply, I have a list of dimension [32, 31, 4] which I would like to reduce to shape [32, 31, 3] in order to replace every array in the last dimension by an array of size (3).
for a in range(len(liste)): #len(list) = 95
for b in range(len(liste[a])): #shape = [32, 31, 3], b travels in the 1st dim.
#print('frame : ', liste[a][b].shape) #[31, 4]
#print('b', b) #32 frames each time ok
for c in range(len(liste[a][b])):
#print('c', c) #31 each time ok
#print('norme du quaternion', np.abs(np.linalg.norm(liste[a][b][c]))) #norm = 1
r = quat2expmap(liste[a][b][c]) #convertion to expmap successful
#print('ExpMap : ', r)
quat = liste[a][b][c]
quat = r #this works
#print('quat', quat)
liste[a][b][c] = r #this doesn't work
To be more precise, I have a dataset of 95 different gestures each represented by 32 frames and quaternions. I converted the quaternions into ExpMap but due to the difference of shapes I am unable to replace the quaternions by their corresponding ExpMap. The error code I receive the most is the following:
ValueError: could not broadcast input array from shape (3) into shape (4)
It comes from the last line of the code.
The weirdest thing is that when I take the quaternion apart and replace it, it works parfectly, yet python would refuse that I do it inside my list. I don't really get why.
Could you lighten me about it? How could I get the proper dimension in my list? I tried all the tricks such as del, remove() but got no result...
You seem to be using numpy arrays (not Python lists). Numpy does not allow changing dimensions on assignment to an element of an array because it would become irregular (some entries with 4 and some with 3).
Also, iterating through numpy arrays using loops is the wrong way to use numpy. In this case you're probably looking at applying the quat2expmap function to the 4th dimension of your matrix to produce a new matrix of shape (95,32,31,3). This will make maximum use of numpy's parallelism and can be written in a couple of lines without any loops.
You could either modify the quat2expmap function so that it works directly on your 4d matrix (will be fastest approach) or use np.apply_along_axis (which is not much faster than loops).
Related
I'm having some trouble understanding the rules for array broadcasting in Numpy.
Obviously, if you perform element-wise multiplication on two arrays of the same dimensions and shape, everything is fine. Also, if you multiply a multi-dimensional array by a scalar it works. This I understand.
But if you have two N-dimensional arrays of different shapes, it's unclear to me exactly what the broadcasting rules are. This documentation/tutorial explains that: In order to broadcast, the size of the trailing axes for both arrays in an operation must either be the same size or one of them must be one.
Okay, so I assume by trailing axis they are referring to the N in a M x N array. So, that means if I attempt to multiply two 2D arrays (matrices) with equal number of columns, it should work? Except it doesn't...
>>> from numpy import *
>>> A = array([[1,2],[3,4]])
>>> B = array([[2,3],[4,6],[6,9],[8,12]])
>>> print(A)
[[1 2]
[3 4]]
>>> print(B)
[[ 2 3]
[ 4 6]
[ 6 9]
[ 8 12]]
>>>
>>> A * B
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: shape mismatch: objects cannot be broadcast to a single shape
Since both A and B have two columns, I would have thought this would work. So, I'm probably misunderstanding something here about the term "trailing axis", and how it applies to N-dimensional arrays.
Can someone explain why my example doesn't work, and what is meant by "trailing axis"?
Well, the meaning of trailing axes is explained on the linked documentation page.
If you have two arrays with different dimensions number, say one 1x2x3 and other 2x3, then you compare only the trailing common dimensions, in this case 2x3. But if both your arrays are two-dimensional, then their corresponding sizes have to be either equal or one of them has to be 1. Dimensions along which the array has size 1 are called singular, and the array can be broadcasted along them.
In your case you have a 2x2 and 4x2 and 4 != 2 and neither 4 or 2 equals 1, so this doesn't work.
From http://cs231n.github.io/python-numpy-tutorial/#numpy-broadcasting:
Broadcasting two arrays together follows these rules:
If the arrays do not have the same rank, prepend the shape of the lower rank array with 1s until both shapes have the same length.
The two arrays are said to be compatible in a dimension if they have the same size in the dimension, or if one of the arrays has size 1 in that dimension.
The arrays can be broadcast together if they are compatible in all dimensions.
After broadcasting, each array behaves as if it had shape equal to the elementwise maximum of shapes of the two input arrays.
In any dimension where one array had size 1 and the other array had size greater than 1, the first array behaves as if it were copied along that dimension
If this explanation does not make sense, try reading the explanation from the documentation or this explanation.
we should consider two points about broadcasting. first: what is possible. second: how much of the possible things is done by numpy.
I know it might look a bit confusing, but I will make it clear by some example.
lets start from the zero level.
suppose we have two matrices. first matrix has three dimensions (named A) and the second has five (named B). numpy tries to match last/trailing dimensions. so numpy does not care about the first two dimensions of B. then numpy compares those trailing dimensions with each other. and if and only if they be equal or one of them be 1, numpy says "O.K. you two match". and if it these conditions don't satisfy, numpy would "sorry...its not my job!".
But I know that you may say comparison was better to be done in way that can handle when they are devisable(4 and 2 / 9 and 3). you might say it could be replicated/broadcasted by a whole number(2/3 in out example). and i am agree with you. and this is the reason I started my discussion with a distinction between what is possible and what is the capability of numpy.
I have 1-dimensional numpy array and want to store sparse updates of it.
Say I have array of length 500000 and want to do 100 updates of 100 elements. Updates are either adds or just changing the values (I do not think it matters).
What is the best way to do it using numpy?
I wanted to just store two arrays: indices, values_to_add and therefore have two objects: one stores dense matrix and other just keeps indices and values to add, and I can just do something like this with the dense matrix:
dense_matrix[indices] += values_to_add
And if I have multiple updates, I just concat them.
But this numpy syntax doesn't work fine with repeated elements: they are just ignored.
Updating pair when we have an update that repeats index is O(n). I thought of using dict instead of array to store updates, which looks fine from the point of view of complexity, but it doesn't look good numpy style.
What is the most expressive way to achieve this? I know about scipy sparse objects, but (1) I want pure numpy because (2) I want to understand the most efficient way to implement it.
If you have repeated indices you could use at, from the documentation:
Performs unbuffered in place operation on operand ‘a’ for elements
specified by ‘indices’. For addition ufunc, this method is equivalent
to a[indices] += b, except that results are accumulated for elements
that are indexed more than once.
Code
a = np.arange(10)
indices = [0, 2, 2]
np.add.at(a, indices, [-44, -55, -55])
print(a)
Output
[ -44 1 -108 3 4 5 6 7 8 9]
In Python/Numpy I can slice arrays in this form:
arr = np.ones((3,4,5))
arr[2]
and the shape will be maintained:
(arr[2]).shape # prints (4, 5)
Which means that, if I want to keep the shape of the array, the following code works for N-dimensional arrays
arr = np.ones((3,4,5,2,2))
(arr[2]).shape # prints (4, 5, 2, 2)
This is great if I want to write functions that work for N-dim arrays preserving their output.
In Julia, however, the same action does not preserve the structure:
arr = ones(3,4,5)
size(arr[3]) # prints () (0-dimensinoal)
size(arr[3,:]) # prints (20,)
because of partial linear indexing. So if want to keep the original dimensions I need to write arr[3,:,:], which only works for 3D arrays. If I want a 4D array I would have to use arr[3,:,:,:] and so on. The code isn't general.
Furthermore, when you get to array that are 5 dimensions or more (which is the case I'm working with now) this notation gets extremely cumbersome.
Is there any way I can write code like I do in Python and make it general? I couldn't even think of a nice clean way with reshape, let alone a way that's as clean as Python.
Notice that in Python the shape is only preserved if you slice the first dimension of the array. In Julia you can use slicedim(A, d, i) to slice dimension d of array A at index i.
Hello I have the following question. I create zero arrays of dimension (40,30,80). Now I need 7*7*7 of these zero arrays in an array. How can I do this?
One of my matrices is created like this:
import numpy as np
zeroMatrix = np.zeros((40,30,80))
My first method was to put the zero matrices in a 7*7*7 list. But i want to have it all in a numpy array. I know that there is a way with structured arrays I think, but i dont know how. If i copy my 7*7*7 list with np.copy() it creates a numpy array with the given shape, but there must be a way to do this instantly, isnt there?
EDIT
Maybe I have to make my question clearer. I have a 7*7 list of my zero matrices. In a for loop all of that arrays will be modified. In another step, this tempory list is appended to an empty list which will have a length of 7 in the end ( So i append the 7*7 list 7 times to the empty list. In the end I have a 7*7*7 List of those matrices. But I think this will be better If I have a numpy array of these zero matrices from the beginning.
Building an array of same-shaped arrays is not well supported by numpy which prefers to create a maximum depth array of minimum depth elements instead.
It turns out that numpy.frompyfunc is quite useful in circumventing this tendency where it is unwanted.
In your specific case one could do:
result = np.frompyfunc(zeroMatrix.copy, 0, 1)(np.empty((7, 7, 7), object))
Indeed:
>>> result.shape
(7, 7, 7)
>>> result.dtype
dtype('O')
>>> result[0, 0, 0].shape
(40, 30, 80)
I'm trying to build a matrix in numpy. The matrix dimensions should be (5001x7). Here is my code:
S=np.array([.0788,.0455,.0222,.0042,.0035,.0029,.0007])
#This is vector S, comprised of 7 scalars.
lamb=list(range(0,5001))
#This is a list of possible values for lambda, a parameter in my data.
M = np.empty([5001,7], order='C')
#This is the empty matrix which is to be filled in the iterations below.
for i in S:
for j in lamb:
np.append(M,((S[i]**2)/(lamb[j]+S[i]**2)))
The problem I'm having is that M remains a matrix of zero vectors.
Important details:
1) I've assigned the final line as:
M=np.append(M,((S[i]**2)/(lamb[j]+S[i]**2)))
I then get an array of values of length 70,014 in a 1d array. I'm not really sure what to make of it.
2) I've already tried switching the dtype parameter between 'float' and 'int' for matrix M.
3) I receive this warning when I run the code:
VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
app.launch_new_instance()
4) I'm working in Python 3.4
I really appreciate your help. Thank you!
1) append adds to the end of the array, which is why your final array has 5001x7x2=70014 elements. Only the first half is zeros. It flattens the array to 1D because you didn't specify an axis to append.
2) A much more "numpy" way to do this whole process is broadcasting
S=np.array([.0788,.0455,.0222,.0042,.0035,.0029,.0007])
lamb=np.arange(0,5001)
M=(S[:,None]**2)/(lamb[None,:]+S[:,None]**2)
np.append makes a copy of the array and appends values to the end of the copy (making the array larger each time), whereas I think you want to modify M in place:
for i in range(len(S)):
for j in range(len(lamb)):
M[j][i] = ((S[i]**2)/(lamb[j]+S[i]**2))