I'm trying to complete the following function, but I have been running into problems with the indexing, resulting in "ValueError: operands could not be broadcast together with shapes (0,9) (5)".
I think my error might be coming from how I'm trying to call the values from ssd_difference[], but I'm not entirely sure.
Also how would I go about using convolve2d based on the hint given below? I understand numpy has a function for it, but I have no idea what I would need to put in to make it work.
Additional information: binomialFilter5() returns a 5x1 numpy array of dtype float representing a binomial filter. I'm also assuming that the "weights[]" are the ssd_difference[] values.
def transitionDifference(ssd_difference):
""" Compute the transition costs between frames, taking dynamics into
account.
Instructions:
1. Iterate through the rows and columns of ssd difference, ignoring the
first two values and the last two values.
1a. For each value at i, j, multiply the binomial filter of length
five (implemented later in the code) by the weights starting two
frames before until two frames after, and take the sum of those
products.
i.e. Your weights for frame i are:
[weight[i - 2, j - 2],
weight[i - 1, j - 1],
weight[i, j],
weight[i + 1, j + 1],
weight[i + 2, j + 2]]
Multiply that by the binomial filter weights at each i, j to get
your output.
It may take a little bit of understanding to get why we are
computing this, the simple explanation is that to change from
frame 4 to 5, lets call this ch(4, 5), and we make this weight:
ch(4, 5) = ch(2, 3) + ch(3, 4) + ch(4, 5) + ch(5, 6) + ch(6, 7)
This accounts for the weights in previous changes and future
changes when considering the current frame.
Of course, we weigh all these sums by the binomial filter, so
that the weight ch(4, 5) is still the most important one, but
hopefully that gives you a better understanding.
Args:
ssd_difference (numpy.ndarray): A difference matrix as produced by your
ssd function.
Returns:
output (numpy.ndarray): A difference matrix that takes preceding and
following frames into account. The output
difference matrix should have the same dtype as
the input, but be 4 rows and columns smaller,
corresponding to only the frames that have valid
dynamics.
Hint: There is an efficient way to do this with 2d convolution. Think about
the coordinates you are using as you consider the preceding and
following frame pairings.
"""
output = np.zeros((ssd_difference.shape[0] - 4,
ssd_difference.shape[1] - 4), dtype=ssd_difference.dtype)
# WRITE YOUR CODE HERE.
for i in range(len(ssd_difference)):
for j in range(len(ssd_difference)):
if i == 0:
if j > 1:
output[i,j] = np.sum( ssd_difference[i-2:i+2]*binomialFilter5())
elif i == ssd_difference.shape[0] - 1:
if j < ssd_difference.shape[1] - 2:
output[i,j] = np.sum( ssd_difference[i-2:i+2]*binomialFilter5())
else:
output[i,j] = np.sum( ssd_difference[i-2:i+2]*binomialFilter5())
# END OF FUNCTION.
return output
As I commented, you really should tell us the line that produced the error message.
But I can guess, since there are just a couple of lines that do an operation that involves broadcasting. Most likely it is:
output[i,j] = np.sum( ssd_difference[i-2:i+2]*binomialFilter5())
You write that binomialFilter5() produces a (5,1) array, but the error talks about a (5,). It probably doesn't matter here, but you really should keep the number of dimensions straight. Sometimes (5,1) is signficantly different from (5,).
output has shape (ssd_difference.shape[0] - 4, ssd_difference.shape[1] - 4). But you are iterating i,j both over range(len(ssd_difference)). output[i,j] will eventually result in an index error. Especially when iterating over a 2d array, it is better to use the correct shape element, rather than len().
But I suspect the immediate error results from ssd_difference[i-2:i+2]. When i==0, this is ssd_difference[-2:2]. This is producing the (0,9) array, since the -2 index means second from the last, which is larger than 2.
I think you are intending to pull 5 rows from this array, to match the 5 values in the other array. A correct iteration, would I think be:
for i in range(output.shape[0]):
for j in range(output.shape[1]):
....
output[i,j] = np.sum(ssd_difference[i:i+5, :] * binomialFilter5())
...
You should test expressions like that individually in an interactive shell, with selected values of i. ssd_difference[i:i+5, :] should have shape (5,9), and binomialFilter5() should be (5,1).
Related
I have a problem where I want to multiply many 2x2 square matrices. The elements in the matrix are of the from a+ib (complex number)
Here's how the code looks like:
Final_list=[]
for x in list_x:
Matrix=np.array([[1,0],[0,1]])
for y in list_y:
Matrix_y=[....] #create a 2x2 matrix from values x and y
Matrix= Matrix.dot(Matrix_y)
value= 5/(Matrix[0,0] + Matrix[1,0]....) #something like this
Final_list.append(abs(value))
Final_list is what I need.
My code works but it takes a while and I feel like there must be a better way to write this. If length of list_x=30000 and list_y=300 it takes a while to compute ~ a minute or so.
I was able to create a large matrix of shape (300,30000,2,2). I did that by array broadcasting and using np.shape. I hoped to multiply all matrices in a column to get an array of matrices of shape (30000,2,2). I am thinking that might help in the computation but havent figured out a way to do it.
Is there a better way to write this instead of using multiple for loops?
Thanks
I don't know a fully vectorized solution. Initially I thought that np.matmul.reduce might work, but reduce seems to be unsupported for non-commutative operations according to the old docs. There is also np.linalg.multi_dot, but it seems like an overkill for 2 by 2 matrices.
Here is a answer that runs in 1.5 seconds on my machine; it is similar to the approach here.
arr = np.random.rand(300, 30_000, 2, 2)
def my_mul(mats):
if len(mats) == 1:
return mats[0,...]
elif len(mats) == 2:
return mats[0,...] # mats[1,...]
else:
mid = len(mats) // 2
return my_mul(mats[:mid,...]) # my_mul(mats[mid:,...])
result = my_mul(arr) # 1.53 s
Please consider the following python code
import matplotlib.pyplot as plt
import numpy as np
#create some data to plot.
dt = 0.001
t = np.arange(0.0,100,dt)
r = np.exp(-t[:1000]/0.05)
x = np.random.randn(len(t))
s = np.convolve(x,r)[:len(x)]*dt
The code compiles and runs and I largely understand what it is doing. However, I am confused about the code '[:len(x)]' is actually doing. If I truncate 's' to 'np.convolve(x,r)*dt', the code fails to compile and there is an error message from 'base.py' as follows:
"raise ValueError(f"x and y must have same first dimension, but "
ValueError: x and y must have same first dimension, but have shapes (100000,) and (100999,)"
What is '[:len(x)]' actually doing and is there something in the language documentation that gives some examples of this sort of context ?
Thanks.
All the objects are of type 'ndarray'.
t is length 100000
t is of shape (100000,)
r is length 1000
r is of shape (1000,)
x is length 100000
x is of shape (100000,)
s is length 100999
s is of shape (100999,)
If we read the docs for np.convolve, we see that with the default parameters, it returns an array that is one shorter than the sum of the lengths of the input array. That is if you call np.convolve(a, b), and len(a) = A and len(b) = B, the output is length A + B - 1.
This is because a convolution can be interpreted as integrating the product of two functions, with one of the functions shifted relative to the other. By default, np.convolve calculates this convolution for all points at which these functions overlap, so the length of the output is approximately the sum of the lengths of the input functions. In your case, x has length 100,000, and r has length 1,000, so the output length is 100,000 + 1,000 - 1 = 100,999.
You can change this behaviour with the mode parameter, so that np.convolve truncates the output automatically, but neither of the alternate options seem to match your use case. You could try supplying mode = same, which ensures the output is the same length as the longest input, and see what happens for your own interest though.
Since t - length 100,000 - and s need to be the same length so you can plot (I assume) s(t), you need to truncate the output s to a length of 100,000 to match.
This is what the notation [:len(x)] does. This is called "slice" notation, and the gist is that A[start:stop] allows you to select the subset of values in A from start (inclusive) to stop (exclusive). If you don't supply a start or end, it defaults to the start or end of the array respectively. So [:len(x)] picks from 0 to len(x) (exclusive) which gives you an array of length len(x). This ensures len(s) = len(x).
For alpha and k fixed integers with i < k also fixed, I am trying to encode a sum of the form
where all the x and y variables are known beforehand. (this is essentially the alpha coordinate of a big iterated matrix-vector multiplication)
For a normal sum varying over one index I usually create a 1d array A and set A[i] equal to the i indexed entry of the sum then use sum(A), but in the above instance the entries of the innermost sum depend on the indices in the previous sum, which in turn depend on the indices in the sum before that, all the way back out to the first sum which prevents me using this tact in a straightforward manner.
I tried making a 2D array B of appropriate length and width and setting the 0 row to be the entries in the innermost sum, then the 1 row as the entries in the next sum times sum(np.transpose(B),0) and so on, but the value of the first sum (of row 0) needs to vary with each entry in row 1 since that sum still has indices dependent on our position in row 1, so on and so forth all the way up to sum k-i.
A sum which allows for a 'variable' filled in by each position of the array it's summing through would thusly do the trick, but I can't find anything along these lines in numpy and my attempts to hack one together have thus far failed -- my intuition says there is a solution that involves summing along the axes of a k-i dimensional array, but I haven't been able to make this precise yet. Any assistance is greatly appreciated.
One simple attempt to hard-code something like this would be:
for j0 in range(0,n0):
for j1 in range(0,n1):
....
Edit: (a vectorized version)
You could do something like this: (I didn't test it)
temp = np.ones(n[k-i])
for j in range(0,k-i):
temp = x[:n[k-i-1-j],:n[k-i-j]].T#(y[:n[k-i-j]]*temp)
result = x[alpha,:n[0]]#(y[:n[0]]*temp)
The basic idea is that you try to press it into a matrix-vector form. (note that this is python3 syntax)
Edit: You should note that you need to change the "k-1" to where the innermost sum is (I just did it for all sums up to index k-i)
This is 95% identical to #sehigle's answer, but includes a generic N vector:
def nested_sum(XX, Y, N, alpha):
intermediate = np.ones(N[-1], dtype=XX.dtype)
for n1, n2 in zip(N[-2::-1], N[:0:-1]):
intermediate = np.sum(XX[:n1, :n2] * Y[:n2] * intermediate, axis=1)
return np.sum(XX[alpha, :N[0]] * Y[:N[0]] * intermediate)
Similarly, I have no knowledge of the expression, so I'm not sure how to build appropriate tests. But it runs :\
How do you convert [1, 2, 3] to [[1],[2],[3]] in python?
Also, say I have a vector of length m with values ranging from 1 to 10, I want to create a matrix of size mx10 such that say if vector y = 1 then the matrix should be [0,1,0,0,0,0,0,0,0,0]. In octave it was possible with,
y_train = zeros(m,output_layer_size);
for i=1:output_layer_size
y_train(find(y==i),i)=1;
end
But similar function gives out VisibleDeprecationWarning warning in python and does give desired output
y_train = np.zeros((y.shape[0],10))
for i in range(10):
y_train[y==i][i]=1
Adding a dimenstion to a vector in numpy is easy. You have a number of options available, depending on what you want to do:
Use np.newaxis, which is often aliased by None, in your index:
v = v[:, None]
OR
v = [None, :]
Using newaxis allows you to control precisely whether the vector becomes a column or a row.
Reshape the vector:
v = v.reshape((1, -1))
OR
v = np.reshape(v, (-1, 1))
I have really shown four options here (np.reshape vs np.ndarray.reshape and row vs column). Using -1 in the new vector's dimensions means "whatever size is necessary to make it the same number of elements as the original". It is much easier than explicitly using the shape.
Use np.expand_dims, which is almost exactly equivalent to np.newaxis, but in functional form.
Construct a new array with ndmin=2:
v = np.array(v, copy=False, ndmin=2)
This method is the least flexible because it does not let you control the position of the new axis. It is usually used when the only thing that matters is the dimensionality and broadcasting takes care of the rest.
The second part of the question appears to be a simple use-case for fancy indexing in Python. Here is as IDEOne link where I unrolled your octave loop. You can rephrase it in Python as:
y_train = np.zeros((y.size, m_output));
y_train[np.arange(y.size), y] = 1
Here is an IDEOne link of the demo.
Transposing 1D array directly will not work. It will return the original array. Try this instead:
np.atleast_2d(x).T
The ones from the comment did not work for me but numpy.where() worked!
b=np.array([[0],[0],[2],[2],[4],[1],[6],[7],[5],[9]])
a=np.random.randint(10,size=(10,10))
for i in range(10):
c=np.zeros((1,10))
c[0][i]=1
a[np.where(b==i)[0]] = c
print a
I am new to python, so please, bear with me!
This function:
def kerf(X,m):
[n_samples, ]= X.shape
n_sa, n_fe = m.shape
ker = np.zeros((n_samples, n_sa))
for i, x_i in enumerate(X):
for j, m_j in enumerate(m):
ker[i, j] = (np.dot(x_i, m_j)) # problem is here!!
return ker
I call it like this:
Z=kerf(myarray[0,[0,1]],myarray[:,[0,1]])
ker[i, j] = (np.dot(x_i, m_j))
ValueError: setting an array element with a sequence.
myarray is basically the same matrix. Why?
When I replace the problem line with:
print(np.dot(x_i, m_j).shape)
it repeatedly prints (2,).
ker[i, j] takes 1 value; 2 values is sequence.
Please give us the dimensions of the arrays at various points, such as myarray (I guessed and tried a (3,4)), and at the problem point. print(...shape) is an essential debugging tool in numpy.
Do you need help on figure out why it's (2,)? May I suggest stepping through the loop in an interactive shell, looking at shapes at various points along the way.
the 2 inputs to the dot look like:
(1.0, array([ 1., 1.]))
a scalar, and a 2 element array - so the dot is also a 2 element array.
You need to explain what size you expect these 2 arrays to be, and what size you expect the dot. Actually we can get the result - it's got to be (1,) or a scalar - 1 value to put in the one slot ker.
You can probably replace the double iteration with a single dot product (or if need be with an einsum call). But let's get this iteration working first.