Develop Reference

Coding an iterated sum of sums in python

For alpha and k fixed integers with i < k also fixed, I am trying to encode a sum of the form
where all the x and y variables are known beforehand. (this is essentially the alpha coordinate of a big iterated matrix-vector multiplication)
For a normal sum varying over one index I usually create a 1d array A and set A[i] equal to the i indexed entry of the sum then use sum(A), but in the above instance the entries of the innermost sum depend on the indices in the previous sum, which in turn depend on the indices in the sum before that, all the way back out to the first sum which prevents me using this tact in a straightforward manner.
I tried making a 2D array B of appropriate length and width and setting the 0 row to be the entries in the innermost sum, then the 1 row as the entries in the next sum times sum(np.transpose(B),0) and so on, but the value of the first sum (of row 0) needs to vary with each entry in row 1 since that sum still has indices dependent on our position in row 1, so on and so forth all the way up to sum k-i.
A sum which allows for a 'variable' filled in by each position of the array it's summing through would thusly do the trick, but I can't find anything along these lines in numpy and my attempts to hack one together have thus far failed -- my intuition says there is a solution that involves summing along the axes of a k-i dimensional array, but I haven't been able to make this precise yet. Any assistance is greatly appreciated.

One simple attempt to hard-code something like this would be:
for j0 in range(0,n0):
for j1 in range(0,n1):
....
Edit: (a vectorized version)
You could do something like this: (I didn't test it)
temp = np.ones(n[k-i])
for j in range(0,k-i):
temp = x[:n[k-i-1-j],:n[k-i-j]].T#(y[:n[k-i-j]]*temp)
result = x[alpha,:n[0]]#(y[:n[0]]*temp)
The basic idea is that you try to press it into a matrix-vector form. (note that this is python3 syntax)
Edit: You should note that you need to change the "k-1" to where the innermost sum is (I just did it for all sums up to index k-i)

This is 95% identical to #sehigle's answer, but includes a generic N vector:
def nested_sum(XX, Y, N, alpha):
intermediate = np.ones(N[-1], dtype=XX.dtype)
for n1, n2 in zip(N[-2::-1], N[:0:-1]):
intermediate = np.sum(XX[:n1, :n2] * Y[:n2] * intermediate, axis=1)
return np.sum(XX[alpha, :N[0]] * Y[:N[0]] * intermediate)
Similarly, I have no knowledge of the expression, so I'm not sure how to build appropriate tests. But it runs :\

How to convert row list to column list

How do you convert [1, 2, 3] to [[1],[2],[3]] in python?
Also, say I have a vector of length m with values ranging from 1 to 10, I want to create a matrix of size mx10 such that say if vector y = 1 then the matrix should be [0,1,0,0,0,0,0,0,0,0]. In octave it was possible with,
y_train = zeros(m,output_layer_size);
for i=1:output_layer_size
y_train(find(y==i),i)=1;
end
But similar function gives out VisibleDeprecationWarning warning in python and does give desired output
y_train = np.zeros((y.shape[0],10))
for i in range(10):
y_train[y==i][i]=1

Adding a dimenstion to a vector in numpy is easy. You have a number of options available, depending on what you want to do:
Use np.newaxis, which is often aliased by None, in your index:
v = v[:, None]
OR
v = [None, :]
Using newaxis allows you to control precisely whether the vector becomes a column or a row.
Reshape the vector:
v = v.reshape((1, -1))
OR
v = np.reshape(v, (-1, 1))
I have really shown four options here (np.reshape vs np.ndarray.reshape and row vs column). Using -1 in the new vector's dimensions means "whatever size is necessary to make it the same number of elements as the original". It is much easier than explicitly using the shape.
Use np.expand_dims, which is almost exactly equivalent to np.newaxis, but in functional form.
Construct a new array with ndmin=2:
v = np.array(v, copy=False, ndmin=2)
This method is the least flexible because it does not let you control the position of the new axis. It is usually used when the only thing that matters is the dimensionality and broadcasting takes care of the rest.
The second part of the question appears to be a simple use-case for fancy indexing in Python. Here is as IDEOne link where I unrolled your octave loop. You can rephrase it in Python as:
y_train = np.zeros((y.size, m_output));
y_train[np.arange(y.size), y] = 1
Here is an IDEOne link of the demo.

Transposing 1D array directly will not work. It will return the original array. Try this instead:
np.atleast_2d(x).T

The ones from the comment did not work for me but numpy.where() worked!
b=np.array([[0],[0],[2],[2],[4],[1],[6],[7],[5],[9]])
a=np.random.randint(10,size=(10,10))
for i in range(10):
c=np.zeros((1,10))
c[0][i]=1
a[np.where(b==i)[0]] = c
print a

Numpy array, indexing and convolving confusion

I'm trying to complete the following function, but I have been running into problems with the indexing, resulting in "ValueError: operands could not be broadcast together with shapes (0,9) (5)".
I think my error might be coming from how I'm trying to call the values from ssd_difference[], but I'm not entirely sure.
Also how would I go about using convolve2d based on the hint given below? I understand numpy has a function for it, but I have no idea what I would need to put in to make it work.
Additional information: binomialFilter5() returns a 5x1 numpy array of dtype float representing a binomial filter. I'm also assuming that the "weights[]" are the ssd_difference[] values.
def transitionDifference(ssd_difference):
""" Compute the transition costs between frames, taking dynamics into
account.
Instructions:
1. Iterate through the rows and columns of ssd difference, ignoring the
first two values and the last two values.
1a. For each value at i, j, multiply the binomial filter of length
five (implemented later in the code) by the weights starting two
frames before until two frames after, and take the sum of those
products.
i.e. Your weights for frame i are:
[weight[i - 2, j - 2],
weight[i - 1, j - 1],
weight[i, j],
weight[i + 1, j + 1],
weight[i + 2, j + 2]]
Multiply that by the binomial filter weights at each i, j to get
your output.
It may take a little bit of understanding to get why we are
computing this, the simple explanation is that to change from
frame 4 to 5, lets call this ch(4, 5), and we make this weight:
ch(4, 5) = ch(2, 3) + ch(3, 4) + ch(4, 5) + ch(5, 6) + ch(6, 7)
This accounts for the weights in previous changes and future
changes when considering the current frame.
Of course, we weigh all these sums by the binomial filter, so
that the weight ch(4, 5) is still the most important one, but
hopefully that gives you a better understanding.
Args:
ssd_difference (numpy.ndarray): A difference matrix as produced by your
ssd function.
Returns:
output (numpy.ndarray): A difference matrix that takes preceding and
following frames into account. The output
difference matrix should have the same dtype as
the input, but be 4 rows and columns smaller,
corresponding to only the frames that have valid
dynamics.
Hint: There is an efficient way to do this with 2d convolution. Think about
the coordinates you are using as you consider the preceding and
following frame pairings.
"""
output = np.zeros((ssd_difference.shape[0] - 4,
ssd_difference.shape[1] - 4), dtype=ssd_difference.dtype)
# WRITE YOUR CODE HERE.
for i in range(len(ssd_difference)):
for j in range(len(ssd_difference)):
if i == 0:
if j > 1:
output[i,j] = np.sum( ssd_difference[i-2:i+2]*binomialFilter5())
elif i == ssd_difference.shape[0] - 1:
if j < ssd_difference.shape[1] - 2:
output[i,j] = np.sum( ssd_difference[i-2:i+2]*binomialFilter5())
else:
output[i,j] = np.sum( ssd_difference[i-2:i+2]*binomialFilter5())
# END OF FUNCTION.
return output

As I commented, you really should tell us the line that produced the error message.
But I can guess, since there are just a couple of lines that do an operation that involves broadcasting. Most likely it is:
output[i,j] = np.sum( ssd_difference[i-2:i+2]*binomialFilter5())
You write that binomialFilter5() produces a (5,1) array, but the error talks about a (5,). It probably doesn't matter here, but you really should keep the number of dimensions straight. Sometimes (5,1) is signficantly different from (5,).
output has shape (ssd_difference.shape[0] - 4, ssd_difference.shape[1] - 4). But you are iterating i,j both over range(len(ssd_difference)). output[i,j] will eventually result in an index error. Especially when iterating over a 2d array, it is better to use the correct shape element, rather than len().
But I suspect the immediate error results from ssd_difference[i-2:i+2]. When i==0, this is ssd_difference[-2:2]. This is producing the (0,9) array, since the -2 index means second from the last, which is larger than 2.
I think you are intending to pull 5 rows from this array, to match the 5 values in the other array. A correct iteration, would I think be:
for i in range(output.shape[0]):
for j in range(output.shape[1]):
....
output[i,j] = np.sum(ssd_difference[i:i+5, :] * binomialFilter5())
...
You should test expressions like that individually in an interactive shell, with selected values of i. ssd_difference[i:i+5, :] should have shape (5,9), and binomialFilter5() should be (5,1).

Getting numbers within a range from a gaussian_kde_resample array

I have a gaussian_kde.resample array. I don't know if it is a numpy array so that I can use numpy functions.
I had the data 0<x<=0.5 of 3000 variables and I used
kde = scipy.stats.gaussian_kde(x) # can also mention bandwidth here (x,bandwidth)
sample = kde.resample(100000) # returns 100,000 values that follow the prob distribution of "x"
This gave me a sample of data that follows the probability distribution of "x". But the problem is, no matter what bandwidth I try to select, I get very few negative values in my "sample". I only want values within the range 0 < sample <= 0.5
I tried to do:
sample = np.array(sample) # to convert this to a numpy array
keep = 0<sample<=0.5
sample = sample[keep] # using the binary conditions
But this does not work! How can I remove the negative values in my array?

Firstly, you can check what type it is by using the 'type' call within python:
x = kde.resample(10000)
type(x)
numpy.ndarray
Secondly, it should be working in the way you wrote, but I would be more explicit in your binary condition:
print x
array([[ 1.42935658, 4.79293343, 4.2725778 , ..., 2.35775067, 1.69647609]])
x.size
10000
y = x[(x>1.5) & (x<4)]
which you can see, does the correct binary conditions and removes the values >1.5 and <4:
print y
array([ 2.95451084, 2.62400183, 2.79426449, ..., 2.35775067, 1.69647609])
y.size
5676

I know I'm answering about 3 years late, but this may be useful for future reference.
The catch is that while kde.resample(100000) technically returns a NumPy array, this array actually contains another array(!), and that gets in the way of all the attempts to use indexing to get subsets of the sample. To get the array that the resample() method probably should have returned all along, do this instead:
sample = kde.resample(100000)[0]
The array variable sample should then have all 100000 samples, and indexing this array should work as expected.
Why SciPy does it this way, I don't know. This misfeature doesn't even appear to be documented.

First of all, the return value of kde.resample is a numpy array, so you do not need to reconvert it.
The problem lies in the line (Edit: No, it doesn't. This should work!)
keep = 0 < sample <= 0.5
It does not do what you would think. Try:
keep = (0 < sample) * (sample <= 0.5)

Selecting data on median value

I want to select one row of an array by the median value in one of the columns.
My method does not work the way I expect it to work, and it could be related to the representation/precision of the value returned by the numpy.median() function.
Here is a minimal working example and a workaround that I found:
import numpy as np
# Create an array with random numbers
some_array = np.random.rand(100)
# Try to select
selection = (some_array == np.median(some_array))
print len(some_array[selection]),len(some_array[~selection]) # Gives: 0, 100 -> selection fails
# Work-around
abs_dist_from_median = np.abs(some_array-np.median(some_array))
selection = (abs_dist_from_median == np.min(abs_dist_from_median))
print len(some_array[selection]),len(some_array[~selection]) # Gives: 1, 99 -> selection succeeded
It seems that the np.median() function returns a different representation off the number, thereby leading to a mismatch in the selection.
I find this behaviour strange, since by definition the median value of an array should be contained in the array. Any help/clarification would be appreciated!

First, the number of values is even such as [1, 2, 3, 4]. the median is (2+3)/2 not 2 or 3. If you change 100 to 101, it works properly. So your second approach is more appropriate on your purpose.
However, the best solution seems to use argsort as
some_array[some_array.argsort()[len(some_array)/2]]
Also, do not use == when you compare two float values. use np.isclose instead.