Develop Reference

Numpy applying a time interval sequence to a multidimensional ndarray (such as coordinates)

EDIT: added prefix / suffix value to interval arrays to make them the same length as their corresponding data arrays, as per #user1319128 's suggestion and indeed interp does the job. For sure his solution was workable and good. I just couldn't see it because I was tired and stupid.
I am sure this is a fairly mundane application, but so I have failed to find or come up with a way to do this without doing it outside of numpy. Maybe my brain just needs a rest, anyway here is the problem with example and solution requirements.
So I have to arrays with different lengths and I want to apply common time intervals between them to these arrays, so that that the result is I have versions of these arrays that are all the same length and their values relate to each other at the same row (if that makes sense). In the example below I have named this functionality "apply_timeintervals_to_array". The example code:
import numpy as np
from colorsys import hsv_to_rgb
num_xy = 20
num_colors = 12
#xy = np.random.rand(num_xy, 2) * 1080
xy = np.array([[ 687.32758344, 956.05651214],
[ 226.97671414, 698.48071588],
[ 648.59878864, 175.4882185 ],
[ 859.56600997, 487.25205922],
[ 794.43015178, 16.46114312],
[ 884.7166732 , 634.59100322],
[ 878.94218682, 835.12886098],
[ 965.47135726, 542.09202328],
[ 114.61867445, 601.74092126],
[ 134.02663822, 334.27221884],
[ 940.6589034 , 245.43354493],
[ 285.87902276, 550.32600784],
[ 785.00104142, 993.19960822],
[1040.49576307, 486.24009511],
[ 165.59409198, 156.79786175],
[1043.54280058, 313.09073855],
[ 645.62878826, 100.81909068],
[ 625.78003257, 252.17917611],
[1056.77009875, 793.02218098],
[ 2.93152052, 596.9795026 ]])
xy_deltas = np.sum((xy[1:] - xy[:-1])**2, axis=-1)
xy_ti = np.concatenate(([0.0],
(xy_deltas) / np.sum(xy_deltas)))
colors_ti = np.concatenate((np.linspace(0, 1, num_colors),
[1.0]))
common_ti = np.unique(np.sort(np.concatenate((xy_ti,
colors_ti))))
common_colors = (np.array(tuple(hsv_to_rgb(t, 0.9, 0.9) for t
in np.concatenate(([0.0],
common_ti,
[1.0]))))
* 255).astype(int)[1:-1]
common_xy = apply_timeintervals_to_array(common_ti, xy)
So one could then use the common arrays for additional computations or for rendering.
The question is what could accomplish the "apply_timeintervals_to_array" functionality, or alternatively a better way to generate the same data.
I hope this is clear enough, let me know if it isn't. Thank you in advance.

I think , numpy.interp should meet your expectations.For example, If a have an 2d array of length 20 , and would like to interpolate at different common_ti values ,whose length is 30 , the code would be as follows.
xy = np.arange(0,400,10).reshape(20,2)
xy_ti = np.arange(20)/19
common_ti = np.linspace(0,1,30)
x=np.interp(common_ti,xy_ti,xy[:,0]) # interpolate the first column
y=np.interp(common_ti,xy_ti,xy[:,1]) #interpolate the second column

Problem in the plotted array, which is the dft of a signal

I have an array including the sample values of a signal (121 samples). However, when I want to plt the Discrete Fourier Transform of it, I take this plot:
This is the related part of my code:
sp = np.fft.fft(flow)
n = np.arange(len(flow))
timeStep = 1
freq = np.fft.fftfreq(n.shape[-1], d=1)
plt.plot(freq, sp.real)
According to the plot every time, the plotted figure has two values. But, this is not sensible and possible. When I print the arrays, everything looks OK. Can anyone help me? Thanks a lot.
P.S.:
The real part of sp matrix is:
[ 4.62700000e+04 -2.64892524e+04 4.94317914e+03 8.58381182e+03
-2.05540197e+03 1.53516262e+03 -1.30716540e+04 1.74769311e+04
-1.13435074e+04 -3.79140600e+03 6.94722233e+03 -2.55937762e+03
2.62187832e+03 -7.91539720e+03 1.07849088e+04 -1.86067707e+02
-8.81467635e+03 5.39181241e+03 4.67386587e+03 -1.16464162e+04
2.25400000e+03 3.43226092e+02 -2.18100065e+03 -6.91513328e+03
7.67106151e+02 6.32196523e+03 -1.11715436e+04 3.84865629e+03
4.89120922e+03 -3.04642885e+03 -1.75000000e+02 2.98504637e+03
2.46837686e+03 -2.87114353e+03 -5.14905071e+02 4.95859846e+03
-2.79387832e+03 -3.71433195e+03 5.20579454e+03 3.77109275e+01
-1.31300000e+03 -2.36758839e+02 4.66440953e+03 4.50017683e+03
-8.51326995e+03 9.20006771e+03 3.47394048e+03 -7.50148888e+03
4.57289385e+03 2.52869599e+03 -3.16622233e+03 -2.08767047e+03
9.15962695e+02 1.44698611e+03 -8.07662141e+03 6.76627369e+03
-8.90969316e+03 6.48281486e+03 -3.46137363e+03 -3.44706367e+03
6.48400000e+03 -3.44706367e+03 -3.46137363e+03 6.48281486e+03
-8.90969316e+03 6.76627369e+03 -8.07662141e+03 1.44698611e+03
9.15962695e+02 -2.08767047e+03 -3.16622233e+03 2.52869599e+03
4.57289385e+03 -7.50148888e+03 3.47394048e+03 9.20006771e+03
-8.51326995e+03 4.50017683e+03 4.66440953e+03 -2.36758839e+02
-1.31300000e+03 3.77109275e+01 5.20579454e+03 -3.71433195e+03
-2.79387832e+03 4.95859846e+03 -5.14905071e+02 -2.87114353e+03
2.46837686e+03 2.98504637e+03 -1.75000000e+02 -3.04642885e+03
4.89120922e+03 3.84865629e+03 -1.11715436e+04 6.32196523e+03
7.67106151e+02 -6.91513328e+03 -2.18100065e+03 3.43226092e+02
2.25400000e+03 -1.16464162e+04 4.67386587e+03 5.39181241e+03
-8.81467635e+03 -1.86067707e+02 1.07849088e+04 -7.91539720e+03
2.62187832e+03 -2.55937762e+03 6.94722233e+03 -3.79140600e+03
-1.13435074e+04 1.74769311e+04 -1.30716540e+04 1.53516262e+03
-2.05540197e+03 8.58381182e+03 4.94317914e+03 -2.64892524e+04]
The flow is:
[ 0. 0. 0. ... 0. 2611. 2984.]

PCA analysis considering N-less relevant components

I am trying to learn the basics of PCA analysis in Python using scikit libraries (in particular sklearn.decomposition and sklearn.preprocessing). The goal is to import data from images into a matrix X (each row is a sample, each column is a feature), then standardize X, use PCA to extract principal components (2 most important, 6 most important....6 less important), project X on these principal components, reverse the previous transformation and plot the result in order to see the difference with respect to the original image/images.
Now let's say that I do not want to consider the 2,3,4... most important principal components but I want to consider the N less relevant components, let's say N=6.
How should the analysis be done?
I mean I can't simply standardize then call PCA().fit_transform and then revert back with inverse_transform() to plot the results.
At the moment I am doing something like this:
X_std = StandardScaler().fit_transform(X) # standardize original data
pca = PCA()
model = pca.fit(X_std) # create model with all components
Xprime = model.components_[range(dim-6, dim, 1),:] # get last 6 PC
And then I stop because I know I should call transform() but I do not understand how to do it...I tried several times withouth being successfull.
Is there someone that can tell me if previous steps are correct and point out the direction to follow?
Thank you very much
EDIT: currently I have adapted this solution as suggested by the first answer to my question:
model = PCA().fit(X_std)
model2pc = model
model2pc.components_[range(2, img_count, 1), :] = 0
Xp_2pc = model2pc.transform(X_std)
Xr_2pc = model2pc.inverse_transform(Xp_2pc)
And then I do the same for 6pc, 60pc, last 6 pc. What I have noticed is that this is very time consuming. I would like to get a model directly extracting the principal components I need (without zeroing out the others) and then perform transform() and inverse_transform() on that with that model.

If you want to ignore all but the last 6 principal components, you can just zero out the ones you don't want to keep.
N = 6
X_std = StandardScaler().fit_transform(X)
pca = PCA()
model = pca.fit(X_std) # create model with all components
model.components_[:-N] = 0
Then, to remove all but the last N components from the data, just do a forward and inverse transform of the data:
Xprime = model.inverse_transform(model.transform(X_std))
Here is an example:
>>> X = np.random.rand(18).reshape(6, 3)
>>> model = PCA().fit(X)
A round-trip transform should give back the original data:
>>> X
array([[0.16594796, 0.02366958, 0.8403745 ],
[0.25219425, 0.22879029, 0.07950927],
[0.69636084, 0.4410933 , 0.97431828],
[0.50121079, 0.44835563, 0.95236146],
[0.6793044 , 0.53847562, 0.27882302],
[0.32886931, 0.0643043 , 0.10597973]])
>>> model.inverse_transform(model.transform(X))
array([[0.16594796, 0.02366958, 0.8403745 ],
[0.25219425, 0.22879029, 0.07950927],
[0.69636084, 0.4410933 , 0.97431828],
[0.50121079, 0.44835563, 0.95236146],
[0.6793044 , 0.53847562, 0.27882302],
[0.32886931, 0.0643043 , 0.10597973]])
Now zero out the first principal component:
>>> model.components_
array([[ 0.22969899, 0.21209762, 0.94986998],
[-0.67830467, -0.66500728, 0.31251894],
[ 0.69795497, -0.71608653, -0.0088847 ]])
>>> model.components_[:-2] = 0
>>> model.components_
array([[ 0. , 0. , 0. ],
[-0.67830467, -0.66500728, 0.31251894],
[ 0.69795497, -0.71608653, -0.0088847 ]])
The round-trip transform now gives a different result since we've removed the first principal component (which contains the greatest amount of variance):
>>> model.inverse_transform(model.transform(X))
array([[ 0.12742811, -0.01189858, 0.68108405],
[ 0.36513945, 0.33308073, 0.54656949],
[ 0.58029482, 0.33392119, 0.49435263],
[ 0.39987803, 0.35478779, 0.53332196],
[ 0.71114004, 0.56787176, 0.41047233],
[ 0.44000711, 0.16692583, 0.56556581]])

How does a numpy function handle a logical if operator for the axis argument?

I stumbled onto this on accident, but can't make sense of what is going on. I am doing a K-means clustering assignment with images and trying to vectorize the code to make it run as fast as possible. I came up with the following code:
image_values =np.array( [[[ 0.36302522 0.51708686 0.20952381]
[ 0.46330538 0.69915968 0.2140056 ]
[ 0.7904762 0.93837535 0.27002802]
[ 0.78375351 0.89187676 0.24201682]
[ 0.57871151 0.79775912 0.24593839]
[ 0.2896359 0.39103645 0.64481789]
[ 0.23809525 0.30924368 0.64257705]]
[[ 0.36302522 0.51708686 0.20952381]
[ 0.46330538 0.69915968 0.2140056 ]
[ 0.7904762 0.93837535 0.27002802]
[ 0.78375351 0.89187676 0.24201682]
[ 0.57871151 0.79775912 0.24593839]
[ 0.2896359 0.39103645 0.64481789]
[ 0.23809525 0.30924368 0.64257705]]
[[ 0.36302522 0.51708686 0.20952381]
[ 0.46330538 0.69915968 0.2140056 ]
[ 0.7904762 0.93837535 0.27002802]
[ 0.78375351 0.89187676 0.24201682]
[ 0.57871151 0.79775912 0.24593839]
[ 0.2896359 0.39103645 0.64481789]
[ 0.23809525 0.30924368 0.64257705]]])
means = np.array([[0.909,0.839,0.6509],[0.813,0.808,0.694],[0.331,0.407,0.597]]) #random centroids
err = 1
while err > .01:
J = [np.sum((image_values-avg)**2, axis = 2) for avg in means]
K = np.argmin(J, axis = 0)
old_means = means
means = np.array([np.mean(image_values[K==i], axis ==True) for i in range(len(means))])
print means
err = abs(sum(old_means)-sum(means))
print err
In each new means calculation, I used my K array to select which pixel values should be included in each mean calculation but I couldn't get the axis to agree. I actually made a typo where instead of axis=3, I typed axis==3 and it worked! I tried a bunch of different numbers, and found out that it doesn't matter what the number is, the result is the same. I tried a bunch of numbers and Booleans with the equal operator they didn't work. I've gone through the documentation, but I couldn't figure it out.
What does numpy do when it gets a logical if in the axis argument of one of its array functions?
Thanks!

I am not entirely sure I fully understood what you're trying to do. Here's what I assume; You have one single image with RGB values and you would like to cluster the pixels within this image. Each centroid will thus define one value for each color channel respectively. I assume that each row in your means matrix is one centroid with the columns being the RGB values.
In your approach, I think you might have a mistake in the way you are subtracting the centroids. You will need to create a distance matrix for each centroid (at the moment your not subtracting each color channel correctly).
Here's one proposition. Please note that with given example data you will run into a NaN error since not all centroids have pixels that are closest to them. You also might need to adjust the stopping criterion to your needs.
err = 1
while err > 0.1:
# There are three centroids. We would like to compute the
# distance for each pixel to each centroid. Here, the image
# is thus replicated three times.
dist = np.tile(image_values, (3,1,1,1))
# The 2D matrix needs to be reshaped to fit the dimensions of
# the dist matrix. With the new shape, the matrix can directly
# be subtracted.
means2 = means.reshape(3,3,1,1)
# Subtract each respective RGB value of the centroid for
# each "replica" of the image
J = np.power(dist - means2, 2)
# Sum the r,g,b channels together to get the total distance for a pixel
J = J.sum(axis=1)
# Check for which cluster the pixel is closest
K = np.argmin(J, axis=0)
# I couldn't think of a better way than this loop
newMeans = np.zeros((3,3))
for i in range(means.shape[0]): # do each centroid
# In axis 1 there are pixels which we would like to
# average for each color channel (axis 0 are the RGB channels)
newMeans[i,:] = image_values[:,K==i].mean(axis=1)
err = np.power(means - newMeans, 2).sum()
means = newMeans

How do I make perspective transform of point with x and y coordinate

So I wrote this little program which allows me to select 4 points on two images.
Usign those points I get a transformation matrix. After that I select a point on one of the images and want to get visualization of where that point will be on other image.
Say my point is marked like this -> (x,y) - so it's a tuple. How should I format this "position" on image so it can be possible to transform it.
I have looked at documentation for perspectiveTransform() method and figured that I should be storing it in following shape:
numpy.array([
[self.points[self.length-1][0]],
[self.points[self.length-1][1]]
], dtype="float32")
Which would give me on a single click this format:
Point= [[ 2300.]
[ 634.]]
This format doesn't seem to work, I use this Transformation matrix:
M = [[ -1.71913123e+00 -4.76850572e+00 5.27968944e+03]
[ 2.07693562e-01 -1.09738424e+01 6.35222770e+03]
[ 1.02865125e-04 -4.80067600e-03 1.00000000e+00]]
in this method (and get following error):
cv2.perspectiveTransform(src, M)
OpenCV Error: Assertion failed (scn + 1 == m.cols) in cv::perspectiveTransform, file C:\builds\master_PackSlaveAddon-win64-vc12-static\opencv\modules\core\src\matmul.cpp
Any advice or tip is welcome.

I figured out the answer.
Found it on this link
The key is to put your point like this:
pts = numpy.array([[x,y]], dtype = "float32")
And then call another numpy.array on existing variable pts:
pts = numpy.array([pts])
The procedure is the same after this.