Related
Let's say that I have this numpy array:
import numpy as np
np.random.seed(0)
data = np.random.normal(size=(5,5))
which result in:
I would like to select all pairs with a specific indexes distance along each row.
For example if I choose a index distance 4 along each row I expect to have:
res[0,0]=1.76,res[0,1]=2.24
res[1,0]=0.40,res[1,1]=1.86
res[2,0]=-0.97,res[2,1]=-0.10
res[3,0]=0.95,res[3,1]=0.41
...
....
I now that I could that with a for cycle but I would like to have something smarter. I was thing to create two list of indexes and then to fill res but also in this I need a cycle.
Best
hstack
I guess that something in the line of
win=3 # Size of window. You say 4, but what you describe is 3 in my view. But you know how to add 1 if needed :D
np.hstack((data[:, :data.shape[1]-win].reshape(-1,1), data[:, win:].reshape(-1,1)))
should do
Result is
array([[ 1.76405235, 2.2408932 ],
[ 0.40015721, 1.86755799],
[-0.97727788, -0.10321885],
[ 0.95008842, 0.4105985 ],
[ 0.14404357, 0.12167502],
[ 1.45427351, 0.44386323],
[ 0.33367433, 0.3130677 ],
[ 1.49407907, -0.85409574],
[-2.55298982, -0.74216502],
[ 0.6536186 , 2.26975462]])
Explanation:
data[:,:data.shape[1]-win] is
array([[ 1.76405235, 0.40015721],
[-0.97727788, 0.95008842],
[ 0.14404357, 1.45427351],
[ 0.33367433, 1.49407907],
[-2.55298982, 0.6536186 ]])
So, just the first columns of data. Number of column, data.shape[1]-win, being the number of possible columns for data's width and win size.
Likewise, data[:, win:] is
array([[ 2.2408932 , 1.86755799],
[-0.10321885, 0.4105985 ],
[ 0.12167502, 0.44386323],
[ 0.3130677 , -0.85409574],
[-0.74216502, 2.26975462]])
Which are this time the last columns (same number of columns), but separated by win indexes.
.reshape(-1,1) flatten vertically those data, if I may use this "flatten vertically" description. For example data[:,:data.shape[1]-win].reshape(-1,1) is the same but with 10 rows of 1 column instead of 5 rows of 2 columns.
array([[ 1.76405235],
[ 0.40015721],
[-0.97727788],
[ 0.95008842],
[ 0.14404357],
[ 1.45427351],
[ 0.33367433],
[ 1.49407907],
[-2.55298982],
[ 0.6536186 ]])
hstack put those two together.
Indexation
Another method, maybe closer to the one you're apparently about to create indexes list, would be
W=data.shape[1]-win # number of pair per row
iy=np.arange(len(data)*2*W)//W//2
ix=np.array([[i,i+win] for i in range(W)]*len(data)).flatten()
data[iy,ix].reshape(-1,2)
That is about 2 times longer in term of cpu time. But it is worth noting that most of cpu time is spend in the creation of indexes ix and iy. So if you have many data sets of the same shape, this option could be faster, since you compute ix and iy once for all
You can take elements by pairs of indices with numpy.take:
np.take(data, [[0, 3], [1, 4]], axis=1).reshape(data.shape[0] * 2, 2)
array([[ 1.76405235, 2.2408932 ],
[ 0.40015721, 1.86755799],
[-0.97727788, -0.10321885],
[ 0.95008842, 0.4105985 ],
[ 0.14404357, 0.12167502],
[ 1.45427351, 0.44386323],
[ 0.33367433, 0.3130677 ],
[ 1.49407907, -0.85409574],
[-2.55298982, -0.74216502],
[ 0.6536186 , 2.26975462]])
I am trying to define a function that finds the minimum value of an array and slices it around that value (plus or minus 5 positions). My array looks something like this:
[[ 0. 9.57705087]
[ 0.0433 9.58249315]
[ 0.0866 9.59745942]
[ 0.1299 9.62194967]
[ 0.1732 9.65324278]
[ 0.2165 9.68725702]
[ 0.2598 9.72263184]
[ 0.3031 9.75256437]
[ 0.3464 9.77025178]
[ 0.3897 9.76889121]
[ 0.433 9.74167982]
[ 0.4763 9.68589645]
[ 0.5196 9.59881999]
[ 0.5629 9.48861383]
[ 0.6062 9.3593597 ]]
However, I am dealing with much larger sets and need a function that can do it automatically without me having to manually find the minimun and then slice the array around that.I want to find the minimun of the array[:,1] values and then apply the slicing to the whole array.
Use np.argmin() to get the index of the minimum value. This will do it using the second column only (you haven't specified if it's the minimum value across columns or not).
your_array[:np.argmin(your_array[:, 1]), :]
To slice it 5 values further than the minimum, use:
your_array[:np.argmin(your_array[:, 1]) + 5, :]
Given your objective array:
import numpy as np
anarray = np.array([[ 0., 9.57705087],
[ 0.0433, 9.58249315],
[ 0.0866, 9.59745942],
[ 0.1299, 9.62194967],
[ 0.1732, 9.65324278],
[ 0.2165, 9.68725702],
[ 0.2598, 9.72263184],
[ 0.3031, 9.75256437],
[ 0.3464, 9.77025178],
[ 0.3897, 9.76889121],
[ 0.433, 9.74167982],
[ 0.4763, 9.68589645],
[ 0.5196, 9.59881999],
[ 0.5629, 0.48861383],
[ 0.6062, 9.3593597]])
This function will do the job:
def slice_by_five(array):
argmin = np.argmin(array[:,1])
if argmin < 5:
return array[:argmin+6,:]
return array[argmin-5:argmin+6,:]
check = slice_by_five(anarray)
print(check)
Output:
[[0.3897 9.76889121]
[0.433 9.74167982]
[0.4763 9.68589645]
[0.5196 9.59881999]
[0.5629 9.48861383]
[0.6062 9.3593597 ]]
The function can certainly be generalized to account for any neighborhood of size n:
def slice_by_n(array, n):
argmin = np.argmin(array[:,1])
if argmin < n:
return array[:argmin+n+1,:]
return array[argmin-n:argmin+n+1,:]
check = slice_by_n(anarray, 2)
print(check)
Output:
[[0.5196 9.59881999]
[0.5629 9.48861383]
[0.6062 9.3593597 ]]
I have simulated 10000 scenarios for 4 variables during 120 months.
Hence, I have a scenarios list of lists of lists on which to get and element I would have to use scenarios[1][1][1], for example, and this would give me a float.
I want to slice this in two, dividing by the second list. Which means I want to keep the 10000 scenarios for 4 variables for the first 60 months.
How would I go about doing this?
My intuition would tell me to do
scenarios[:][0:60]
but this does not work. Instead of cutting the second list, it cuts the first. What is wrong?
Example:
Q = data.cov().as_matrix() # monthly covariance matrix Q
r=[0.00565,0.00206,0.00368,0.00021] # monthly return
scenarios = [[]]*10000
for i in range(10000):
scenarios[i] = np.random.multivariate_normal(r, Q, size = 120) # monthly scenarios
In my case, Q=
2.167748064990633258e-03 -8.736421379048196659e-05 1.457397098602368978e-04 2.799384719379381381e-06
-8.736421379048196659e-05 9.035930360181909865e-04 3.196576120840064102e-04 3.197146643002681875e-06
1.457397098602368978e-04 3.196576120840064102e-04 2.390042779951682440e-04 2.312645986876262622e-06
2.799384719379381381e-06 3.197146643002681875e-06 2.312645986876262622e-06 4.365866475269951553e-06
Use a list comprehension:
early_scenarios = [x[:60] for x in scenarios]
So, you are trying to use multidimensional slicing on Python list objects, but fundamentally, list objects do not have dimensions. They have no inherent knowledge of their contents, other than the total number of them. But, you *shouldn't be working with list objects at all! Instead, replace this:
scenarios = [[]]*10000
for i in range(10000):
scenarios[i] = np.random.multivariate_normal(r, Q, size = 120) # monthly scenarios
With this:
scenarios = np.random.multivariate_normal(r, Q, size=(1000, 120))
In a REPL:
>>> scenarios = np.random.multivariate_normal(r, Q, size=(1000, 120))
>>> scenarios.shape
(1000, 120, 4)
Then, you can slice to your heart's content in N dimensions using:
scenarios[:, 0:60]
Or, a more wieldy slice:
>>> scenarios[500:520, 0:60]
array([[[-0.05785267, 0.01122828, 0.00786622, -0.00204875],
[ 0.01682276, 0.00163375, 0.00439909, -0.0022255 ],
[ 0.02821342, -0.01634708, 0.01175085, -0.00194007],
...,
[ 0.04918003, -0.02146014, 0.00071328, -0.00222226],
[-0.03782566, -0.00685615, -0.00837397, -0.00095019],
[-0.06164655, 0.02817698, 0.01001757, -0.00149662]],
[[ 0.00071181, -0.00487313, -0.01471801, -0.00180559],
[ 0.05826763, 0.00978292, 0.02442642, -0.00039461],
[ 0.04382627, -0.00804489, 0.00046985, 0.00086524],
...,
[ 0.01231702, 0.01872649, 0.01534518, -0.0022179 ],
[ 0.04212831, -0.05289387, -0.03184881, -0.00078165],
[-0.04361605, -0.01297212, 0.00135886, 0.0057856 ]],
[[ 0.00232622, 0.01773357, 0.00795682, 0.00016406],
[-0.04367355, -0.02387383, -0.00448453, 0.0008559 ],
[ 0.01256918, 0.06565425, 0.05170755, 0.00046948],
...,
[ 0.04457427, -0.01816762, 0.00068176, 0.00186112],
[ 0.00220281, -0.01119046, 0.0103347 , -0.00089715],
[ 0.02178122, 0.03183001, 0.00959293, -0.00057862]],
...,
[[ 0.06338153, 0.01641472, 0.01962643, -0.00256244],
[ 0.07537754, -0.0442643 , -0.00362656, 0.00153777],
[ 0.0505006 , 0.0070783 , 0.01756948, 0.0029576 ],
...,
[ 0.03524508, -0.03547517, -0.00664972, -0.00095385],
[-0.03699107, 0.02256328, 0.00300107, 0.00253193],
[-0.0199608 , -0.00536222, 0.01370301, -0.00131981]],
[[ 0.08601913, -0.00364473, 0.00946769, 0.00045275],
[ 0.01943327, 0.07420857, 0.00109217, -0.00183334],
[-0.04481884, -0.02515305, -0.02357894, -0.00198166],
...,
[-0.01221928, -0.01241903, 0.00928084, 0.00066379],
[ 0.10871802, -0.01264407, 0.00601223, 0.00090526],
[-0.02603179, -0.00413112, -0.006037 , 0.00522712]],
[[-0.02929114, 0.02188803, -0.00427137, 0.00250174],
[ 0.02479416, -0.01470632, -0.01355196, 0.00338125],
[-0.01915726, -0.00869161, 0.01451885, -0.00137969],
...,
[ 0.05398784, -0.00834729, -0.00437888, 0.00081602],
[ 0.00626345, -0.0261016 , -0.01484753, 0.00060499],
[ 0.05427697, 0.04006612, 0.03371313, -0.00203731]]])
>>>
You need to explicitly slice each secondary list, either in a loop or in list comprehensions. I built a 10x10 set of lists so you have to change the indexing to fit your problem:
x = []
for a in range(10):
x.append([10*a+n for n in range(10)])
# x is now a list of 10 lists, each of which has 10 elements
print(x)
x1 = [a[:5] for a in x]
# x1 is a list of containing the low elements of the secondary lists
x2 = [a[5:] for a in x]
# x2 is a list containing the high elements of the secondary lists
print(x1, x2)
Python slicing doesn't consider all dimension like this. Your expression makes a copy of the entire list, scenarios[:], and then takes the first 60 elements of the copy. You need to write a comprehension to grab the elements you want.
Perhaps
[scenarios[x][y][z]
for x in range(len(scenarios))
for y in range(60)
for z in range(len(scenarios[0][0])) ]
Assume I have a multidimensional Numpy Array. Now I want to:
Slice out a certain row range defined by startIndex and endIndex.
Get a array with the original array minus the slice (so the left over).
The code below does this trick, however is this the most performance one?
Because my array is very big, can I (memory neutral) slice out the original array so that afterwards the original array is the left over. So except some overhead for the header of the new array this will cost no additional memory?
Is my snippet below (with creating new arrays), the the most efficient solution if we retain the original array?
Example:
import numpy as np
X = np.random.random((6, 2))
print('Orig',X)
startIndex = 2
endIndex = 4
print('Slice ',X[startIndex:endIndex])
print('LeftOver ',np.concatenate((X[:startIndex-1],X[endIndex:])))
Output:
Orig [[ 0.94661646 0.3911347 ]
[ 0.6807441 0.676658 ]
[ 0.81109554 0.18089991]
[ 0.6161699 0.19907537]
[ 0.12859196 0.34866049]
[ 0.22283545 0.04949782]]
Slice [[ 0.81109554 0.18089991]
[ 0.6161699 0.19907537]]
LeftOver [[ 0.94661646 0.3911347 ]
[ 0.12859196 0.34866049]
[ 0.22283545 0.04949782]]
Concatenate make a copy, and you need it if order matters.
But if your slices are slim, and order doesn't matter, a more economic way can be:
import numpy as np
size=6
X = np.random.random((size, 2))
print('Orig\n',X)
startIndex = 3
endIndex = 5
Slice=X[startIndex:endIndex].copy()
length = min(endIndex-startIndex,size-endIndex) # to check overlap
X[startIndex:startIndex+length]=X[-length:]
Left=X[:size-len(Slice)]
print('Slice\n',Slice)
print('LeftOver\n',Left)
because at most 2x the size of the slice is copied, not the whole array.
it gives:
Orig
[[ 0.39351322 0.42100711]
[ 0.14793363 0.12149344]
[ 0.94524844 0.22004186]
[ 0.816418 0.35630767]
[ 0.37781821 0.12336287]
[ 0.65995888 0.23812275]]
Slice
[[ 0.816418 0.35630767]
[ 0.37781821 0.12336287]]
LeftOver
[[ 0.39351322 0.42100711]
[ 0.14793363 0.12149344]
[ 0.94524844 0.22004186]
[ 0.65995888 0.23812275]]
I am trying to find mirror images in a numpy array. In particular, (x,y) == (y,x) but I want to rule out tuples with identical values (x,x).
Given a numpy array pckList with the size (198L,3L) containing floats.
I have the following code:
np.sum([x==pckLst[:,2] for x in pckLst[:,1]])
Which returns a given number, lets say 73
np.sum([x==pckLst[:,2] for x in pckLst[:,1]] and [x==pckLst[:,1] for x in pckLst[:,1]])
Returns a larger number, lets say 266.
Can someone please explain how this comes about?
I thought the first line returns True, when seen as tuples (x,y) == (any,y) and the second line returns only true when (x,y) == (y,x).
Is this correct?
EDIT:
Further explaination:
pckLst=[[ 112.066, 6.946, 6.938],
[ 111.979, 6.882, 7.634],
[ 112.014, 6.879, 7.587],
[ 112.005, 6.887, 7.554],
[ 111.995, 6.88, 6.88 ],
[ 112.048, 6.774, 6.88 ],
[ 111.808, 7.791, 7.566],
[ 111.802, 6.88, 6.774]]
Now I would like to find [ 112.048, 6.774, 6.88 ], since (6.88, 6.774) == (6.774, 6.88). However, [ 111.995, 6.88, 6.88 ] should not be considered a match.
Rather than commenting on your code here is a simpler implementation
a=np.array([[1,1,10],[1,2,20],[2,1,30],[1,3,40],[2,3,50]])
xy= a[:,:2].tolist()
[[x,y,z] for [x,y,z] in a if [y,x] in xy and x!=y]
[[1, 2, 20], [2, 1, 30]]
The arguments to "and" in your example are python-lists. The truth value of a list is True if it is not empty. Thats why you get a bigger sum in the latter case.
This will return the sum of elements with (x,y) == (y,x). It obviously only works if your just interested in the sum and not particular indices:
import numpy
pckLst = numpy.array([[ 112.066, 6.946, 6.938],
[ 111.979, 6.882, 7.634],
[ 112.014, 6.879, 7.587],
[ 112.005, 6.887, 7.554],
[ 111.995, 6.88, 6.88 ],
[ 112.048, 6.774, 6.88 ],
[ 111.808, 7.791, 7.566],
[ 111.802, 6.88, 6.774]])
coords = pckLst[:,1:]
equal_ids = numpy.ravel(coords[:,:1] != coords[:,1:])
unequal_coords = coords[equal_ids]
flipped = numpy.fliplr(unequal_coords)
coords_tuple_set = set(tuple(map(tuple, unequal_coords)))
flipped_tuple_set = set(tuple(map(tuple, flipped)))
print coords_tuple_set
print flipped_tuple_set
# need to devide by two, because we get (x,y) and (y,x) by the intersection
print "number of mirrored points:",
print len(coords_tuple_set.intersection(flipped_tuple_set))/2