Related
Let's say that I have this numpy array:
import numpy as np
np.random.seed(0)
data = np.random.normal(size=(5,5))
which result in:
I would like to select all pairs with a specific indexes distance along each row.
For example if I choose a index distance 4 along each row I expect to have:
res[0,0]=1.76,res[0,1]=2.24
res[1,0]=0.40,res[1,1]=1.86
res[2,0]=-0.97,res[2,1]=-0.10
res[3,0]=0.95,res[3,1]=0.41
...
....
I now that I could that with a for cycle but I would like to have something smarter. I was thing to create two list of indexes and then to fill res but also in this I need a cycle.
Best
hstack
I guess that something in the line of
win=3 # Size of window. You say 4, but what you describe is 3 in my view. But you know how to add 1 if needed :D
np.hstack((data[:, :data.shape[1]-win].reshape(-1,1), data[:, win:].reshape(-1,1)))
should do
Result is
array([[ 1.76405235, 2.2408932 ],
[ 0.40015721, 1.86755799],
[-0.97727788, -0.10321885],
[ 0.95008842, 0.4105985 ],
[ 0.14404357, 0.12167502],
[ 1.45427351, 0.44386323],
[ 0.33367433, 0.3130677 ],
[ 1.49407907, -0.85409574],
[-2.55298982, -0.74216502],
[ 0.6536186 , 2.26975462]])
Explanation:
data[:,:data.shape[1]-win] is
array([[ 1.76405235, 0.40015721],
[-0.97727788, 0.95008842],
[ 0.14404357, 1.45427351],
[ 0.33367433, 1.49407907],
[-2.55298982, 0.6536186 ]])
So, just the first columns of data. Number of column, data.shape[1]-win, being the number of possible columns for data's width and win size.
Likewise, data[:, win:] is
array([[ 2.2408932 , 1.86755799],
[-0.10321885, 0.4105985 ],
[ 0.12167502, 0.44386323],
[ 0.3130677 , -0.85409574],
[-0.74216502, 2.26975462]])
Which are this time the last columns (same number of columns), but separated by win indexes.
.reshape(-1,1) flatten vertically those data, if I may use this "flatten vertically" description. For example data[:,:data.shape[1]-win].reshape(-1,1) is the same but with 10 rows of 1 column instead of 5 rows of 2 columns.
array([[ 1.76405235],
[ 0.40015721],
[-0.97727788],
[ 0.95008842],
[ 0.14404357],
[ 1.45427351],
[ 0.33367433],
[ 1.49407907],
[-2.55298982],
[ 0.6536186 ]])
hstack put those two together.
Indexation
Another method, maybe closer to the one you're apparently about to create indexes list, would be
W=data.shape[1]-win # number of pair per row
iy=np.arange(len(data)*2*W)//W//2
ix=np.array([[i,i+win] for i in range(W)]*len(data)).flatten()
data[iy,ix].reshape(-1,2)
That is about 2 times longer in term of cpu time. But it is worth noting that most of cpu time is spend in the creation of indexes ix and iy. So if you have many data sets of the same shape, this option could be faster, since you compute ix and iy once for all
You can take elements by pairs of indices with numpy.take:
np.take(data, [[0, 3], [1, 4]], axis=1).reshape(data.shape[0] * 2, 2)
array([[ 1.76405235, 2.2408932 ],
[ 0.40015721, 1.86755799],
[-0.97727788, -0.10321885],
[ 0.95008842, 0.4105985 ],
[ 0.14404357, 0.12167502],
[ 1.45427351, 0.44386323],
[ 0.33367433, 0.3130677 ],
[ 1.49407907, -0.85409574],
[-2.55298982, -0.74216502],
[ 0.6536186 , 2.26975462]])
I am new to Python. I would like to create a new array, that contains all values from an existing array with the step.
I tried to implement it but I think there is another way to have better performance. Any try or recommendation is highly appreciated.
Ex: Currently, I have:
An array: 115.200 values (2D dimension)
Step: 10.000
....
array([[ 0.2735, -0.308 ],
[ 0.287 , -0.3235],
[ 0.2925, -0.324 ],
[ 0.312 , -0.329 ],
[ 0.3275, -0.345 ],
[ 0.3305, -0.352 ],
[ 0.332 , -0.3465],
...
[ 0.3535, -0.353 ],
[ 0.361 , -0.3445],
[ 0.3545, -0.329 ]])
Expectation: A new array is sliced the array above by step of 10.000.
Below is my code:
for x in ecg_data:
number_samples_by_duration_exp_temp = 10000
# len(ecg_property.sample) = 115200
times = len(ecg_property.sample) / number_samples_by_duration_exp_temp
index_by_time = [int(y)*number_samples_by_duration_exp_temp for y in np.arange(1, times, 1)]
list = []
temp = 0
for z in index_by_time:
arr_samples_by_duration = ecg_property.sample[temp:z]
list.append(arr_samples_by_duration)
temp = z
numpy can not be used for this purpose as len(ecg_property.sample) #115,200 is not fully divisible by number_samples_by_duration_exp_temp #10,000 and numpy cannot allow elements of varying lengths :)
You can try list comprehension.
result_list = [ecg_property.sample[temp :temp+step] for temp in np.arange(times)*step ]
where
step=10000 and times = len(ecg_property.sample) / step
It can be further modified if needed and as per requirement.
(You can try out each step in above line of code in this answer and see the output to understand each step )
Hope this works out.
ty!
I am trying to define a function that finds the minimum value of an array and slices it around that value (plus or minus 5 positions). My array looks something like this:
[[ 0. 9.57705087]
[ 0.0433 9.58249315]
[ 0.0866 9.59745942]
[ 0.1299 9.62194967]
[ 0.1732 9.65324278]
[ 0.2165 9.68725702]
[ 0.2598 9.72263184]
[ 0.3031 9.75256437]
[ 0.3464 9.77025178]
[ 0.3897 9.76889121]
[ 0.433 9.74167982]
[ 0.4763 9.68589645]
[ 0.5196 9.59881999]
[ 0.5629 9.48861383]
[ 0.6062 9.3593597 ]]
However, I am dealing with much larger sets and need a function that can do it automatically without me having to manually find the minimun and then slice the array around that.I want to find the minimun of the array[:,1] values and then apply the slicing to the whole array.
Use np.argmin() to get the index of the minimum value. This will do it using the second column only (you haven't specified if it's the minimum value across columns or not).
your_array[:np.argmin(your_array[:, 1]), :]
To slice it 5 values further than the minimum, use:
your_array[:np.argmin(your_array[:, 1]) + 5, :]
Given your objective array:
import numpy as np
anarray = np.array([[ 0., 9.57705087],
[ 0.0433, 9.58249315],
[ 0.0866, 9.59745942],
[ 0.1299, 9.62194967],
[ 0.1732, 9.65324278],
[ 0.2165, 9.68725702],
[ 0.2598, 9.72263184],
[ 0.3031, 9.75256437],
[ 0.3464, 9.77025178],
[ 0.3897, 9.76889121],
[ 0.433, 9.74167982],
[ 0.4763, 9.68589645],
[ 0.5196, 9.59881999],
[ 0.5629, 0.48861383],
[ 0.6062, 9.3593597]])
This function will do the job:
def slice_by_five(array):
argmin = np.argmin(array[:,1])
if argmin < 5:
return array[:argmin+6,:]
return array[argmin-5:argmin+6,:]
check = slice_by_five(anarray)
print(check)
Output:
[[0.3897 9.76889121]
[0.433 9.74167982]
[0.4763 9.68589645]
[0.5196 9.59881999]
[0.5629 9.48861383]
[0.6062 9.3593597 ]]
The function can certainly be generalized to account for any neighborhood of size n:
def slice_by_n(array, n):
argmin = np.argmin(array[:,1])
if argmin < n:
return array[:argmin+n+1,:]
return array[argmin-n:argmin+n+1,:]
check = slice_by_n(anarray, 2)
print(check)
Output:
[[0.5196 9.59881999]
[0.5629 9.48861383]
[0.6062 9.3593597 ]]
I have simulated 10000 scenarios for 4 variables during 120 months.
Hence, I have a scenarios list of lists of lists on which to get and element I would have to use scenarios[1][1][1], for example, and this would give me a float.
I want to slice this in two, dividing by the second list. Which means I want to keep the 10000 scenarios for 4 variables for the first 60 months.
How would I go about doing this?
My intuition would tell me to do
scenarios[:][0:60]
but this does not work. Instead of cutting the second list, it cuts the first. What is wrong?
Example:
Q = data.cov().as_matrix() # monthly covariance matrix Q
r=[0.00565,0.00206,0.00368,0.00021] # monthly return
scenarios = [[]]*10000
for i in range(10000):
scenarios[i] = np.random.multivariate_normal(r, Q, size = 120) # monthly scenarios
In my case, Q=
2.167748064990633258e-03 -8.736421379048196659e-05 1.457397098602368978e-04 2.799384719379381381e-06
-8.736421379048196659e-05 9.035930360181909865e-04 3.196576120840064102e-04 3.197146643002681875e-06
1.457397098602368978e-04 3.196576120840064102e-04 2.390042779951682440e-04 2.312645986876262622e-06
2.799384719379381381e-06 3.197146643002681875e-06 2.312645986876262622e-06 4.365866475269951553e-06
Use a list comprehension:
early_scenarios = [x[:60] for x in scenarios]
So, you are trying to use multidimensional slicing on Python list objects, but fundamentally, list objects do not have dimensions. They have no inherent knowledge of their contents, other than the total number of them. But, you *shouldn't be working with list objects at all! Instead, replace this:
scenarios = [[]]*10000
for i in range(10000):
scenarios[i] = np.random.multivariate_normal(r, Q, size = 120) # monthly scenarios
With this:
scenarios = np.random.multivariate_normal(r, Q, size=(1000, 120))
In a REPL:
>>> scenarios = np.random.multivariate_normal(r, Q, size=(1000, 120))
>>> scenarios.shape
(1000, 120, 4)
Then, you can slice to your heart's content in N dimensions using:
scenarios[:, 0:60]
Or, a more wieldy slice:
>>> scenarios[500:520, 0:60]
array([[[-0.05785267, 0.01122828, 0.00786622, -0.00204875],
[ 0.01682276, 0.00163375, 0.00439909, -0.0022255 ],
[ 0.02821342, -0.01634708, 0.01175085, -0.00194007],
...,
[ 0.04918003, -0.02146014, 0.00071328, -0.00222226],
[-0.03782566, -0.00685615, -0.00837397, -0.00095019],
[-0.06164655, 0.02817698, 0.01001757, -0.00149662]],
[[ 0.00071181, -0.00487313, -0.01471801, -0.00180559],
[ 0.05826763, 0.00978292, 0.02442642, -0.00039461],
[ 0.04382627, -0.00804489, 0.00046985, 0.00086524],
...,
[ 0.01231702, 0.01872649, 0.01534518, -0.0022179 ],
[ 0.04212831, -0.05289387, -0.03184881, -0.00078165],
[-0.04361605, -0.01297212, 0.00135886, 0.0057856 ]],
[[ 0.00232622, 0.01773357, 0.00795682, 0.00016406],
[-0.04367355, -0.02387383, -0.00448453, 0.0008559 ],
[ 0.01256918, 0.06565425, 0.05170755, 0.00046948],
...,
[ 0.04457427, -0.01816762, 0.00068176, 0.00186112],
[ 0.00220281, -0.01119046, 0.0103347 , -0.00089715],
[ 0.02178122, 0.03183001, 0.00959293, -0.00057862]],
...,
[[ 0.06338153, 0.01641472, 0.01962643, -0.00256244],
[ 0.07537754, -0.0442643 , -0.00362656, 0.00153777],
[ 0.0505006 , 0.0070783 , 0.01756948, 0.0029576 ],
...,
[ 0.03524508, -0.03547517, -0.00664972, -0.00095385],
[-0.03699107, 0.02256328, 0.00300107, 0.00253193],
[-0.0199608 , -0.00536222, 0.01370301, -0.00131981]],
[[ 0.08601913, -0.00364473, 0.00946769, 0.00045275],
[ 0.01943327, 0.07420857, 0.00109217, -0.00183334],
[-0.04481884, -0.02515305, -0.02357894, -0.00198166],
...,
[-0.01221928, -0.01241903, 0.00928084, 0.00066379],
[ 0.10871802, -0.01264407, 0.00601223, 0.00090526],
[-0.02603179, -0.00413112, -0.006037 , 0.00522712]],
[[-0.02929114, 0.02188803, -0.00427137, 0.00250174],
[ 0.02479416, -0.01470632, -0.01355196, 0.00338125],
[-0.01915726, -0.00869161, 0.01451885, -0.00137969],
...,
[ 0.05398784, -0.00834729, -0.00437888, 0.00081602],
[ 0.00626345, -0.0261016 , -0.01484753, 0.00060499],
[ 0.05427697, 0.04006612, 0.03371313, -0.00203731]]])
>>>
You need to explicitly slice each secondary list, either in a loop or in list comprehensions. I built a 10x10 set of lists so you have to change the indexing to fit your problem:
x = []
for a in range(10):
x.append([10*a+n for n in range(10)])
# x is now a list of 10 lists, each of which has 10 elements
print(x)
x1 = [a[:5] for a in x]
# x1 is a list of containing the low elements of the secondary lists
x2 = [a[5:] for a in x]
# x2 is a list containing the high elements of the secondary lists
print(x1, x2)
Python slicing doesn't consider all dimension like this. Your expression makes a copy of the entire list, scenarios[:], and then takes the first 60 elements of the copy. You need to write a comprehension to grab the elements you want.
Perhaps
[scenarios[x][y][z]
for x in range(len(scenarios))
for y in range(60)
for z in range(len(scenarios[0][0])) ]
I have a 3d, 3x3x3 array of integers. Numpy will print these as a block of the first 3x3, then below it the 2nd 3x3, then below that the 3rd 3x3.
If I wanted to print these 3 3x3 blocks BESIDE each other, rather than underneath each other, how would I tell numpy to print differently?
class MyArray(numpy.array):
def __str__(self):
print [[[d for d in c] for c in b] for b in a]
This essentially subclasses numpy.array and just changes the __str__ function (which is called when getting the string representation of an object) which converts it into a regular array and prints that.
import numpy as np
arr=np.random.random((3,3,3))
print(arr)
# [[[ 0.05733376 0.00646892 0.96180769]
# [ 0.11560363 0.56058966 0.83942817]
# [ 0.5520361 0.17355794 0.87699437]]
# [[ 0.90999361 0.03036473 0.5064459 ]
# [ 0.76169531 0.48234618 0.56884999]
# [ 0.93220906 0.9460365 0.65307273]]
# [[ 0.04400683 0.58783221 0.74281147]
# [ 0.69999475 0.14870245 0.32175415]
# [ 0.20044376 0.11985585 0.69949965]]]
for rows in zip(*arr):
print('\t'.join(map(str,rows)))
# [ 0.05733376 0.00646892 0.96180769] [ 0.90999361 0.03036473 0.5064459 ] [ 0.04400683 0.58783221 0.74281147]
# [ 0.11560363 0.56058966 0.83942817] [ 0.76169531 0.48234618 0.56884999] [ 0.69999475 0.14870245 0.32175415]
# [ 0.5520361 0.17355794 0.87699437] [ 0.93220906 0.9460365 0.65307273] [ 0.20044376 0.11985585 0.69949965]
For convenience, you could wrap it in a function:
def format_arr(arr):
result=[]
for x in zip(*arr):
result.append('\t'.join(map(str,x)))
return '\n'.join(result)
print(format_arr(arr))