Related
I have a function taking two vectors with same size, making some calculations with them, and returning a third vector. Let's now consider that I have a multiple dimension array containing many vectors that I want to pass to my function as the first argument, and a fixed vector that I want to pass as the second argument. Below is an example. Is there a way to simplify the code by removing the loops?
def foo(x, y):
result = np.zeros(x.shape)
for i in range(y.size-1):
result[i+1] = result[i] + (x[i+1] + x[i]) / (y[i+1] + y[i])
return result
a = np.arange(2*3*4).reshape(2,3,4)
b = np.arange(4)*10
c = np.ones(a.shape)*-9999.
for i in range(a.shape[0]):
for j in range(a.shape[1]):
c[i, j, :] = foo(a[i, j, :], b)
Thanks for your help!
EDIT: Below is the real function I'm trying to implement.
def press2alt(press, Tv):
"""
Convert pressure level to altitude level with hydrostatic atmosphere calculation (hypsometric
equation). The altitude reference (z = 0 km) is taken for the largest pressure level.
:param press: pressure level profile [hPa]
:param Tv: virtual temperature profile [K]
:return: altitude level profile [km]
"""
press_c = np.copy(press)
if press[0] < press[1]:
press_c = press_c[::-1] # high press first
Tv = Tv[::-1]
alt = np.zeros(press_c.size)
for i in range(alt.size-1):
alt[i+1] = alt[i] + DRY_AIR_GAS_CST/STD_GRAV_ACC*(Tv[i+1]-Tv[i])* \
(np.log(press_c[i])-np.log(press_c[i+1]))/(np.log(Tv[i+1])-np.log(Tv[i]))
if press[0] < press[1]:
alt = alt[::-1]
return alt
# Get altitude at each pressure level
z = np.ones(tv.shape)*FILL_VALUE_FLOAT
for i_month in range(tv.shape[0]):
for i_lat in range(tv.shape[2]):
for i_lon in range(tv.shape[3]):
z[i_month, :, i_lat, i_lon] = \
press2alt(pressure_level, tv[i_month, :, i_lat, i_lon])
The c from your sample (which you should have shown) is:
In [164]: c
Out[164]:
array([[[0. , 0.1 , 0.2 , 0.3 ],
[0. , 0.9 , 1.26666667, 1.52666667],
[0. , 1.7 , 2.33333333, 2.75333333]],
[[0. , 2.5 , 3.4 , 3.98 ],
[0. , 3.3 , 4.46666667, 5.20666667],
[0. , 4.1 , 5.53333333, 6.43333333]]])
np.vectorize with signature turns out to be easier to use than I first thought:
In [165]: f = np.vectorize(foo, signature="(n),(n)->(n)")
In [166]: f(a, b)
Out[166]:
array([[[0. , 0.1 , 0.2 , 0.3 ],
[0. , 0.9 , 1.26666667, 1.52666667],
[0. , 1.7 , 2.33333333, 2.75333333]],
[[0. , 2.5 , 3.4 , 3.98 ],
[0. , 3.3 , 4.46666667, 5.20666667],
[0. , 4.1 , 5.53333333, 6.43333333]]])
But vectorize does not improve speed:
In [167]: %%timeit
...: c = np.ones(a.shape)*-9999.
...: for i in range(a.shape[0]):
...: for j in range(a.shape[1]):
...: c[i, j, :] = foo(a[i, j, :], b)
...:
57 µs ± 126 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
In [168]: timeit f(a, b)
206 µs ± 3.05 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
In other recent cases I've found that vectorize does improve in relative performance with larger arrays, but that wasn't with signature.
The function can be rewritten to accept arrays of any size, as long as the iteration on the last dimension is correct. Basically I use ... in the indexing:
def myfoo(a, b):
result = np.zeros(a.shape)
for i in range(a.shape[-1] - 1):
result[..., i + 1] = result[..., i] + (a[..., i + 1] +
a[...,i]) / ( b[..., i + 1] + b[..., i])
return result
In [182]: myfoo(a, b)
Out[182]:
array([[[0. , 0.1 , 0.2 , 0.3 ],
[0. , 0.9 , 1.26666667, 1.52666667],
[0. , 1.7 , 2.33333333, 2.75333333]],
[[0. , 2.5 , 3.4 , 3.98 ],
[0. , 3.3 , 4.46666667, 5.20666667],
[0. , 4.1 , 5.53333333, 6.43333333]]])
In [183]: timeit myfoo(a, b)
65.8 µs ± 483 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
This doesn't help with speed, possibly because the last axis, size 4, is comparable to the 2*3 iterations of the first. I expect it will do better, relatively, if the initial dimensions get much larger.
We may be able to improve speed by replacing the i iteration on:
(a[..., i + 1] + a[...,i]) / ( b[..., i + 1] + b[..., i])
with
(a[...,1:]+a[...,:-1])/(b[...,1:]+b[...,:-1])
edit
In [192]: ab = (a[..., 1:] + a[..., :-1]) / (b[..., 1:] + b[..., :-1])
In [193]: ab
Out[193]:
array([[[0.1 , 0.1 , 0.1 ],
[0.9 , 0.36666667, 0.26 ],
[1.7 , 0.63333333, 0.42 ]],
[[2.5 , 0.9 , 0.58 ],
[3.3 , 1.16666667, 0.74 ],
[4.1 , 1.43333333, 0.9 ]]])
In [194]: ab.cumsum(axis=2)
Out[194]:
array([[[0.1 , 0.2 , 0.3 ],
[0.9 , 1.26666667, 1.52666667],
[1.7 , 2.33333333, 2.75333333]],
[[2.5 , 3.4 , 3.98 ],
[3.3 , 4.46666667, 5.20666667],
[4.1 , 5.53333333, 6.43333333]]])
Those are the values - except for the leading 0's column.
In [195]: timeit((a[..., 1:] + a[..., :-1]) / (b[..., 1:] + b[..., :-1])).cumsum
...: (axis=2)
18.8 µs ± 36.8 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
I have a np.arrray like this:
[[ 1.3 , 2.7 , 0.5 , NaN , NaN],
[ 2.0 , 8.9 , 2.5 , 5.6 , 3.5],
[ 0.6 , 3.4 , 9.5 , 7.4 , NaN]]
And a function to compute the distance between two rows:
def nan_manhattan(X, Y):
nan_diff = np.absolute(X - Y)
length = nan_diff.size
return np.nansum(nan_diff) * length / (length - np.isnan(nan_diff).sum())
I need all pairwise distances, and I don't want to use a loop. How do I do that?
Leveraging broadcasting -
def manhattan_nan(a):
s = np.nansum(np.abs(a[:,None,:] - a), axis=-1)
m = ~np.isnan(a)
k = m.sum(1)
r = a.shape[1]/np.minimum.outer(k,k)
out = s*r
return out
Benchmarking
From OP's comments, the use-case seems to be a tall array. Let's reproduce one for benchmarking re-using given sample data :
In [2]: a
Out[2]:
array([[1.3, 2.7, 0.5, nan, nan],
[2. , 8.9, 2.5, 5.6, 3.5],
[0.6, 3.4, 9.5, 7.4, nan]])
In [3]: a = np.repeat(a, 100, axis=0)
# #Dani Mesejo's soln
In [4]: %timeit pdist(a, nan_manhattan)
1.02 s ± 35.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# Naive for-loop version
In [18]: n = a.shape[0]
In [19]: %timeit [[nan_manhattan(a[i], a[j]) for i in range(j+1,n)] for j in range(n)]
991 ms ± 45.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# With broadcasting
In [9]: %timeit manhattan_nan(a)
8.43 ms ± 49.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Use pdist:
import numpy as np
from scipy.spatial.distance import pdist, squareform
def nan_manhattan(X, Y):
nan_diff = np.absolute(X - Y)
length = nan_diff.size
return np.nansum(nan_diff) * length / (length - np.isnan(nan_diff).sum())
arr = np.array([[1.3, 2.7, 0.5, np.nan, np.nan],
[2.0, 8.9, 2.5, 5.6, 3.5],
[0.6, 3.4, 9.5, 7.4, np.nan]])
result = squareform(pdist(arr, nan_manhattan))
print(result)
Output
[[ 0. 14.83333333 17.33333333]
[14.83333333 0. 19.625 ]
[17.33333333 19.625 0. ]]
I want to convert a pandas Series of strings of list of numbers into a numpy array. What I have is something like:
ds = pd.Series(['[1 -2 0 1.2 4.34]', '[3.3 4 0 -1 9.1]'])
My desired output:
arr = np.array([[1, -2, 0, 1.2, 4.34], [3.3, 4, 0, -1, 9.1]])
What I have done so far is to convert the pandas Series to a Series of a list of numbers as:
ds1 = ds.apply(lambda x: [float(number) for number in x.strip('[]').split(' ')])
but I don't know how to go from ds1 to arr.
Use Series.str.strip + Series.str.split and create a new np.array with dtype=float:
arr = np.array(ds.str.strip('[]').str.split().tolist(), dtype='float')
Result:
print(arr)
array([[ 1. , -2. , 0. , 1.2 , 4.34],
[ 3.3 , 4. , 0. , -1. , 9.1 ]])
You can try to remove the "[]" from the Series object first, then things will become easier, https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.split.html.
ds1 = ds.str.strip("[]")
# split and exapand the data, conver to numpy array
arr = ds1.str.split(" ", expand=True).to_numpy(dtype=float)
Then arr will be the right format you want,
array([[ 1. , -2. , 0. , 1.2 , 4.34],
[ 3.3 , 4. , 0. , -1. , 9.1 ]])
Then I did a little profiling in comparison with Shubham's colution.
# Shubham's way
%timeit arr = np.array(ds.str.strip('[]').str.split().tolist(), dtype='float')
332 µs ± 5.72 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# my way
%timeit ds.str.strip("[]").str.split(" ", expand=True).to_numpy(dtype=float)
741 µs ± 4.21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
Obviously, his solution is much faster! Cheers!
I want to group a table such that the two first columns remain as they were when grouped, the 3d is the grouping mean, and the 4th the grouping dispersion, defined in the code. This is how I currently do it:
x = pd.DataFrame(np.array(((1,1,1,1),(1,1,10,2),(2,2,2,2),(2,2,8,3))))
0 1 2 3
0 1 1 1 1
1 1 1 10 2
2 2 2 2 2
3 2 2 8 3
g = x.groupby(0)
res = g.mean()
res[3] = g.apply(lambda x: ((x[2]+x[3]).max()-(x[2]-x[3]).min())*0.5)
res
1 2 3
0
1 1.0 5.5 6.0
2 2.0 5.0 5.5
I am looking to speed this up anyway possible. In particular if I could get rid of apply and use g only once that would be great.
For testing purposes, this runs on data sizes of:
A few to 60 rows
1-5 groups (there could be a single group)
4 columns
Here is a mid-sized sample:
array([[ 0.00000000e+000, 4.70221520e-003, 1.14943038e-003,
3.44829114e-009],
[ 1.81557753e-011, 4.94065646e-324, 4.70221520e-003,
1.14943038e-003],
[ 2.36416931e-008, 1.97231804e-011, 9.88131292e-324,
8.43322640e-003],
[ 1.74911362e-003, 3.43575891e-009, 1.12130677e-010,
1.48219694e-323],
[ 8.43322640e-003, 1.74911362e-003, 3.42014182e-009,
1.11974506e-010],
[ 1.97626258e-323, 4.70221520e-003, 1.14943038e-003,
3.48747627e-009],
[ 1.78945412e-011, 2.47032823e-323, 4.70221520e-003,
1.14943038e-003],
[ 2.32498418e-008, 1.85476266e-010, 2.96439388e-323,
4.70221520e-003],
[ 1.14943038e-003, 3.50053798e-009, 1.85476266e-011,
3.45845952e-323],
[ 4.70221520e-003, 1.14943038e-003, 4.53241298e-008,
3.00419304e-010],
[ 3.95252517e-323, 4.70221520e-003, 1.14943038e-003,
3.55278482e-009],
[ 1.80251583e-011, 4.44659081e-323, 4.70221520e-003,
1.14943038e-003],
[ 1.09587738e-008, 1.68496045e-011, 4.94065646e-323,
4.70221520e-003],
[ 1.14943038e-003, 3.48747627e-009, 1.80251583e-011,
5.43472210e-323],
[ 4.70221520e-003, 1.14943038e-003, 3.90545096e-008,
2.63846519e-010],
[ 5.92878775e-323, 8.43322640e-003, 1.74911362e-003,
3.15465136e-009],
[ 1.04009792e-010, 6.42285340e-323, 8.43322640e-003,
1.74911362e-003],
[ 2.56120209e-010, 4.15414486e-011, 6.91691904e-323,
8.43322640e-003],
[ 1.74911362e-003, 3.43575891e-009, 1.12286848e-010,
7.41098469e-323],
[ 8.43322640e-003, 1.74911362e-003, 5.91887557e-009,
1.45863583e-010],
[ 7.90505033e-323, 8.43322640e-003, 1.74911362e-003,
3.34205639e-009],
[ 1.07133209e-010, 8.39911598e-323, 8.43322640e-003,
1.74911362e-003],
[ 1.21188587e-009, 7.07453993e-011, 8.89318163e-323,
8.43322640e-003],
[ 1.74911362e-003, 3.38890765e-009, 1.12130677e-010,
9.38724727e-323],
[ 8.43322640e-003, 1.74911362e-003, 1.79596488e-009,
8.38637515e-011]])
You can use syntacting sugar - .groupby with Series:
res[3] = ((x[2] + x[3]).groupby(x[0]).max() - (x[2] - x[3]).groupby(x[0]).min())*.5
print (res)
1 2 3
0
1 1.0 5.5 6.0
2 2.0 5.0 5.5
I get this timigs with you array:
In [279]: %%timeit
...: res = x.groupby(0).mean()
...: res[3] = ((x[2] + x[3]).groupby(x[0]).max() - (x[2] - x[3]).groupby(x[0]).min())*.5
...:
4.26 ms ± 62.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [280]: %%timeit
...: g = x.groupby(0)
...: res = g.mean()
...: res[3] = g.apply(lambda x: ((x[2]+x[3]).max()-(x[2]-x[3]).min())*0.5)
...:
11 ms ± 76.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Also if turn off sorting by grouping column if possible:
In [283]: %%timeit
...: res = x.groupby(0, sort=False).mean()
...: res[3] = ((x[2] + x[3]).groupby(x[0], sort=False).max() - (x[2] - x[3]).groupby(x[0], sort=False).min())*.5
...:
4.1 ms ± 50.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
arr is a n-dimensional numpy array.
How to change sign of every element of arr with an odd sum of indices?
For example, arr[0, 1, 2] needs a sign change because it has a sum of indices 0 + 1 + 2 = 3, which is odd.
When I convert arr to a list, I notice that every second element in the list are elements that needs a sign change.
Another example:
Original array:
[[[ 0 1 2]
[ 3 4 5]
[ 6 7 8]]
[[ 9 10 11]
[12 13 14]
[15 16 17]]
[[18 19 20]
[21 22 23]
[24 25 26]]]
Array with signs changed:
[[[ 0 -1 2]
[ -3 4 -5]
[ 6 -7 8]]
[[ -9 10 -11]
[12 -13 14]
[-15 16 -17]]
[[18 -19 20]
[-21 22 -23]
[24 -25 26]]]
np.negative is silghtly faster than multiplying (as it is a ufunc)
N = 5
arr = np.arange(N ** 3).reshape(N, N, N)
%timeit arr.ravel()[1::2] *= -1
%timeit np.negative(arr.ravel()[1::2], out = arr.ravel()[1::2])
The slowest run took 8.74 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 3.39 µs per loop
The slowest run took 5.57 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 3.12 µs per loop
N = 25
arr = np.arange(N ** 3).reshape(N, N, N)
%timeit arr.ravel()[1::2] *= -1
%timeit np.negative(arr.ravel()[1::2], out = arr.ravel()[1::2])
The slowest run took 7.03 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 10.8 µs per loop
The slowest run took 5.27 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 8.63 µs per loop
N = 101
arr = np.arange(N ** 3).reshape(N, N, N)
%timeit arr.ravel()[1::2] *= -1
%timeit np.negative(arr.ravel()[1::2], out = arr.ravel()[1::2])
1000 loops, best of 3: 663 µs per loop
1000 loops, best of 3: 512 µs per loop
Greatly simplified by hpaulj's suggestion.
Program:
import numpy as np
def change_sign(arr):
"""Switch sign of every second element of arr in-place
Note
----
Modifies the input array (arr).
"""
# arr.reshape(-1) makes a 1D view of arr
#
# [1::2] select every other element of arr,
# starting from the 1st element.
#
# *= -1 changes sign of selected elements.
arr.reshape(-1)[1::2] *= -1
return arr
def main():
N = 3
arr = np.arange(N ** 3).reshape(N, N, N)
print("original array:")
print(arr)
print("change signs")
print(change_sign(arr))
main()
Result:
original array:
[[[ 0 1 2]
[ 3 4 5]
[ 6 7 8]]
[[ 9 10 11]
[12 13 14]
[15 16 17]]
[[18 19 20]
[21 22 23]
[24 25 26]]]
change signs
[[[ 0 -1 2]
[ -3 4 -5]
[ 6 -7 8]]
[[ -9 10 -11]
[ 12 -13 14]
[-15 16 -17]]
[[ 18 -19 20]
[-21 22 -23]
[ 24 -25 26]]]