Develop Reference

very strange results of numpy.astype with numba

WHY? very strange...
In python, if we test np.astype() with numba, the following will print some results as
x: [-6. -5. -4. -3. -2. -1. 0. 1. 2. 3. 4. 5.]
x-int: [-6 -5 -4 -3 -2 -1 0 1 2 3 4 5]
#numba.njit
def tt():
nn = 3
x = np.linspace(0, 4*nn-1, 4*nn)-2*nn
print(x)
print(x.astype(np.int32))
BUT, if I change the line of x to be x = np.linspace(0, 8*nn-1, 8*nn)-4*nn, the result will be strange as
x: [-12. -11. -10. -9. -8. -7. -6. -5. -4. -3. -2. -1. 0. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.]
x-int: [-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 0 2 3 4 5 6 7 8 9 10 11]
There are two 0 in x-int? why?

tl;dr: This is a reported bug of Numba.
The issue come from a slightly inaccuracy in the Numba linspace function related to floating-point rounding. Here is an example to highlight the issue:
def tt_classic():
nn = 3
return np.linspace(0, 8*nn-1, 8*nn)-4*nn
#numba.njit
def tt_numba():
nn = 3
return np.linspace(0, 8*nn-1, 8*nn)-4*nn
print(tt_classic()[13])
print(tt_numba()[13])
Here is the result:
1.0
0.9999999999999982
As you can see, the Numba implementation does not return a exact value. While this problem cannot be avoided for big values, it can be considered as a bug for such small value since they can be represented exactly (without any loss of precision) on any IEEE-754 platform.
As a result, the cast will then truncate the floating point number 0.9999999999999982 to 0 (and not to the nearest integer). If you want a safe conversion (ie. workaround), you can explicitly tell Numpy/Numba to do it. Here is an example:
#numba.njit
def tt():
nn = 3
x = np.linspace(0, 4*nn-1, 4*nn)-2*nn
np.round(x, 0, x)
print(x)
print(x.astype(np.int32))
This bug as been reported on the Numba bug tracker here.
You may also be interested in this related Numba issue.

Efficient NumPy way of rolling nanstd

I have a 1-D NumPy array where I create a rolling window and then compute the np.nanstd:
import numpy as np
def rolling_window(a, window):
a = np.asarray(a)
shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
strides = a.strides + (a.strides[-1],)
return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
if __name__ == "__main__":
n = 100_000_000
nan_indices = np.random.choice(np.arange(n), size=1000, replace=False)
T = np.random.rand(n)
T[nan_indices] = np.nan
m = 50
np.nanstd(rolling_window(T, m), axis=T.ndim)
However, I noticed that not only is this extremely time consuming, it also uses a lot of memory. Is there a way to improve both the memory and speed performance (Numba is an option)?

NumPy vectorized
After grueling through the math, here's what I ended up with few np.convolve and some masking to get a vectorized NumPy solution -
def nanstd(a, W):
k = np.ones(W, dtype=int)
m = ~np.isnan(a)
a0 = np.where(m, a,0)
n = np.convolve(m,k,'valid')
c1 = np.convolve(a0, k,'valid')
f2 = c1**2
p2 = f2/n**2
f1 = np.convolve((a0**2)*m,k,'valid')+n*p2
out = np.sqrt((f1 - (2/n)*f2)/n)
return out
Complete Explanation is at the end of this post.
Pandas equivalent
Here's the equivalent pandas version, which isn't too bad on performance -
import pandas as pd
def pdroll(T,m):
return pd.Series(T).rolling(m).std(ddof=0).values[m-1:]
Benchmarking
Using benchit package (few benchmarking tools packaged together; disclaimer: I am its author) to benchmark proposed solutions.
def setup(n):
nan_indices = np.random.choice(np.arange(n), size=10, replace=False)
T = np.random.rand(n)
T[nan_indices] = np.nan
return T
import benchit
f = {'rolling': lambda T,m: np.nanstd(rolling_window(T, m), axis=T.ndim),
'pdroll': pdroll, 'conv':nanstd}
in_={(n,w):(setup(n),w) for n in 10**np.arange(2,6) for w in [5,10,20,50,80,100]}
t = benchit.timings(f, in_, multivar=True)
t.plot(logx=True, sp_ncols=2, save='timings.png', dpi=200)
NumPy one is good on smaller window sizes, while pandas is better on larger ones.
NumPy vectorized : Explanation on NumPy version nanstd
Basically, np.nanstd is computing std ignoring NaNs. Now, std can be computed based on mean.
Thus, for an array a with no NaNs, it would be :
np.sqrt(np.mean((a-np.mean(a))**2)) # (1)
Let's prove it :
In [43]: a = np.arange(1,6).astype(float)
In [44]: np.nanstd(a)
Out[44]: 1.4142135623730951
In [45]: np.sqrt(np.mean((a-np.mean(a))**2))
Out[45]: 1.4142135623730951
Now, let's say, we have a NaN in it :
In [46]: a[2] = np.nan
In [47]: a
Out[47]: array([ 1., 2., nan, 4., 5.])
The std with nanstd would be :
In [48]: np.nanstd(a)
Out[48]: 1.5811388300841898
Let's figure out the equivalent one based on (1).
Let's start with (a-np.mean(a))**2.
This one : ?
In [72]: (a-np.mean(a))**2
Out[72]: array([nan, nan, nan, nan, nan])
No!
This one : ?
In [73]: (a0 - np.sum(a0)/n)**2
Out[73]: array([4., 1., 9., 1., 4.])
No! Because a is :
In [76]: a
Out[76]: array([ 1., 2., nan, 4., 5.])
We need to make the NaN position one 0.
This one : ?
In [75]: m*((a0 - np.sum(a0)/n)**2)
Out[75]: array([4., 1., 0., 1., 4.])
Yes!
Then, what's np.mean((a-np.mean(a))**2)? It would be, sum of those in [75] divided by n :
In [77]: np.sum(m*((a0-np.sum(a0)/n)**2))/n
Out[77]: 2.5
Hence, the final std value :
In [78]: np.sqrt(np.sum(m*((a0-np.sum(a0)/n)**2))/n)
Out[78]: 1.5811388300841898
Summarizing :
In [55]: m = ~np.isnan(a) # (2)
...: a0 = np.where(m, a,0)
...: n = m.sum()
...: out0 = np.sqrt(np.sum(m*((a0-np.sum(a0)/n)**2))/n)
In [56]: out0
Out[56]: 1.5811388300841898
Next part is incorporating the sliding nature. So, we need to do (2) in a sliding nature. So, the first two steps remains.
Hence, it starts off with :
m = ~np.isnan(a)
a0 = np.where(m, a,0)
But the last two would change, let's see how.
Let's focus on the final step to compute out0. We have :
m*((a0-np.sum(a0)/n)**2)
Then, we compute the summation :
np.sum(m*((a0-np.sum(a0)/n)**2))
We have : (a-b)**2 = a**2 + b**2 - 2*a*b. So, earlier step becomes
np.sum(m*(a0**2 + (np.sum(a0)/n)**2 - 2*a0*np.sum(a0)/n))
Further re-arranging leads to :
np.sum(m*(a0**2 + (np.sum(a0)/n)**2) - np.sum(2*a0*np.sum(a0)/n))
np.sum(m*(a0**2 + (np.sum(a0)/n)**2)) - np.sum(2*a0*np.sum(a0)/n)
np.sum(m*(a0**2 + (np.sum(a0)/n)**2)) - 2*np.sum(a0*np.sum(a0))/n
np.sum(m*(a0**2 + (np.sum(a0)/n)**2)) - (2/n)*np.sum(a0*np.sum(a0)) # (3)
Let's focus on the first two parts for the summation.
Also, let's take a sample case to make things concrete. We will setup two datasets - One for the complete array setup and another a windowed version of the complete one.
Setup :
#=========================== 1. Complete setup
a = np.arange(1,10).astype(float)
a[[2,5]] = np.nan
W = 5
k = np.ones(W, dtype=int)
m_comp = ~np.isnan(a)
a0_comp = np.where(m_comp, a,0)
n_comp = np.convolve(m_comp,k,'valid')
c1 = np.convolve(a0_comp, k,'valid')
c2 = np.convolve((a0_comp**2)*m_comp,k,'valid')
#=========================== 2. Windowed setup
a1 = np.arange(1,6).astype(float)
a1[2] = np.nan
m = ~np.isnan(a1)
a0 = np.where(m, a1,0)
n = m.sum()
out0 = np.sqrt(np.sum(m*((a0-np.sum(a0)/n)**2))/n)
From windowed setup, we have :
In [51]: np.sum(m*(a0**2 + (np.sum(a0)/n)**2))
Out[51]: 82.0
In [52]: np.sum(m*(a0**2) + m*((np.sum(a0)/n)**2))
Out[52]: 82.0
In [53]: np.sum(m*(a0**2)) + np.sum(m*((np.sum(a0)/n)**2))
Out[53]: 82.0
First summation part :
In [86]: np.sum(m*(a0**2))
Out[86]: 46.0
# complete setup version :
In [87]: c2
Out[87]: array([ 46., 45., 90., 154., 219.])
Second summation part :
In [54]: np.sum(m*((np.sum(a0)/n)**2))
Out[54]: 36.0
# complete setup version :
In [55]: n_comp*(c1/n_comp)**2
Out[55]:
array([ 36. , 40.33333333, 85.33333333, 144. ,
210.25 ])
The remaining piece of puzzle fromn (3) is :
In [79]: (2/n)*np.sum(a0*np.sum(a0))
Out[79]: 72.0
Let's focus on the meat of it :
In [80]: np.sum(a0*np.sum(a0))
Out[80]: 144.0
On the complete setup, it would correspond to :
In [81]: c1**2
Out[81]: array([144., 121., 256., 576., 841.])
Thus, for the entire remaining piece :
In [82]: (2/n)*np.sum(a0*np.sum(a0))
Out[82]: 72.0
# complete setup version :
In [83]: (2/n_comp)*c1**2
Out[83]:
array([ 72. , 80.66666667, 170.66666667, 288. ,
420.5 ])
Hence, (3) and its complete version counterpart would be :
In [89]: np.sum(m*(a0**2 + (np.sum(a0)/n)**2)) - (2/n)*np.sum(a0*np.sum(a0))
Out[89]: 10.0
In [90]: c2 + n_comp*(c1/n_comp)**2 - (2/n_comp)*c1**2
Out[90]: array([10. , 4.66666667, 4.66666667, 10. , 8.75 ])
To get the final std values, we need to divide by the count of valid ones per window and then apply sqrt :
In [99]: np.sqrt((c2 + n_comp*(c1/n_comp)**2 - (2/n_comp)*c1**2)/n_comp)
Out[99]: array([1.58113883, 1.24721913, 1.24721913, 1.58113883, 1.47901995])
Hence, with some cleanup, we end up with the final nanstd version.

2D local maxima and minima in Python

I have a data frame, df, representing a correlation matrix, with this heatmap with example extrema. Every point has, obviously, (x,y,value):
I am looking into getting the local extrema. I looked into argrelextrema, I tried it on individual rows and the results were as expected, but that didn't work for 2D. I have also looked into scipy.signal.find_peaks, but this is for a 1D array.
Is there anything in Python that will return the local extrema over/under certain values(threshold)?
Something like an array of (x, y, value)? If not then can you point me in the right direction?

This is a tricky question, because you need to carefully define the notion of how "big" a maximum or minimum needs to be before it is relevant. For example, imagine that you have a patch containing the following 5x5 grid of pixels:
im = np.array([[ 0 0 0 0 0
0 5 5 5 0
0 5 4 5 0
0 5 5 5 0
0 0 0 0 0. ]])
This might be looked at as a local minimum, because 4 is less than the surrounding 5s. OTOH, it might be looked at as a local maximum, where the single lone 4 pixel is just "noise", and the 3x3 patch of average 4.89-intensity pixels is actually a single local maximum. This is commonly known as the "scale" at which you are viewing the image.
In any case, you can estimate the local derivative in one direction by using a finite difference in that direction. The x direction might be something like:
k = np.array([[ -1 0 1
-1 0 1
-1 0 1. ]])
Applying this filter to the image patch defined above gives:
>>> cv2.filter2D(im, cv2.CV_64F, k)[1:-1,1:-1]
array([[ 9., 0., -9.],
[ 14., 0., -14.],
[ 9., 0., -9.]])
Applying a similar filter in the y direction will transpose this. The only point in here with a 0 in both the x and the y directions is the very middle, which is the 4 that we decided was a local minimum. This is tantamount to checking that the gradient is 0 in both x and y.
This whole process can be extended to find the larger single local maximum that we have identified. You'll use a larger filter, e.g.
k = np.array([[ -2, -1, 0, 1, 2],
[ -2, -1, 0, 1, 2], ...
Since the 4 makes the local maximum an approximate thing, you'll need to use some "approximate" logic. i.e. you'll look for values that are "close" to 0. Just how close depends on just how fudgy you are willing to allow the local extrema to be. To sum up, the two fudge factors here are 1. filter size and 2. ~=0 fudge factor.

Maximizing the value of an equation given a constraint in python

There are four variables
(S1, S2, S3, S4)
with the constraint
(S1+S2+S3+S4=100).
There are four given constants (C1, C2, C3, C4). I want to maximize the value of (S1/C1 + S2/C2 + S3/C3 + S4/C4). Here is my code in python:
#!/usr/bin/env python3
import numpy as np
from scipy.optimize import minimize
S0 = [25, 25, 25, 25]
C = [89415,8991,10944,15164]
def objective(S, C):
total = 0
for index in range(4):
total = total + S[index]/C[index]
return -total
def constraint(S):
return (100 - S[0] - S[1] - S[2] - S[3])
b = (0.0, 100.0)
boundaries = (b,b,b,b)
con = ({'type':'eq', 'fun':constraint})
solution = minimize(objective,S0,args=(C),method='SLSQP',bounds=boundaries,constraints=con)
print (solution)
My code is simply returning the initial guess for S as the final result
fun: -0.0069931517268763755
jac: array([-1.11838453e-05, -1.11222384e-04, -9.13742697e-05, -6.59456709e-05])
message: 'Optimization terminated successfully.'
nfev: 6
nit: 1
njev: 1
status: 0
success: True
x: array([25., 25., 25., 25.])
Where am I going wrong?

It looks like the differences in the output values of your functions are within the default tolerance for the optimizer to stop optimizing your function between iterations. Setting your tolerance to a smaller value like 1e-12 helps with this:
solution = minimize(objective,S0,args=(C),method='SLSQP',bounds=boundaries,constraints=con, tol=1e-12)
Result:
fun: -0.01112223334445557
jac: array([ -1.11837871e-05, -1.11222267e-04, -9.13742697e-05,
-6.59456709e-05])
message: 'Optimization terminated successfully.'
nfev: 192
nit: 32
njev: 32
status: 0
success: True
x: array([ 0.00000000e+00, 1.00000000e+02, 3.01980663e-14,
0.00000000e+00])
which is approximately equal to the absolute maximum solution [0,100,0,0].

scipy.optimize with SLSQP. 'Singular matrix C in LSQ subproblem'

I'm trying to minimize a dot product of 2 vectors but it doesn't work and I have no idea why. Can someone please help me?
I have a matrix c of this form:
c = [[c11, c12, c13, c14, c15],
[c21, c22, c23, c24, c25]]
I want to get a matrix p of this form:
p = [[p11, p12, p13, p14, p15],
[p21, p22, p23, p24, p25]]
I want to maximize this value :
c11*p11 + c12*p12 +c13*p13 + c14*p14 + c15*p15 + c21*p21 + c22*p22 +c23*p23 + c24*p24 + c25*p25
To get that I convert the c and p to 1-D vector and do the dot product so that my function to maximize is:
f(p) = c.dot(p)
The constraints are:
c11 + c12 + c13 + c14 + c15 = 1
c21 + c22 + c23 + c24 + c25 = 1
every element in p must be between 0.01 and 0.99.
I have tried scipy.optimize.linprog and it works:
from scipy.optimize import linprog
c = np.array([0. , 0. , 0. , 0. , 0. , 0. , 20094.21019108, 4624.08079143, 6625.51724138, 3834.81081081])
A_eq = np.array([[1,1,1,1,1,0,0,0,0,0],
[0,0,0,0,0,1,1,1,1,1]])
b_eq = np.array([1, 1])
res = linprog(-c, A_eq=A_eq, b_eq=b_eq, bounds=(0.01, 0.99))
res
Out[561]:
fun: -19441.285871873002
message: 'Optimization terminated successfully.'
nit: 13
slack: array([0.03, 0.98, 0.98, 0.98, 0.98, 0.98, 0.03, 0.98, 0.98, 0.98, 0. ,
0. , 0.95, 0. , 0. , 0. , 0. , 0. , 0. , 0. ])
status: 0
success: True
x: array([0.96, 0.01, 0.01, 0.01, 0.01, 0.01, 0.96, 0.01, 0.01, 0.0
But I'm trying to use scipy.optimize.minimize with SLSQP instead and that's where I get this 'Singular matrix C in LSQ subproblem' . Here is what I've done:
from scipy.optimize import minimize
def build_objective(ck, sign = -1.00):
"""
Builds the objective fuction for matrix ck
"""
# Here I turn my c matrix to a 1-D matrix
ck = np.concatenate(ck)
def objective(P):
return sign*(ck.dot(P))
return objective
def build_constraint_rows(ck):
"""
Builds the constraint functions that specify that the sum of the proportions for
each bin equals 1
"""
ncol = ck.shape[1]
nrow = ck.shape[0]
constrain_dict = []
for i in range(nrow):
vector = np.zeros((nrow,ncol))
vector[i, :] = 1
vector = np.concatenate(vector)
def row_constrain(P):
return 1 - vector.dot(P)
constrain_dict.append({'type': 'eq', 'fun': row_constrain})
return constrain_dict
# Matrix: Notice that this is not in vector form yet
c = np.array([[0. , 0. , 0. , 0., 0.],
[0. , 20094.21019108, 4624.08079143, 6625.51724138, 3834.81081081]])
# I need some initial p matrix for the function 'minimize'. I look for the value of the row that is the highest and assign it a proportion p of 0.96 and the rest 0.01 so the sum in 1 per row
P_initial = np.ones(c.shape)*0.01
nrow = test.shape[0]
for i in range(nrow):
index= np.where(c[i,] == np.max(c[i,]))[0]
if index.shape[0] > 1:
index = int(np.random.choice(index, size = 1))
else:
index = int(index)
P_initial[i,index] = 0.96
# I turn the P_initial to vector form
P_initial = np.concatenate(P_initial)
# These are the bounds of each p value
b = (0.01,0.99)
bnds = (b,)*c.size
# I then use my previous functions
objective_fun = build_objective(c)
cons = build_constraint_rows(c)
res = minimize(objective_fun,P_initial,method='SLSQP',\
bounds=bnds,constraints=cons)
This is my final result:
res
Out[546]:
fun: -19434.501741138763
jac: array([0. , 0.,0. , 0. ,0. , 0., -20094.21020508, -4624.08056641, -6625.51708984, -3834.81079102])
message: 'Singular matrix C in LSQ subproblem'
nfev: 24
nit: 2
njev: 2
status: 6
success: False
x: array([0.96 , 0.01 , 0.01 , 0.01 , 0.01 ,
0.01020202, 0.95962502, 0.01006926, 0.01001178, 0.01009192])
Please help me understand what I'm doing wrong.
Thank you in advanced,
Karol