I'm performing Data Science and am calculating the Log Likelihood of a Poisson Distribution of arrival times.
def LogLikelihood(arrival_times, _lambda):
"""Calculate the likelihood that _lambda predicts the arrival_times well."""
ll = 0
for t in arrival_times:
ll += -(_lambda) + t*np.log(_lambda) - np.log(factorial(t))
return ll
Mathematically, the expression is on the last line:
Is there a more pythonic way to perform this sum? Perhaps in one line?
Seems perfectly Pythonic to me; but since numpy is already here, why not to vectorize the whole thing?
return (
-_lambda
+ arrival_times * np.log(_lambda)
- np.log(np.vectorize(np.math.factorial)(arrival_times))
).sum()
If you have scipy available, use loggamma which is more robust than chaining log and factorial:
from scipy import special
def loglikeli(at,l):
return (np.log(l)*at-l-special.loggamma(at+1)).sum()
### example
rng = np.random.default_rng()
at = rng.integers(1,3,10).cumsum()
l = rng.uniform(0,1)
### check against OP's impementation
np.isclose(loglikeli(at,l),LogLikelihood(at,l))
# True
This looks ugly to me but it fits a single line:
def LogLikelihood(arrival_times, _lambda):
return np.cumsum(list(map(lambda t: -(_lambda) + t*np.log(_lambda) - np.log(factorial(t)),arrival_times)))[-1]
Related
What is the fastest way to remove the N-Rightmost number in python integer?
Here are some of my code:
def f_int(dividend,n):
return int(dividend/(10 ** n))
def f_str_to_int(dividend,n):
return int(str(dividend)[:-n])
If we don't care whether the output is in str on int, we can skip the int() in def_f_str_to_int():
def f_str(dividend,n):
return str(dividend)[:-n]
We also can increase speed by asking the input in the form of power of ten:
divisor_in_power_of_ten = 10 ** n #outside function run time
def f_int_hack (dividend,divisor_in_power_of_ten):
return int(dividend/(divisor_in_power_of_ten))
My question is, is there a faster way (maybe using bit manipulation)? I need to optimize it since it will be used as part of a real-time web. Note: Restricted only using python, not JS or Cython (it's okay to extend your answer in JS and Cython, but that is not the main question).
Here is my result:
FUNCTION: f_int_hack Used 200170 times
MEDIAN 1.1000000004202093e-06
MEAN 1.5179877104460484e-06
STDEV 3.600025074234889e-05
FUNCTION: f_int Used 199722 times
MEDIAN 1.8999999999991246e-06
MEAN 2.420203582980709e-06
STDEV 3.482858342790541e-05
FUNCTION: f_str Used 200132 times
MEDIAN 1.4999999997655777e-06
MEAN 1.7462234924949252e-06
STDEV 1.4733864640549157e-05
FUNCTION: f_str_to_int Used 199639 times
MEDIAN 2.000000000279556e-06
MEAN 2.751038624717222e-06
STDEV 6.383386278143267e-05
Edit:
Function for the benchmark (edit accordingly, since I edit some of them):
import time
import random
import statistics
def benchmark(functions, iteration, *args):
times = {f.__name__: [] for f in functions}
for i in range(iteration):
func = random.choice(functions)
t0 = time.perf_counter()
func(*args)
t1 = time.perf_counter()
times[func.__name__].append(t1 - t0)
for name, numbers in times.items():
print('FUNCTION:', name, 'Used', len(numbers), 'times')
print('\tMEDIAN', statistics.median(numbers))
print('\tMEAN ', statistics.mean(numbers))
print('\tSTDEV ', statistics.stdev(numbers))
if __name__=="__main__":
# Variables
divident = 12345600000
n = 3
iteration = 1000000
# The functions to compare
def f_int(divident,n):
return int(divident/(10 ** n))
def f_str_to_int(divident,n):
return int(str(divident)[:-n])
functions = f_int, f_str_to_int
benchmark(functions, iteration, divident, n)
More clarification: input is python integer. Actually, i didn't care with the output format (wether it's str or int), but let's make the output in int first and str as "bonus question".
Edit:
From comment section a//b:
FUNCTION: f_int_double_slash Used 166028 times
MEDIAN 1.2000000002565514e-06
MEAN 1.4399938564575845e-06
STDEV 1.2767417156171526e-05
YES, IT'S FASTER THAN int(a/b)
Edit:
From comment, if we accept float, the fastest way is:
def f_float_hack(dividend,divisor_in_power_of_ten):
return dividend//divisor_in_power_of_ten
With result:
FUNCTION: f_float_hack Used 142983 times
MEDIAN 7.000000001866624e-07
MEAN 9.574040270508322e-07
STDEV 3.725603760159355e-05
Improvement of int double slash using precomputing power of ten:
FUNCTION: f_int_double_slash_hack Used 143082 times
MEDIAN 7.999999995789153e-07
MEAN 1.1596266476572136e-06
STDEV 4.9442788346866335e-05
Current result: float_hack is the fastest (if we accept float).
As I'm really struggleing to get from R-code, to Python code, I would like to ask some help. The code I want to use has been provided to my from withing the mathematics forum of stackexchange.
https://math.stackexchange.com/questions/2205573/curve-fitting-on-dataset
I do understand what is going on. But I'm really having a hard time trying to solve the R-code, as I have never seen anything of it. I have written the function to return the sum of squares. But I'm stuck at how I could use a function similar to the optim function. And also I don't really like the guesswork at the initial values. I would like it better to run and re-run a type of optim function untill I get the wanted result, because my needs for a nearly perfect curve fit are really high.
def model (par,x):
n = len(x)
res = []
for i in range(1,n):
A0 = par[3] + (par[4]-par[1])*par[6] + (par[5]-par[2])*par[6]**2
if(x[i] == par[6]):
res[i] = A0 + par[1]*x[i] + par[2]*x[i]**2
else:
res[i] = par[3] + par[4]*x[i] + par[5]*x[i]**2
return res
This is my model function...
def sum_squares (par, x, y):
ss = sum((y-model(par,x))^2)
return ss
And this is the sum of squares
But I have no idea on how to convert this:
#I found these initial values with a few minutes of guess and check.
par0 <- c(7,-1,-395,70,-2.3,10)
sol <- optim(par= par0, fn=sqerror, x=x, y=y)$par
To Python code...
I wrote an open source Python package (BSD license) that has a genetic algorithm (Differential Evolution) front end to the scipy Levenberg-Marquardt solver, it functions similarly to what you describe in your question. The github URL is:
https://github.com/zunzun/pyeq3
It comes with a "user-defined function" example that's fairly easy to use:
https://github.com/zunzun/pyeq3/blob/master/Examples/Simple/FitUserDefinedFunction_2D.py
along with command-line, GUI, cluster, parallel, and web-based examples. You can install the package with "pip3 install pyeq3" to see if it might suit your needs.
Seems like I have been able to fix the problem.
def model (par,x):
n = len(x)
res = np.array([])
for i in range(0,n):
A0 = par[2] + (par[3]-par[0])*par[5] + (par[4]-par[1])*par[5]**2
if(x[i] <= par[5]):
res = np.append(res, A0 + par[0]*x[i] + par[1]*x[i]**2)
else:
res = np.append(res,par[2] + par[3]*x[i] + par[4]*x[i]**2)
return res
def sum_squares (par, x, y):
ss = sum((y-model(par,x))**2)
print('Sum of squares = {0}'.format(ss))
return ss
And then I used the functions as follow:
parameter = sy.array([0.0,-8.0,0.0018,0.0018,0,200])
res = least_squares(sum_squares, parameter, bounds=(-360,360), args=(x1,y1),verbose = 1)
The only problem is that it doesn't produce the results I'm looking for... And that is mainly because my x values are [0,360] and the Y values only vary by about 0.2, so it's a hard nut to crack for this function, and it produces this (poor) result:
Result
I think that the range of x values [0, 360] and y values (which you say is ~0.2) is probably not the problem. Getting good initial values for the parameters is probably much more important.
In Python with numpy / scipy, you would definitely want to not loop over values of x but do something more like
def model(par,x):
res = par[2] + par[3]*x + par[4]*x**2
A0 = par[2] + (par[3]-par[0])*par[5] + (par[4]-par[1])*par[5]**2
res[np.where(x <= par[5])] = A0 + par[0]*x + par[1]*x**2
return res
It's not clear to me that that form is really what you want: why should A0 (a value independent of x added to a portion of the model) be so complicated and interdependent on the other parameters?
More importantly, your sum_of_squares() function is actually not what least_squares() wants: you should return the residual array, you should not do the sum of squares yourself. So, that should be
def sum_of_squares(par, x, y):
return (y - model(par, x))
But most importantly, there is a conceptual problem that is probably going to plague this model: Your par[5] is meant to represent a breakpoint where the model changes form. This is going to be very hard for these optimization routines to find. These routines generally make a very small change to each parameter value to estimate to derivative of the residual array with respect to that variable in order to figure out how to change that variable. With a parameter that is essentially used as an integer, the small change in the initial value will have no effect at all, and the algorithm will not be able to determine the value for this parameter. With some of the scipy.optimize algorithms (notably, leastsq) you can specify a scale for the relative change to make. With leastsq that is called epsfcn. You may need to set this as high as 0.3 or 1.0 for fitting the breakpoint to work. Unfortunately, this cannot be set per variable, only per fit. You might need to experiment with this and other options to least_squares or leastsq.
"""Some simulations to predict the future portfolio value based on past distribution. x is
a numpy array that contains past returns.The interpolated_returns are the returns
generated from the cdf of the past returns to simulate future returns. The portfolio
starts with a value of 100. portfolio_value is filled up progressively as
the program goes through every loop. The value is multiplied by the returns in that
period and a dollar is removed."""
portfolio_final = []
for i in range(10000):
portfolio_value = [100]
rand_values = np.random.rand(600)
interpolated_returns = np.interp(rand_values,cdf_values,x)
interpolated_returns = np.add(interpolated_returns,1)
for j in range(1,len(interpolated_returns)+1):
portfolio_value.append(interpolated_returns[j-1]*portfolio_value[j-1])
portfolio_value[j] = portfolio_value[j]-1
portfolio_final.append(portfolio_value[-1])
print (np.mean(portfolio_final))
I couldn't find a way to write this code using numpy. I was having a look at iterations using nditer but I was unable to move ahead with that.
I guess the easiest way to figure out how you can vectorize your stuff would be to look at the equations that govern your evolution and see how your portfolio actually iterates, finding patterns that could be vectorized instead of trying to vectorize the code you already have. You would have noticed that the cumprod actually appears quite often in your iterations.
Nevertheless you can find the semi-vectorized code below. I included your code as well such that you can compare the results. I also included a simple loop version of your code which is much easier to read and translatable into mathematical equations. So if you share this code with somebody else I would definitely use the simple loop option. If you want some fancy-pants vectorizing you can use the vector version. In case you need to keep track of your single steps you can also add an array to the simple loop option and append the pv at every step.
Hope that helps.
Edit: I have not tested anything for speed. That's something you can easily do yourself with timeit.
import numpy as np
from scipy.special import erf
# Prepare simple return model - Normal distributed with mu &sigma = 0.01
x = np.linspace(-10,10,100)
cdf_values = 0.5*(1+erf((x-0.01)/(0.01*np.sqrt(2))))
# Prepare setup such that every code snippet uses the same number of steps
# and the same random numbers
nSteps = 600
nIterations = 1
rnd = np.random.rand(nSteps)
# Your code - Gives the (supposedly) correct results
portfolio_final = []
for i in range(nIterations):
portfolio_value = [100]
rand_values = rnd
interpolated_returns = np.interp(rand_values,cdf_values,x)
interpolated_returns = np.add(interpolated_returns,1)
for j in range(1,len(interpolated_returns)+1):
portfolio_value.append(interpolated_returns[j-1]*portfolio_value[j-1])
portfolio_value[j] = portfolio_value[j]-1
portfolio_final.append(portfolio_value[-1])
print (np.mean(portfolio_final))
# Using vectors
portfolio_final = []
for i in range(nIterations):
portfolio_values = np.ones(nSteps)*100.0
rcp = np.cumprod(np.interp(rnd,cdf_values,x) + 1)
portfolio_values = rcp * (portfolio_values - np.cumsum(1.0/rcp))
portfolio_final.append(portfolio_values[-1])
print (np.mean(portfolio_final))
# Simple loop
portfolio_final = []
for i in range(nIterations):
pv = 100
rets = np.interp(rnd,cdf_values,x) + 1
for i in range(nSteps):
pv = pv * rets[i] - 1
portfolio_final.append(pv)
print (np.mean(portfolio_final))
Forget about np.nditer. It does not improve the speed of iterations. Only use if you intend to go one and use the C version (via cython).
I'm puzzled about that inner loop. What is it supposed to be doing special? Why the loop?
In tests with simulated values these 2 blocks of code produce the same thing:
interpolated_returns = np.add(interpolated_returns,1)
for j in range(1,len(interpolated_returns)+1):
portfolio_value.append(interpolated_returns[j-1]*portfolio[j-1])
portfolio_value[j] = portfolio_value[j]-1
interpolated_returns = (interpolated_returns+1)*portfolio - 1
portfolio_value = portfolio_value + interpolated_returns.tolist()
I assuming that interpolated_returns and portfolio are 1d arrays of the same length.
I am trying to speed up my code which currently takes a little over an hour to run in Python / Numpy. The majority of computation time occurs in the function pasted below.
I'm trying to vectorize Z, but I'm finding it rather difficult for a triple for loop. Could I possible implement the numpy.diff function somewhere? Take a look:
def MyFESolver(KK,D,r,Z):
global tdim
global xdim
global q1
global q2
for k in range(1,tdim):
for i in range(1,xdim-1):
for j in range (1,xdim-1):
Z[k,i,j]=Z[k-1,i,j]+r*q1*Z[k-1,i,j]*(KK-Z[k-1,i,j])+D*q2*(Z[k-1,i-1,j]-4*Z[k-1,i,j]+Z[k-1,i+1,j]+Z[k-1,i,j-1]+Z[k-1,i,j+1])
return Z
tdim = 75 xdim = 25
I agree, it's tricky because the BCs on all four sides, ruin the simple structure of the Stiffness matrix. You can get rid of the space loops as such:
from pylab import *
from scipy.sparse.lil import lil_matrix
tdim = 3; xdim = 4; r = 1.0; q1, q2 = .05, .05; KK= 1.0; D = .5 #random values
Z = ones((tdim, xdim, xdim))
#Iterate in time
for k in range(1,tdim):
Z_prev = Z[k-1,:,:] #may need to flatten
Z_up = Z_prev[1:-1,2:]
Z_down = Z_prev[1:-1,:-2]
Z_left = Z_prev[:-2,1:-1]
Z_right = Z_prev[2:,1:-1]
centre_term = (q1*r*(Z_prev[1:-1,1:-1] + KK) - 4*D*q2)* Z_prev[1:-1,1:-1]
Z[k,1:-1,1:-1]= Z_prev[1:-1,1:-1]+ centre_term + q2*(Z_up+Z_left+Z_right+Z_down)
But I don't think you can get rid of the time loop...
I think the expression:
Z_up = Z_prev[1:-1,2:]
makes a copy in numpy, whereas what you want is a view - if you can figure out how to do this - it should be even faster (how much?)
Finally, I agree with the rest of the answerers - from experience, this kind of loops are better done in C and then wrapped into numpy. But the above should be faster than the original...
This looks like an ideal case for Cython. I'd suggest writing that function in Cython, it'll probably be hundreds of times faster.
I would like to know how I can round a number in numpy to an upper or lower threshold which is function of predefined step size. Hopefully stated in a clearer way, if I have the number 123 and a step size equal to 50, I need to round 123 to the closest of either 150 or 100, in this case 100. I came out with function below which does the work but I wonder if there is a better, more succint, way to do this.
Thanks in advance,
Paolo
def getRoundedThresholdv1(a, MinClip):
import numpy as np
import math
digits = int(math.log10(MinClip))+1
b = np.round(a, -digits)
if b > a: # rounded-up
c = b - MinClip
UpLow = np.array((b,c))
else: # rounded-down
c = b + MinClip
UpLow = np.array((c,b))
AbsDelta = np.abs(a - UpLow)
return UpLow[AbsDelta.argmin()]
getRoundedThresholdv1(143, 50)
The solution by pb360 is much better, using the second argument of builtin round in python3.
I think you don't need numpy:
def getRoundedThresholdv1(a, MinClip):
return round(float(a) / MinClip) * MinClip
here a is a single number, if you want to vectorize this function you only need to replace round with np.round and float(a) with np.array(a, dtype=float)
Summary: This is a correct way to do it, the top answer has cases that do not work:
def round_step_size(quantity: Union[float, Decimal], step_size: Union[float, Decimal]) -> float:
"""Rounds a given quantity to a specific step size
:param quantity: required
:param step_size: required
:return: decimal
"""
precision: int = int(round(-math.log(step_size, 10), 0))
return float(round(quantity, precision))
My reputation is too low to post a comment on the top answer from Ruggero Turra and point out the issue. However it has cases which did not work for example:
def getRoundedThresholdv1(a, MinClip):
return round(float(a) / MinClip) * MinClip
getRoundedThresholdv1(quantity=13.200000000000001, step_size=0.0001)
Returns 13.200000000000001 right back whether using numpy or the standard library round. I didn't even find this by stress testing the function. It just came up when using it in production code and spat an error.
Note full credit for this answer comes out of an open source github repo which is not mine found here
Note that round() in Ruggero Turra his answer rounds to the nearest even integer. Meaning:
a= 0.5
round(a)
Out: 0
Which may not be what you expect.
In case you want 'classical' rounding, you can use this function, which supports both scalars and Numpy arrays:
import Numpy as np
def getRoundedThresholdv1(a, MinClip):
scaled = a/MinClip
return np.where(scaled % 1 >= 0.5, np.ceil(scaled), np.floor(scaled))*MinClip
Alternatively, you could use Numpy's method digitize. It requires you to define the array of your steps. digitize will kind of ceil your value to the next step. So in order to round in a 'classical' way we need an intermediate step.
You can use this:
import Numpy as np
def getRoundedThresholdv1(a, MinClipBins):
intermediate = (MinClipBins[1:] + MinClipBins[:-1])/2
return MinClipBins[np.discritize(a, intermediate)]
You can then call it like:
bins = np.array([0, 50, 100, 150])
test1 = getRoundedThresholdv1(74, bins)
test2 = getRoundedThresholdv1(125, bins)
Which gives:
test1 = 50
test2 = 150