I am using PyMC3 to calculate something which I won't get into here but you can get the idea from this link if interested.
The '2-lambdas' case is basically a switch function, which needs to be compiled to a Theano function to avoid dtype errors and looks like this:
import theano
from theano.tensor import lscalar, dscalar, lvector, dvector, argsort
#theano.compile.ops.as_op(itypes=[lscalar, dscalar, dscalar], otypes=[dvector])
def lambda_2_distributions(tau, lambda_1, lambda_2):
"""
Return values of `lambda_` for each observation based on the
transition value `tau`.
"""
out = zeros(num_observations)
out[: tau] = lambda_1 # lambda before tau is lambda1
out[tau:] = lambda_2 # lambda after (and including) tau is lambda2
return out
I am trying to generalize this to apply to 'n-lambdas', where taus.shape[0] = lambdas.shape[0] - 1, but I can only come up with this horribly slow numpy implementation.
#theano.compile.ops.as_op(itypes=[lvector, dvector], otypes=[dvector])
def lambda_n_distributions(taus, lambdas):
out = zeros(num_observations)
np_tau_indices = argsort(taus).eval()
num_taus = taus.shape[0]
for t in range(num_taus):
if t == 0:
out[: taus[np_tau_indices[t]]] = lambdas[t]
elif t == num_taus - 1:
out[taus[np_tau_indices[t]]:] = lambdas[t + 1]
else:
out[taus[np_tau_indices[t]]: taus[np_tau_indices[t + 1]]] = lambdas[t]
return out
Any ideas on how to speed this up using pure Theano (avoiding the call to .eval())? It's been a few years since I've used it and so don't know the right approach.
Using a switch function is not recommended, as it breaks the nice geometry of the parameters space and makes sampling using modern sampler like NUTS difficult.
Instead, you can try model it using a continuous relaxation of a switch function. The main idea here would be to model the rate before the first switch point as a baseline; and add the prediction from a logistic function after each switch point:
def logistic(L, x0, k=500, t=np.linspace(0., 1., 1000)):
return L/(1+tt.exp(-k*(t_-x0)))
with pm.Model() as m2:
lambda0 = pm.Normal('lambda0', mu, sd=sd)
lambdad = pm.Normal('lambdad', 0, sd=sd, shape=nbreak-1)
trafo = Composed(pm.distributions.transforms.LogOdds(), Ordered())
b = pm.Beta('b', 1., 1., shape=nbreak-1, transform=trafo,
testval=[0.3, 0.5])
theta_ = pm.Deterministic('theta', tt.exp(lambda0 +
logistic(lambdad[0], b[0]) +
logistic(lambdad[1], b[1])))
obs = pm.Poisson('obs', theta_, observed=y)
trace = pm.sample(1000, tune=1000)
There are a few tricks I used here as well, for example, the composite transformation that is not on the PyMC3 code base yet. You can have a look at the full code here: https://gist.github.com/junpenglao/f7098c8e0d6eadc61b3e1bc8525dd90d
If you have more question, please post to https://discourse.pymc.io with your model and (simulated) data. I check and answer on the PyMC3 discourse much more regularly.
Related
I am trying to implement a Expectation Maximization algorithm using TensorFlow and TensorFlow Probability. It worked very well until I tried to implement Missing Data (data can contain NaN values in some random dimensions).
The problem is that with Missing Data I can no longer do all the operations as vector operations, I have to work with indexing and for-loops, like this:
# Here we iterate through all the data samples
for i in range(n):
# x_i is the sample i
x_i = tf.expand_dims(x[:, i], 1)
gamma.append(estimate_gamma(x_i, pi, norm, ber))
est_x_n_i = []
est_xx_n_i = []
est_x_b_i = []
for j in range(k):
mu_k = norm.mean()[j, :]
sigma_k = norm.covariance()[j, :, :]
rho_k = ber.mean()[j, :]
est_x_n_i.append(estimate_x_norm(x_i[:d, :], mu_k, sigma_k))
est_xx_n_i.append(estimate_xx_norm(x_i[:d, :], mu_k, sigma_k))
est_x_b_i.append(estimate_x_ber(x_i[d:, :], rho_k))
est_x_n.append(tf.convert_to_tensor(est_x_n_i))
est_xx_n.append(tf.convert_to_tensor(est_xx_n_i))
est_x_b.append(tf.convert_to_tensor(est_x_b_i))
What I found out was that these operations are not very efficient. While the first samples took about less than 1 second per sample, after 50 samples it took about 3 seconds per sample. I guess that this was happening because I was creating different tensors inside the session and that was messing up the memory or something.
I am quite new using TensorFlow and a lot of people only use TensorFlow for Deep Learning and Neural Networks so I couldn't find a solution for this.
Then I tried to implement the previous for-loop and the functions called inside that loop using only numpy arrays and numpy operations. But this returned the following error:
You must feed a value for placeholder tensor 'Placeholder_4' with
dtype double and shape [8,18]
This error happens because when it tries to execute the numpy functions inside the loop, the placeholder has not been fed yet.
pi_k, mu_k, sigma_k, rho_k, gamma_ik, exp_loglik = exp_max_iter(x, pi, dist_norm, dist_ber)
pi, mu, sigma, rho, responsability, NLL[i + 1] = sess.run([pi_k, mu_k, sigma_k, rho_k, gamma_ik, exp_loglik],{x: samples})
Is there any way to solve this? Thanks.
To answer your title question "Is there a way to call a Numpy function inside a TensorFlow session?", I've put in place below some sample code to execute a "numpy function" (sklearn.mixture.GaussianMixture) given missing data by directly calling the function or via Tensorflow's py_function. I am sensing this may not 100% be what you are looking for... in the case that you are just trying to implement EM..? the existing implementation of Gaussian Mixture Model in Tensorflow may be of some help:
documentation on tf.contrib.factorization.gmm:
https://www.tensorflow.org/api_docs/python/tf/contrib/factorization/gmm
implementation:
https://github.com/tensorflow/tensorflow/blob/r1.14/tensorflow/contrib/factorization/python/ops/gmm_ops.py#L462-L506
Sample code to call a 'numpy function' directly and within Tensorflow graph:
import numpy as np
np.set_printoptions(2)
import tensorflow as tf
from sklearn.mixture import GaussianMixture as GMM
def myfunc(x,istf=True):
#strip nans
if istf:
mask = ~tf.is_nan(x)
x = tf.boolean_mask(x,mask)
else:
ind=np.where(~np.isnan(x))
x = x[ind]
x = np.expand_dims(x,axis=-1)
gmm = GMM(n_components=2)
gmm.fit(x)
m0,m1 = gmm.means_[:,0]
return np.array([m0,m1])
# create data with nans
np.random.seed(42)
x = np.random.rand(5,28,1)
c = 5
x.ravel()[np.random.choice(x.size, c, replace=False)] = np.nan
# directly call "numpy function"
for ind in range(x.shape[0]):
val = myfunc(x[ind,:],istf=False)
print(val)
[0.7 0.26]
[0.15 0.72]
[0.77 0.2 ]
[0.65 0.23]
[0.35 0.87]
# initialization
sess = tf.InteractiveSession()
tf.global_variables_initializer().run()
# create graph
X = tf.placeholder(tf.float32, [28,1])
Y = tf.py_function(myfunc,[X],[tf.float32],name='myfunc')
# call "numpy function" in tensorflow graph
for ind in range(x.shape[0]):
val = sess.run(Y, feed_dict={X: x[ind,:],})
print(val)
[array([0.29, 0.76], dtype=float32)]
[array([0.72, 0.15], dtype=float32)]
[array([0.77, 0.2 ], dtype=float32)]
[array([0.23, 0.65], dtype=float32)]
[array([0.35, 0.87], dtype=float32)]
You can convert your numpy function into tensorflow function then it might not create problem when calling inside a session a simple function is following. Make an IOU function in numpy and then call it via tf.numpy_functionhere
def IOU(Pred, GT, NumClasses, ClassNames):
ClassIOU=np.zeros(NumClasses)#Vector that Contain IOU per class
ClassWeight=np.zeros(NumClasses)#Vector that Contain Number of pixel per class Predicted U Ground true (Union for this class)
for i in range(NumClasses): # Go over all classes
Intersection=np.float32(np.sum((Pred==GT)*(GT==i)))# Calculate class intersection
Union=np.sum(GT==i)+np.sum(Pred==i)-Intersection # Calculate class Union
if Union>0:
ClassIOU[i]=Intersection/Union# Calculate intesection over union
ClassWeight[i]=Union
# b/c we will only take the mean over classes that are actually present in the GT
present_classes = np.unique(GT)
mean_IOU = np.mean(ClassIOU[present_classes])
# append it in final results
ClassNames = np.append(ClassNames, 'Mean')
ClassIOU = np.append(ClassIOU, mean_IOU)
ClassWeight = np.append(ClassWeight, np.sum(ClassWeight))
return mean_IOU
# an now call as
NumClasses=6
ClassNames=['Background', 'Class_1', 'Class_1',
'Class_1 ', 'Class_1', 'Class_1 ']
x = tf.numpy_function(IOU, [y_pred, y_true, NumClasses, ClassNames],
tf.float64, name=None)
I am trying to define a piecewise function to be fitted by lmfit library in Python. The issue I am having is a parameter I have defined for the function will not evaluate alongside the data I am submitting.
I have one example of a case somewhat similar to mine here. However, the vectorize function the answer describes wasn't producing values I wanted, and when reading the documentation, it didn't seem to be the answer to my solution. I also used scipy.optimize.leastsq, but I got the same issue with lmfit described below.
I have a my residual function defined such as
from lmfit import minimize, Parameters, Model
def residual(params, y, x):
param1 = params['one']
param2 = params['two']
if(param2 < x):
p = 1
else:
p = param1*x + param2
return p - y
params = Parameters()
params.add('one', value=1)
params.add('two', value=2)
out = minimize(residual, params,args=(y,x))
I also tried defining the function such that
def f(param1,param2,x):
if(param2 < x):
p = 1
else:
p = param1*x + param2
return p
def residual(params, y, x):
param1 = params['one']
param2 = params['two']
return f(param1,param2,x) - y
I have also tried inline using a lambda function.
I am getting an error 'The truth value of an array with more than one element is ambiguous.' When I got the error, it made sense why it happened, because (param2 < x) would produce a logical array. However, I can't seem to find a way to define the function in a piecewise fashion with the given case to get it fitted with the lmfit.minimize() function. I have seen the answer done in Matlab, in which it's nlinfit function seems to evaluate the data element-wise without issue (I tried searching if Python has an equivalent operation to define element-wise computation such as .* or .+, but that doesn't seem to exist as explicitly).
lmfit also seems to operate a bit differently compared to nlinfit, because we have to always have our residuals return (model - y) while nlinfit outputs the result once the function is given, which I am not sure could be another issue.
So to reiterate, my main question is if there is a method of defining the piecewise function such that it can compare the parameter to the data set.
Any help or explanation would be appreciated, thank you!
In place of (param2 < x) (where param2 is a float and x is an numpy array), you want to use numpy.where. You might try:
def residual(params, y, x):
param1 = params['one']
param2 = params['two']
p = param1 * x + param2
p[np.where(param2 < x)] = 1.0
return p - y
I should also warn you about a potential problem with this approach to having a variable be a boundary for a piecewise function.
In non-linear fits, variables are always floating point (continuous, non-discrete) values. As the fit proceeds, it will make small adjustments in the values and see how that small change alters the result. In your approach, the parameter 'two' is used as both the transition between pieces and the offset for the line -- that is good.
If a parameter is used only as the transition, it may not work. Consider, say, x=np.array([0, 1., 2., 3., 4., ..., 20.0]). Having two = 10.5 and two=10.4 would then give the same result. In that case, the fit would not be able to alter the value of two: it would try a very small change, see no change in the result and give up.
So, either make sure that two is also used elsewhere in your real model (assuming your real model is more complicated than the example given), or consider using a more gentle transition rather than a hard change in pieces. I find an error-function of width ~spacing between x points often works. Depending on the nature of your problem, you might try something like this:
from scipy.special import erf, erfc
def residual(params, y, x):
param1 = params['one']
param2 = params['two']
dx = (max(x) - min(x))/(len(x)-1)
xhi = (erf((x-param2)/dx) + 1)/2.0
xlo = (erfc((x-param2)/dx) + 1)/2.0
p = xlo*1.0 + xhi*(param1*x + param2)
# note: did you really want?
# p = xlo*param + xhi*(param1*x + param2)
# p = param2 + xhi*param1*x
return p - y
Hope that helps.
I am trying to create a fairly generic genetic algorithm implementation I'm TensorFlow. I have an implementation that is slow and am trying to increase its speed. I will provide a really simple example of where the program is getting slow and would welcome recommendations of improving the speed of the current implementation.
Let us say that we create the following:
W = tf.Variable(tf.convert_to_tensor(Warr, dtype=tf.float32))
X = tf.placeholder(dtype=tf.float32, shape=(3, None))
y = tf.placeholder(dtype=tf.float32, shape=(None) )
And we want to find W for the condition:
Warr = np.array([[0.1, 0, 0]])
Xarr = np.random.random((3, 100))
yarr = np.dot(Warr, Xarr)
A naive implementation (like the one I have created) goes thus:
1 a cost function is created for this implementation:
yHat = tf.matmul(W, X)
costFunction = tf.reduce_mean( tf.sqrt((y - yHat)*(y - yHat)) )
Note that the cost function can be arbitrarily complex and is not known apriori. Hence, it is something that will be passed into a class. Note that the rest of the code are excerpts within a class, but the main idea is easy to follow:
2 A population is generated (within a class).
self.population = []
for i in tqdm(range(self.GAconfig['numChildren'])):
temp = []
for j, v in enumerate(locVars):
v = (v + (np.random.random( v.shape ) - 0.5) * 2)
v = tf.Variable(tf.convert_to_tensor(v, dtype=tf.float32))
temp.append(v)
self.population.append( temp )
Finding the cost function for this population is a rather arduous task. First copy the weights in the population into the original weight tensor and then calculate the original cost function:
for ps in self.population:
for i, v in enumerate(self.variables):
sess.run(tf.assign( self.variables[i], ps[i] ))
result = sess.run(self.costFunction, feed_dict={
self.X : X, self.y : y
})
This implementation is obviously slow. One possible way would be to to generate a set of cost function tensors rather than weight variables, which can all be updated at once.
However, this is the point at which I am not sure what a "good implementation" would be that can improve the speed of the current implementation. Ant help will be greatly appreciated ...
Note: The full implementation is available here:
https://github.com/sankhaMukherjee/tfNNGA
It is at its very early stages, so the code at the moment is very bad.
The implementation of the GA function can be found in the file src/lib/GA/GA.py
Crossover is found within the GA class
This is called from within the file src/moduleGA/moduleGA.py
Have you tried next functions?
copy_variable_to_graph
copy_op_to_graph
get_copied_op
source
Very simple regression task. I have three variables x1, x2, x3 with some random noise. And I know target equation: y = q1*x1 + q2*x2 + q3*x3. Now I want to find target coefs: q1, q2, q3 evaluate the
performance using the mean Relative Squared Error (RSE) (Prediction/Real - 1)^2 to evaluate the performance of our prediction methods.
In the research, I see that this is ordinary Least Squares Problem. But I can't get from examples on the internet how to solve this particular problem in Python. Let say I have data:
import numpy as np
sourceData = np.random.rand(1000, 3)
koefs = np.array([1, 2, 3])
target = np.dot(sourceData, koefs)
(In real life that data are noisy, with not normal distribution.) How to find this koefs using Least Squares approach in python? Any lib usage.
#ayhan made a valuable comment.
And there is a problem with your code: Actually there is no noise in the data you collect. The input data is noisy, but after the multiplication, you don't add any additional noise.
I've added some noise to your measurements and used the least squares formula to fit the parameters, here's my code:
data = np.random.rand(1000,3)
true_theta = np.array([1,2,3])
true_measurements = np.dot(data, true_theta)
noise = np.random.rand(1000) * 1
noisy_measurements = true_measurements + noise
estimated_theta = np.linalg.inv(data.T # data) # data.T # noisy_measurements
The estimated_theta will be close to true_theta. If you don't add noise to the measurements, they will be equal.
I've used the python3 matrix multiplication syntax.
You could use np.dot instead of #
That makes the code longer, so I've split the formula:
MTM_inv = np.linalg.inv(np.dot(data.T, data))
MTy = np.dot(data.T, noisy_measurements)
estimated_theta = np.dot(MTM_inv, MTy)
You can read up on least squares here: https://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)#The_general_problem
UPDATE:
Or you could just use the builtin least squares function:
np.linalg.lstsq(data, noisy_measurements)
In addition to the #lhk answer I have found great scipy Least Squares function. It is easy to get the requested behavior with it.
This way we can provide a custom function that returns residuals and form Relative Squared Error instead of absolute squared difference:
import numpy as np
from scipy.optimize import least_squares
data = np.random.rand(1000,3)
true_theta = np.array([1,2,3])
true_measurements = np.dot(data, true_theta)
noise = np.random.rand(1000) * 1
noisy_measurements = true_measurements + noise
#noisy_measurements[-1] = data[-1] # (1000 * true_theta) - uncoment this outliner to see how much Relative Squared Error esimator works better then default abs diff for this case.
def my_func(params, x, y):
res = (x # params) / y - 1 # If we change this line to: (x # params) - y - we will got the same result as np.linalg.lstsq
return res
res = least_squares(my_func, x0, args=(data, noisy_measurements) )
estimated_theta = res.x
Also, we can provide custom loss with loss argument function that will process the residuals and form final loss.
I'm a bit new to Python and PyMC, and making rapid progress. But I'm just confused about the use of setting deterministic values of a 2D matrix. I have a model below, that I cannot get to parse correctly. The problem relates to setting the value theta in the model.
import numpy as np
import pymc
define known variables
N = 2
T = 10
tau = 1
define model... which I cannot get to parse correctly. It's the allocation of theta that I'm having trouble with. The aim to to get samples of D and x. Theta is just an intermediate variable, but I need to keep it as it's used in more complex variations of the model.
def NAFCgenerator():
D = np.empty(T, dtype=object)
theta = np.empty([N,T], dtype=object)
x = np.empty([N,T], dtype=object)
# true location of signal
for t in range(T):
D[t] = pymc.DiscreteUniform('D_%i' % t, lower=0, upper=N-1)
for t in range(T):
for n in range(N):
#pymc.deterministic(plot=False)
def temp_theta(dt=D[t], n=n):
return dt==n
theta[n,t] = temp_theta
x[n,t] = pymc.Normal('x_%i,%i' % (n,t),
mu=theta[n,t], tau=tau)
return locals()
** EDIT **
Explicit indexing is useful for me as I'm learning both PyMC and Python. But it seems that extracting MCMC samples is a bit clunky, e.g.
D0values = pymc_generator.trace('D_0')[:]
But I am probably missing something. But did I managed to get a vectorised version working
# Approach 1b - actually quite promising
def NAFCgenerator():
# NOTE TO SELF. It's important to declare these as objects
D = np.empty(T, dtype=object)
theta = np.empty([N,T], dtype=object)
x = np.empty([N,T], dtype=object)
# true location of signal
D = pymc.Categorical('D', spatial_prior, size=T)
# displayed stimuli
#pymc.deterministic(plot=False)
def theta(D=D):
theta = np.zeros([N,T])
theta[0,D==0]=1
theta[1,D==1]=1
return theta
#for n in range(N):
x = pymc.Normal('x', mu=theta, tau=tau)
return locals()
Which seems easier to get at MCMC samples using this for example
Dvalues = pymc_generator.trace('D')[:]
In PyMC2, when creating deterministic nodes with decorators, the default is to take the node name from the function name. The solution is simple: specify the node name as a parameter for the decorator.
#pymc.deterministic(name='temp_theta_%d_%d'%(t,n), plot=False)
def temp_theta(dt=D[t], n=n):
return dt==n
theta[n,t] = temp_theta
Here is a notebook that puts this in context.