I need to connect two functions: first I define f(x), then I define g(t), and finally I need to set that x = g(t).
MWE
import tensorflow as tf
import numpy as np
session = tf.Session()
# f(x)
x = tf.placeholder(shape=[None,4], dtype=tf.float64)
f = tf.square(x)
# g(t)
t = tf.placeholder(shape=[None,2], dtype=tf.float64)
g = tf.matmul(t,np.ones((2,4)))
session.run(tf.global_variables_initializer())
t_eval = np.asmatrix(np.random.rand(10,2))
# f(g(t))
session.run(f, {x: session.run(g, {t: t_eval})})
# but I want to do (does not work because I need to assign a value to placeholder x)
x = g
session.run(f, {t: t_eval})
Basically I want to replace placeholder x with tensor g. How can I do that?
You don't need x in your code at all.
f = tf.square(g) # replace your definition of f with this
session.run(f, {t: t_eval})
Related
I have a #tf.function decorated function. Inside the function, I would like to draw from a distribution and compute some value several times (let's call this proceduce f(x)).
How can I do this in Tensorflow 2.0? I can't use numpy arrays as I would like to use the #tf.function decorator.
A numpy implementation would look like:
reps = 4
store = np.zeros((n, reps))
for i in range(reps):
store[:, i] = f(x) #f(x) is shape (n,)
The goal would then be to compute the row means of store.
This should be easy but I haven't been able to work out how to do it!
Something like this maybe:
import tensorflow as tf
def f():
return tf.random.normal((10,))
#tf.function
def store_this():
reps = 4
n = 10
store = tf.zeros((n, reps))
values = [f() for _ in range(reps)]
indices = tf.stack([tf.tile(tf.range(n), multiples=[reps]), tf.repeat(tf.range(reps), repeats=n)], axis=-1)
return tf.tensor_scatter_nd_update(store, indices, tf.reshape(values, [-1]))
store_this()
If f takes a one-dimensional tensor as input, this is a shorter alternative:
#tf.function
def f(x):
return tf.random.normal((10,))
x = tf.constant([1.0, 2.0])
reps = 4
def store_this(fp, x, reps):
return tf.transpose(tf.map_fn(fp, tf.tile(tf.expand_dims(x, 0),[reps,1])))
store_this(f, x, reps)
In PyTorch, there are two ways of calculating second gradients. The first method is to use torch.autograd.grad function, and the other is to use backward function. I use the following examples to illustrate it:
Method 1:
x=torch.tensor([3.0], requires_grad=True)
y = torch.pow(x, 2)
grad_1 = torch.autograd.grad(y, x, create_graph=True)
print(grad_1[0].item())
grad_2 = torch.autograd.grad(grad_1[0], x)
print(grad_2)
The result makes sense for me, and the second gradient of the function is 2.
Method 2:
x=torch.tensor([3.0], requires_grad=True)
y = torch.pow(x, 2) # y=x**2
y.backward(retain_graph=True)
print(x.grad)
y.backward()
print(x.grad)
When calculating the first gradient, I use create_graph=True to make sure that we can use back prorogation method to calculate the second gradient. However, the result is is 12, which is wrong. I was wondering what's wrong with the second method?
Use the grad method from torch.autograd to differentiate your function. So the steps would be:
>>> import torch
>>> from torch.autograd import grad
>>> x = torch.tensor([3.0], requires_grad=True)
>>> y = torch.pow(x,2)
>>> z = grad(y, x, create_graph=True)
>>> print(grad(z, x, create_graph=True))
>>> (tensor([2.], grad_fn=<MulBackward0>),)
Similarly, you can loop through to make the nth derivative.
I am trying to use scipy.odeint() method in order to solve an second order partial derivative function.
I can do that for a single value of constant k, which is a constant of the function I have.
But I want to try this solution for many values of k.
To do so, I included the values that I want in a list k, and going through a loop I want to plug in these values for the final solution as arguments.
However, I am getting an error
error: Extra arguments must be in a tuple
import numpy as np
from scipy.integrate import odeint
### Code with a single value of K.THAT WORKS FINE!!!! ###
k = 1 #attributes to be changed
t = [0.1,0.2,0.3] #Data
init = [45,0] #initial values
#Function to apply an integration
def f(init, t, args=(k,)):
dOdt = init[1]
dwdt = -np.cos(init[0]) + k*dOdt
return [dOdt, dwdt]
#integrating function that returns a list of 2D numpy arrays
zCH = odeint(f,init,t)
################################################################
### Code that DOES NOT WORK!###
k = [1,2,3] #attributes to be changed
t = [0.1,0.2,0.3] #Data
init = [45,0] #initial values
#Function to apply an integration
def f(init, t, args=(k,)):
dOdt = init[1]
dwdt = -np.cos(init[0]) + k*dOdt
return [dOdt, dwdt]
solutions = []
for i in k:
#integrating function that returns a list of 2D numpy arrays
zCH = odeint(f,init,t,(k[i-1]))
solutions.append(zCH)```
It has to do with the way you are passing k into your function f().
The following changes the value of k on each iteration
k_list = [1,2,3] #attributes to be changed
t = [0.1,0.2,0.3] #Data
init = [45,0] #initial values
#Function to apply an integration
def f(init, t, args=(k,)):
dOdt = init[1]
dwdt = -np.cos(init[0]) + k*dOdt
return [dOdt, dwdt]
solutions = []
for k in k_list:
#integrating function that returns a list of 2D numpy arrays
zCH = odeint(f, init, t)
solutions.append(zCH)
I try to implement Polynomial Regression with Gradient Descent. I want to fit the following function:
The code I use is:
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg
from sklearn.preprocessing import PolynomialFeatures
np.random.seed(seed=42)
def create_data():
x = PolynomialFeatures(degree=5).fit_transform(np.linspace(-10,10,100).reshape(100,-1))
l = lambda x_i: (1/3)*x_i**3-2*x_i**2+2*x_i+2
data = l(x[:,1])
noise = np.random.normal(0,0.1,size=np.shape(data))
y = data+noise
y= y.reshape(100,1)
return {'x':x,'y':y}
def plot_function(x,y):
fig = plt.figure(figsize=(10,10))
plt.plot(x[:,1],[(1/3)*x_i**3-2*x_i**2+2*x_i+2 for x_i in x[:,1]],c='lightgreen',linewidth=3,zorder=0)
plt.scatter(x[:,1],y)
plt.show()
def w_update(y,x,batch,w_old,eta):
derivative = np.sum([(y[i]-np.dot(w_old.T,x[i,:]))*x[i,:] for i in range(np.shape(x)[0])])
print(derivative)
return w_old+eta*(1/batch)*derivative
# initialize variables
w = np.random.normal(size=(6,1))
data = create_data()
x = data['x']
y = data['y']
plot_function(x,y)
# Update w
w_s = []
Error = []
for i in range(500):
error = (1/2)*np.sum([(y[i]-np.dot(w.T,x[i,:]))**2 for i in range(len(x))])
Error.append(error)
w_prime = w_update(y,x,np.shape(x)[0],w,0.001)
w = w_prime
w_s.append(w)
# Plot the predicted function
plt.plot(x[:,1],np.dot(x,w))
plt.show()
# Plot the error
fig3 = plt.figure()
plt.scatter(range(len(Error[10:])),Error[10:])
plt.show()
But as result I receive smth. strange which is completely out of bounds...I have also tried to alter the number of iterations as well as the parameter theta but it did not help. I assume I have made an mistake in the update of w.
I have found the solution. The Problem is indeed in the part where I calculate the weights. Specifically in:
np.sum([(y[d]-np.dot(w_old.T,x[d,:]))*x[d,:] for d in range(np.shape(x)[0])])
which should be like:
np.sum([-(y[d]-np.dot(w.T.copy(),x[d,:]))*x[d,:].reshape(np.shape(w)) for d in range(len(x))],axis=0)
We have to add np.sum(axis=0) to get the dimensionality we want --> Dimensionality must be equal to w. The numpy sum documentation sais
The default, axis=None, will sum all of the elements of the input
array.
This is not what we want to achieve. Adding axis = 0 sums over the first axis of our array which is of dimensionality (100,7,1) hence the 100 elements of dimensionality (7,1) are summed up and the resulting array is of dimensionality (7,1) which is exactly what we want. Implementing this and cleaning up the code yields:
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg
from sklearn.preprocessing import PolynomialFeatures
from sklearn.preprocessing import MinMaxScaler
np.random.seed(seed=42)
def create_data():
x = PolynomialFeatures(degree=6).fit_transform(np.linspace(-2,2,100).reshape(100,-1))
x[:,1:] = MinMaxScaler(feature_range=(-2,2),copy=False).fit_transform(x[:,1:])
l = lambda x_i: np.cos(0.8*np.pi*x_i)
data = l(x[:,1])
noise = np.random.normal(0,0.1,size=np.shape(data))
y = data+noise
y= y.reshape(100,1)
# Normalize Data
return {'x':x,'y':y}
def plot_function(x,y,w,Error,w_s):
fig,ax = plt.subplots(nrows=1,ncols=2,figsize=(40,10))
ax[0].plot(x[:,1],[np.cos(0.8*np.pi*x_i) for x_i in x[:,1]],c='lightgreen',linewidth=3,zorder=0)
ax[0].scatter(x[:,1],y)
ax[0].plot(x[:,1],np.dot(x,w))
ax[0].set_title('Function')
ax[1].scatter(range(iterations),Error)
ax[1].set_title('Error')
plt.show()
# initialize variables
data = create_data()
x = data['x']
y = data['y']
w = np.random.normal(size=(np.shape(x)[1],1))
eta = 0.1
iterations = 10000
batch = 10
def stochastic_gradient_descent(x,y,w,eta):
derivative = -(y-np.dot(w.T,x))*x.reshape(np.shape(w))
return eta*derivative
def batch_gradient_descent(x,y,w,eta):
derivative = np.sum([-(y[d]-np.dot(w.T.copy(),x[d,:]))*x[d,:].reshape(np.shape(w)) for d in range(len(x))],axis=0)
return eta*(1/len(x))*derivative
def mini_batch_gradient_descent(x,y,w,eta,batch):
gradient_sum = np.zeros(shape=np.shape(w))
for b in range(batch):
choice = np.random.choice(list(range(len(x))))
gradient_sum += -(y[choice]-np.dot(w.T,x[choice,:]))*x[choice,:].reshape(np.shape(w))
return eta*(1/batch)*gradient_sum
# Update w
w_s = []
Error = []
for i in range(iterations):
# Calculate error
error = (1/2)*np.sum([(y[i]-np.dot(w.T,x[i,:]))**2 for i in range(len(x))])
Error.append(error)
# Stochastic Gradient Descent
"""
for d in range(len(x)):
w-= stochastic_gradient_descent(x[d,:],y[d],w,eta)
w_s.append(w.copy())
"""
# Minibatch Gradient Descent
"""
w-= mini_batch_gradient_descent(x,y,w,eta,batch)
"""
# Batch Gradient Descent
w -= batch_gradient_descent(x,y,w,eta)
# Show predicted weights
print(w_s)
# Plot the predicted function and the Error
plot_function(x,y,w,Error,w_s)
As result we receive:
Which surely can be improved by altering eta and the number of iterations as well as switching to Stochastic or Mini Batch Gradient Descent or more sophisticated optimization algorithms.
Say I have a certain the Tensor x whose dimensions are not defined upon graph initialization.
I can get its shape using:
x_shape = tf.shape(input=x)
Now if I want to create a variable based on the values defined in x_shape using:
y = tf.get_variable(variable_name="y", shape=[x_shape[0], 10])
I get an error, since the values passed to the argument shape must be int and not Tensor. How can I create such a dynamically shaped variable without using placeholders?
I'm running out of time so this is quick and dirty, but maybe it helps you to get to your solution...
It's based on this (dynamic size for tf.zeros) but extends the idea to tf.Variables. Since your variable needs to be initialized anyway - I choose 0s...
import tensorflow as tf
I1_ph = tf.placeholder(name = "I1",shape=(None,None,None),dtype=tf_dtype)
zerofill = tf.fill(tf.shape(I1_ph), 0.0)
myVar = tf.Variable(0.0)
updateMyVar = tf.assign(myVar,zerofill,validate_shape=False)
res, = sess.run([updateMyVar], { I1_ph:np.zeros((1,2,2)) } )
print ("dynamic variable shape",res.shape)
res, = sess.run([updateMyVar], { I1_ph:np.zeros((3,5,2)) } )
print ("dynamic variable shape",res.shape)
You can use x.get_shape():
y = tf.get_variable('y', shape=[x.get_shape()[0], 10])
import tensorflow as tf
x = tf.zeros(shape=[10,20])
x_shape = x.get_shape().as_list()
y = tf.get_variable(shape=x_shape, name = 'y')
Update
You can't create tf.Variable with unknown size. For example this code is not valid:
y = tf.get_variable(shape=[None, 10], name = 'y')
First argument is variable name.
x = tf.zeros(shape=[10,20])
x_shape = x.shape
variable_name ='y'
y = tf.get_variable(variable_name, shape=[x_shape[0], x_shape[1]])
To the best of my knowledge, you cannot create a Variable with a dynamic shape through the shape argument, instead you have to pass this dynamic shape through the initializer of the tf.Variable.
This should work:
zero_init = tf.fill([x_shape[0], 10], tf.constant(0))
# Initialize
y = tf.get_variable(
"my_var", shape=None, validate_shape=False, initializer=zero_init
)
Note that the shape has to be defined before, or with the first execution of tf.Session.run(...). So if your x is a placeholder, you will need to feed a value for it.