The following code does exactly what I want; however, the for loop is far too slow. On my machine, the wall time for the for loop is 1min 5s. I'm looking for an alternative to the for loop that is much faster.
# Imports
from sympy.solvers.solveset import solveset_real
from sympy import Symbol, Eq
# Define variables
initial_value = 1
rate = Symbol('r')
decay_obs_window = 1480346
target_decay = .15
# Solver to calculate decay rate
decay_rate = solveset_real(Eq((initial_value - rate * decay_obs_window), target_decay), rate).args[0]
# Generate weights
weights = []
for i in range(5723673):
# How to handle data BEYOND decay_obs_window
if i > decay_obs_window and target_decay == 0:
# Record a weight of zero
weights.append(0)
elif i > decay_obs_window and target_decay > 0:
# Record the final target weight
weights.append(decayed_weight)
# How to handle data WITHIN decay_obs_window
else:
# Calculate the new slightly decayed weight
decayed_weight = 1 - (decay_rate * i)
weights.append(decayed_weight)
weights[0:10]
I wrote this list comprehension with the hope of improving the execution time. While it works perfectly, it does not yield any appreciable runtime improvement over the for loop 😞:
weights = [0 if i > decay_obs_window and target_decay == 0 else decayed_weight if i > decay_obs_window and target_decay > 0 else (decayed_weight := 1 - (decay_rate * i)) for i in range(len(weights_df))]
I'm interested in any approaches that would help speed this up. Thank you 🙏!
FINAL SOLUTION:
This was the final solution that I settled on. On my machine, the wall time to execute the entire thing is only 425 ms. It's a slightly modified version of Aaron's proposed solution.
import numpy as np
from sympy.solvers.solveset import solveset_real
from sympy import Symbol, Eq
# Define variables
initial_value = 1
rate = Symbol('r')
decay_obs_window = 1480346
target_decay = .15
# Instantiate weights array
weights = np.zeros(5723673)
# Solver to calculate decay rate
decay_rate = solveset_real(Eq((initial_value - rate * decay_obs_window), target_decay), rate).args[0]
# Fix a bug where numpy doesn't like sympy floats :(
decay_rate = float(decay_rate)
# How to weight observations WITHIN decay_obs_window
weights[:decay_obs_window + 1] = 1 - np.arange(decay_obs_window + 1) * decay_rate
# How to weight observations BEYOND decay_obs_window
weights[decay_obs_window + 1 : 5723673] = target_decay
weights
TLDR; None of the variables you test against in your if statements ever change during the loop, so you can easily kick the conditional logic out of the loop, and decide beforehand. I also am a huge proponent of numpy and vectorization.
Looking at the logic there aren't too many possible outcomes of what weights ends up looking like. As RufusVS mentioned, you can separate out the first section where no additional logic is being calculated. It is also a simple linear function, so why not compute it with numpy which is great for linear algebra:
import numpy as np
weights = np.zeros(5723673)
#Fix a bug where numpy doesn't like sympy floats :(
decay_rate = float(decay_rate)
weights[:decay_obs_window + 1] = 1 - np.arange(decay_obs_window + 1) * decay_rate
Then you can decide what to do with the remaining values based on the value of target_decay outside of any loops because it never changes:
if target_decay == 0:
pass #weights array started out filled with 0's so we don't need to do anything
elif target_decay > 0:
#fill the rest of the array with the last value of the window
weights[decay_obs_window + 1 : 5723673] = weights[decay_obs_window + 1]
pass
else: #target_decay < 0:
#continue calculating the linear function
weights[decay_obs_window + 1 : 5723673] = 1 - np.arange(decay_obs_window + 1, 5723673) * decay_rate
By breaking it into two loops, you eliminate a lot of comparison to the break point:
# Imports
from sympy.solvers.solveset import solveset_real
from sympy import Symbol, Eq
# Define variables
initial_value = 1
rate = Symbol('r')
decay_obs_window = 1480346
target_decay = .15
# Solver to calculate decay rate
decay_rate = solveset_real(Eq((initial_value - rate * decay_obs_window), target_decay), rate).args[0]
# Generate weights
weights = []
for i in range(decay_obs_window+1):
# Calculate the new slightly decayed weight
decayed_weight = 1 - (decay_rate * i)
weights.append(decayed_weight)
for i in range(decay_obs_window+1, 5723673):
# How to handle data BEYOND decay_obs_window
if target_decay == 0:
weights.append(0)
elif target_decay > 0:
# Record the final target weight
weights.append(decayed_weight)
else:
# Calculate the new slightly decayed weight
decayed_weight = 1 - (decay_rate * i)
weights.append(decayed_weight)
weights[0:10]
Modified to include #MarkSouls comment, and my own further observation:
# Imports
from sympy.solvers.solveset import solveset_real
from sympy import Symbol, Eq
# Define variables
initial_value = 1
rate = Symbol('r')
decay_obs_window = 1480346
target_decay = .15
# Solver to calculate decay rate
decay_rate = solveset_real(Eq((initial_value - rate * decay_obs_window), target_decay), rate).args[0]
TOTAL_ENTRIES = 5723673
# Generate weights
weights = [0]* TOTAL_ENTRIES
for i in range(decay_obs_window+1):
# Calculate the new slightly decayed weight
decayed_weight = 1 - (decay_rate * i)
weights[i]=decayed_weight
if target_decay == 0:
pass
elif target_decay > 0:
for i in range(decay_obs_window+1, TOTAL_ENTRIES):
# Record the final target weight
weights[i]=decayed_weight
else:
for i in range(decay_obs_window+1, TOTAL_ENTRIES):
decayed_weight = 1 - (decay_rate * i)
weights[i]=decayed_weight
weights[0:10]
I have what I believe to be an absolutely optimal method here:
# Imports
from sympy.solvers.solveset import solveset_real
from sympy import Symbol, Eq
# Define variables
initial_value = 1
rate = Symbol('r')
decay_obs_window = 1480346
target_decay = .15
# Solver to calculate decay rate
decay_rate = solveset_real(Eq((initial_value - rate * decay_obs_window), target_decay), rate).args[0]
# Generate weights
wLength = 5723673
weights = [1 - (decay_rate * i) for i in range(decay_obs_window + 1)]
extend_length = wLength - decay_obs_window - 1
if target_decay == 0:
weights.extend(0 for _ in range(extend_length))
elif target_decay > 0:
decayed_weight = weights[-1]
weights.extend(decayed_weight for _ in range(extend_length))
This brings all the branching logic out of the loop, so it's only calculated once instead of ~ 1.5 million times.
That said, this still represents nearly no improvement in speed over what you already have. The fact of the matter is that most of your time is spent calculating 1 - (decay_rate * i), and there's nothing you can do in python to speed this up.
If you really need more performance you're probably at the point of needing to figure out how to call a C (or Rust) library.
Numpy is perfect for this. We can use the fromfunction method to create an array. First import the function:
from numpy import fromfunction
Then replace
weights = [1 - (decay_rate * i) for i in range(decay_obs_window + 1)]
with
weights = fromfunction(lambda i: 1 - (decay_rate * i), (decay_obs_window + 1, )).tolist()
This is likely to represent the fastest you can do this in Python.
Related
I have the following code where I have implemented gradient descent for a function using pyTorch. How do I add noise to the code so that it identifies both local minima?
import torch
startVal = -5.0
alpha = 0.001
space = " "
progressionCheck = True
x = torch.tensor(startVal, requires_grad=True)
def function(a):
f = a**4 - a**3 - a**2 + a - 1
return f
for i in range(1000):
function(x).backward()
newVal = x - alpha * (x.grad)
progressionCheck = function(newVal) < function(startVal)
x = newVal.detach().clone().requires_grad_()
print(x)
print("The minimum value occurs at" + space + str(float(x)))
print("The minimum value is" + space + str(function(float(x))))
I assume you intend to disturb the gradients by some noise. To do so, you could specify a distribution e.g. as follows
low, high = -0.1, 0.1
dist = torch.distributions.uniform.Uniform(low, high)
and then sample from it to update the gradients, i.e. adjust
newVal = x - alpha * (x.grad)
to
newVal = x - alpha * (x.grad) * dist.sample([1]).item()
Altneratively, sample the noise in advance
noise = dist.sample([1000])
and then index it
newVal = x - alpha * (x.grad) * noise[i]
However, I doubt this fulfils the purpose and don't see how you could avoid multiple runs coupled with using varying start values (or, less beautiful, very large noise or step size) to find multiple local minima.
In this paper, a very simple model is described to illustrate how the ant colony algorithm works. In short, it assumes two nodes which are connected via two links one of which is shorter. Then, given a pheromone increment and a pheromone evaporation dynamics, one expects that all ants eventually pick the shorter path.
Now, I'm trying to replicate the simulation of this paper corresponding to scenario above whose result should be (more or less) like below.
Here is an implementation of mine (taking the same specification as that of the test above).
import random
import matplotlib.pyplot as plt
N = 10
l1 = 1
l2 = 2
ru = 0.5
Q = 1
tau1 = 0.5
tau2 = 0.5
epochs = 150
success = [0 for x in range(epochs)]
def compute_probability(tau1, tau2):
return tau1/(tau1 + tau2), tau2/(tau1 + tau2)
def select_path(prob1, prob2):
if prob1 > prob2:
return 1
if prob1 < prob2:
return 2
if prob1 == prob2:
return random.choice([1,2])
def update_accumulation(link_id):
global tau1
global tau2
if link_id == 1:
tau1 += Q / l1
return tau1
if link_id == 2:
tau2 += Q / l2
return tau2
def update_evapuration():
global tau1
global tau2
tau1 *= (1-ru)
tau2 *= (1-ru)
return tau1, tau2
def report_results(success):
plt.plot(success)
plt.show()
for epoch in range(epochs-1):
temp = 0
for ant in range(N-1):
prob1, prob2 = compute_probability(tau1, tau2)
selected_path = select_path(prob1,prob2)
if selected_path == 1:
temp += 1
update_accumulation(selected_path)
update_evapuration()
success[epoch] = temp
report_results(success)
However, what I get is fairly weird as below.
It seems that my understanding of how pheromone should be updated is flawed.
So, can one address what I am missing in this implementation?
Three problems in the proposed approach:
As #Mark mentioned in his comment, you need a weighted random choice. Otherwise the proposed approach will likely always pick one of the paths and the plot will result in a straight line as you show above. However, I think this was part of the solution, because even with this, you will likely still get a straight line because of early convergence, which led two problem two.
Ant Colony Optimization is a metaheuristic that needs several (hyper) parameters configured to guide the search for a certain solution (e.g., tau from above or number of ants). Fine tuning this parameters is important because you can converge early on a particular result (which is fine to some extent - if you want to use it as an heuristic). But the purpose of a metaheuristic is to provide you with some middle ground between the exact and heuristic algorithms, which makes the continous exploration/exploitation an important part of its workings. This means the parameters need to be careful optimised for your problem size/type.
Given that the ACO uses a probabilistic approach for guiding the search (and as the plot from the referenced paper is showing), you will need to run the experiment several times and compute some statistic on those numbers. In my case below, I computed the average over 100 samples.
import random
import matplotlib.pyplot as plt
N = 10
l1 = 1.1
l2 = 1.5
ru = 0.05
Q = 1
tau1 = 0.5
tau2 = 0.5
samples = 10
epochs = 150
success = [0 for x in range(epochs)]
def compute_probability(tau1, tau2):
return tau1/(tau1 + tau2), tau2/(tau1 + tau2)
def weighted_random_choice(choices):
max = sum(choices.values())
pick = random.uniform(0, max)
current = 0
for key, value in choices.items():
current += value
if current > pick:
return key
def select_path(prob1, prob2):
choices = {1: prob1, 2: prob2}
return weighted_random_choice(choices)
def update_accumulation(link_id):
global tau1
global tau2
if link_id == 1:
tau1 += Q / l1
else:
tau2 += Q / l2
def update_evaporation():
global tau1
global tau2
tau1 *= (1-ru)
tau2 *= (1-ru)
def report_results(success):
plt.ylim(0.0, 1.0)
plt.xlim(0, 150)
plt.plot(success)
plt.show()
for sample in range(samples):
for epoch in range(epochs):
temp = 0
for ant in range(N):
prob1, prob2 = compute_probability(tau1, tau2)
selected_path = select_path(prob1, prob2)
if selected_path == 1:
temp += 1
update_accumulation(selected_path)
update_evaporation()
ratio = ((temp + 0.0) / N)
success[epoch] += ratio
# reset pheromone values here to evaluate new sample
tau1 = 0.5
tau2 = 0.5
success = [x / samples for x in success]
for x in success:
print(x)
report_results(success)
The code above should return something close to the desired plot.
I am trying to show the gray scott model of diffusion. I keep getting a bunch of runtime warning errors even though I feel like my code is really close to correct. Is there something wrong with my discretization?
import numpy as np
import matplotlib.pyplot as plt
#parameters
N=128
F=.042
k=.062
Du=(2**-5)*(N**2/6.25)
Dv=(1**-5)*(N**2/6.25)
tend=1000
dt=tend/N
t=0
#start arrays
U=np.ones((N,N))
V=np.zeros((N,N))
#Initial Value Boxes (20x20 in middle)
low=int(((N/2)-10))
high=int(((N/2)+10))+1
U[low:high,low:high]=.5
V[low:high,low:high]=.25
#Random Noise
U+=.01*np.random.random((N,N))
V+=.01*np.random.random((N,N))
#Laplace
def Laplace(f):
return np.roll(f,1)+np.roll(f,-1)+np.roll(f,1,axis=False)+np.roll(f,-1,axis=False)-4*f
#Solve
pstep=100
for t in range(tend):
U+=((Du*Laplace(U))-(U*V*V)+(F*(1-U)))
V+=((Dv*Laplace(V))+(U*V*V)-(V*(F+k)))
if t%pstep ==0:
print(t//pstep)
plt.imshow(U, interpolation='bicubic',cmap=plt.cm.jet)
Ok, I got it to work by changing a few things in the calculation, but mostly altering the numeric stability by massively decreasing the diffusion coefficients, and decreasing the timestep. The net result of this, is that the whole simulation changes less between each step, so the value of the change is much smaller.
The errors you were getting were due to overflow of the floats in the calculation of dU and dV, by slowing the whole thing down (more timesteps) you don't need such massive numbers in dU and dV
import numpy as np
import matplotlib.pyplot as plt
# parameters
N = 128
F = .042
k = .062
# Du=(2**-5)*(N**2/6.25) # These were way too high for the
# numeric stability given the timestep
Du = 0.1
# Dv=(1**-5)*(N**2/6.25)
Dv = 0.5
tend = 1000
dt = tend / N
t = 0
dt = 0.1 # Timestep -
# the smaller you go here, the faster you can let the diffusion coefficients be
# start arrays
U = np.ones((N, N))
V = np.zeros((N, N))
# Initial Value Boxes (20x20 in middle)
low = int(((N / 2) - 10))
high = int(((N / 2) + 10)) + 1
U[low:high, low:high] = .5
V[low:high, low:high] = .25
# Random Noise
U += .01 * np.random.random((N, N))
V += .01 * np.random.random((N, N))
# Laplace
def Laplace(f):
return np.roll(f, 1) + np.roll(f, -1) + np.roll(f, 1, axis=False) + np.roll(f,-1, axis=False) - 4 * f
# Solve
pstep = 100
for t in range(tend):
dU = ((Du * Laplace(U)) - (U * V * V) + (F * (1 - U))) * dt
dV = ((Dv * Laplace(V)) + (U * V * V) - (V * (F + k))) * dt
U += dU
V += dV
if t % pstep == 0:
print(t // pstep)
plt.imshow(U, interpolation='bicubic', cmap=plt.cm.jet, vmin=0, vmax=1)
Of course the changes I made alter the physics a bit, and you will need to alter your t and pstep so those values make sense. And also check how you were calculating Du and Dv. If those values are actually supposed to be ~8000, then you need a much much smaller timestep.
For anyone else's reference, the Gray Scott model is explained here
I have a given dataset in the matrix y and I want to train different SOMs with it. The SOM is one-dimensional (a line), and its number of neurons varies. I train a SOM of size N=2 at first, and N=NMax at last, giving a total of NMax-2+1 SOMs. For each SOM, I want to store the weights once the training is over before moving on to the next SOM.
The whole point of using PyOpenCL here is that each one of the outer loops is independent of the others. Namely, for each value of N, the script doesn't care about what happens when N takes other values. One could have the same result running the script NMax-2+1 times changing the value of N manually.
With this in mind, I was hoping to be able to perform each one of these independent iterations at the same time using the GPU, so that the time spent reduces significantly. The increase in speed will be less than 1/(NMax-2+1) though, because each iteration is more expensive that the previous ones, as for larger values of N, more calculations are made.
Is there a way to 'translate' this code to run on the GPU? I've never used OpenCL before, so let me know if this is too broad or silly so I can ask a more specific question. The code is self-contained, so feel free to try it out.The four constants declared at the beginning can be changed to whatever you like (given that NMax > 1 and all the others are strictly positive).
import numpy as np
import time
m = 3 # Dimension of datapoints
num_points = 2000 # Number of datapoints
iterMax = 150 # Maximum number of iterations
NMax = 3 # Maximum number of neurons
#%%
np.random.seed(0)
y = np.random.rand(num_points,m) # Generate always the same dataset
sigma_0 = 5 # Initial value of width of the neighborhood function
eta_0 = 1 # Initial value of learning rate
w = list(range(NMax - 1))
wClusters = np.zeros((np.size(y,axis = 0),NMax - 1)) # Clusters for each N
t_begin = time.clock() # Start time
for N in range(NMax-1): # Number of neurons for this iteration
w[N] = np.random.uniform(0,1,(N+2,np.size(y,axis=1))) - 0.5 # Initialize weights
iterCount = 1
while iterCount < iterMax:
# Mix up the input patterns
mixInputs = y[np.random.permutation(np.size(y,axis = 0)),:]
# Sigma reduction
sigma = sigma_0 - (sigma_0/(iterMax + 1)) * iterCount
s2 = 2*sigma**2
# Learning rate reduction
eta = eta_0 - (eta_0/(iterMax + 1)) * iterCount
for selectedInput in mixInputs: # Pick up one pattern
# Search winning neuron
aux = np.sum((selectedInput - w[N])**2, axis = -1)
ii = np.argmin(aux) # Neuron 'ii' is the winner
jjs = abs(ii - list(range(N+2)))
dists = np.min(np.vstack([jjs , abs(jjs-(N+2))]), axis = 0)
# Update weights
w[N] = w[N] + eta * np.exp((-dists**2)/s2).T[:,np.newaxis] * (selectedInput - w[N])
print(N+2,iterCount)
iterCount += 1
# Assign each datapoint to its nearest neuron
for kk in range(np.size(y,axis = 0)):
aux = np.sum((y[kk,] - w[N])**2,axis=-1)
ii = np.argmin(aux) # Neuron 'ii' is the winner
wClusters[kk,N] = ii + 1
t_end = time.clock() # End time
#%%
print(t_end - t_begin)
I'm trying to give a somewhat complete answer.
First of all:
Can this code be adapted to be run on the GPU using (py)OpenCL?
Most probably yes.
Can this been done automatically?
No (afaik).
Most of the questions I get about OpenCL are along the lines of: "Is it worth porting this piece of code to OpenCL for a speedup gain?" You are stating, that your outer loop is independent on the results of other runs, which makes the code basically parallelizable. In a straightforward implementation, each OpenCL working element would execute the same code with slightly different input parameters. Not regarding overhead by data transfer between host and device, the running time of this approach would be equal to the running time of the slowest iteration. Depending on the iterations in your outer loop, this could be a massive speed gain. As long as the numbers stay relatively small, you could try the multiprocessing module in python to parallelize these iterations on the CPU instead of the GPU.
Porting to the GPU usually only makes sense, if a huge number of processes are to be run in parallel (about 1000 or more). So in your case, if you really want an enormous speed boost, see if you can parallelize all calculations inside the loop. For example, you have 150 iterations and 2000 data points. If you could somehow parallelize these 2000 data points, this could offer a much bigger speed gain, which could justify the work of porting the whole code to OpenCL.
TL;DR:
Try parallelizing on CPU first. If you find the need to run more than several 100s of processes at the same time, move to GPU.
Update: Simple code for parallelizing on CPU using multiprocessing (without callback)
import numpy as np
import time
import multiprocessing as mp
m = 3 # Dimension of datapoints
num_points = 2000 # Number of datapoints
iterMax = 150 # Maximum number of iterations
NMax = 10 # Maximum number of neurons
#%%
np.random.seed(0)
y = np.random.rand(num_points,m) # Generate always the same dataset
sigma_0 = 5 # Initial value of width of the neighborhood function
eta_0 = 1 # Initial value of learning rate
w = list(range(NMax - 1))
wClusters = np.zeros((np.size(y,axis = 0),NMax - 1)) # Clusters for each N
def neuron_run(N):
w[N] = np.random.uniform(0,1,(N+2,np.size(y,axis=1))) - 0.5 # Initialize weights
iterCount = 1
while iterCount < iterMax:
# Mix up the input patterns
mixInputs = y[np.random.permutation(np.size(y,axis = 0)),:]
# Sigma reduction
sigma = sigma_0 - (sigma_0/(iterMax + 1)) * iterCount
s2 = 2*sigma**2
# Learning rate reduction
eta = eta_0 - (eta_0/(iterMax + 1)) * iterCount
for selectedInput in mixInputs: # Pick up one pattern
# Search winning neuron
aux = np.sum((selectedInput - w[N])**2, axis = -1)
ii = np.argmin(aux) # Neuron 'ii' is the winner
jjs = abs(ii - list(range(N+2)))
dists = np.min(np.vstack([jjs , abs(jjs-(N+2))]), axis = 0)
# Update weights
w[N] = w[N] + eta * np.exp((-dists**2)/s2).T[:,np.newaxis] * (selectedInput - w[N])
print(N+2,iterCount)
iterCount += 1
# Assign each datapoint to its nearest neuron
for kk in range(np.size(y,axis = 0)):
aux = np.sum((y[kk,] - w[N])**2,axis=-1)
ii = np.argmin(aux) # Neuron 'ii' is the winner
wClusters[kk,N] = ii + 1
t_begin = time.clock() # Start time
#%%
def apply_async():
pool = mp.Pool(processes=NMax)
for N in range(NMax-1):
pool.apply_async(neuron_run, args = (N,))
pool.close()
pool.join()
print "Multiprocessing done!"
if __name__ == '__main__':
apply_async()
t_end = time.clock() # End time
print(t_end - t_begin)
I am trying to implement gradient descent on a dataset. Even though I tried everything, I couldn't make it work. So, I created a test case. I am trying my code on a random data and try to debug.
More specifically, what I am doing is, I am generating random vectors between 0-1 and random labels for these vectors. And try to over-fit the training data.
However, my weight vector gets bigger and bigger in each iteration. And then, I have infinities. So, I do not actually learn anything. Here is my code:
import numpy as np
import random
def getRandomVector(n):
return np.random.uniform(0,1,n)
def getVectors(m, n):
return [getRandomVector(n) for i in range(n)]
def getLabels(n):
return [random.choice([-1,1]) for i in range(n)]
def GDLearn(vectors, labels):
maxIterations = 100
stepSize = 0.01
w = np.zeros(len(vectors[0])+1)
for i in range(maxIterations):
deltaw = np.zeros(len(vectors[0])+1)
for i in range(len(vectors)):
temp = np.append(vectors[i], -1)
deltaw += ( labels[i] - np.dot(w, temp) ) * temp
w = w + ( stepSize * (-1 * deltaw) )
return w
vectors = getVectors(100, 30)
labels = getLabels(100)
w = GDLearn(vectors, labels)
print w
I am using LMS for loss function. So, in all iterations, my update is the following,
where w^i is the ith weight vector and R is the stepSize and E(w^i) is the loss function.
Here is my loss function. (LMS)
and here is how I derivated the loss function,
,
Now, my questions are:
Should I expect good results in this random scenario using Gradient Descent? (What is the theoretical bounds?)
If yes, what is my bug in my implementation?
PS: I tried several other maxIterations and stepSize parameters. Still not working.
PS2: This is the best way I can ask the question here. Sorry if the question is too specific. But it made me crazy. I really want to learn the problem.
Your code has a couple of faults:
In GetVectors() method, you did not actually use the input variable m;
In GDLearn() method, you have a double loop, but you use the same variable i as the loop variables in both loops. (I guess the logic is still right, but it's confusing).
The prediction error (labels[i] - np.dot(w, temp)) has the wrong sign.
Step size does matters. If I am using 0.01 as step size, the cost is increasing in each iteration. Changing it to be 0.001 solved the problem.
Here is my revised code based on your original code.
import numpy as np
import random
def getRandomVector(n):
return np.random.uniform(0,1,n)
def getVectors(m, n):
return [getRandomVector(n) for i in range(m)]
def getLabels(n):
return [random.choice([-1,1]) for i in range(n)]
def GDLearn(vectors, labels):
maxIterations = 100
stepSize = 0.001
w = np.zeros(len(vectors[0])+1)
for iter in range(maxIterations):
cost = 0
deltaw = np.zeros(len(vectors[0])+1)
for i in range(len(vectors)):
temp = np.append(vectors[i], -1)
prediction_error = np.dot(w, temp) - labels[i]
deltaw += prediction_error * temp
cost += prediction_error**2
w = w - stepSize * deltaw
print 'cost at', iter, '=', cost
return w
vectors = getVectors(100, 30)
labels = getLabels(100)
w = GDLearn(vectors, labels)
print w
Running result -- you can see the cost is decreasing with each iteration but with a diminishing return.
cost at 0 = 100.0
cost at 1 = 99.4114482617
cost at 2 = 98.8476022685
cost at 3 = 98.2977744556
cost at 4 = 97.7612851154
cost at 5 = 97.2377571222
cost at 6 = 96.7268325883
cost at 7 = 96.2281642899
cost at 8 = 95.7414151147
cost at 9 = 95.2662577529
cost at 10 = 94.8023744037
......
cost at 90 = 77.367904046
cost at 91 = 77.2744249433
cost at 92 = 77.1823702888
cost at 93 = 77.0917090883
cost at 94 = 77.0024111475
cost at 95 = 76.9144470493
cost at 96 = 76.8277881325
cost at 97 = 76.7424064707
cost at 98 = 76.6582748518
cost at 99 = 76.5753667579
[ 0.16232142 -0.2425511 0.35740632 0.22548442 0.03963853 0.19595213
0.20080207 -0.3921798 -0.0238925 0.13097533 -0.1148932 -0.10077534
0.00307595 -0.30111942 -0.17924479 -0.03838637 -0.23938181 0.1384443
0.22929163 -0.0132466 0.03325976 -0.31489526 0.17468025 0.01351012
-0.25926117 0.09444201 0.07637793 -0.05940019 0.20961315 0.08491858
0.07438357]