I am solving some partial differential equation discretizing the time and space, to avoid complexity I avoid this thing and just consider that I solve in an iterative way the problem using a function that I called "computation" . The point is that I want to take (and store in some matrix called "Cn") some values given by "y" of the loop "while", but without take all the values of the iteration in time.
To be precise: I am doing a loop "while" for the time evolution taking some time steep dt. I am runing it from t=1 up to t=100 using dt=0.001. My solution "y" is computed to each time steep. The point is that I want to store "y" at some particular values of the time "t", not at each time steep of the loop, i.e for instance I want to store the values at t=1.0,2.0,3.0,...,100.0 using the values that I compute inside the loop while. But I don't want to store the values of "y" at t=1.001,1.002,1.003 etc
I show you the code that I did
import numpy as np
import math
from matplotlib import pyplot as plt
import matplotlib.animation as animation
# grid in 1D
xmin = 0.0
xmax = 100.0
Nx = 120
dx = (xmax-xmin)/Nx
x = np.linspace(xmin,xmax,Nx)
# timing of the numerical simulation
t_initial = 1.0
t_final = 100.0
t = t_initial
dt = 10**(-2)
#initial profile
y = np.exp(-0.5*x**2)
#number of time points to store the numerical solution
dt_solution = 1.0 #time steep to save the numerical data inside the loop while
Nt = int((t_final-t_initial)/dt_solution)
def computation(t,y):
return np.log(t)+y
Cn = np.zeros((Nt, len(x))) # holds the numerical solution
Cn[0,:] = y #we put the initial y
ite = 0
while t<t_final:
t += dt #WE MAKE THE TIME STEEP
ite +=1
y = computation(t,y)
#Cn[ite,:] = y #I WANT TO SAVE THE VECTOR Y FOR THE TIMES t THAT I AM INTERESTD, NOT THE ONES GIVEN INSIDE THE LOOP WHILE
Someone knows how to do that? I was thinking maybe solve this problem using two loops, but I would like to know if its possible to use some more efficient way. Thanks! (I hope that my question is clear, if not please tell me)
You could use a modulo operator. This operator shows the remainder when one number is divided by another. For example:
10%2 = 0 # 10 is exactly divisible by 2
11%2 = 1 # 11 is divisible by 2 with remainder 1
We can use this with an if condition within the while loop.
#...
t = 0
dt=0.001 #timestep for iteration
# set the condition threshold
threshold = dt/10
# choose the step you want to save values at
store_step = 0.1
while t<100:
t += dt
y = computation(t,y)
if (t%store_step<threshold) or (t%store_step>(store_step-threshold)):
# store y values
Cn[ite,:] = y
Note If your timestep is an integer you could use: if (t%1==0) as your condition.
Add this to your while loop where you want to save y:
if int(t % 1) == 0:
Cn[ite,:] = y
So this only saves y when t is divisible by 1, i.e., t is 1.000, 2.000...
Likewise, if you have other conditions in which you want to only save y under, simply check against that condition in a way that can be computed. If not, a static list or set is a viable option as well.
Related
I have audio data that I normalized between -4 and 4; however, I need the "center" of the wave / data (mean?) to be centered at "1" (please see the attached figure)
How can I manipulate my list of data where the mean is translated / converted to be ~1? While still maintaining a normalization of -4 to 4
Is there an easy way to do this using numpy?
In the future it would be nice if you provided an example to tinker with. The second problem is that you don't tell us what type of function to use or what properties it should preserve. Since there clearly is no linear function that does what you want I am going with a quadratic one. But first here is my "sound" to tinker with
np.random.seed(1)
x = np.linspace(0,0.01,10**4)
y = 0.1*np.sin(50*2*np.pi*x/(0.01))+np.random.random()*np.sin(50*2*np.pi*x/(0.01))*((0.0067<x)&(x<0.007))+((0.0067<x)&(x<0.007))*0.2
y -= y.mean()
plt.plot(x,y)
It is not quite your original one but has the property that after centering it is not between -1 and 1. Since it seems more clean I will center to zero and range -1 and 1 but that should make no real difference. My plan was to pick the unique quadratic function that maps the maximum to 1, the minimum to -1 and pick a good value to send 0 to i.e. one for which the mean is zero.
import sympy
a,b,c = sympy.symbols('a, b, c')
def change_scale(arr, lamb=0):
m = arr.min()
M = arr.max()
eqns = [a*x**2+b*x+c-y for x,y in zip([m,0,M],[-1,lamb,1])]
sols = sympy.solve(eqns,[a,b,c])
d,e,f = map(np.float64, sols.values())
return d*arr**2+e*arr+f
y2 = change_scale(y)
print(y2.mean())
plt.plot(x,y2)
To me this almost looks good. The only problem is that now the mean is -0.015. Well but for that we have that lambda parameter.
A quick check shows that 0.5 would be to big and -0.5 would be to small meaning this is a perfect job for scipy.optimize.bisect.
from scipy.optimize import bisect
lamb = bisect(lambda x: change_scale(y,x).mean(),0.5,-0.5)
gives us the perfect value for lambda. The picture looks like so
Which is exactly what I expected since. The mean was slightly too small so we moved things up a bit. Now the mean is zero, the max is 1 and the min is -1.
As a final sanity check we can plot the function we actually applied to our values:
I'm facing a problem while trying to implement the coupled differential equation below (also known as single-mode coupling equation) in Python 3.8.3. As for the solver, I am using Scipy's function scipy.integrate.solve_bvp, whose documentation can be read here. I want to solve the equations in the complex domain, for different values of the propagation axis (z) and different values of beta (beta_analysis).
The problem is that it is extremely slow (not manageable) compared with an equivalent implementation in Matlab using the functions bvp4c, bvpinit and bvpset. Evaluating the first few iterations of both executions, they return the same result, except for the resulting mesh which is a lot greater in the case of Scipy. The mesh sometimes even saturates to the maximum value.
The equation to be solved is shown here below, along with the boundary conditions function.
import h5py
import numpy as np
from scipy import integrate
def coupling_equation(z_mesh, a):
ka_z = k # Global
z_a = z # Global
a_p = np.empty_like(a).astype(complex)
for idx, z_i in enumerate(z_mesh):
beta_zf_i = np.interp(z_i, z_a, beta_zf) # Get beta at the desired point of the mesh
ka_z_i = np.interp(z_i, z_a, ka_z) # Get ka at the desired point of the mesh
coupling_matrix = np.empty((2, 2), complex)
coupling_matrix[0] = [-1j * beta_zf_i, ka_z_i]
coupling_matrix[1] = [ka_z_i, 1j * beta_zf_i]
a_p[:, idx] = np.matmul(coupling_matrix, a[:, idx]) # Solve the coupling matrix
return a_p
def boundary_conditions(a_a, a_b):
return np.hstack(((a_a[0]-1), a_b[1]))
Moreover, I couldn't find a way to pass k, z and beta_zf as arguments of the function coupling_equation, given that the fun argument of the solve_bpv function must be a callable with the parameters (x, y). My approach is to define some global variables, but I would appreciate any help on this too if there is a better solution.
The analysis function which I am trying to code is:
def analysis(k, z, beta_analysis, max_mesh):
s11_analysis = np.empty_like(beta_analysis, dtype=complex)
s21_analysis = np.empty_like(beta_analysis, dtype=complex)
initial_mesh = np.linspace(z[0], z[-1], 10) # Initial mesh of 10 samples along L
mesh = initial_mesh
# a_init must be complex in order to solve the problem in a complex domain
a_init = np.vstack((np.ones(np.size(initial_mesh)).astype(complex),
np.zeros(np.size(initial_mesh)).astype(complex)))
for idx, beta in enumerate(beta_analysis):
print(f"Iteration {idx}: beta_analysis = {beta}")
global beta_zf
beta_zf = beta * np.ones(len(z)) # Global variable so as to use it in coupling_equation(x, y)
a = integrate.solve_bvp(fun=coupling_equation,
bc=boundary_conditions,
x=mesh,
y=a_init,
max_nodes=max_mesh,
verbose=1)
# mesh = a.x # Mesh for the next iteration
# a_init = a.y # Initial guess for the next iteration, corresponding to the current solution
s11_analysis[idx] = a.y[1][0]
s21_analysis[idx] = a.y[0][-1]
return s11_analysis, s21_analysis
I suspect that the problem has something to do with the initial guess that is being passed to the different iterations (see commented lines inside the loop in the analysis function). I try to set the solution of an iteration as the initial guess for the following (which must reduce the time needed for the solver), but it is even slower, which I don't understand. Maybe I missed something, because it is my first time trying to solve differential equations.
The parameters used for the execution are the following:
f2 = h5py.File(r'path/to/file', 'r')
k = np.array(f2['k']).squeeze()
z = np.array(f2['z']).squeeze()
f2.close()
analysis_points = 501
max_mesh = 1e6
beta_0 = 3e2;
beta_low = 0; # Lower value of the frequency for the analysis
beta_up = beta_0; # Upper value of the frequency for the analysis
beta_analysis = np.linspace(beta_low, beta_up, analysis_points);
s11_analysis, s21_analysis = analysis(k, z, beta_analysis, max_mesh)
Any ideas on how to improve the performance of these functions? Thank you all in advance, and sorry if the question is not well-formulated, I accept any suggestions about this.
Edit: Added some information about performance and sizing of the problem.
In practice, I can't find a relation that determines de number of times coupling_equation is called. It must be a matter of the internal operation of the solver. I checked the number of callings in one iteration by printing a line, and it happened in 133 ocasions (this was one of the fastests). This must be multiplied by the number of iterations of beta. For the analyzed one, the solver returned this:
Solved in 11 iterations, number of nodes 529.
Maximum relative residual: 9.99e-04
Maximum boundary residual: 0.00e+00
The shapes of a and z_mesh are correlated, since z_mesh is a vector whose length corresponds with the size of the mesh, recalculated by the solver each time it calls coupling_equation. Given that a contains the amplitudes of the progressive and regressive waves at each point of z_mesh, the shape of a is (2, len(z_mesh)).
In terms of computation times, I only managed to achieve 19 iterations in about 2 hours with Python. In this case, the initial iterations were faster, but they start to take more time as their mesh grows, until the point that the mesh saturates to the maximum allowed value. I think this is because of the value of the input coupling coefficients in that point, because it also happens when no loop in beta_analysisis executed (just the solve_bvp function for the intermediate value of beta). Instead, Matlab managed to return a solution for the entire problem in just 6 minutes, aproximately. If I pass the result of the last iteration as initial_guess (commented lines in the analysis function, the mesh overflows even faster and it is impossible to get more than a couple iterations.
Based on semi-random inputs, we can see that max_mesh is sometimes reached. This means that coupling_equation can be called with a quite big z_mesh and a arrays. The problem is that coupling_equation contains a slow pure-Python loop iterating on each column of the arrays. You can speed the computation up a lot using Numpy vectorization. Here is an implementation:
def coupling_equation_fast(z_mesh, a):
ka_z = k # Global
z_a = z # Global
a_p = np.empty(a.shape, dtype=np.complex128)
beta_zf_i = np.interp(z_mesh, z_a, beta_zf) # Get beta at the desired point of the mesh
ka_z_i = np.interp(z_mesh, z_a, ka_z) # Get ka at the desired point of the mesh
# Fast manual matrix multiplication
a_p[0] = (-1j * beta_zf_i) * a[0] + ka_z_i * a[1]
a_p[1] = ka_z_i * a[0] + (1j * beta_zf_i) * a[1]
return a_p
This code provides a similar output with semi-random inputs compared to the original implementation but is roughly 20 times faster on my machine.
Furthermore, I do not know if max_mesh happens to be big with your inputs too and even if this is normal/intended. It may make sense to decrease the value of max_mesh in order to reduce the execution time even more.
is it possible to do the following without loop (so it improves the speed)? i have looked sklearn, sm, and pd, unfortunately don't think they have any direct solution.
i have
x = np.array(range(1000)) # ie a standard discrete time series
y = np.append(np.zeros(600),np.random.random(400)) #it has a lot of zeros
y = np.random.permutation(y) #the number of zeros in b/w the non zero is random
z = np.empty(1000) # z will contain predicted values from the reg analysis
rolling_window=20
i wish to obtain z, where z(i) = a(i)+b(i)x(i) for i within range(1000)
and a(i) and b(i) are obtained by regressing Ys vs Xs for the i b/w (i-rolling_window, i), but only uses Ys that are non zero (hence need to assign weight = 0 for Ys that are zero in the regression. preferably use a weighting method rather than getting rid of the zeros together, because i dont wish to loop)
many thanks in advance
I want to be able to compute following integral using numpys trapz function
numpy.trapz([-1, 1]) # returns 0
But I don't want to allow negative areas. Is there an efficient way to do this or do I have to look for the minimum point and transform the array by hand?
Does numpy.trapz(numpy.abs([-1, 1])) make sense?
If you want to discard negative contributions to the integrated area we can simply grab the np.trapz source code and rewrite it:
def abstrapz(y, x=None, dx=1.0):
y = np.asanyarray(y)
if x is None:
d = dx
else:
x = np.asanyarray(x)
d = np.diff(x)
ret = (d * (y[1:] +y[:-1]) / 2.0)
return ret[ret>0].sum() #The important line
A quick test:
np.trapz([-1,0,1])
0.0
abstrapz([-1,0,1])
0.5
If you just want to avoid areas where y is less then zero simply mask 'y' values less then zero to zero:
arr = np.array([-2,-1,0.5,1,2,1,0,5,3,0])
np.trapz(arr)
10.5
arr[arr<0] = 0
np.trapz(arr)
12.5
This isn't the best way to do it, but it is a good approximation. If this is what you mean I can update this.
I had to change your example slightly as trapz([-1,1]) will always return 0 by definition. We do remove some functionality this way, if you need to do this on multidimensional arrays it is easy to add it back in.
I'm trying to (in python) fit a series of an arbitrary number of gaussian functions (determined by a simple algorithm still being improved) to a data set. For my current sample data set, I have 174 gaussian functions. I have a procedure for doing the fit, but it's basically complicated guess-and-check, and consumes all 4GB of memory available.
Is there any way to accomplish this using something in scipy or numpy?
Here is what I'm trying to use, where wavelength[] is the list of x-coordinates, and fluxc[] is the list of y-coordinates:
#Pick a gaussian
for repeat in range(0,2):
for f in range(0,len(centroid)):
#Iterate over every other gaussian
for i in range(0,len(centroid)):
if i!= f:
#For every wavelength,
for w in wavelength:
#Append the value of each to an list, called others
others.append(height[i]*math.exp(-(w-centroid[i])**2/(2*width[i]**2)))
#Optimize the centroid of the current gaussian
prev = centroid[f]
best = centroid[f]
#Pick an order of magnitude
for p in range (int(round(math.log10(centroid[i]))-3-repeat),int(round(math.log10(centroid[i])))-6-repeat,-1):
#Pick a value of that order of magnitude
for m in range (-5,9):
#Change the value of the current item
centroid[f] = prev + m * 10 **(p)
#Increment over all wavelengths, make a list of the new values
variancy = 0
residual = 0
test = []
#Increment across every wavelength and evaluate if this change gets R^2 any larger
for k in range(0,len(wavelength)):
test.append(height[i]*math.exp(-(wavelength[k]-centroid[f])**2/(2*width[i]**2)))
residual += (test[k]+others[k]-cflux[k])**2
variancy += (test[k]+others[k]-avgcflux)**2
rsquare = 1-(residual/variancy)
#Check the R^2 value for this new fit
if rsquare > bestr:
bestr = rsquare
best = centroid[f]
centroid[f] = best
#Optimize the height of the current gaussian
prev = height[f]
best = height[f]
#Pick an order of magnitude
for p in range (int(round(math.log10(height[i]))-repeat),int(round(math.log10(height[i])))-3-repeat,-1):
#Pick a value of that order of magnitude
for m in range (-5,9):
#Change the value of the current item
height[f] = prev + m * 10 **(p)
#Increment over all wavelengths, make a list of the new values
variancy = 0
residual = 0
test = []
#Increment across every wavelength and evaluate if this change gets R^2 any larger
for k in range(0,len(wavelength)):
test.append(height[f]*math.exp(-(wavelength[k]-centroid[i])**2/(2*width[i]**2)))
residual += (test[k]+others[k]-cflux[k])**2
variancy += (test[k]+others[k]-avgcflux)**2
rsquare = 1-(residual/variancy)
#Check the R^2 value for this new fit
if rsquare > bestr:
bestr = rsquare
best = height[f]
height[f] = best
#Optimize the width of the current gaussian
prev = width[f]
best = width[f]
#Pick an order of magnitude
for p in range (int(round(math.log10(width[i]))-repeat),int(round(math.log10(width[i])))-3-repeat,-1):
#Pick a value of that order of magnitude
for m in range (-5,9):
if prev + m * 10**(p) == 0:
m+=1
#Change the value of the current item
width[f] = prev + m * 10 **(p)
#Increment over all wavelengths, make a list of the new values
variancy = 0
residual = 0
test = []
#Increment across every wavelength and evaluate if this change gets R^2 any larger
for k in range(0,len(wavelength)):
test.append(height[i]*math.exp(-(wavelength[k]-centroid[i])**2/(2*width[f]**2)))
residual += (test[k]+others[k]-cflux[k])**2
variancy += (test[k]+others[k]-avgcflux)**2
rsquare = 1-(residual/variancy)
#Check the R^2 value for this new fit
if rsquare > bestr:
bestr = rsquare
best = width[f]
width[f] = best
count += 1
#print '{} of {} peaks optimized, iteration {} of {}'.format(f+1,len(centroid),repeat+1,2)
complete = round(100*(count/(float(len(centroid))*2)),2)
print '{}% completed'.format(complete)
print 'New R^2 = {}'.format(bestr)
Yes, it can likely be done better (easier) using scipy. But firstly, refactor your code into smaller functions; it justs makes it a lot easier to read and understand what's going on.
As for the memory consumption: you're probably overextending a list far too much somewhere (others is a candidate: I never see it cleared (or initialized!), while it gets filled in a quadruple loop). That, or your data is simply that large (in which case you really should be using numpy arrays, just to speed up things). I can't tell, because you're introducing various variables without giving any idea of the size (how big is wavelengths? How large does others get? What and where are all the initializations of your data arrays?)
Also, fitting 174 Gaussians is just a bit crazy; either look into another way of determining whatever you want to get out of your data, or split things up. From the wavelengths variable, it appears you're trying to fit lines in a high resolution spectrum; perhaps isolating most of the lines and fitting those isolated groups separately is better. If they all overlap, I doubt any normal fitting technique is going to help you.
Lastly, perhaps a package like pandas can help (e.g., the computation subpackage).
Perhaps very lastly, since I see a lot that can be improved in the code. At some point codereview may also be useful. Though for now I guess your memory usage is the most problematic part.