I am trying to implement Quantitative Mobility Spectrum Analysis (QMSA) in Python 2.7 + NumPy + MatPlotLib. QMSA is a tool for analysis of magnetic field-dependent Hall-effect measurements. In a semiconductor, more than one type of carrier (electron or hole) with different concentration and mobility can be found. QMSA can seperate the number of different types of carriers, their density, mobility, and sign. There are some versions of it like improved version i-QMSA. However, even the building the standart QMSA from zero is a hard work to do.
The job is highly know. There are alot of scientific articles about the subject. However, because of copyrights of each article is held by a publisher I can not share with you. Mostly, you can reach them with an university account. There are some thesis about it like:
1.) digital.library.txstate.edu/bitstream/handle/10877/4390/CUNNINGHAM-THESIS.pdf?sequence=1
2.) wrap.warwick.ac.uk/56132/1/WRAP_thesis_Kiatgamolchai_2000.pdf
3.) etd.lib.nsysu.edu.tw/ETD-db/ETD-search/getfile?URN=etd-0629104-152644&filename=etd-0629104-152644.pdf (I think it is in Chinese)
4.) fbetezbankasi.gazi.edu.tr/pdf-indir/22233741 (I think it is in Turkish. Equations which is given in QMSA manual is listed in thesis at pages 73-75)
Code will I am trying to do Successive-over-relaxation (SOR) iterative approach as originally done. Firstly, I prepare I simple code to produce artificial experimental data of magnetic field dependent conductivity tensor sigmaxx(B) and sigmaxy(B). With this experimental input and a predefine mobility values, code is running...
for i in range (0,n,1):
bxx[i] = data[i][1]
bxy[i] = data[i][2]
for j in range (0,m,1):
if data[i][0] == 0:
data[i][0] = 0.001
Axx[j,i]=1/(1+(mobin[j]**2)*(data[i][0]**2))
Axy[j,i]=(mobin[j]*data[i][0])/(1+(mobin[j]**2)*(data[i][0]**2))
Here, bxx, bxy, mobin and data[i][0] is experimental sigmaxx, experimental sigmaxy, predefined mobility values taken from a text file and experimental magnetic field points, respectively. Therefore we are trying to solve two equations with SOR in the form of Ax=b. For XX part of the problem A, x and b are renamed as Axx, solxx and bxx. For XY part of the problem A, x and b are renamed as Axy, solxy and bxy.
For the SOR, you need a parameter called omega. I found optimum omega values with GAuss-Seidel (here I am showing this for XX part of the conductivity. Same procedures are also done for XY):
print "Iterations for finding omega..."
for it_count in range(1,501):
for i in range(0,n,1):
s1xx = np.dot(Axx[i, :i], solxx_new[:i])
s2xx = np.dot(Axx[i, i + 1:], solxx[i + 1:])
solxx_new[i] = (bxx[i] - s1xx - s2xx) / Axx[i, i]
dx = sqrt(np.dot(solxx_new-solxx,solxx_new-solxx))
for i in range(0,n,1):
solxx[i] = solxx_new[i]
if it_count == k: dx1 = dx
if it_count == k + 1:
dx2 = dx
omegaxx = 2.0/(1.0 + sqrt(abs(1.0 - (dx2/dx1))))
break
print "I think best omega value for these XX calculations is ", omegaxx
This "finding optimum omega" procedure is taken from Kiusalaas, Jaan. Numerical methods in engineering with Python 3. Cambridge university press, 2013, page 83.
After finding omega, this time same iterations are done with SOR:
for it_count in range(ITERATION_LIMIT):
for i in range(0,n,1):
s1xx = np.dot(Axx[i, :i], solxx_new[:i])
s2xx = np.dot(Axx[i, i + 1:], solxx[i + 1:])
solxx_new[i] = (1-omegaxx)*solxx[i-1]+omegaxx*(bxx[i] - s1xx - s2xx) / Axx[i, i]
if np.allclose(solxx, solxx_new, rtol=1e-9):
break
print "Iteration:",it_count
for i in range(0,n,1):
solxx[i] = solxx_new[i]
Then, I calculated conductivity spectrum values for each mobility with:
for i in range (0,n,1):
if i == 0:
deltamob = 100
else:
deltamob = mobin[i] - mobin[i-1]
sn[i] = abs((solxx[i] - solxy[i]))/(2*deltamob*1.6e-19)
sp[i] = abs((solxx[i] + solxy[i]))/(2*deltamob*1.6e-19)
x[i] = mobin[i]
B[i] = data[i][0]
Then x vs sn and x vs sp must be your mobility spectrums. Only thing that I can get is a single Gaussian like peak. And even without any hole carrier, electron and hole spectrums are identical. Problem is solxx and solxy is getting larger values after each iteration. Problem may caused by the SOR code is written for Python 3. But I am using Python 2.7.
I can send files if necessary.
Thanks for any responses.
Related
so I am using this python based ldpc package https://hichamjanati.github.io/pyldpc/ In this package for some constraints of my project I cannot use the SNR values. I can only apply bit flips with certain probabilities to the entire encoded message. But with this package, it seems SNR value is a must.
from pyldpc import make_ldpc, encode, decode, get_message
n = 15
d_v = 4
d_c = 5
snr = 20
H, G = make_ldpc(n, d_v, d_c, systematic=True, sparse=True)
k = G.shape[1]
print('K:', k)
v = np.random.randint(2, size=k)
print('V:', v)
y = encode(G, v, snr)
print('Encode:', y)
if y[1] == 1:
y[1] = 0
else:
y[1] = 1
print('Corrupted:', y)
d = decode(H, y, snr)
print('Decode:', y)
x = get_message(G, d)
print('Get message:', x)
(ignore the snr variable I turned it off in a way).
When I went ahead and forcibly turned off the SNR argument and applied bit-flip errors on my own the decoder was not able to correct it, giving a message
UserWarning: Decoding stopped before convergence. You may want to increase maxiter
warnings.warn("Decoding stopped before convergence.")
Is there a proper way to implement simple bit flips in this package? Thanks in advance.
Bitflip almost equal to inverse of sign of soft data after
encode(G, v, snr)
I think you've made too much errors and decoder can't decode it.
p.s. mention that this package use SP-decoding, which very sensitive to correctness of SNR estimation (and constructed to work only with gaussian noise, not with channel with erasures or inversions of analog input data).
Better to try using of MS-decoder which not sensitive to snr estimation.
I am trying to make a lumped rainfall-runoff balance model with a lot parameters (from 37 to 1099) in python. As input it will receive daily rainfall and temperature data and then provides output as a daily flows.
I am stuck on the optimisation method for the model's calibration. I choosed differential evolution algorithm because it is easy to use and can be applied to this kind of problem. The algorithm I wrote works well and it seems to minimise the objective function (which is Nash-Sutcliff model Eficiency - NSE). The problem starts with higher number of parameters which significantly slows the whole algorithm.
The DE algorithm I wrote:
import numpy as np
import flow # a python file from where I get observed daily flows as a np.array
def differential_evolution(func, bounds, popsize=10, mutate=0.8, CR=0.85, maxiter=50):
#--- INITIALIZE THE FIRST POPULATION WITHIN THE BOUNDS-------------------+
bounds = [(0, 250)] * 1 + [(0, 5)] * 366 + [(0, 2)] * 366 + [(0, 100)] * 366
dim = len(bounds)
pop_norm = np.random.rand(popsize, dim)
min_bound, max_bound = np.asarray(bounds).T
difference = np.fabs(min_bound - max_bound)
population = min_bound + pop_norm * difference
# Computed value of objective function for intial population
fitness = np.asarray([func(x, flow.l_flow) for x in population])
best_idx = np.argmin(fitness)
best = population[best_idx]
#--- MUTATION -----------------------------------------------------------+
# This is the part which take to much time to complete
for i in range(maxiter):
print('Generation: ', i)
for j in range(popsize):
# Random selection of three individuals to make a noice vector
idxs = list(range(0, popsize))
idxs.remove(j)
x_1, x_2, x_3 = pop_norm[np.random.choice(idxs, 3, replace=True)]
noice_vector = np.clip(x_1 + mutate * (x_2 - x_3), 0, 1)
#--- RECOMBINATION ------------------------------------------------------+
cross_points = np.random.rand(dim) < CR
if not np.any(cross_points):
cross_points[np.random.randint(0, dim)] = True
trial_vector_norm = np.where(cross_points, noice_vector, pop_norm[j])
trial_vector = min_bound + trial_vector_norm * difference
crit = func(trial_vector, flow.l_flow)
# Check for better fitness of objective function
if crit < fitness[j]:
fitness[j] = crit
pop_norm[j] = trial_vector_norm
if crit < fitness[best_idx]:
best_idx = j
best = trial_vector
return best, fitness[best_idx]
The rainfall-runoff model itself is a function which works basically on lists and via for loop it iteraters over each row to compute daily flows by simple equation.
The objective function NSE is vectorised by numpy arrays:
import model # a python file where rainfall-runoff model function is defined
def nse_min(parameters, observations):
# Modeled flows from model function
Q_modeled = np.array(model.model(parameters))
# Computation of the NSE fraction
numerator = np.subtract(observations, Q_modeled) ** 2
denominator = np.subtract(observations, np.sum(observations)/len(observations)) ** 2
return np.sum(numerator) / np.sum(denominator)
Is there any chance to speed it up? I found out about the numba library which "compiles python code to a machine code" and then let you compute on CPU more efficiently or on GPU using CUDA cores. But I do not study anything related to IT and have no idea how CPU/GPU works, therefore I do not know how to use numba properly. Can anyone help me with it? Or can anyone suggest different optimisation method?
What I use:
Python 3.7.0 64-bit,
Windows 10 Home x64,
Intel Core(TM) i7-7700HQ CPU # 2.80 Ghz,
NVIDIA GeForce GTX 1050 Ti 4GB GDDR5,
16 GB RAM DDR4.
I am a python beginner who study a water management and use python sometimes just for some sripts which make my life easier in data processing. Thank you for your help in advance.
You can use the python library multiprocessing. It just makes more processes to run your function.
you can use it like this.
from multiprocessing import Process
def f(name):
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
I have come to the conclusion that I simply do not understand python as well as I thought.
after a great many tries in python, I tried writing the same thing in Matlab and it just worked.
What my conclusion is, is that the way structures work is just a lot different from what I expect and I cannot see what that difference is.
For example in python, it could have a structure that looks like [[1], [2], [3]] and in Matlab, it would be [1,2,3]. running a loop over i in python would yield only [1] and the same in Matlab would be the sequence.
I remedied this by using np.hstack to get [1,2,3], so I fixed that issue, but I suspect that the rest of my issue right now is also structure-based. in the Matlab code, I get a coupling and the numbers converge. However, in my python code, all of them diverge.
Is there a great resource on how data structures work in python, that isn't the python doc, maybe something that compares Matlab and Python structures? or does anyone have an idea of how I should restructure my Python code?
EDIT: the code is an attempt at euler integration of coupled oscillators, where each oscillator couples to all of its neigbours
it takes in a frequency, w, as this is a solution for a coupling constant of K = 0
a loop runs from each oscillator i, over each neighbour j.
dTheta[i] is the frequency of the current oscillator in the loop.
k/N indicates the coupling strength based on the number of neighbours and theta[j,c] and theta[i,c] is respectively the previous angles for neighbour and current oscillator
a new angle is then assigned based on the integration step of the frequency
in Matlab, I wrote the following
%% Initialize items
k = 1; %coupling factor
N = 20 ; %Number of oscillators
tend = 10;
dt= tend/200;
t = 0:dt:tend;
theta=zeros(N,length(t));
theta(:,1)=abs(2*pi*rand(N,1));
dTheta=zeros(N,1);
w = .1.*ones(N,1); %Set the frequency of the oscillators
y = zeros(size(theta));
%% Calculations
for c=2:length(t)
dTheta=w;
for i=1:N
for j=1:N
dTheta(i)=dTheta(i)+((k/N)*sin(theta(j,c-1)-theta(i,c-1))); %Genereate delta theta.
end
end
theta(:,c)=theta(:,c-1)+(dTheta*dt); %Euler forward step
c/length(t);
end
for c=1:length(t)
y(:,c)=sin((5*t(c))+theta(:,c)); %Generate the y.
end
in python I have
import numpy as np
import matplotlib.pyplot as plt
k = 1
N = 10
tend = 20
dt = tend*4
t = np.linspace(0,tend,dt)
theta = np.zeros((N,len(t)))
theta[:,0] = 2*np.pi*abs(np.random.randint(0, high=N ,size=(N)))/10
dTheta = np.hstack(np.zeros((N,1)))
w = np.hstack(20*np.ones((N,1)))
y = np.zeros(theta.shape)
for c in range(1,len(t)):
dTheta=w
for i in range(N):
for j in range(N):
dTheta[i] = dTheta[i] + ((k/N)*np.sin(theta[j,c-1] - theta[i,c-1]))
theta[:,c] = theta[:,c-1] + (dTheta*dt)
c/len(t)
for c in range(len(t)):
y[:,c] = np.sin((5*t[c]) + theta[:,c])
plt.figure()
for c in range(N):
plt.plot(t,y[c,:])
plt.figure()
for c in range(N):
plt.plot(t,theta[c,:])
plt.show()
In the following code I am trying to implement the following
write a function naturalSpline that implements cubic spline interpolation with natural boundary conditions
Use a tridiagonal solver to solve the arising tridiagonal system for the first derivatives.
The prototype of the function should read yy=naturalSpline(x,y,xx) where (x,y) are the input points and data, and xx are the points where the data should be interpolated.
I figured first I would start with the second bullet point, creating the tridiagonal solver. So this is just the Thomas algorithm. I spent some time to create this part of the code and I have formatted it below. But now I am trying to finish the first and third bullet points but I am not sure how to use what I have done already to finish those. Looking for some help with this! Thanks in advance.
import numpy as np
def TDMA(a,b,c,d):
n = len(d)
w= np.zeros(n-1,float)
g= np.zeros(n, float)
p = np.zeros(n,float)
w[0] = c[0]/b[0]
g[0] = d[0]/b[0]
for i in range(1,n-1):
w[i] = c[i]/(b[i] - a[i-1]*w[i-1])
for i in range(1,n):
g[i] = (d[i] - a[i-1]*g[i-1])/(b[i] - a[i-1]*w[i-1])
p[n-1] = g[n-1]
for i in range(n-1,0,-1):
p[i-1] = g[i-1] - w[i-1]*p[i]
return p
A = np.array([[10,2,0,0],[3,10,4,0],[0,1,7,5], [0,0,3,4]],dtype=float)
a = np.array([3.,1,3])
b = np.array([10.,10.,7.,4.])
c = np.array([2.,4.,5.])
d = np.array([3,4,5,6.])
print (TDMA(a, b, c, d))
Which gives the correct output, I even tested it against np.linalg.solve(a,b,c,d) to make sure it was correct
[ 0.14877589 0.75612053 -1.00188324 2.25141243]
For each interval [x_k, x_(k+1)], you can solve the four equations
p_k(x_k) = f(x_k) = y_k
p_k'(x_k) = f'(x_k) = d_k
p_k(x_(k+1)) = f(x_(k+1)) = y_(k+1)
p_k'(x_(k+1)) = f'(x_(k+1)) = d_(k+1)
(without checking your code, I assume that this is what you did).
From this, you can construct a dict
{'polynomials': [ [a_0, ..., d_0], ..., [a_24, ..., d_24] ],
'knots': [x_0, ..., x_24]}
For each x of your 250 point, you check for which k the point x is in the interval [x_k, x_(k+1)] and evaluate p_k(x).
All of this is straight forward mathematics and python coding. If something is not clear, you are better of learning more about both fields, instead of getting specialized advise on this website.
I am aware of scipy.solve_bvp but it requires that you interpolate your variables which I do not want to do.
I have a boundary value problem of the following form:
y1'(x) = -c1*f1(x)*f2(x)*y2(x) - f3(x)
y2'(x) = f4(x)*y1 + f1(x)*y2(x)
y1(x=0)=0, y2(x=1)=0
I have values for x=[0, 0.0001, 0.025, 0.3, ... 0.9999999, 1] on a non-uniform grid and values for all of the variables/functions at only those values of x.
How can I solve this BVP?
This is a new function, and I don't have it on my scipy version (0.17), but I found the source in scipy/scipy/integrate/_bvp.py (github).
The relevant pull request is https://github.com/scipy/scipy/pull/6025, last April.
It is based on a paper and MATLAB implementation,
J. Kierzenka, L. F. Shampine, "A BVP Solver Based on Residual
Control and the Maltab PSE", ACM Trans. Math. Softw., Vol. 27,
Number 3, pp. 299-316, 2001.
The x mesh handling appears to be:
while True:
....
solve_newton
....
insert_1, = np.nonzero((rms_res > tol) & (rms_res < 100 * tol))
insert_2, = np.nonzero(rms_res >= 100 * tol)
nodes_added = insert_1.shape[0] + 2 * insert_2.shape[0]
if m + nodes_added > max_nodes:
status = 1
if verbose == 2:
nodes_added = "({})".format(nodes_added)
print_iteration_progress(iteration, max_rms_res, m,
nodes_added)
...
if nodes_added > 0:
x = modify_mesh(x, insert_1, insert_2)
h = np.diff(x)
y = sol(x)
where modify_mesh add nodes to x based on:
insert_1 : ndarray
Intervals to each insert 1 new node in the middle.
insert_2 : ndarray
Intervals to each insert 2 new nodes, such that divide an interval
into 3 equal parts.
From this I deduce that
you can track the addition of nodes with the verbose parameter
nodes are added, but not removed. So the out mesh should include all of your input points.
I assume nodes are added to improve resolution in certain segments of the problem
This is based on reading the code, and not verified with test code. You may be the only person to be asking about this function on SO, and one of the few to have actually used it.