Below code is running so slow. I tried using numpy.argwhere instead of "if statement" to speed up the code and I got a pretty efficient result but it's still very slow. I also tried numpy.frompyfunc and numpy.vectorize but I failed. What would you suggest to speed up the code below?
import numpy as np
import time
time1 = time.time()
n = 1000000
k = 10000
velos = np.linspace(-1000, 1000, n)
line_centers = np.linspace(-1000, 1000, k)
weights = np.random.random_sample(k)
rvs = np.arange(-60, 60, 2)
m = len(rvs)
w = np.arange(10)
M = np.zeros((n, m))
for l, lc in enumerate(line_centers):
vi = velos - lc
for j in range(m - 1):
w = np.argwhere((vi < rvs[j + 1]) & (vi > rvs[j])).T[0]
M[w, j] = weights[l] * (rvs[j + 1] - vi[w]) / (rvs[j + 1] - rvs[j])
M[w, j + 1] = weights[l] * (vi[w] - rvs[j]) / (rvs[j + 1] - rvs[j])
time2 = time.time()
print(time2 - time1)
EDIT:
The size of the array M was incorrect. I fixed it.
This seems like a situation where a c++ interface could come in handy. With Pybind11 you can create c++ functions which take numpy arrays as argument, manipulate them and return them back to python. That would speed up you loops. Take a look at it!
Of course it is slow, you have two nested loops! You need to rethink your algorithm using vector operations, as in, no iteration over indices, but implement in terms of index or boolean arrays, and index shifts.
You have not given any background information, so it is incredibly hard for anyone to suggest something meaningful (given the soup of indices in the example). A few quick suggestions based on quickly gleaning over your example.
An expression like this (rvs[j + 1] - rvs[j]) is easily replaced with numpy.ediff1d.
You seem to be iterating through n in blocks of m, maybe numpy.nditer will be of use.
I have a hunch that your inner loop has an error, are you sure you really mean to iterate over range(m - 1)? That would mean you are iterating from 0 to m-2 (inclusive), I doubt you meant that.
We can help with more concrete answers if you provide more background information.
Related
I want to append in separate list for example: list1 for j=1, list 2 for j=2 and so on, similarly (total[j]= (sum(sum_list[j]/mcSteps-j)
Here is code :
sum_list=[]
cor =[]
for j in range(10):
for i in range(mcSteps-j):
sum_list.append(Energy_list[i]*Energy_list[i+j])
total[j]=sum(sum_list[j])
deltat[j]=total[j]/(mcSteps-j)
cor[j].append(Divide*(deltat[j]-E1mean*E1mean))
print('cor[j]')
I would recommend considering numpy for something like this (from what it looks like). You can do many of these operations in far fewer lines – and it maintains wall-clock efficiency compared to Python lists and loops when you get to rather large numerical computations (around N=10_000).
Anywho, a solution to what you've asked for appears to be...
from typing import List
def compute_delta_t(ls: List, steps: int, offset: int) -> float:
energies = [ls[step] * ls[step + offset] for step in range(steps)]
delta_t = sum(energies) / (steps - offset)
return delta_t
delta_ts = [
compute_delta_t(Energy_list, mcSteps, offset)
for offset in range(10)
]
corrs = [Divide * (delta_t - (E1mean ** 2)) for delta_t in delta_ts]
The above approach should be rather efficient Python code (albeit getting untenable and unreadable).
numpy, an alternative for doing [matrix] math (among many other things) in Python.
import numpy as np
def compute_delta_t(ls: np.ndarray, mcSteps: int, offset: int) -> np.ndarray:
energies = ls[:(mcSteps - offset)] * ls[offset:(mcSteps + offset)]
delta_t = energies.sum() / (mcSteps - offset)
return delta_t
delta_ts = np.asarray([
compute_delta_t(Energy_list, mcSteps, offset)
for offset in range(10)
])
corrs = Divide * (delta_ts - (E1mean ** 2))
This numpy-based approach looks very similar to the Python approach explicitly because I'm not sure the each np.ndarray would have the same size. If they did, then most of what you're looking for could be accomplished in around 2-3 lines of code. đŸ˜…
(For those who know what I'm talking about, I'm ignoring the possibility of padding with 0s.)
This question already has answers here:
Binary random array with a specific proportion of ones?
(6 answers)
Closed 4 years ago.
I want to randomly produce an array of n ones and m zeros.
I thought of this solution:
produce the ones array (np.ones)
produce the zeros array (np.zeros)
combine them to one array (np.hstack)
shuffle the resulting array (np.random.shuffle)
Seems to be not natural as a solution. Some pythonic ideas?
Your solution seems reasonable. It states exactly what it's doing, and does it clearly.
Let's compare your implementation:
a = np.hstack((np.ones(n), np.zeros(m)))
np.random.shuffle(a)
… with an obvious alternative:
a = np.ones(n+m)
a[:m] = 0
np.random.shuffle(a)
That might save a bit of time not allocating and moving hunks of data around, but it takes a bit more thought to understand.
And doing it in Python instead of in NumPy:
a = np.array([1]*n + [0]*m)
np.random.shuffle(a)
… might be a little more concise, but it seems less idiomatically NumPy (in the same way that np.array([1]*n) is less idiomatic than np.ones(n)), and it's going to be slower and use more memory for no good reason. (You could improve the memory by using np.fromiter, but then it's pretty clearly not going to be more concise.)
Of course if you're doing this more than once, the real answer is to factor it out into a function. Then the function's name will explain what it does, and almost any solution that isn't too tortured will be pretty easy to understand…
I'd make an array of n ones and m zeros as
a = np.array([1] * n + [0] * m)
Then I'd call np.random.shuffle() on it.
Use numpy.random.permutation:
a = numpy.random.permutation([1] * n + [0] * m)
or, using arrays instead of an initial list:
a = numpy.random.permutation(numpy.concatenate(np.ones(n), np.zeros(m)))
(I don't know enough about numpy to comment on the difference between concatenate and hstack; they seem to produce the same results here.)
I think your solution is suitable, in that it's readable and pythonic. You didn't say whether memory or performance are considerations. It's possible that np.random.shuffle is as good as O(m + n), but the other answers suggest that it does more than a single pass to shuffle the values. You could do it in O(m + n) with only a single pass and no memory overhead like this:
import random
m = 600 # zeros
n = 400 # ones
result = []
while m + n > 0:
if (m > 0 and random.random() < float(m)/float(m + n)):
result.append(0)
m -= 1
else:
result.append(1)
n -= 1
I am trying to fill an array with calculated values from functions defined earlier in my code. I started with a code that has a similar structure to the following:
from numpy import cos, sin, arange, zeros
a = arange(1000)
b = arange(1000)
def defcos(x):
return cos(x)
def defsin(x):
return sin(x)
a_len = len(a)
b_len = len(b)
result = zeros((a_len,b_len))
for i in xrange(b_len):
for j in xrange(a_len):
a_res = defcos(a[j])
b_res = defsin(b[i])
result[i,j] = a_res * b_res
I tried to use array representations of the functions, which ended up in the following change for the loop
a_res = defsin(a)
b_res = defcos(b)
for i in xrange(b_len):
for j in xrange(a_len):
result[i,j] = a_res[i] * b_res[j]
This is already significantly faster, than the first version. But is there a way to avoid the loop entirely? I have encountered those loops a couple of times in the past but never botheres as it was not critical in terms of speed. But this time it is the core component of something, which is looped through a couple of times more. :)
Any help would be appreciated, thanks in advance!
Like so:
from numpy import newaxis
a_res = sin(a)
b_res = cos(b)
result = a_res[:, newaxis] * b_res
To understand how this works, have a look at the rules for array broadcasting. And please don't define useless functions like defsin, just use sin itself! Another minor detail, you get i from range(b_len), but you use it to index a_res! This is a bug if a_len != b_len.
import numpy as np
import matplotlib.pyplot as pp
curve = np.genfromtxt('C:\Users\latel\Desktop\kool\Neuro\prax2\data\curve.csv',dtype = 'float', delimiter = ',')
curve_abs2 = np.empty_like(curve)
z = 1j
N = len(curve)
for i in range(0,N-1):
curve_abs2[i] =0
for k in range(0,N-1):
curve_abs2[i] += (curve[i]*np.exp((-1)*z*(np.pi)*i*((k-1)/N)))
for i in range(0,N):
curve_abs2[i] = abs(curve_abs2[i])/(2*len(curve_abs2))
#curve_abs = (np.abs(np.fft.fft(curve)))
#pp.plot(curve_abs)
pp.plot(curve_abs2)
pp.show()
The code behind # gives me 3 values. But this is just ... different
Wrong ^^ this code: http://www.upload.ee/image/3922681/Ex5problem.png
Correct using numpy.fft.fft(): http://www.upload.ee/image/3922682/Ex5numpyformulas.png
There are several problems:
You are assigning complex values to the elements of curve_abs2, so it should be declared to be complex, e.g. curve_abs2 = np.empty_like(curve, dtype=np.complex128). (And I would recommend using the name, say, curve_fft instead of curve_abs2.)
In python, range(low, high) gives the sequence [low, low + 1, ..., high - 2, high - 1], so instead of range(0, N - 1), you must use range(0, N) (which can be simplified to range(N), if you want).
You are missing a factor of 2 in your formula. You could fix this by using z = 2j.
In the expression that is being summed in the inner loop, you are indexing curve as curve[i], but this should be curve[k].
Also in that expression, you don't need to subtract 1 from k, because the k loop ranges from 0 to N - 1.
Because k and N are integers and you are using Python 2.7, the division in the expression (k-1)/N will be integer division, and you'll get 0 for all k. To fix this and the previous problem, you can change that term to k / float(N).
If you fix those issues, when the first double loop finishes, the array curve_abs2 (now a complex array) should match the result of np.fft.fft(curve). It won't be exactly the same, but the differences should be very small.
You could eliminate that double loop altogether using numpy vectorized calculations, but that is a topic for another question.
I have this code:
for j in xrange (j_start, self.max_j):
for i in xrange (0, self.max_i):
new_i = round (i + ((j - j_start) * discriminant))
if new_i >= self.max_i:
continue
self.grid[new_i, j] = standard[i]
and I want to speed it up by throwing away slow native python loops. There is possibility to use numpy vector operations instead, they are really fast. How to do that?
j_start, self.max_j, self.max_i, discriminant
int, int, int, float (constants).
self.grid
two-dimensional numpy array (self.max_i x self.max_j).
standard
one-dimensional numpy array (self.max_i).
Here is a complete solution, perhaps that will help.
jrange = np.arange(self.max_j - j_start)
joffset = np.round(jrange * discriminant).astype(int)
i = np.arange(self.max_i)
for j in jrange:
new_i = i + joffset[j]
in_range = new_i < self.max_i
self.grid[new_i[in_range], j+j_start] = standard[i[in_range]]
It may be possible to vectorize both loops but that will, I think, be tricky.
I haven't tested this but I believe it computes the same result as your code.