I have a function f which I am using to evolve a Numpy array z repeatedly. Thus my code looks like this:
for t in range(100):
z = f(z)
However, now I want to combine elements of array while evolving. For example, in simple Python, I would like to do something like this:
N = len(z)
for t in range(100):
for i in range(len(z)):
z_new[i] = f(z[i]) + z[(i-1)%N] + z[(i+1)%N]
z = z_new
How can achieve the same thing in Numpy vector operations so that I wouldn't have to compromise with the great speed that Numpy gives me?
Thanks in advance
You can roll the data back and forth to achieve the same result.
Z = f(z)
Z = np.roll(Z, 1) + z
Z = np.roll(Z, -2) + z
z = np.roll(Z, 1)
I had also first thought about slicing but went with np.roll when I found it.
Prompted by #hpaulj's comment I came up with a slice solution:
q = np.array([1,2,3,4,5])
Q = f(q)
# operate on the middle
Q[1:] += q[:-1]
Q[:-1] += q[1:]
# operate on the ends
Q[0] += q[-1]
Q[-1] += q[0]
q = Q.copy()
Related
I have two two-dimensional arrays res1, res2 of size 1501x780.
I have to take slices from them, process these slices with a function and add the result to the array.
The slice on res1 starts at (0,0) and goes up to (100,100). (size 100x100) It is fixed and does not move.
The second slice of the res2 array also starts at (0,0) and ends at (100,100).
We got 2 slices and calculated the value using a certain formula, and added this value to the resulting array.
Then, with a fixed slice of res1, we change the value of res2 to (0,1),(100,101). That is, we shift the slice to the right by one pixel, leaving it at the same height. We do all the same manipulations. Then again we shift to the right by (0,2),(100,102). And so 100 elements. That is, up to (0,100)(100,200).
Then we move it down one pixel, by (1.0)(0.100).
And we also run from (1,0...100) to (1,0...200)
Shift by (2,0),(2,100) and so on until (100,100),(200,200).
slice of res1 remains fixed in all this.
As a result, the resulting two-dimensional array of elements should be obtained ...
Thank you very much Robin De Schepper for his help, I understood how to interact with slices, but I didn’t describe my task in such a way. And if you add a second cycle for j in range(100) to its function, then I don't see how I can run through the slice values I need., and how to enter the result of the calculation into a two-dimensional array, because the declaration is ahead of results = [[]] doesn't work the way I imagined.
For some reason, while I was describing my question and thinking in parallel, I came to the conclusion that this is not possible to implement.
To obtain an (x, y) matrix of results by performing an operation on (size_x, size_y) slices of input arrays res1 and res2, you can use a nested for loop and numpy:
import numpy as np
def operation(a, b):
return a + b
res1 = np.array(res1)
res2 = np.array(res2)
x = 100
y = 100
size_x = 100
size_y = 100
start_x = 0
start_y = 0
results = np.empty((x, y))
for i in range(x):
for j in range(y):
slice1 = res1[(start_x + i):(start_x + i + size_x), (start_y + j):(start_y + j + size_y)]
slice2 = res2[(start_x + i):(start_x + i + size_x), (start_y + j):(start_y + j + size_y)]
results[i, j] = operation(slice1, slice2)
There are probably smarter numpy ways of doing this
I'm trying to create a matrix with values based on x,y values I have stored in a tuple. I use a loop to iterate over the tuple and perform a simple calculation on the data:
import numpy as np
# Trying to fit quadratic equation to the measured dots
N = 6
num_of_params = 3
# x values
x = (1,4,3,5,2,6)
# y values
y = (3.96, 24.96,14.15,39.8,7.07,59.4)
# X is a matrix N * 3 with the x values to the power of {0,1,2}
X = np.zeros((N,3))
Y = np.zeros((N,1))
print X,"\n\n",Y
for i in range(len(x)):
for p in range(num_of_params):
X[i][p] = x[i]**(num_of_params - p - 1)
Y[i] = y[i]
print "\n\n"
print X,"\n\n",Y
Is this can be achieved in an easier way? I'm looking for some way to init the matrix like X = np.zeros((N,3), read_values_from = x)
Is it possible? Is there another simple way?
Python 2.7
Extend array version of x to 2D with a singleton dim (dim with length=1) along the second one using np.newaxis/None. This lets us leverage NumPy broadcasting to get the 2D output in a vectorized manner. Similar philosophy for y.
Hence, the implementation would be -
X = np.asarray(x)[:,None]**(num_of_params - np.arange(num_of_params) - 1)
Y = np.asarray(y)[:,None]
Or use the built-in outer method for np.power to get X that takes care of the array conversion under the hoods -
X = np.power.outer(x, num_of_params - np.arange(num_of_params) - 1)
Alternatively, for Y, use np.expand_dims -
Y = np.expand_dims(y,1)
I have a function, roughness that is called quite often in a larger piece of code. I need some help with replacing this double for-loop with a simpler vectorized version. Here is the code below:
def roughness(c,d,e,f,z,ndim,half_tile,dx):
imin=0-half_tile
imax=half_tile
z_calc = np.zeros((ndim,ndim), dtype=float)
for j in range(ndim):
y=(j-half_tile)*dx
for i in range(ndim):
x=(i-half_tile)*dx
z_calc[i,j] = c*x*y + d*x + e*y + f - z[i,j]
z_min=z_calc[z_calc!=0].min()
z_max=z_calc[z_calc!=0].max()
# Calculate some statistics for the difference tile
difference = np.reshape(z_calc,ndim*ndim)
mean = np.mean(difference)
var = stats.tvar(difference,limits=None)
skew = stats.skew(difference,axis=None)
kurt = stats.kurtosis(difference, axis=None)
return(z_min,z_max,mean,var,skew,kurt)
After the main calculations, the various stats are calculated on them. The values, of c,d,e,f, ndim,half_tile are all single integer values, and the variable z is an array with size ndim x ndim I have tried to vectorize this before, but the values do not come out correctly, although the code does run.
Here is my attempt:
def roughness(c,d,e,f,z,ndim,half_tile,dx):
z_calc = np.zeros((ndim,ndim), dtype=float)
x = np.zeros((ndim,ndim), dtype=float)
y = np.zeros((ndim,ndim), dtype=float)
x,y = np.mgrid[1:ndim+1,1:ndim+1]
x = (x-half_tile)*dx
y = (y-half_tile)*dx
z_calc = c*x*y + d*x + e*y + f - z
z_min=z_calc[z_calc!=0].min()
z_max=z_calc[z_calc!=0].max()
# Calculate some statistics for the difference tile
difference = np.reshape(z_calc,ndim*ndim)
mean = np.mean(difference)
var = stats.tvar(difference,limits=None)
skew = stats.skew(difference,axis=None)
kurt = stats.kurtosis(difference, axis=None)
return(z_min,z_max,mean,var,skew,kurt)
Aside from getting the correct values, I would really like to know if I did the vectorization of the nested for loop correctly, which i'm assuming I didn't.
Sorry for the seemingly elementary question. What I'm trying to implement is summarized in the following steps:
Generate input variables: x, y.
Let z = F(x,y).
Plot z's for particular combinations of x and y.
For example:
zlist = []
for _ in range(100):
x = np.random.random()*1.
y = np.random.random()*.5
if x < .5:
z = y / 2
else:
z = y * 2
zlist.append(z)
Now if I want to plot z for all the x between (0, 0.3), I presumably would need some marker on each element in zlist indicating its inputs variables. How would I attach such marker and then access it from the list when plotting?
I don't actually know anything about Numpy, so someone please comment and tell me if I'm making a fool out of myself. It seems like vanilla python behavior, though.
Rather than appending z, let's append (z,x) instead. Now zlist is a list of tuples, and you can loop through and plot by checking zlist[i][1].
zlist = []
for _ in range(100):
x = np.random.random()*1.
y = np.random.random()*.5
if x < .5:
z = y / 2
else:
z = y * 2
zlist.append((z,x))
for value in zlist:
if value[1] > 0 and value[1] < 0.3:
# Plot value[0]
# Or if you prefer list comprehensions:
# [value[0] for value in zlist if value[1] >0 and value[1] < 0.3]
# that will return a list with only the z values in zlist.
With numpy it's almost always much more efficient to perform operations on vectors and
arrays rather than on built-in Python sequence types such as lists. Here's one
way you can quickly find F(x, y) for every combination of two sets of random x
and y values without looping in Python. The result is going to be an nx-by-ny
array Z, where Z[i, j] = F(x[i], y[j]).
First of all, you can generate all of your x, y inputs as vectors:
nx = 100
ny = 200
x = np.random.random(size=nx) * 1.
y = np.random.random(size=ny) * 5.
For the result to be an nx-by-ny array, you could take these two vectors and
multiply them by ones to get two 2D nx-by-ny arrays containing the x and y
values in the rows and columns respectively. You can do this by taking advantage of numpy's
broadcasting rules:
x_arr = x[:,np.newaxis] * np.ones((nx,ny))
y_arr = y[np.newaxis,:] * np.ones((nx,ny))
The function you will apply to each x,y pair depends on the x value.
Fortunately, you can use np.where(<condition>, <do_this>, <do_that>) to apply
different operations to the values in your input depending on some condition:
Z = np.where(x_arr < 0.5, y_arr / 2., y_arr * 2.)
We can check that all the results are correct:
for i in xrange(nx):
for j in xrange(ny):
if x[i] < 0.5:
assert Z[i, j] == y[j] / 2.
else:
assert Z[i, j] == y[j] * 2
There's actually an even cleaner way to compute Z without expanding x and y into 2D arrays. Using the same broadcasting trick we used to get
x_arr and y_arr, you can pass x and y directly to np.where():
x2 = x[:,np.newaxis]
y2 = y[np.newaxis,:]
Z2 = np.where(x2 < 0.5, y2 / 2., y2 * 2.)
assert np.all(Z == Z2)
I would like to filter the values of a numpy meshgrid:
X,Y = np.mgrid[-10:10,-10:10]
in this case, I would like to remove all coordinates for which x**2 + y**2 <= 2. However, when I try to filter the array directly, for example
filter(lambda x,y: x**2 + y**2 >= 2, np.meshgrid[-10:10,-10:10])
I get errors because I'm not properly dealing with the array's structure.
Any tips for doing this right would be appreciated!
I was able to achieve the result that I needed using numpy.where, by filtering each array individually, but referencing both in the where condition:
X,Y = np.mgrid[-10:10,-10:10]
X,Y = np.where(X**2 + Y**2 > 2, X, 0), np.where(X**2 + Y**2 > 2, Y, 0)
This results in new 2D arrays, which is what I needed for matplotlib. Thanks to everyone who took the time to look at this question!
X,Y = np.mgrid[-10:10,-10:10]
idx = (X**2 + Y**2 > 2)
X, Y = X[idx], Y[idx]
The problem is that you no longer have 2D arrays, which may be an issue for things like matplotlib.
Seeing your own answer, and that you basically want to replace with 0 entries not fulfilling the condition, it is probably going to be cleaner and more efficient to do:
idx = X**2 + Y**2 > 2
X[~idx] = 0
Y[~idx] = 0