Develop Reference

How can I access the neighboring elements of the matrix using numpy?

I have made a code that calculates the dissolution of fluids, the problem is that the code is very poor, so I have been looking at that with numpy I can optimize it but I have been stuck without knowing how to do the following code using numpy and the roll function. Basically I have a matrix that the index and cannot be more than 1024, for this I use% to calculate what index it is. But this takes a long time.
I tried using numpy, using roll, rotating the matrix and then I don't have to calculate the module. But I don't know how to take the values ​​of the neighbors.
def evolve(grid, dt, D=1.0):
xmax, ymax = grid_shape
new_grid = [[0.0,] * ymax for x in range(xmax)]
for i in range(xmax):
for j in range(ymax):
grid_xx = grid[(i+1)%xmax][j] + grid[(i-1)%xmax][j] - 2.0 * grid[i][j]
grid_yy = grid[i][(j+1)%ymax] + grid[i][(j-1)%ymax] - 2.0 * grid[i][j]
new_grid[i][j] = grid[i][j] + D * (grid_xx + grid_yy) * dt
return new_grid

You have to rewrite the evolve function from (almost) zero using numpy.
Here the guidelines:
First, grid must be a 2D numpy array, not a list of lists.
Your teacher suggested the roll function: look at its docs and try to understand how it works. roll will solve the problem of finding neighbour entries in the matrix by shifting (or rolling) the matrix over one of the axis. You can then create shifted versions of grid in the four directions and use them, instead of searching for neighbours.
Once you have the shifted grids, you'll see that you will not need the for loops to calculate each cell of new_grid: you can use vectorized calculation, which is faster.
So the code will look like this:
def evolve(grid, dt, D=1.0):
if not isinstance(grid, np.ndarray): #ensuring that is a numpy array.
grid = np.array(grid)
u_grid = np.roll(grid, 1, axis=0)
d_grid = np.roll(grid, -1, axis=0)
r_grid = np.roll(grid, 1, axis=1)
l_grid = np.roll(grid, -1, axis=1)
new_grid = grid + D * (u_grid + d_grid + r_grid + l_grid - 4.0*grid) * dt
return new_grid
With a 1024 x 1024 matrix, each numpy evolve takes (on my machine) ~0.15 seconds to return the new_grid. Your evolve with the for loops takes ~3.85 seconds.

How to fix: Having problems with the resulting array of a for-loop equation

I have the following for-loop:
T = 100
mu = 10
var = 3
e = np.array(np.random.normal(mu, var, T+1))
yt = np.array([10])
for i in range(1,T+1):
ythat = 10 + (0.5 * (yt[i-1] - 10)) + e
yt = np.append(yt,ythat)
len(yt)
The result of len(yt) is 10101, however I am only trying to append the numpy array with 100 iterations of the loop.
Note: My goal with the 'yt[i-1]' in the middle is to get the result of the previous iteration of the 'ythat' formula.

I think you should use a list as calling append on a numpy array will not always give you what you expect but is also pretty computationally expensive. Change your code to:
T = 100
mu = 10
var = 3
e = np.array(np.random.normal(mu, var, T+1))
yt = [10]
for i in range(1,T+1):
ythat = 10 + (0.5 * (yt[i-1] - 10)) + e
yt.append(ythat)
len(yt)
Note that yt cannot be converted to a numpy array as the dimensions do not match for all of its elements.

You need to specify the axis to append on. Without an axis, np.append() will flatten the arrays and concatenate them. To add a new row to the array, use
yt = np.append(yt, ythat, axis=0)
You also need to make the original yt array 2-dimensional:
yt = np.array([10 + e])

Create a matrix using values from a tuple with numpy

I'm trying to create a matrix with values based on x,y values I have stored in a tuple. I use a loop to iterate over the tuple and perform a simple calculation on the data:
import numpy as np
# Trying to fit quadratic equation to the measured dots
N = 6
num_of_params = 3
# x values
x = (1,4,3,5,2,6)
# y values
y = (3.96, 24.96,14.15,39.8,7.07,59.4)
# X is a matrix N * 3 with the x values to the power of {0,1,2}
X = np.zeros((N,3))
Y = np.zeros((N,1))
print X,"\n\n",Y
for i in range(len(x)):
for p in range(num_of_params):
X[i][p] = x[i]**(num_of_params - p - 1)
Y[i] = y[i]
print "\n\n"
print X,"\n\n",Y
Is this can be achieved in an easier way? I'm looking for some way to init the matrix like X = np.zeros((N,3), read_values_from = x)
Is it possible? Is there another simple way?
Python 2.7

Extend array version of x to 2D with a singleton dim (dim with length=1) along the second one using np.newaxis/None. This lets us leverage NumPy broadcasting to get the 2D output in a vectorized manner. Similar philosophy for y.
Hence, the implementation would be -
X = np.asarray(x)[:,None]**(num_of_params - np.arange(num_of_params) - 1)
Y = np.asarray(y)[:,None]
Or use the built-in outer method for np.power to get X that takes care of the array conversion under the hoods -
X = np.power.outer(x, num_of_params - np.arange(num_of_params) - 1)
Alternatively, for Y, use np.expand_dims -
Y = np.expand_dims(y,1)

Combining different elements of array in Numpy while doing vector operations

I have a function f which I am using to evolve a Numpy array z repeatedly. Thus my code looks like this:
for t in range(100):
z = f(z)
However, now I want to combine elements of array while evolving. For example, in simple Python, I would like to do something like this:
N = len(z)
for t in range(100):
for i in range(len(z)):
z_new[i] = f(z[i]) + z[(i-1)%N] + z[(i+1)%N]
z = z_new
How can achieve the same thing in Numpy vector operations so that I wouldn't have to compromise with the great speed that Numpy gives me?
Thanks in advance

You can roll the data back and forth to achieve the same result.
Z = f(z)
Z = np.roll(Z, 1) + z
Z = np.roll(Z, -2) + z
z = np.roll(Z, 1)
I had also first thought about slicing but went with np.roll when I found it.
Prompted by #hpaulj's comment I came up with a slice solution:
q = np.array([1,2,3,4,5])
Q = f(q)
# operate on the middle
Q[1:] += q[:-1]
Q[:-1] += q[1:]
# operate on the ends
Q[0] += q[-1]
Q[-1] += q[0]
q = Q.copy()

Using combinations or another trick to iterate though 3 different arrays?

consider my code
a,b,c = np.loadtxt ('test.dat', dtype='double', unpack=True)
a,b, and c are the same array length.
for i in range(len(a)):
q[i] = 3*10**5*c[i]/100
x[i] = q[i]*math.sin(a)*math.cos(b)
y[i] = q[i]*math.sin(a)*math.sin(b)
z[i] = q[i]*math.cos(a)
I am trying to find all the combinations for the difference between 2 points in x,y,z to iterate this equation (xi-xj)+(yi-yj)+(zi-zj) = r
I use this combination code
for combinations in it.combinations(x,2):
xdist = (combinations[0] - combinations[1])
for combinations in it.combinations(y,2):
ydist = (combinations[0] - combinations[1])
for combinations in it.combinations(z,2):
zdist = (combinations[0] - combinations[1])
r = (xdist + ydist +zdist)
This takes a long time for python for a large file I have and I am wondering if there is a faster way to get my array for r preferably using a nested loop?
Such as
if i in range(?):
if j in range(?):

Since you're apparently using numpy, let's actually use numpy; it'll be much faster. It's almost always faster and usually easier to read if you avoid python loops entirely when working with numpy, and use its vectorized array operations instead.
a, b, c = np.loadtxt('test.dat', dtype='double', unpack=True)
q = 3e5 * c / 100 # why not just 3e3 * c?
x = q * np.sin(a) * np.cos(b)
y = q * np.sin(a) * np.sin(b)
z = q * np.cos(a)
Now, your example code after this doesn't do what you probably want it to do - notice how you just say xdist = ... each time? You're overwriting that variable and not doing anything with it. I'm going to assume you want the squared euclidean distance between each pair of points, though, and make a matrix dists with dists[i, j] equal to the distance between the ith and jth points.
The easy way, if you have scipy available:
# stack the points into a num_pts x 3 matrix
pts = np.hstack([thing.reshape((-1, 1)) for thing in (x, y, z)])
# get squared euclidean distances in a matrix
dists = scipy.spatial.squareform(scipy.spatial.pdist(pts, 'sqeuclidean'))
If your list is enormous, it's more memory-efficient to not use squareform, but then it's in a condensed format that's a little harder to find specific pairs of distances with.
Slightly harder, if you can't / don't want to use scipy:
pts = np.hstack([thing.reshape((-1, 1)) for thing in (x, y, z)])
sqnorms = np.sum(pts ** 2, axis=1)
dists = sqnorms.reshape((-1, 1)) - 2 * np.dot(pts, pts.T) + sqnorms
which basically implements the formula (a - b)^2 = a^2 - 2 a b + b^2, but all vector-like.

Apologies for not posting a full solution, but you should avoid nesting calls to range(), as it will create a new tuple every time it gets called. You are better off either calling range() once and storing the result, or using a loop counter instead.
For example, instead of:
max = 50
for number in range (0, 50):
doSomething(number)
...you would do:
max = 50
current = 0
while current < max:
doSomething(number)
current += 1

Well, the complexity of your calculation is pretty high. Also, you need to have huge amounts of memory if you want to store all r values in a single list. Often, you don't need a list and a generator might be enough for what you want to do with the values.
Consider this code:
def calculate(x, y, z):
for xi, xj in combinations(x, 2):
for yi, yj in combinations(y, 2):
for zi, zj in combinations(z, 2):
yield (xi - xj) + (yi - yj) + (zi - zj)
This returns a generator that computes only one value each time you call the generator's next() method.
gen = calculate(xrange(10), xrange(10, 20), xrange(20, 30))
gen.next() # returns -3
gen.next() # returns -4 and so on