Develop Reference

Python numpy: Simplify operation on multiple matrices

I have 3 numpy matrices:
One contains pixels positions in X (x_pos), another pixel positions in Y (y_pos) and a last one containing pixel values (p_value)
I would like to use these 3 matrices to build a results image
With loops I have this result:
#Resulting image
res = np.zeros((128,128,3), dtype = np.uint8)
for i in range(x_pos.shape[0]):
for j in range(x_pos.shape[1]):
# Get coordinates
x = x_pos[i][j]
y = y_pos[i][j]
res[y,x] = p_value[i][j]
With large matrices (2048*2048) this code already takes a lot of time. Is it possible to optimize this code without using a nested loop?
I specify that the positions in the pos_x and pos_y matrices do not necessarily follow each other, there may be holes or duplicate values

It should be possible using np.meshgrid
i = np.arange(0, x.shape[0])
j = np.arange(0, x.shape[1])
i_1, j_1 = np.meshgrid(i, j, indexing='ij')
res[y_1.ravel(),x_1.ravel()] = p_value[i_1.ravel(),j_1.ravel()]

First use consistent numpy 2d array indexing:
x = x_pos[i,j]
y = y_pos[i,j]
res[y,x] = p_value[i,j]
Now instead of scalar i,j use arrays
i = np.arange(n); j = np.arange(m)
You didn't provida [mcve] so I won't try to demonstrate that th

Thanks to #hpaulj and #ai2ys answer the problem is solved.
Here is a comparison of the results in terms of execution speed:
import numpy as np
import cv2
import time
m_size = 4096
m_x = np.random.randint(0,m_size,(m_size,m_size), dtype = np.uint16)
m_y = np.random.randint(0,m_size,(m_size,m_size), dtype = np.uint16)
p_value = np.ones((m_size,m_size), dtype = np.uint8)
#Meshgrid method:
out = np.zeros((m_size,m_size),dtype=np.uint8)
start = time.time()
i = np.arange(0, m_x.shape[0])
j = np.arange(0, m_x.shape[1])
i_1, j_1 = np.meshgrid(i, j, indexing='ij')
out[m_x.ravel(),m_y.ravel()] = p_value[i_1.ravel(),j_1.ravel()]
end = time.time()
print("Meshgrid: {} s".format(end - start))
#No for loop method:
out = np.zeros((m_size,m_size),dtype=np.uint8)
start = time.time()
i = np.arange(m_x.shape[0])
j = np.arange(m_y.shape[1])
x = m_x[i,j]
y = m_y[i,j]
out[x,y] = p_value[i,j]
end = time.time()
print("No loop: {} s".format(end - start))
#For loop method:
out = np.zeros((m_size,m_size),dtype=np.uint8)
start = time.time()
for i in range(m_x.shape[0]):
for j in range(m_y.shape[1]):
x = m_x[i,j]
y = m_y[i,j]
out[x,y] = p_value[i,j]
end = time.time()
print("Nested loop: {} s".format(end - start))
#Output:
Meshgrid: 0.4837045669555664 s
No loop: 0.3600656986236572 s
Nested loop: 13.10097336769104 s

Python compiler giving IndexError when trying to find the similarity matrix

I am trying to make a movie recommendation system which requires me to find the user-user similarity matrix for the top 100 users.
On running the code I get:
similarMatrix[row] = top100_similar
IndexError: index 663 is out of bounds for axis 0 with size 617
Code:
def getUser_UserSimilarity(sparseMatrix, top = 100):
startTimestamp20 = datetime.now()
row_index, col_index = sparseMatrix.nonzero() #this will give indices of rows in "row_index" and indices of columns in
#"col_index" where there is a non-zero value exist.
rows = np.unique(row_index)
similarMatrix = np.zeros(61700).reshape(617,100) # 617*100 = 61700. As we are building similarity matrix only
#for top 100 most similar users.
timeTaken = []
howManyDone = 0
for row in rows[:top]:
howManyDone += 1
startTimestamp = datetime.now().timestamp() #it will give seconds elapsed
sim = cosine_similarity(sparseMatrix.getrow(row), sparseMatrix).ravel()
top100_similar_indices = sim.argsort()[-top:]
top100_similar = sim[top100_similar_indices]
similarMatrix[row] = top100_similar
timeforOne = datetime.now().timestamp() - startTimestamp
timeTaken.append(timeforOne)
if howManyDone % 20 == 0:
print("Time elapsed for {} users = {}sec".format(howManyDone, (datetime.now() - startTimestamp20)))
print("Average Time taken to compute similarity matrix for 1 user = "+str(sum(timeTaken)/len(timeTaken))+"seconds")
fig = plt.figure(figsize = (12,8))
plt.plot(timeTaken, label = 'Time Taken For Each User')
plt.plot(np.cumsum(timeTaken), label='Cumulative Time')
plt.legend(loc='upper left', fontsize = 15)
plt.xlabel('Users', fontsize = 20)
plt.ylabel('Time(Seconds)', fontsize = 20)
plt.tick_params(labelsize = 15)
plt.show()
return similarMatrix
simMatrix = getUser_UserSimilarity(TrainUISparseData, 100)
Please tell me where exactly I need to make the changes.

The error is due to the following line
similarMatrix = np.zeros(61700).reshape(617,100)
Your similarMatrix is of smaller dimension than your sparseMatrix. Thats why you are getting index error.
You need to make the dimensions of similarMatrix equal to the dimensions of sparseMatrix. So modify the code as below
similarMatrix = np.zeros(sparseMatrix.shape[0]*100).reshape(sparseMatrix.shape[0],100)
Or for more simple structure
n_cols = 100
n_rows = sparseMatrix.shape[0]
similarMatrix = np.zeros(n_rows*n_cols).reshape(n_rows, n_cols)

scipy.integrate.solve_ivp diverges on a state space simulation well finished in MATLAB

I tried to simulate a state space model with MATLAB ode45, then I tried the same work in Python with scipy.integrate.solve_ivp. As it is obviously shown in this post pictures. Python simulation diverges for no good reason. The solvers message is "Required step size is less than spacing between numbers." but adding time steps is not a solution.
Here is the MATLAB code for the time interval of half a second following a link for the plot of 173rd state:
[1]: https://i.stack.imgur.com/mMdNQ.png
C_static=csvread('C_static.csv');
M_static=csvread('M_static.csv');
B_static=csvread('B_static.csv');
CY_static=csvread('CY_static.csv');
DY_static=csvread('DY_static.csv');
dynamoterm = csvread('dynamoterm.csv');
C_static=0*C_static;
n2panto=dynamoterm(6,1);
n2cw=dynamoterm(6,2);
k_dynamic = KCdyna(0,dynamoterm);
K_total = K_static;
K_total(n2cw+1:n2panto+1,n2cw+1:n2panto+1)=K_total(n2cw+1:n2panto+1,n2cw+1:n2panto+1)+k_dynamic;
A_static = [0*K_static, eye(length(B_static));
-M_static\K_static, -M_static\C_static];
Bu = [0*B_static;
M_static\B_static];
inc0 = -A_static\Bu;
M_in=inv(M_static);
M_cwp=M_in(:,n2cw+1:n2panto+1);
timer=tic;
[T, Y] = ode45(#(t,X) Asol(X,t,A_static,M_cwp,Bu,n2cw,n2panto,dynamoterm),[0,0.5],inc0);
output=[CY_static,0*CY_static]*Y'+DY_static*ones(1,length(T));
figure
plot(T,output(173,:));
stopwatch=toc(timer);
function dx=Asol(X,t,A_static,M_cwp,Bu,n2cw,n2panto,dynamoterm)
[k_dynamic]=KCdyna(t,dynamoterm);
A=A_static;
A(n2panto+4:2*(n2panto+3),n2cw+1:n2panto+1)=A_static(n2panto+4:2*(n2panto+3),n2cw+1:n2panto+1)-M_cwp*k_dynamic;
dx=A*X+Bu;
end
[![MATLAB simulation plot of the 173rd state][1]][1]
Here is my similar work in Python for the time interval of half a second following a link for the plot of 173rd state:
[2]: https://i.stack.imgur.com/LOg2j.png
from KCdyna2 import K_dyn
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
# Imports matrices via .csv file
M = np.genfromtxt('Excel\dyn\M_static.csv', delimiter=',')
C = np.genfromtxt('Excel\dyn\C_static.csv', delimiter=',')
C = np.zeros(np.shape(C))
K_static = np.genfromtxt('Excel\dyn\K_static.csv', delimiter=',')
B = np.genfromtxt('Excel\dyn\B_static.csv', delimiter=',')
dyn_trm = np.genfromtxt('Excel\dyn\dynamoterm.csv', delimiter=',')
# Slice addresses
n2cw = int(dyn_trm[5, 1]) # Slice beginning
n2panto = int(dyn_trm[5, 0]) # Slice finishing
# Time interval for solution
time_interval = [0, 0.5]
times = np.linspace(time_interval[0], time_interval[1], 50000)
M_inv = np.linalg.inv(M)
K_total = K_static
K_total[n2cw:n2panto + 1,
n2cw:n2panto + 1] += K_dyn(0, dyn_trm)
# System dynamics matrix
A_static = np.block([[np.zeros((len(M_inv), len(M_inv)), dtype='uint8'), np.eye(len(M_inv), dtype='uint8')],
[np.matmul(-M_inv, K_total), np.matmul(-M_inv, C)]])
Bu = np.block([[np.zeros((len(B), 1), dtype='uint8')],
[np.matmul(M_inv, B).reshape(len(B), 1)]])
inc0 = np.matmul(-np.linalg.inv(A_static), Bu)
M_inv = np.linalg.inv(M)
M_cwp = M_inv[:, n2cw:n2panto + 1]
def SttSpcEq(t, x, M_cwp, A_st, Bu, dynamoterm, n2cw,n2panto):
K_dynamic = K_dyn(t, dynamoterm)
A = A_st
A[n2panto + 3:2*(n2panto+3),
n2cw:n2panto + 1] -= np.matmul(M_cwp, K_dynamic)
return (np.matmul(A, x.reshape(len(Bu), 1)) + Bu).reshape(len(Bu), )
soln = solve_ivp(SttSpcEq,
time_interval,
inc0.reshape(len(inc0),),
method='RK45', t_eval=times, args=(M_cwp, A_static, Bu, dyn_trm, n2cw, n2panto))
print(soln.message)
plt.plot(soln.t, soln.y[172])
plt.show() ```
[![Python simulation plot of the 173rd state][2]][2]

Python's fsolve not working

I'm currently trying to find the intercept of 2 equations from my code (pasted below). I'm using fsolve and have used it successfully in one part but I can't get it to work for the second.
Confusingly it's not showing up an error, if you paste this code into your notebook and run it you'll see 2 grphs, on the first graph there's a line at an angle which should be stopping at the eqm line.
The section which wont work is def q_eqm(x_q). Thank you for your help
import numpy as np
import scipy.optimize as opt
import matplotlib.pyplot as plt
AC_LK = np.array([4.02232,1206.53,220.291])
AC_HK = np.array([4.0854,1348.77,219.976])
P_Tot = 1 # Bara
N_Size = 11 # 1001 = 0.1% accuracy for xA
xf = 0.7
q = 0.7
xA = np.linspace(0,1,N_Size)
yA = np.linspace(0.00,0.00,N_Size)
T = np.linspace(0.00,0.00,N_Size)
x = np.array([xA[0:N_Size],yA[0:N_Size],T[0:N_Size]]) # x[xA,yA,T]
F = np.empty((1))
def xA_T(N):
xA_Ant = x[0,N]
def P_Ant(T):
PA = pow(10,AC_LK[0]-(AC_LK[1]/(T+AC_LK[2])))*xA_Ant
PB = pow(10,AC_HK[0]-(AC_HK[1]/(T+AC_HK[2])))*(1-xA_Ant)
F[0] = P_Tot - (PA + PB)
return F
return x
TGuess = [100]
T = opt.fsolve(P_Ant,TGuess)
x[2,N] = T
return x
for N in range(0,len(xA)):
xA_T(N)
x[1,N] = pow(10,AC_LK[0]-(AC_LK[1]/(x[2,N]+AC_LK[2])))*x[0,N]/P_Tot
q_int = ((-q*0)/(1-q)) + (xf/(1-q))
Eqm_Poly = np.polyfit(x[0,0:N_Size], x[1,0:N_Size], 6)
q_Poly = np.polyfit([xf,0], [xf,q_int], 1)
F = np.empty((1))
def q_Eqm(x_q):
y_q = q_Poly[0]*x_q + q_Poly[1]
eqm_y = (Eqm_Poly[0]*pow(x_q,6)+Eqm_Poly[1]*pow(x_q,5)+Eqm_Poly[2]*pow(x_q,4)+Eqm_Poly[3]*pow(x_q,3)+Eqm_Poly[4]*pow(x_q,2)+Eqm_Poly[5]*pow(x_q,1)+Eqm_Poly[6]*pow(x_q,0))
F[0] = y_q - eqm_y
return F
x_qGuess = [0]
x_q = opt.fsolve(q_Eqm,x_qGuess)
print(x,Eqm_Poly,x_q,q_int)
plt.plot(x[0,0:N_Size],x[1,0:N_Size],'k-',linewidth=1)
plt.plot([xf,xf],[0,xf],'b-',linewidth=1)
plt.plot([xf,x_q],[xf,(q_Poly[0]*x_q + q_Poly[1])],'r-',linewidth=1)
plt.legend(['Eqm','Feed'])
plt.xlabel('xA')
plt.ylabel('yA')
plt.xlim([0.00, 1])
plt.ylim([0.00, 1])
plt.savefig('x.png')
plt.savefig('x.eps')
plt.show()
plt.plot(x[0,0:N_Size],x[2,0:N_Size],'r--',linewidth=3)
plt.plot(x[1,0:N_Size],x[2,0:N_Size],'b--',linewidth=3)
plt.legend(['xA','yA'])
plt.xlabel('Mol Frac')
plt.ylabel('Temp degC')
plt.xlim([0, 1])
plt.savefig('Txy.png')
plt.savefig('Txy.eps')
plt.show()

The answer turns out to be relatively simple:
#F = np.empty((1)) # remove this
def q_Eqm(x_q):
y_q = q_Poly[0]*x_q + q_Poly[1]
eqm_y = (Eqm_Poly[0]*pow(x_q,6)+Eqm_Poly[1]*pow(x_q,5)+Eqm_Poly[2]*pow(x_q,4)+Eqm_Poly[3]*pow(x_q,3)+Eqm_Poly[4]*pow(x_q,2)+Eqm_Poly[5]*pow(x_q,1)+Eqm_Poly[6]*pow(x_q,0))
return y_q - eqm_y
The original code defines a global F, which is modified in the function and then returned. So in each iteration the function returns different values but they are the same object. This seems to confuse fsolve (I guess it internally stores references to the results rather than values). Removing this F and simply returning the result of the subtraction resolves the problem.

Numpy convert list of 3D variable size volumes to 4D array

I'm working on a neural network where I am augmenting data via rotation and varying the size of each input volume.
Let me back up, the input to the network is a 3D volume. I generate variable size 3D volumes, and then pad each volume with zero's such that the input volume is constant. Check here for an issue I was having with padding (now resolved).
I generate a variable size 3D volume, append it to a list, and then convert the list into a numpy array. At this point, padding hasn't occured so converting it into a 4D tuple makes no sense...
input_augmented_matrix = []
label_augmented_matrix = []
for i in range(n_volumes):
if i % 50 == 0:
print ("Augmenting step #" + str(i))
slice_index = randint(0,n_input)
z_max = randint(5,n_input)
z_rand = randint(3,5)
z_min = z_max - z_rand
x_max = randint(75, n_input_x)
x_rand = randint(60, 75)
x_min = x_max - x_rand
y_max = randint(75, n_input_y)
y_rand = randint(60, 75)
y_min = y_max - y_rand
random_rotation = randint(1,4) * 90
for j in range(2):
temp_volume = np.empty((z_rand, x_rand, y_rand))
k = 0
for z in range(z_min, z_max):
l = 0
for x in range(x_min, x_max):
m = 0
for y in range(y_min, y_max):
if j == 0:
#input volume
try:
temp_volume[k][l][m] = input_matrix[z][x][y]
except:
pdb.set_trace()
else:
#ground truth volume
temp_volume[k][l][m] = label_matrix[z][x][y]
m = m + 1
l = l + 1
k = k + 1
temp_volume = np.asarray(temp_volume)
temp_volume = np.rot90(temp_volume,random_rotation)
if j == 0:
input_augmented_matrix.append(temp_volume)
else:
label_augmented_matrix.append(temp_volume)
input_augmented_matrix = np.asarray(input_augmented_matrix)
label_augmented_matrix = np.asarray(label_augmented_matrix)
The dimensions of input_augmented_matrix at this point is (N,)
Then I pad with the following code...
for i in range(n_volumes):
print("Padding volume #" + str(i))
input_augmented_matrix[i] = np.lib.pad(input_augmented_matrix[i], ((0,n_input_z - int(input_augmented_matrix[i][:,0,0].shape[0])),
(0,n_input_x - int(input_augmented_matrix[i][0,:,0].shape[0])),
(0,n_input_y - int(input_augmented_matrix[i][0,0,:].shape[0]))),
'constant', constant_values=0)
label_augmented_matrix[i] = np.lib.pad(label_augmented_matrix[i], ((0,n_input_z - int(label_augmented_matrix[i][:,0,0].shape[0])),
(0,n_input_x - int(label_augmented_matrix[i][0,:,0].shape[0])),
(0,n_input_y - int(label_augmented_matrix[i][0,0,:].shape[0]))),
'constant', constant_values=0)
At this point, the dimensions are still (N,) even though every element of the list is constant. For example input_augmented_matrix[0] = input_augmented_matrix[1]
Currently I just loop through and create a new array, but it takes too long and I would prefer some sort of method that automates this. I do it with the following code...
input_4d = np.empty((n_volumes, n_input_z, n_input_x, n_input_y))
label_4d = np.empty((n_volumes, n_input_z, n_input_x, n_input_y))
for i in range(n_volumes):
print("Converting to 4D tuple #" + str(i))
for j in range(n_input_z):
for k in range(n_input_x):
for l in range(n_input_y):
input_4d[i][j][k][l] = input_augmented_matrix[i][j][k][l]
label_4d[i][j][k][l] = label_augmented_matrix[i][j][k][l]
Is there a cleaner and faster way to do this?

As I understood this part
k = 0
for z in range(z_min, z_max):
l = 0
for x in range(x_min, x_max):
m = 0
for y in range(y_min, y_max):
if j == 0:
#input volume
try:
temp_volume[k][l][m] = input_matrix[z][x][y]
except:
pdb.set_trace()
else:
#ground truth volume
temp_volume[k][l][m] = label_matrix[z][x][y]
m = m + 1
l = l + 1
k = k + 1
You just want to do this
temp_input = input_matrix[z_min:z_max, x_min:x_max, y_min:y_max]
temp_label = label_matrix[z_min:z_max, x_min:x_max, y_min:y_max]
and then
temp_input = np.rot90(temp_input, random_rotation)
temp_label = np.rot90(temp_label, random_rotation)
input_augmented_matrix.append(temp_input)
label_augmented_matrix.append(temp_label)
Here
input_augmented_matrix[i] = np.lib.pad(
input_augmented_matrix[i],
((0,n_input_z - int(input_augmented_matrix[i][:,0,0].shape[0])),
(0,n_input_x - int(input_augmented_matrix[i][0,:,0].shape[0])),
(0,n_input_y - int(input_augmented_matrix[i][0,0,:].shape[0]))),
'constant', constant_values=0)
Better to do this, because shape property gives you size of array by all dimensions
ia_shape = input_augmented_matrix[i].shape
input_augmented_matrix[i] = np.lib.pad(
input_augmented_matrix[i],
((0, n_input_z - ia_shape[0]),
(0, n_input_x - ia_shape[1])),
(0, n_input_y - ia_shape[2]))),
'constant',
constant_values=0)
I guess now you're ready to refactor the last part of your code with magic indexing of NumPy.
My common suggestions:
use functions for repeated parts of code to avoid such indents like in your cascade of loops;
if you need so lot of nested loops, think about recursion, if you can't deal without them;
explore abilities of NumPy in official documentation: they're really exciting ;) For example, indexing is helpful for this task;
use PyLint and Flake8 packages to inspect quality of your code.
Do you want to write neural network by yourself, or you just want to solve some patterns recognition task? SciPy library may contain what you need and it's based on NumPy.