Develop Reference

How to calculate pairwise inside a matrix

how to perform pair operations on more than 2 lists
Example
If my matrix have 2 lists (L,M) I calculate the dot product and the results are [[M.M M.L , L.M LL]]
How to calculate the same operation for matrices that have more than 2 lists in a way that the result is a symmetric matrice
x = np.array([[1, 3, 5],[1, 4, 5],[2,6,10]])
How to perform pairwise analysis ?

Solution 1: An alternative to the brute force below is using np.einsum, but it is not simple to use that function. This link has an explanation on how to use it, https://ajcr.net/Basic-guide-to-einsum/. See Solution 2 on how matrix is defined.
np.einsum('ij,jk', matrix,matrix.T)
Out[35]:
array([[35, 38],
[38, 42]])
matrix = np.array([L, M, N]) # matrix with 3 lists
np.einsum('ij,jk', matrix,matrix.T)
Out[37]:
array([[ 35, 38, 70],
[ 38, 42, 76],
[ 70, 76, 140]])
Solution 2 for smaller matrices. Explanation below:
def dot_pairwise(matrix):
return [[np.dot(i, j) for j in matrix] for i in matrix]
dot_pairwise(matrix)
Explanation:
import numpy as np
L = np.array([1, 3, 5])
M = np.array([1, 4, 5])
N = np.array([2, 6, 10])
matrix = np.array([L, M, N]) # matrix with 3 lists
# matrix = np.array([L, M]) # matrix with 2 lists to replicate your example
# Initialize an empty result list
result = []
for i in matrix:
row = [] # Initialize an empty row
for j in matrix:
# Calculate the dot product between the ith and jth lists using numpy.dot
print(i,j) # to print the matrices
dot_product = np.dot(i, j)
row.append(dot_product) # Add the dot product to the row
result.append(row) # Add the row to the result
print(result) # [[LL, LM, LN], [ML, MM, MN], [NL, NM, NN]]
This is the result using L, M matrix:
[1 3 5] [1 3 5] LL
[1 3 5] [1 4 5] LM
[1 4 5] [1 3 5] ML
[1 4 5] [1 4 5] MM
[[35, 38], [38, 42]] # dot products

Alternative from this answer, slightly changed:
np.tensordot(x, x, axes=(1,1))

Construct equivalent transform for vectorized Matrix

Equivalent transform for vectorized solution
For a given symmetric 4x4 matrix Q and a 3x4 matrix P the 3x3 matrix C is obtained through
C=P # Q # P.T
It can be shown that the output C will be symmetric again. The same problem can be formulated using only the unique elements in Q and C exploiting their symmetry. To do so, the matrices are vectorized as seen below.
I want to construct a matrix B that maps the vectorized matrices onto each other like so:
c = B # q
B must be a 6x10 and should be constructable from P only. How can I get B from P?
I tried this, but it doesnt seem to work. Maybe someone has experienced a similar problem?
import numpy as np
def vectorize(A, ord='c'):
"""
Symmetric matrix to vector e.g:
[[1, 2, 3],
[2, 4, 5],
[3, 5, 6]] -> [1, 2, 3, 4, 5, 6] (c-order, row-col)
-> [1, 2, 4, 3, 5, 6] (f-order, col-row)
"""
# upper triangle mask
m = np.triu(np.ones_like(A, dtype=bool)).flatten(order=ord)
return A.flatten(order=ord)[m]
def B(P):
B = np.zeros((6, 10))
counter = 0
# the i,j entry in C depends on the i, j columns in P
for i in range(3):
for j in range(i, 3):
coeffs = np.outer(P[i], P[j])
B[counter] = vectorize(coeffs)
counter += 1
return B
if __name__ == '__main__':
# original transform
P = np.arange(12).reshape((3, 4))
# calculated transform for vectorized matrix
_B = B(P)
# some random symmetric matrix
Q = np.array([[1, 2, 3, 4],
[2, 5, 6, 7],
[3, 6, 8, 9],
[4, 7, 9, 10]])
# if B is an equivilant transform to P, these should be similar
C = P # Q # P.T
c = _B # vectorize(Q)
print(f"q: {vectorize(Q)}\n"
f"C: {vectorize(C)}\n"
f"c: {c}")
Output:
q: [ 1 2 3 4 5 6 7 8 9 10]
C: [ 301 949 2973 1597 4997 8397]
c: [ 214 542 870 1946 3154 5438] <-- not the same

import numpy as np
def vec_from_mat(A, order='c'):
"""
packs the unique elements of symmetric matrix A into a vector
:param A: symmetric matrix
:return:
"""
return A[np.triu_indices(A.shape[0])].flatten(order=order)
def B_from_P(P):
"""
returns a 6x10 matrix that maps the 10 unique elements of a symmetric 4x4 matrix Q on the 6 unique elements of a
3x3 matrix C to linearize the equation C=PTQP to c=Bv
:param P: 3x4 matrix
:return: B with shape (6, 10)
"""
n, m = P.shape
b1, b2 = (n * (n + 1) // 2), (m * (m + 1) // 2)
B = np.zeros((b1, b2))
for a, (i, j) in enumerate(zip(*np.triu_indices(n))):
coeffs = np.outer(P[i], P[j])
# collect coefficients from lower and upper triangle of symmetric matrix
B[a] = vec_from_mat(coeffs) + vec_from_mat(np.triu(coeffs.T, k=1))
return B

Remove and append middle column from a 3D numpy array

Suppose I have a 3D numpy array A, say given below:
A = np.array( [[[1,2,3], [4,5,6]] , [[7,8,9] , [10,11,12], [13,14,15]] ] , ndmin = 3 )
The only thing given about A is that it is a 3D arrays which is an array of arbitrary number of 2D arrays, where each 2D array is an array of arbitrary number of 1D arrays, and each 1D array has exactly 3 elements.
I want to remove the middle element from each 1D array from this 3D array, basically get the new array A1, and the removed column as X given below:
A1 = np.array( [[[1,3], [4,6]] , [[7,9] , [10,12], [13,15]] ] , ndmin = 3 )
X = np.array( [ [[2],[5]], [[8],[11],[14]] ], ndmin = 3 )
I want to write a function that given A it outputs (A1, X) and another function which given (A1, X) outputs A. I believe it should be possible to write the first function via array slicing, but I am not able to do so. Also how do I write the second function.

For you ragged array, it is better to store in a list of np.arrays with shape n by 3:
A = [np.array([[1,2,3],
[4,5,6]]) ,
np.array([[7,8,9],
[10,11,12],
[13,14,15]])]
Now you could:
def remove_middle(arr):
x = [a[:, 1] for a in arr]
arr_new = [np.delete(a, 1, axis = 1) for a in arr]
return arr_new, x
def insert_middle(arr, x):
return [np.concatenate([a[:, :1], xx.reshape(-1, 1), a[:, 1:]], axis = 1) for a, xx in zip(arr, x)]
remove_middle(A)
([array([[1, 3],
[4, 6]]),
array([[ 7, 9],
[10, 12],
[13, 15]])],
[array([2, 5]), array([ 8, 11, 14])])
insert_middle(*remove_middle(A))
# gets back the original A
[array([[1, 2, 3],
[4, 5, 6]]),
array([[ 7, 8, 9],
[10, 11, 12],
[13, 14, 15]])]

Without the ndmin=3 argument I can solve your answer using two nested list comprehensions in which the first indexes the middle argument and the second one deletes the middle argument of the inner arrays.
import numpy
A = np.array([[[1,2,3], [4,5,6]] , [[7,8,9] , [10,11,12], [13,14,15]]])
middle = [[array1d[1] for array1d in array2d] for array2d in A]
without_middle = [[np.delete(array1d, 1) for array1d in array2d] for array2d in A]

With your strange data, this absurd-looking quintuple-nested list-comprehension is the best I could come up with :P
A1 = [[[[[e for i, e in enumerate(d) if i != 1] for d in c] for c in b] for b in a] for a in A]
X = [[[[[e for i, e in enumerate(d) if i == 1] for d in c] for c in b] for b in a] for a in A]
Output:
>>> A1
[[[[[1, 3], [4, 6]], [[7, 9], [10, 12], [13, 15]]]]]
>>> X
[[[[[2], [5]], [[8], [11], [14]]]]]

Here is a solution. Note that lists are returned as raw python list which you can use as you want.
I changed your definition of A to a more suitable object.
import numpy as np
def f(A):
A1 = A.tolist()
X = []
for i in range(len(A1)):
temp = []
for j in range(len(A1[i])):
temp.append([A1[i][j].pop(1)])
X.append(temp)
return (A1, X)
def g(A1, X):
A = A1
for i in range(len(A1)):
for j in range(len(A1[i])):
A[i][j].insert(1, X[i][j][0])
return A
def main():
#A = np.array( [ [ [1,2,3], [4,5,6] ] , [ [7,8,9] , [10,11,12], [13,14,15] ] ] , ndmin = 3 )
A = np.asarray([ [ [1,2,3], [4,5,6] ] , [ [7,8,9] , [10,11,12], [13,14,15] ] ])
B, X = f(A)
print(g(B,X))
if __name__ == '__main__':
main()
Finally please note that this is one solution among many possible alternatives.

More "pythonic" way to show a 4d matrix in 2d

I would like to plot a 4d matrix as a 2d matrix with indices:
[i][j][k][l] --> [i * nj + j][ k * nl + l]
I have a working version here.
This is working as I want, but it's not very elegant. I looked into "reshape" but this is not exactly what I'm looking for, or perhaps I am using it incorrectly.
Given a 4d array "r" with shape (100000,4), the relevant snippet I want to replace is:
def transform(i,j,k,l, s1, s2):
return [i * s1 + j, k * s2 + l]
nx = 5
ny = 11
iedges=np.linspace(0,100, nx)
jedges=np.linspace(0, 20, ny)
bins = ( iedges,jedges,iedges,jedges )
H, edges = np.histogramdd(r, bins=bins )
H2 = np.zeros(( (nx-1)*(ny-1),(nx-1)*(ny-1)))
for i in range(nx-1):
for j in range(ny-1):
for k in range(nx-1):
for l in range(ny-1):
x,y = transform(i,j,k,l,ny-1,ny-1)
H2[x][y] = H[i][j][k][l]
In this case the values of H2 will correspond to the values of H, but the entry i,j,k,l will display as i*ny + j, k * ny + l.
Example plot:

Are you sure reshape doesn't work?
I ran your code on a small random r. The nonzero terms of H are:
In [13]: np.argwhere(H)
Out[13]:
array([[0, 9, 3, 1],
[1, 1, 1, 2],
[1, 2, 1, 3],
[2, 2, 2, 3],
[3, 1, 1, 8]])
and for the transformed H2:
In [14]: np.argwhere(H2)
Out[14]:
array([[ 9, 31],
[11, 12],
[12, 13],
[22, 23],
[31, 18]])
And one of the H indices transforms to H2 indices with:
In [16]: transform(0,9,3,1,4,10)
Out[16]: [9, 31]
If I simply reshape H, I get the same array as H2:
In [17]: H3=H.reshape(40,40)
In [18]: np.argwhere(H3)
Out[18]:
array([[ 9, 31],
[11, 12],
[12, 13],
[22, 23],
[31, 18]])
In [19]: np.allclose(H2,H3)
Out[19]: True
So without delving into the details of your code, it looks to me like a simple reshape.

Looks like you can calculate i,j,k,l from x,y? This should be something like:
from functools import partial
def get_ijkl(x, y, s1, s2):
# "Reverse" of `transform`
i, j = divmod(x, s1)
k, l = divmod(y, s2)
return (i, j, k, l)
def get_2d_val(x, y, s1, s2, four_dim_array):
return four_dim_array[get_ijkl(x, y, s1, s2)]
smaller_shape = ((nx-1)*(ny-1), (nx-1)*(ny-1))
Knowing this there are several approaches possible:
numpy.fromfunction:
H3 = np.fromfunction(
partial(get_2d_val, s1=ny-1, s2=ny-1, four_dim_array=H),
shape=smaller_shape,
dtype=int,
)
assert np.all(H2 == H3)
by indexing:
indices_to_take = np.array([
[list(get_ijkl(x, y, ny-1, ny-1)) for x in range(smaller_shape[0])] for y in range(smaller_shape[1])
]).transpose()
H4 = H[tuple(indices_to_take)]
assert np.all(H2 == H4)
as answered by #hpaulj you can simply reshape array and it will be faster. But If you have some different transform and can calculate appropriate "reverse" function then using fromfunction or custom indexing will get useful

What is a pythonic way of finding maximum values and their indices for moving subarrays for numpy ndarray?

I have numpy ndarrays which could be 3 or 4 dimensional. I'd like to find maximum values and their indices in a moving subarray window with specified strides.
For example, suppose I have a 4x4 2d array and my moving subarray window is 2x2 with stride 2 for simplicity:
[[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9,10,11,12],
[13,14,15,16]].
I'd like to find
[[ 6 8],
[14 16]]
for max values and
[(1,1), (3,1),
(3,1), (3,3)]
for indices as output.
Is there a concise, efficient implementation for this for ndarray without using loops?

Here's a solution using stride_tricks:
def make_panes(arr, window):
arr = np.asarray(arr)
r,c = arr.shape
s_r, s_c = arr.strides
w_r, w_c = window
if c % w_c != 0 or r % w_r != 0:
raise ValueError("Window doesn't fit array.")
shape = (r / w_r, c / w_c, w_r, w_c)
strides = (w_r*s_r, w_c*s_c, s_r, s_c)
return np.lib.stride_tricks.as_strided(arr, shape, strides)
def max_in_panes(arr, window):
w_r, w_c = window
r, c = arr.shape
panes = make_panes(arr, window)
v = panes.reshape((-1, w_r * w_c))
ix = np.argmax(v, axis=1)
max_vals = v[np.arange(r/w_r * c/w_c), ix]
i = np.repeat(np.arange(0,r,w_r), c/w_c)
j = np.tile(np.arange(0, c, w_c), r/w_r)
rel_i, rel_j = np.unravel_index(ix, window)
max_ix = i + rel_i, j + rel_j
return max_vals, max_ix
A demo:
>>> vals, ix = max_in_panes(x, (2,2))
>>> print vals
[[ 6 8]
[14 16]]
>>> print ix
(array([1, 1, 3, 3]), array([1, 3, 1, 3]))
Note that this is pretty untested, and is designed to work with 2d arrays. I'll leave the generalization to n-d arrays to the reader...