Develop Reference

Element-wise numpy matrix multiplication

I have two numpy arrays A and B, both with the dimension [2,2,n], where n is a very large number. I want to matrix multiply A and B in the first two dimensions to get C, i.e. C=AB, where C has the dimension [2,2,n].
The simplest way to accomplish this is by using for loop, i.e.
for i in range(n):
C[:,:,i] = np.matmul(A[:,:,i],B[:,:,i])
However, this is inefficient since n is very large. What's the most efficient way to do this with numpy?

You can do the following:
new_array = np.einsum('ijk,jlk->ilk', A, B)

What you want is the the default array multiplication in Numpy
In [22]: a = np.arange(8).reshape((2,2,2))+1 ; a[:,:,0], a[:,:,1]
Out[22]:
(array([[1, 3],
[5, 7]]),
array([[2, 4],
[6, 8]]))
In [23]: aa = a*a ; aa[:,:,0], aa[:,:,1]
Out[23]:
(array([[ 1, 9],
[25, 49]]),
array([[ 4, 16],
[36, 64]]))
Notice that I emphasized array because Numpy's arrays look like matrices but are indeed Numpy's ndarrays.
Post Scriptum
I guess that what you really want are matricesarrays with shape (n,2,2), so that you can address individual 2×2 matrices using a single index, e.g.,
In [27]: n = 3
...: a = np.arange(n*2*2)+1 ; a_22n, a_n22 = a.reshape((2,2,n)), a.reshape((n,2,2))
...: print(a_22n[0])
...: print(a_n22[0])
[[1 2 3]
[4 5 6]]
[[1 2]
[3 4]]
Post Post Scriptum
Re semantically correct:
In [13]: import numpy as np
...: n = 3
...: a = np.arange(2*2*n).reshape((2,2,n))+1
...: p = lambda t,a,n:print(t,*(a[:,:,i]for i in range(n)),sep=',\n')
...: p('Original array', a, n)
...: p('Using `einsum("ijk,jlk->ilk", ...)`', np.einsum('ijk,jlk->ilk', a, a), n)
...: p('Using standard multiplication', a*a, n)
Original array,
[[ 1 4]
[ 7 10]],
[[ 2 5]
[ 8 11]],
[[ 3 6]
[ 9 12]]
Using `einsum("ijk,jlk->ilk", ...)`,
[[ 29 44]
[ 77 128]],
[[ 44 65]
[104 161]],
[[ 63 90]
[135 198]]
Using standard multiplication,
[[ 1 16]
[ 49 100]],
[[ 4 25]
[ 64 121]],
[[ 9 36]
[ 81 144]]

method for cells adjacent/connected to vertex in fipy?

Is there such a function or easy method?
The only functions I have found so far are mesh.vertexCoords and mesh.faceVertexIDs but I couldn't figure quit out if they might help me.

As the comments suggest, the vertex to cell data shouldn't usually be required in a finite volume scheme. However, the following is a solution for finding the vertex to cell IDs given the cell to vertex IDs. The cell to vertex data is available in FiPy with the mesh._cellVertexIDs array.
The following uses sparse matrices to represent the cell to vertex link and then a transpose to find the vertex to cell links.
from fipy import Grid2D
import numpy as np
from scipy.sparse import coo_matrix
import itertools
def lists_to_numpy(x):
"""List of lists of different length to Numpy array. See
https://stackoverflow.com/questions/38619143/convert-python-sequence-to-numpy-array-filling-missing-values
>>> print(lists_to_numpy([[0], [0, 1], [0, 1, 2]]))
array([[ 0, -1, -1],
[ 0, 1, -1],
[ 0, 1, 2]])
"""
return np.array(list(itertools.zip_longest(*x, fillvalue=-1))).T
def invert_sparse_bool(x, mshape):
"""Invert a sparse bool matrix represented by a 2D array and return as
inverted 2D array.
>>> a = numpy.array([[0, 2], [1, 3], [0, 3], [3, 4]])
>>> print(invert_sparse_bool(a, (4, 5)))
[[ 0 2 -1]
[ 1 -1 -1]
[ 0 -1 -1]
[ 1 2 3]
[ 3 -1 -1]]
"""
arr1 = np.indices(x.shape)[0]
arr2 = np.stack((arr1, x), axis=-1)
arr3 = np.reshape(arr2, (-1, 2))
lists = coo_matrix(
(np.ones(len(arr3), dtype=int),
(arr3[:, 0], arr3[:, 1])),
shape=mshape
).tolil().T.rows
return lists_to_numpy(lists)
m = Grid2D(nx=3, ny=3)
cellVertexIDs = m._cellVertexIDs.swapaxes(0, 1)
vertexCellIDs = invert_sparse_bool(
cellVertexIDs,
(m.numberOfCells, m.numberOfVertices)
)
print('cellVertexIDs:', m._cellVertexIDs)
print('vertexCellIDs:', vertexCellIDs)
Note that the m._cellVertexIDs are of shape (maxNumberOfVerticesPerCell, numberOfCells), but it's a little easier to implement when they are reshaped. The new vertexCellIDs array are shaped as (numberOfVertices, maxNumberOfCellsPerVertex). The vertexCellIDs do need a fill value as each vertex won't be connected to the same number of cells.
The output from this is
cellVertexIDs: [[ 1 5 4 0]
[ 2 6 5 1]
[ 3 7 6 2]
[ 5 9 8 4]
[ 6 10 9 5]
[ 7 11 10 6]
[ 9 13 12 8]
[10 14 13 9]
[11 15 14 10]]
vertexCellIDs: [[ 0 -1 -1 -1]
[ 0 1 -1 -1]
[ 1 2 -1 -1]
[ 2 -1 -1 -1]
[ 0 3 -1 -1]
[ 0 1 3 4]
[ 1 2 4 5]
[ 2 5 -1 -1]
[ 3 6 -1 -1]
[ 3 4 6 7]
[ 4 5 7 8]
[ 5 8 -1 -1]
[ 6 -1 -1 -1]
[ 6 7 -1 -1]
[ 7 8 -1 -1]
[ 8 -1 -1 -1]]
which makes sense to me for a 3x3 mesh with 9 cells and 16 vertices and an ordered numbering system for both cells and vertices (left to right, bottom to top).

Convert matrix to tuples

Say I generate a sequence of values, tile them by the range provided and then increment each value in each row by that current row ID, then mask some values outside of a desired range like below:
>>> range = 5
>>> matrix = np.arange(-5, 10, 1)
>>> matrix = np.tile(matrix, (range, 1))
>>> matrix = np.add(matrix, np.arange(0, range)[:, None])
>>> matrix = ma.masked_outside(matrix, 0, 10)
[[-- -- -- -- -- 0 1 2 3 4 5 6 7 8 9]
[-- -- -- -- 0 1 2 3 4 5 6 7 8 9 10]
[-- -- -- 0 1 2 3 4 5 6 7 8 9 10 --]
[-- -- 0 1 2 3 4 5 6 7 8 9 10 -- --]
[-- 0 1 2 3 4 5 6 7 8 9 10 -- -- --]]
How would you best convert the above output to a matrix of the format [non-masked value, row-id], i.e.:
[0,0], [1, 0], [2,0] ... [10, 4]
Also, is the original code too wasteful to achieve the final desired step?

Playing around with your matrix I produced this:
In [50]: np.stack((matrix.compressed(), np.where(~matrix.mask)[0]),1)
Out[50]:
array([[ 0, 0],
[ 1, 0],
[ 2, 0],
[ 3, 0],
[ 4, 0],
[ 5, 0],
[ 6, 0],
[ 7, 0],
[ 8, 0],
[ 9, 0],
....
We could probably skip the masked array step, creating the mask directly. The compressed for example is produced by matrix.data[~matrix.mask].
In [52]: mask = ~matrix.mask
In [53]: data = matrix.data
In [54]: np.stack((data[mask], np.where(mask)[0]), 1)

python - numpy fancy broadcasting for special case riddle

I want to do some forces calculations between vertices and because the forces are symmetrical I have a list of vertice-pairs that need those forces added. I am sure it's possible with fancy indexing, but I really just can get it to work with a slow python for-loop. for symmetric reasons, the right-hand side of the index array needs a negative sign when adding the forces.
consider you have the vertice index array:
>>> I = np.array([[0,1],[1,2],[2,0]])
I = [[0 1]
[1 2]
[2 0]]
and the x,y forces array for each pair:
>>> F = np.array([[3,6],[4,7],[5,8]])
F = [[3 6]
[4 7]
[5 8]]
the wanted operation could be described as:
"vertice #0 sums the force vectors (3,6) and (-5,-8),
vertice #1 sums the force vectors (-3,-6) and (4,7),
vertice #2 sums the force vectors (-4,-7) and (5,8)"
Desired results:
[ 3 6 ] [ 0 0 ] [-5 -8 ] [-2 -2 ] //resulting force Vertice #0
A = [-3 -6 ] + [ 4 7 ] + [ 0 0 ] = [ 1 1 ] //resulting force Vertice #1
[ 0 0 ] [-4 -7 ] [ 5 8 ] [ 1 1 ] //resulting force Vertice #2
edit:
my ugly for-loop solution:
import numpy as np
I = np.array([[0,1],[1,2],[2,0]])
F = np.array([[3,6],[4,7],[5,8]])
A = np.zeros((3,2))
A_x = np.zeros((3,2))
A_y = np.zeros((3,2))
for row in range(0,len(F)):
A_x[I[row][0],0]= F[row][0]
A_x[I[row][1],1]= -F[row][0]
A_y[I[row][0],0]= F[row][1]
A_y[I[row][1],1]= -F[row][1]
A = np.hstack((np.sum(A_x,axis=1).reshape((3,1)),np.sum(A_y,axis=1).reshape((3,1))))
print(A)
A= [[-2. -2.]
[ 1. 1.]
[ 1. 1.]]

Your current "push-style" interpretation of I is
For row-index k in I, take the forces from F[k] and add/subtract them to out[I[k], :]
I = np.array([[0,1],[1,2],[2,0]])
out = numpy.zeros_like(F)
for k, d in enumerate(I):
out[d[0], :] += F[k]
out[d[1], :] -= F[k]
out
# array([[-2, -2],
# [ 1, 1],
# [ 1, 1]])
However you can also change the meaning of I on its head and make it "pull-style", so it says
For row-index k in I, set vertex out[k] to be the difference of F[I[k]]
I = np.array([[0,2],[1,0],[2,1]])
out = numpy.zeros_like(F)
for k, d in enumerate(I):
out[k, :] = F[d[0], :] - F[d[1], :]
out
# array([[-2, -2],
# [ 1, 1],
# [ 1, 1]])
In which case the operation simplifies quite easily to mere fancy indexing:
out = F[I[:, 0], :] - F[I[:, 1], :]
# array([[-2, -2],
# [ 1, 1],
# [ 1, 1]])

You can preallocate an array to hold the shuffled forces and then use the index like so:
>>> N = I.max() + 1
>>> out = np.zeros((N, 2, 2), F.dtype)
>>> out[I, [1, 0]] = F[:, None, :]
>>> np.diff(out, axis=1).squeeze()
array([[-2, -2],
[ 1, 1],
[ 1, 1]])
or, equivalently,
>>> out = np.zeros((2, N, 2), F.dtype)
>>> out[[[1], [0]], I.T] = F
>>> np.diff(out, axis=0).squeeze()
array([[-2, -2],
[ 1, 1],
[ 1, 1]])

The way I understand the question, the values in the I array represent the vortex number, or the name of the vortex. They are not an actual positional index. Based on this thought, I have a different solution that uses the original I array. It does not quite come without loops, but should be OK for a reasonable number of vertices:
I = np.array([[0,1],[1,2],[2,0]])
F = np.array([[3,6],[4,7],[5,8]])
pos = I[:, 0]
neg = I[:, 1]
A = np.zeros_like(F)
unique = np.unique(I)
for i, vortex_number in enumerate(unique):
A[i] = F[np.where(pos==vortex_number)] - F[np.where(neg==vortex_number)]
# produces the expected result
# [[-2 -2]
# [ 1 1]
# [ 1 1]]
Maybe this loop can also be replaced by some numpy magic.

How to sort numpy array by absolute value of a column?

What I have now:
import numpy as np
# 1) Read CSV with headers
data = np.genfromtxt("big.csv", delimiter=',', names=True)
# 2) Get absolute values for column in a new ndarray
new_ndarray = np.absolute(data["target_column_name"])
# 3) Append column in new_ndarray to data
# I'm having trouble here. Can't get hstack, concatenate, append, etc; to work
# 4) Sort by new column and obtain a new ndarray
data.sort(order="target_column_name_abs")
I would like:
A solution for 3): To be able to add this new "abs" column to the original ndarray or
Another approach to be able to sort a csv file by the absolute values of a column.

Here is a way to do it.
First, let's create a sample array:
In [39]: a = (np.arange(12).reshape(4, 3) - 6)
In [40]: a
Out[40]:
array([[-6, -5, -4],
[-3, -2, -1],
[ 0, 1, 2],
[ 3, 4, 5]])
Ok, lets say
In [41]: col = 1
which is the column we want to sort by,
and here is the sorting code - using Python's sorted:
In [42]: b = sorted(a, key=lambda row: np.abs(row[col]))
Let's convert b from list to array, and we have:
In [43]: np.array(b)
Out[43]:
array([[ 0, 1, 2],
[-3, -2, -1],
[ 3, 4, 5],
[-6, -5, -4]])
Which is the array with the rows sorted according to
the absolute value of column 1.

Here's a solution using pandas:
In [117]: import pandas as pd
In [118]: df = pd.read_csv('test.csv')
In [119]: df
Out[119]:
a b
0 1 -3
1 2 2
2 3 -1
3 4 4
In [120]: df['c'] = abs(df['b'])
In [121]: df
Out[121]:
a b c
0 1 -3 3
1 2 2 2
2 3 -1 1
3 4 4 4
In [122]: df.sort_values(by='c')
Out[122]:
a b c
2 3 -1 1
1 2 2 2
0 1 -3 3
3 4 4 4