Develop Reference

Split a 2d array creating array from row to row values with unitary displacement

I want to split a 2D array this way:
Example:
From this 4x4 2D array:
np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]])
Create these five 2x2 2D arrays, with unitary displacement (shift):
np.array([[1,2],[3,4]])
np.array([[4,5],[6,7]])
np.array([[7,8],[9,10]])
np.array([[10,11],[12,13]])
np.array([[13,14],[15,16]])
In a general case, from an NXN 2D array (square arrays) create Y 2D arrays of MXM shape, as many as possible.
Just to be more precise: to create the output array, not necessarily it will be made of all values from the row.
Example:
From a 2D 8x8 array, with values from 1 to 64, if I want to split this array in 2D 2x2 arrays, the first row from 8x8 array is a row from 1 to 8, and the first output 2D 2x2 array will be np.array([[1,2],[3,4]]), and the second output 2D 2x2 array will be np.array([[4,5],[6,7]])... It continues until the last output 2D array, that will be np.array([[61,62],[63,64]]). Look that each 2D 2x2 array was not filled with all the values from the row (CORRECT). And that exists a unitary displacement (shift) from previous array to next array.
There is a Numpy method that do this?
MSeifert answered here (How to split an 2D array, creating arrays from "row to row" values) a question that solves almost 95% of this question, except the unitary displacement (shift) part.
So, from the 4x4 2D array example:
np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]])
Instead of create these FOUR 2x2 2D arrays (without unitary shift/displacement):
np.array([[1,2],[3,4]])
np.array([[5,6],[7,8]])
np.array([[9,10],[11,12]])
np.array([[13,14],[15,16]])
Create these FIVE 2x2 2D arrays (with unitary shift/displacement):
np.array([[1,2],[3,4]])
np.array([[4,5],[6,7]])
np.array([[7,8],[9,10]])
np.array([[10,11],[12,13]])
np.array([[13,14],[15,16]])
And, of course, it should work for the general case of, given a square NXN 2D array, to create Y MXM square 2D arrays.
Example: from a 60x60 square 2d array, create Y MXM square 2D arrays (10x10, for example).
Plus: I need to know what is the rule that relates the number of points of the original square 2D array (4x4 2D array, in the example), with points of mini square 2D arrays (2X2 2D arrays, in the example). In the case, given 16 points (4x4 2D array), it is possible create 5 2x2 2D arrays (each one with 4 points).

The condition for the subarrays to exactly fit is (M+1)*(M-1) divides (N+1)*(N-1), the number of subarrays you can put is the quotient of these numbers. Note that these numbers are equal to M*M-1 and N*N-1. In this form the rule also applies to non square matrices.
Examples
M N M*M-1 N*N-1 Y
-----------------------------
3 5 8 24 3
3 7 8 48 6
5 7 24 48 2
4 11 15 120 8
Implementation: Please note that this returns overlapping views into the original array. If you want to modify them you may want to make a copy. Also note that this implementation fits as many subsquares as fit, any leftover elements in the larger matrix are dropped.
Update: I've added two functions which calculate given N all possible M and vice versa.
Output:
# Testing predictions ...
# ok
# M = 105
# solutions: [105, 1273, 1377, 4135, 4239, 5407, 5511, 5513]
# this pattern repeats at offsets 5512, 11024, 16536, ...
# N = 1000001
# solutions: [2, 3, 4, 5, 7, 9, 11, 31, 49, 1000001]
# example N, M = (5, 3)
# [[[ 0 1 2]
# [ 3 4 5]
# [ 6 7 8]]
# [[ 8 9 10]
# [11 12 13]
# [14 15 16]]
# [[16 17 18]
# [19 20 21]
# [22 23 24]]]
Code:
import numpy as np
import sympy
import itertools as it
import functools as ft
import operator as op
def get_subsquares(SqNN, M0, M1=None):
M1 = M0 if M1 is None else M1
N0, N1 = SqNN.shape
K = (N0*N1-1) // (M0*M1-1)
SqNN = SqNN.ravel()
s, = SqNN.strides
return np.lib.stride_tricks.as_strided(SqNN, (K, M0, M1),
(s*(M0*M1-1), s*M1, s))
def get_M_for_N(N):
"""Given N return all possible M
"""
assert N >= 2
f = 1 + (N & 1)
factors = sympy.factorint((N+1)//f)
factors.update(sympy.factorint((N-1)//f))
if f == 2:
factors[2] += 2
factors = [ft.reduce(op.mul, fs) for fs in it.product(
*([a**k for k in range(n+1)] for a, n in factors.items()))]
return [fs + 1 for fs in sorted(set(factors) & set(fs - 2 for fs in factors)) if (N*N-1) % (fs * (fs+2)) == 0]
def get_N_for_M(M):
"""Given M return all possible N in the form period, smallest
smallest is a list of all solutions between M and M*M if M is even
and between M and (M*M+1) / 2 if M is odd, all other solutions can be
obtained by adding multiples of period
"""
assert M >= 2
f = 1 + (M & 1)
factors = sympy.factorint((M+1)//f)
factors.update(sympy.factorint((M-1)//f))
factors = [k**v for k, v in factors.items()]
rep = (M+1)*(M-1) // f
f0 = [ft.reduce(op.mul, fs) for fs in it.product(*zip(it.repeat(1), factors))]
f1 = [rep // (f*a) for a in f0]
inv = [f if b==1 else f*b + 2 if a==1 else 2 * sympy.mod_inverse(a, b)
for a, b in zip(f1, f0)]
if f==1:
inv[1:-1] = [a%b for a, b in zip(inv[1:-1], f0[1:-1])]
return rep, sorted(a*b - 1 for a, b in zip(f1, inv))
def test_predict(N):
def brute_force(a, b):
return [i for i in range(a, b) if (i*i-1) % (a*a-1) == 0]
for x in range(2, N+1):
period, pred = get_N_for_M(x)
assert brute_force(x, period*4+2) \
== [a + b for b in range(0, 4*period, period) for a in pred]
def brute_force(b):
return [i for i in range(2, b+1) if (b*b-1) % (i*i-1) == 0]
for x in range(2, N+1):
assert brute_force(x) == get_M_for_N(x)
print('ok')
# test
print("Testing predictions ...")
test_predict(200)
print()
# examples
M = 105
period, pred = get_N_for_M(M)
print(f"M = {M}")
print(f"solutions: {pred}")
print(f"this pattern repeats at offsets {period}, {2*period}, {3*period}, ...")
print()
N = 1000001
pred = get_M_for_N(N)
print(f"N = {N}")
print(f"solutions: {pred}")
print()
N, M = 5, 3
SqNN = np.arange(N*N).reshape(N, N)
print(f"example N, M = ({N}, {M})")
print(get_subsquares(SqNN, M))

Compute x**k with x, k being arrays of arbitrary dimensionality

I have two numpy arrays: One array x with shape (n, a0, a1, ...) and one array k with shape (n, b0, b1, ...). I would like to compute and array of exponentials such that the output has dimension (a0, a1, ..., b0, b1, ...) and
out[i0, i1, ..., j0, j1, ...] == prod(x[:, i0, i1, ...] ** k[:, j0, j1, ...])
If there is only one a_i and one b_j, broadcasting does the trick via
import numpy
x = numpy.random.rand(2, 31)
k = numpy.random.randint(1, 10, size=(2, 101))
out = numpy.prod(x[..., None]**k[:, None], axis=0)
If x has a few dimensions more, more Nones have to be added:
x = numpy.random.rand(2, 31, 32, 33)
k = numpy.random.randint(1, 10, size=(2, 101))
out = numpy.prod(x[..., None]**k[:, None, None, None], axis=0)
If x has a few dimensions more, more Nones have to be added at other places:
x = numpy.random.rand(2, 31)
k = numpy.random.randint(1, 10, size=(2, 51, 51))
out = numpy.prod(x[..., None, None]**k[:, None], axis=0)
How to make the computation of out generic with respect to the dimensionality of the input arrays?

Here's one using reshaping of the two arrays so that they are broadcastable against each other and then performing those operations and prod reduction along the first axis -
k0_shp = [k.shape[0]] + [1]*(x.ndim-1) + list(k.shape[1:])
x0_shp = list(x.shape) + [1]*(k.ndim-1)
out = (x.reshape(x0_shp) ** k.reshape(k0_shp)).prod(0)
Here's another way to reshape both inputs to 3D allowing one singleton dim per input and such that they are broadcastable against each other, perform prod reduction to get 2D array, then reshape back to multi-dim array -
s = x.shape[1:] + k.shape[1:] # output shape
out = (x.reshape(x.shape[0],-1,1)**k.reshape(k.shape[0],1,-1)).prod(0).reshape(s)
It must be noted that reshaping merely creates a view into the input array and as such is virtually free both memory-wise and performance-wise.

Without understanding fully the math of what you're doing, it seems that you need a constant number of None's for the number of dimensions of each x and k.
does something like this work?
out = numpy.prod(x[[...]+[None]*(k.ndim-1)]**k[[slice(None)]+[None]*(x.ndim-1)])
Here are the slices separately so they're a bit easier to read:
x[ [...] + [None]*(k.ndim-1) ]
k[ [slice(None)] + [None]*(x.ndim-1) ]
Compatibility Note:
[...] seems to only be valid in python 3.x If you are using 2.7 (I haven't tested lower) substitute [Ellipsis] instead:
x[ [Ellipsis] + [None]*(k.ndim-1) ]

Numpy array and column extracted from a matrix, different shape

I'm trying to do an integration with numpy:
A = n.trapz(B,C)
but I have some issues with B and C shapes
B is a filled array inizialized with numpy zeros function
B=np.zeros((N,1))
C is a column extracted from a matrix, always inizialized with numpy:
C = D[:,0]
D = np.zeros((N,2))
the problem is that:
n.shape(B) # (N,1)
n.shape(C) # (N,)
how can I manage this?

Try
B = np.zeros(N)
np.trapz(B, C)
Also, you np.trapz accepts multi-dimensional arrays, so arrays of shape (N, 1) are ok; you just need to specify an axis to handle it properly.
B = np.zeros((N, 1))
C = D[:, 0]
np.trapz(B, C.reshape(N, 1), axis=1)

Efficiently generating a Cauchy matrix from two Numpy arrays

A Cauchy matrix (Wikipedia article) is a matrix determined by two vectors (arrays of numbers). Given two vectors x and y, the Cauchy matrix C generated by them is defined entry-wise as
C[i][j] := 1/(x[i] - y[j])
Given two Numpy arrays x and y, what is an efficient way to generate a Cauchy matrix?

This is the most efficient way I found, using array broadcasting to take advantage of vectorization.
1.0 / (x.reshape((-1,1)) - y)
Edit: #HYRY and #shx2 have suggested that, instead of x.reshape((-1,1)), which makes a copy, you can use x[:,np.newaxis], which returns a view of the same array. #HYRY also suggests 1.0/np.subtract.outer(x,y), which is slightly slower for me but maybe more explicit.
Example:
>>> x = numpy.array([1,2,3,4]) #x
>>> y = numpy.array([5,6,7]) #y
>>>
>>> #transpose x, to nx1
... x = x.reshape((-1,1))
>>> x
array([[1],
[2],
[3],
[4]])
>>>
>>> #array of differences x[i] - y[j]
... #an nx1 array minus a 1xm array is an nxm array
... diff_matrix = x-y
>>> diff_matrix
array([[-4, -5, -6],
[-3, -4, -5],
[-2, -3, -4],
[-1, -2, -3]])
>>>
>>> #apply the multiplicative inverse to each entry
... cauchym = 1.0/diff_matrix
>>> cauchym
array([[-0.25 , -0.2 , -0.16666667],
[-0.33333333, -0.25 , -0.2 ],
[-0.5 , -0.33333333, -0.25 ],
[-1. , -0.5 , -0.33333333]])
I tried a few other methods, all of which were significantly slower.
This is the naive approach, which costs list comprehension:
cauchym = numpy.array([[ 1.0/(x_i-y_j) for y_j in y] for x_i in x])
This one generates the matrix as a 1-dimensional array (saving the cost of nested Python lists) and reshapes it to a matrix afterward. It also moves the division to a single Numpy operation:
cauchym = 1.0/numpy.array([(x_i-y_j) for x_i in x for y_j in y]).reshape([len(x),len(y)])
Using numpy.repeat and numpy.tile (which respectively tile the array horizontally and vertically). This way makes unnecessary copies:
lenx = len(x)
leny = len(y)
xm = numpy.repeat(x,leny) #the i'th row is s_i
ym = numpy.tile(y,lenx)
cauchym = (1.0/(xm-ym)).reshape([lenx,leny]);

I created a function hope it helps u to understand in a better way.
# Creating a function in order to form a cauchy matrix
def cauchy_matrix(arr1,arr2):
"""
Enter two arrays in order to get a cauchy matrix.The input array should be a 1-D array.
arr1 = First 1-D array
arr2 = Second 1-D array
It returns the cauchy matrix having shape equal to m*n, where m is size of arr1 and n is size of arr2.
"""
my_list = []
try:
for i in range(len(arr1)):
for j in range(len(arr2)):
z = 1/(arr1[i]-arr2[j])
my_list.append(z)
return np.array(my_list).reshape(arr1.shape[0],arr2.shape[0])
except ZeroDivisionError:
print("Check if both the arrays has '0' as one of it's element. One array can have a zero but both the arrays having '0' is not acceptable!")

numpy array each element multiplication with matrix

I have a matrix
A = [[ 1. 1.]
[ 1. 1.]]
and two arrays (a and b), every array contains 20 float numbers How can I multiply the using formula:
( x' = A * ( x )
y' ) y
Is this correct? m = A * [a, b]

Matrix multiplication with NumPy arrays can be done with np.dot.
If X has shape (i,j) and Y has shape (j,k) then np.dot(X,Y) will be the matrix product and have shape (i,k). The last axis of X and the second-to-last axis of Y is multiplied and summed over.
Now, if a and b have shape (20,), then np.vstack([a,b]) has shape (2, 20):
In [66]: np.vstack([a,b]).shape
Out[66]: (2, 20)
You can think of np.vstack([a, b]) as a 2x20 matrix with the values of a on the first row, and the values of b on the second row.
Since A has shape (2,2), we can perform the matrix multiplication
m = np.dot(A, np.vstack([a,b]))
to arrive at an array of shape (2, 20).
The first row of m contains the x' values, the second row contains the y' values.
NumPy also has a matrix subclass of ndarray (a special kind of NumPy array) which has convenient syntax for doing matrix multiplication with 2D arrays. If we define A to be a matrix (rather than a plain ndarray which is what np.array(...) creates), then matrix multiplication can be done with the * operator.
I show both ways (with A being a plain ndarray and A2 being a matrix) below:
import numpy as np
A = np.array([[1.,1.],[1.,1.]])
A2 = np.matrix([[1.,1.],[1.,1.]])
a = np.random.random(20)
b = np.random.random(20)
c = np.vstack([a,b])
m = np.dot(A, c)
m2 = A2 * c
assert np.allclose(m, m2)