Im trying to solve linear systems of the form Ax = b where A is an nxn matrix of real numbers and b a 1xn vector of real numbers, using the A = LU algorithm.
I've implemented the necessary functions, but Im not sure in which function or functions the problem lies.
import numpy as np
def forward_sub(L, b):
"""Given a lower triangular matrix L and right-side vector b,
compute the solution vector y solving Ly = b."""
y = []
for i in range(len(b)):
y.append(b[i])
for j in range(i):
y[i]=y[i]-(L[i, j]*y[j])
y[i] = y[i]/L[i, i]
return y
def backward_sub(U, y):
"""Given a lower triangular matrix U and right-side vector y,
compute the solution vector x solving Ux = y."""
x = [0 for i in range(len(U))]
for i in range(len(U)-1, 0, -1):
x[i] = U[i, i]/y[i]
for j in range (i-1, 0, -1):
U[i, i] += U[j, i]*x[i]
return x
def lu_factor(A):
#LU decompostion using Doolittles method
L = np.zeros_like(A)
U = np.zeros_like(A)
N = np.size(A,0)
for k in range(N):
L[k, k] = 1
U[k, k] = (A[k, k] - np.dot(L[k, :k], U[:k, k])) / L[k, k]
for j in range(k+1, N):
U[k, j] = (A[k, j] - np.dot(L[k, :k], U[:k, j])) / L[k, k]
for i in range(k+1, N):
L[i, k] = (A[i, k] - np.dot(L[i, :k], U[:k, k])) / U[k, k]
return (L, U)
def lu_solve(L, U, b):
# Step 1: Solve Uy = b using forward substitution
# Step 2: Solve Lx = y using backward substitution
y = forward_sub(L,b)
x = backward_sub(U,y)
return x
def linear_solve(A, b):
# ...
L, U = lu_factor(A)
x = lu_solve(L,U,b)
return x
b = [6,-4,27]
A = np.matrix([[1,1,1],[0,2,5],[2,5,-1]])
print(linear_solve(A,b))
Choosing A and b as above gives x = [0,-0.5,-0.42] as my solution vector, however it should give x = [5,3,-2]
A is an integer matrix. That makes the L and U integer matrices as well, but the right results are:
L:
[[1. 0. 0. ]
[0. 1. 0. ]
[2. 1.5 1. ]]
U:
[[ 1. 1. 1. ]
[ 0. 2. 5. ]
[ 0. 0. -10.5]]
Some fractional values are required. In general that is the case for LU decomposition even if the input consists of integers. There is some division going on after all.
Changing the data type fixes that. For example:
A = np.matrix([[1.,1,1],
[0,2,5],
[2,5,-1]])
backward_sub is broken, I'm not sure how exactly but in any case it's a strange implementation. This one seems to work:
def backward_sub(U, y):
"""Given a lower triangular matrix U and right-side vector y,
compute the solution vector x solving Ux = y."""
x = np.zeros_like(y)
for i in range(len(x), 0, -1):
x[i-1] = (y[i-1] - np.dot(U[i-1, i:], x[i:])) / U[i-1, i-1]
return x
Result is [ 5. 3. -2.], try it one ideone
Related
I try to code an LUP (or PLU it's the same) factorization in python. I have a code which works for small matrix (under a 4x4 size). However when I have tried it with a random generated matrix the decomposition has failed.
import numpy as np
def LUP_factorisation(A):
"""Find P, L and U : PA = LU"""
U = A.copy()
shape_a = U.shape
n = shape_a[0]
L = np.eye(n)
P = np.eye(n)
for i in range(n):
print(U)
k = i
comp = abs(U[i, i])
for j in range(i, n):
if abs(U[j, i]) > comp:
k = j
comp = abs(U[j, i])
line_u = U[k, :].copy()
U[k, :] = U[i, :]
U[i, :] = line_u
print(U)
line_p = P[k, :].copy()
P[k, :] = P[i, :]
P[i, :] = line_p
for j in range(i + 1, n):
g = U[j, i] / U[i, i]
L[j, i] = g
U[j, :] -= g * U[i, :]
return L, U, P
if __name__ == "__main__":
A = np.array(
[[1.0, 2.2, 58, 9.5, 42.65], [6.56, 58.789954, 4.45, 23.465, 6.165], [7.84516, 8.9864, 96.546, 4.654, 7.6514],
[45.65, 47.985, 1.56, 3.9845, 8.6], [455.654, 102.615, 63.965, 5.6, 9.456]])
L, U, P = LUP_factorisation(A)
print(L # U)
print(P # A)
With the example I gave it works: we have PA = LU. But when i do for example :
A = np.random.rand(10, 10)
Then, i don't obtain a good result because PA is different of LU. Any ideas ? Thanks.
As #MattTimmermans writes you should swap rows in both L and U.
Normally this is implicitly handled by storing LU in A and then the swaps are automatically applied to both L and U. See https://en.wikipedia.org/wiki/LU_decomposition#C_code_example
But you have split them so you have to add
line_l = L[k, :].copy()
L[k, :] = L[i, :]
L[i, :] = line_l
Only testing it with diagonally dominant matrices is really bad; and only testing linear algebra routines with random matrices is known to be bad as their properties are very specific - and not "random". See work by Trefethen and his students, e.g. http://dspace.mit.edu/handle/1721.1/14322
The goal of testing should be to find bugs - not to make test-cases so simple that it works.
Make sure that the diagonal of the input matrix A is dominant. So add some value to the diagonal of A, e.g.
A = A + np.eye(A.shape)
or
A = A + 100* np.eye(A.shape)
I hope that helps !
I'm currently trying to implement the Householder based QR decomposition for rectangular matrices as described in http://eprints.ma.man.ac.uk/1192/1/qrupdating_12nov08.pdf (pages 3, 4, 5).
Apparently I got some of the pseudocode wrong though, since (1) my results differ from numpy.qr.linalg() and (2) the matrix R produced by my routines is not an upper triangular matrix.
My code (also available under https://pyfiddle.io/fiddle/afcc2e0e-0857-4cb2-adb5-06ff9b80c9d3/?i=true)
import math
import argparse
import numpy as np
from typing import Union
def householder(alpha: float, x: np.ndarray) -> Union[np.ndarray, int]:
"""
Computes Householder vector for alpha and x.
:param alpha:
:param x:
:return:
"""
s = math.pow(np.linalg.norm(x, ord=2), 2)
v = x
if s == 0:
tau = 0
else:
t = math.sqrt(alpha * alpha + s)
v_one = alpha - t if alpha <= 0 else -s / (alpha + t)
tau = 2 * v_one * v_one / (s + v_one * v_one)
v /= v_one
return v, tau
def qr_decomposition(A: np.ndarray, m: int, n: int) -> Union[np.ndarray, np.ndarray]:
"""
Applies Householder-based QR decomposition on specified matrix A.
:param A:
:param m:
:param n:
:return:
"""
H = []
R = A
Q = A
I = np.eye(m, m)
for j in range(0, n):
# Apply Householder transformation.
x = A[j + 1:m, j]
v_householder, tau = householder(np.linalg.norm(x), x)
v = np.zeros((1, m))
v[0, j] = 1
v[0, j + 1:m] = v_householder
res = I - tau * v * np.transpose(v)
R = np.matmul(res, R)
H.append(res)
return Q, R
m = 10
n = 8
A = np.random.rand(m, n)
q, r = np.linalg.qr(A)
Q, R = qr_decomposition(A, m, n)
print("*****")
print(Q)
print(q)
print("-----")
print(R)
print(r)
So I'm unclear about how to introduce zeroes to my R matrix/about which part of my code is incorrect. I'd be happy about any pointers!
Thanks a lot for your time.
There were a bunch of problems/missing details in the notes you linked to. After consulting a few other sources (including this very useful textbook), I was able to come up with a working implementation of something similar.
The working algorithm
Heres the code for a working version of qr_decomposition:
import numpy as np
from typing import Union
def householder(x: np.ndarray) -> Union[np.ndarray, int]:
alpha = x[0]
s = np.power(np.linalg.norm(x[1:]), 2)
v = x.copy()
if s == 0:
tau = 0
else:
t = np.sqrt(alpha**2 + s)
v[0] = alpha - t if alpha <= 0 else -s / (alpha + t)
tau = 2 * v[0]**2 / (s + v[0]**2)
v /= v[0]
return v, tau
def qr_decomposition(A: np.ndarray) -> Union[np.ndarray, np.ndarray]:
m,n = A.shape
R = A.copy()
Q = np.identity(m)
for j in range(0, n):
# Apply Householder transformation.
v, tau = householder(R[j:, j])
H = np.identity(m)
H[j:, j:] -= tau * v.reshape(-1, 1) # v
R = H # R
Q = H # Q
return Q[:n].T, R[:n]
m = 5
n = 4
A = np.random.rand(m, n)
q, r = np.linalg.qr(A)
Q, R = qr_decomposition(A)
with np.printoptions(linewidth=9999, precision=20, suppress=True):
print("**** Q from qr_decomposition")
print(Q)
print("**** Q from np.linalg.qr")
print(q)
print()
print("**** R from qr_decomposition")
print(R)
print("**** R from np.linalg.qr")
print(r)
Output:
**** Q from qr_decomposition
[[ 0.5194188817843675 -0.10699353671401633 0.4322294754656072 -0.7293293270703678 ]
[ 0.5218635773595086 0.11737804362574514 -0.5171653705211056 0.04467925806590414]
[ 0.34858177783013133 0.6023104248793858 -0.33329256746256875 -0.03450824948274838]
[ 0.03371048915852807 0.6655221685383623 0.6127023580593225 0.28795294754791 ]
[ 0.5789790833500734 -0.411189947884951 0.24337120818874305 0.618041080584351 ]]
**** Q from np.linalg.qr
[[-0.5194188817843672 0.10699353671401617 0.4322294754656068 0.7293293270703679 ]
[-0.5218635773595086 -0.11737804362574503 -0.5171653705211053 -0.044679258065904115]
[-0.3485817778301313 -0.6023104248793857 -0.33329256746256863 0.03450824948274819 ]
[-0.03371048915852807 -0.665522168538362 0.6127023580593226 -0.2879529475479097 ]
[-0.5789790833500733 0.41118994788495106 0.24337120818874317 -0.6180410805843508 ]]
**** R from qr_decomposition
[[ 0.6894219296137802 1.042676051151294 1.3418719684631446 1.2498925815126485 ]
[ 0.00000000000000000685 0.7076056836914905 0.29883043386651403 0.41955370595004277 ]
[-0.0000000000000000097 -0.00000000000000007292 0.5304551654027297 0.18966088433421135 ]
[-0.00000000000000000662 0.00000000000000008718 0.00000000000000002322 0.6156558913022807 ]]
**** R from np.linalg.qr
[[-0.6894219296137803 -1.042676051151294 -1.3418719684631442 -1.2498925815126483 ]
[ 0. -0.7076056836914905 -0.29883043386651376 -0.4195537059500425 ]
[ 0. 0. 0.53045516540273 0.18966088433421188]
[ 0. 0. 0. -0.6156558913022805 ]]
This version of qr_decomposition near exactly reproduces the output of np.linalg.qr. The differences are commented on below.
Numerical precision of the output
The values in the outputs of np.linalg.qr and qr_decomposition match to high precision. However, the combination of computations that qr_decomposition uses to produce the zeros in R don't exactly cancel, so the zeros aren't actually quite equal to zero.
It turns out that np.linalg.qr isn't doing any fancy floating point tricks to ensure that the zeros in its output are 0.0. It just calls np.triu, which forcibly sets those values to 0.0. So to achieve the same results, just change the return line in qr_decomposition to:
return Q[:n].T, np.triu(R[:n])
Signs (+/-) in the output
Some of the +/- signs in Q and R are different in the outputs of np.linalg.qr and qr_decomposition, but this isn't really an issue as there are many valid choices for the signs (see this discussion of the uniqueness of Q and R). You can exactly match the sign convention that np.linalg.qr by using an alternative algorithm to generate v and tau:
def householder_vectorized(a):
"""Use this version of householder to reproduce the output of np.linalg.qr
exactly (specifically, to match the sign convention it uses)
based on https://rosettacode.org/wiki/QR_decomposition#Python
"""
v = a / (a[0] + np.copysign(np.linalg.norm(a), a[0]))
v[0] = 1
tau = 2 / (v.T # v)
return v,tau
Exactly matching the output of np.linalg.qr
Putting it all together, this version of qr_decomposition will exactly match the output of np.linalg.qr:
import numpy as np
from typing import Union
def qr_decomposition(A: np.ndarray) -> Union[np.ndarray, np.ndarray]:
m,n = A.shape
R = A.copy()
Q = np.identity(m)
for j in range(0, n):
# Apply Householder transformation.
v, tau = householder_vectorized(R[j:, j, np.newaxis])
H = np.identity(m)
H[j:, j:] -= tau * (v # v.T)
R = H # R
Q = H # Q
return Q[:n].T, np.triu(R[:n])
m = 5
n = 4
A = np.random.rand(m, n)
q, r = np.linalg.qr(A)
Q, R = qr_decomposition(A)
with np.printoptions(linewidth=9999, precision=20, suppress=True):
print("**** Q from qr_decomposition")
print(Q)
print("**** Q from np.linalg.qr")
print(q)
print()
print("**** R from qr_decomposition")
print(R)
print("**** R from np.linalg.qr")
print(r)
Output:
**** Q from qr_decomposition
[[-0.10345123000824041 0.6455437884382418 0.44810714367794663 -0.03963544711256745 ]
[-0.55856415402318 -0.3660716543156899 0.5953932791844518 0.43106504879433577 ]
[-0.30655198880585594 0.6606757192118904 -0.21483067305535333 0.3045011114089389 ]
[-0.48053620675695174 -0.11139783377793576 -0.6310958848894725 0.2956864520726446 ]
[-0.5936453158283703 -0.01904935140131578 -0.016510508076204543 -0.79527388379824 ]]
**** Q from np.linalg.qr
[[-0.10345123000824041 0.6455437884382426 0.44810714367794663 -0.039635447112567376]
[-0.5585641540231802 -0.3660716543156898 0.5953932791844523 0.4310650487943359 ]
[-0.30655198880585594 0.6606757192118904 -0.21483067305535375 0.30450111140893893 ]
[-0.48053620675695186 -0.1113978337779356 -0.6310958848894725 0.29568645207264455 ]
[-0.5936453158283704 -0.01904935140131564 -0.0165105080762043 -0.79527388379824 ]]
**** R from qr_decomposition
[[-1.653391466100325 -1.0838054573405895 -1.0632037969249921 -1.1825735233596888 ]
[ 0. 0.7263519982452554 0.7798481878600413 0.5496287509656425 ]
[ 0. 0. -0.26840760341581243 -0.2002757085967938 ]
[ 0. 0. 0. 0.48524469321440966]]
**** R from np.linalg.qr
[[-1.6533914661003253 -1.0838054573405895 -1.0632037969249923 -1.182573523359689 ]
[ 0. 0.7263519982452559 0.7798481878600418 0.5496287509656428]
[ 0. 0. -0.2684076034158126 -0.2002757085967939]
[ 0. 0. 0. 0.4852446932144096]]
Aside from the inevitable rounding error in the trailing digits, the outputs now match.
The Wikipedia entry for the Arnoldi method provides a Python example that produces basis of the Krylov subspace of a matrix A. Supposedly, if A is Hermitian (i.e. if A == A.conj().T) then the Hessenberg matrix h generated by this algorithm is tridiagonal (source). However, when I use the Wikipedia code on a real-world Hermitian matrix, the Hessenberg matrix is not at all tridiagonal. When I perform the computation on the real part of A (so that A == A.T) then I do get a tridiagonal Hessenberg matrix, so there seems to be a problem with the imaginary components of A. Does anybody know why the Wikipedia code doesn't produce the expected results?
Working example:
import numpy as np
import matplotlib.pyplot as plt
from scipy.linalg import circulant
def arnoldi_iteration(A, b, n):
m = A.shape[0]
h = np.zeros((n + 1, n), dtype=np.complex)
Q = np.zeros((m, n + 1), dtype=np.complex)
q = b / np.linalg.norm(b) # Normalize the input vector
Q[:, 0] = q # Use it as the first Krylov vector
for k in range(n):
v = A.dot(q) # Generate a new candidate vector
for j in range(k + 1): # Subtract the projections on previous vectors
h[j, k] = np.dot(Q[:, j], v)
v = v - h[j, k] * Q[:, j]
h[k + 1, k] = np.linalg.norm(v)
eps = 1e-12 # If v is shorter than this threshold it is the zero vector
if h[k + 1, k] > eps: # Add the produced vector to the list, unless
q = v / h[k + 1, k] # the zero vector is produced.
Q[:, k + 1] = q
else: # If that happens, stop iterating.
return Q, h
return Q, h
# Construct matrix A
N = 2**4
I = np.eye(N)
k = np.fft.fftfreq(N, 1.0 / N) + 0.5
alpha = np.linspace(0.1, 1.0, N)*2e2
c = np.fft.fft(alpha) / N
C = circulant(c)
A = np.einsum("i, ij, j->ij", k, C, k)
# Show that A is Hermitian
print(np.allclose(A, A.conj().T))
# Arbitrary (random) initial vector
np.random.seed(0)
v = np.random.rand(N)
# Perform Arnoldi iteration with complex A
_, h = arnoldi_iteration(A, v, N)
# Perform Arnoldi iteration with real A
_, h2 = arnoldi_iteration(np.real(A), v, N)
# Plot results
plt.subplot(121)
plt.imshow(np.abs(h))
plt.title("Complex A")
plt.subplot(122)
plt.imshow(np.abs(h2))
plt.title("Real A")
plt.tight_layout()
plt.show()
Result:
After browsing through some conference presentation slides, I realised that at some point Q had to be conjugated when A is complex. The correct algorithm is posted below for reference, with the code change marked (note that this correction has also been submitted to the Wikipedia entry):
import numpy as np
def arnoldi_iteration(A, b, n):
m = A.shape[0]
h = np.zeros((n + 1, n), dtype=np.complex)
Q = np.zeros((m, n + 1), dtype=np.complex)
q = b / np.linalg.norm(b)
Q[:, 0] = q
for k in range(n):
v = A.dot(q)
for j in range(k + 1):
h[j, k] = np.dot(Q[:, j].conj(), v) # <-- Q needs conjugation!
v = v - h[j, k] * Q[:, j]
h[k + 1, k] = np.linalg.norm(v)
eps = 1e-12
if h[k + 1, k] > eps:
q = v / h[k + 1, k]
Q[:, k + 1] = q
else:
return Q, h
return Q, h
I have three numpy matrices x, r and r. Whose values are:
x = np.array([[4,2],
[0,-1],
[-2,5],
[2,6]])
y = np.array([[1,7],
[2,6],
[5,2]])
r = np.array([[2,2,1],
[2,3,1],
[9,5,1],
[2,0,4]])
What I'm going to do is:(it's hard to describe by words so I use code to present what I want to do)
K = r.shape[1]
D = x.shape[1]
v = np.zeros((K, D, D))
for k in range(K):
v[k] = (r[:, k] * (x - y[k]).transpose() # (x - y[k]))
print(v)
The final v is what I need and v is equals to
[[[103. 38.]
[ 38. 216.]]
[[100. 46.]
[ 46. 184.]]
[[111. -54.]
[-54. 82.]]]
Is there any elegant or pythonic way to achieve this without for loops?
Thanks
This should work for you:
A = x[np.newaxis,...]-y[:,np.newaxis,:] # equivalent to (x-y[k]) for all k
B = A.swapaxes(1,2) # equivalent to (x-y[k]).transpose() for all k
C = r.T[:,np.newaxis,:]*B # equivalent to r[:, k] * (x - y[k]).transpose()
D = C#A # equivalent to r[:, k] *(x - y[k]).transpose() # (x - y[k])
Or in monster unreadable form
((r.T[:,np.newaxis,:]*(x[np.newaxis,...]
-y[:,np.newaxis,:]).swapaxes(1,2))#
(x[np.newaxis,...]-y[:,np.newaxis,:]))
proof:
>>> (v==((r.T[:,np.newaxis,:]*(x[np.newaxis,...]
-y[:,np.newaxis,:]).swapaxes(1,2))#
(x[np.newaxis,...]-y[:,np.newaxis,:]))).all()
True
I am learning some techniques for doing statistics with missing data from a book (Statistical Analysis with Missing Data by Little and Rubin). One particularly useful function for working with monotone non-response data is the Sweep Operator (details on page 148-151). I know that the R module gmm has the swp function which does this but I was wondering if anyone has implemented this function in Python, ideally for Numpy matrices to hold the input data. I searched StackOverflow and also did several web searches without success. Thanks for any help.
Here is the definition.
A PxP symmetric matrix G is said to be swept on row and column k if it is replaced by another symmetric PxP matrix H with elements defined as follows:
h_kk = -1/g_kk
h_jk = h_kj = g_jk/g_kk for j != k
h_jl = g_jl - g_jk g_kl / g_kk j != k, l != k
G = [g11, g12, g13
g12, g22, g23
g13, g23, g33]
H = SWP(1,G) = [-1/g11, g12/g11, g13/g11
g12/g11, g22-g12^2/g11, g23-g13*g12/g11
g13/g11, g23-g13*g12/g11, g33-g13^2/g11]
kvec = [k1,k2,k3]
SWP[kvec,G] = SWP(k1,SWP(k2,SWP(k3,G)))
Inverse function
H = RSW(k,G)
h_kk = -1/g_kk
h_jk = h_kj = -g_jk/g_kk for j != k
h_jl = g_jk g_kl / g_kk j != k, l != k
G == SWP(k,RSW(k,G)) == RSW(k,SWP(k,G))
def sweep(g, k):
g = np.asarray(g)
n = g.shape[0]
if g.shape != (n, n):
raise ValueError('Not a square array')
if not np.allclose(g - g.T, 0):
raise ValueError('Not a symmetrical array')
if k >= n:
raise ValueError('Not a valid row number')
# Fill with the general formula
h = g - np.outer(g[:, k], g[k, :]) / g[k, k]
# h = g - g[:, k:k+1] * g[k, :] / g[k, k]
# Modify the k-th row and column
h[:, k] = g[:, k] / g[k, k]
h[k, :] = h[:, k]
# Modify the pivot
h[k, k] = -1 / g[k, k]
return h
I have no way of testing the above code, but I found an alternativee description here, which is valid for non-symmetrical matrices, which can be calculated as follows:
def sweep_non_sym(a, k):
a = np.asarray(a)
n = a.shape[0]
if a.shape != (n, n):
raise ValueError('Not a square array')
if k >= n:
raise ValueError('Not a valid row number')
# Fill with the general formula
b = a - np.outer(a[:, k], a[k, :]) / a[k, k]
# b = a - a[:, k:k+1] * a[k, :] / a[k, k]
# Modify the k-th row and column
b[k, :] = a[k, :] / a[k, k]
b[:, k] = -a[:, k] / a[k, k]
# Modify the pivot
b[k, k] = 1 / a[k, k]
return b
This one does give the correct results for the examples in that link:
>>> a = [[2,4],[3,1]]
>>> sweep_non_sym(a, 0)
array([[ 0.5, 2. ],
[-1.5, -5. ]])
>>> sweep_non_sym(sweep_non_sym(a, 0), 1)
array([[-0.1, 0.4],
[ 0.3, -0.2]])
>>> np.dot(a, sweep_non_sym(sweep_non_sym(a, 0), 1))
array([[ 1.00000000e+00, 0.00000000e+00],
[ 5.55111512e-17, 1.00000000e+00]])