Develop Reference

scipy.stats.multivariable_norm.pdf: "The input matrix must be symmetric positive semidefinite."

So I have the following code below.
L = np.array([1,2,3])
M = np.array([1,2,3])
Q = np.random.uniform(0,10,size=(3,3))
S = Q.T*Q
print(sp.stats.multivariate_normal.pdf(L,M,S))
Clearly S is a symmetric positive semidefinite matrix. I can prove it using linear algebra theory. However, scipy complains that it isn't when running the above code. What can I do to solve this problem?

Like the comment by Mechanic Pig says, replace * (element-wise multiplication on Numpy arrays) with #.
import scipy as sp
import numpy as np
L = np.array([1, 2, 3])
M = np.array([1, 2, 3])
Q = np.random.uniform(0, 10, size=(3, 3))
S = Q.T # Q
print(sp.stats.multivariate_normal.pdf(L, M, S))
prints, in my case, 0.0003568248543110567.
You can verify your "covariance" is positive-definite by just comparing
np.linalg.eig(Q.T * Q)[0], using element-wise multiply (will likely have negative values), vs
np.linalg.eig(Q.T # Q)[0] (will have no negative values).

Building N-th order Markovian transition matrix from a given sequence

I am trying to create a function which can transform a given input sequence to a transition matrix of the requested order. I found an implementation for the first-order Markovian transition matrix.
Now, I want to be able to come up with a solution which can calculate 2nd and 3rd order transition matrices.
Example of the 1st order matrix implementation:
import numpy as np
# sequence with 3 states -> 0, 1, 2
a = [0, 1, 0, 0, 0, 2, 2, 1, 1, 1, 0, 0, 0, 0, 0, 1, 2, 2, 2, 0, 0, 2]
def transition_matrix_first_order(seq):
M = np.full((3, 3), fill_value = 1/3, dtype= np.float64)
for (i,j) in zip(seq, seq[1:]):
M[i, j] += 1
M = M / M.sum(axis = 1, keepdims = True)
return M
print(transition_matrix_first_order(a))
Which gives me this:
[[0.61111111 0.19444444 0.19444444]
[0.38888889 0.38888889 0.22222222]
[0.22222222 0.22222222 0.55555556]]
When making a 2nd order matrix, it should have unique_state_count ** order rows and unique_state_count columns. In the example above, I have 3 unique states, so the matrix will have 9x3 structure.
Desirable function sample:
cal_tr_matrix(seq, unique_state_count, order)

I think you have a slight misunderstanding about the Markov chains and their transition matrices.
First of all, the estimated transition matrix your function produces is unfortunately not correct. Why? Let's refresh.
A discrete Markov chain in discrete time with N different states has a transition matrix P of size N x N, where a (i, j) element is P(X_1=j|X_0=i), i.e. the probability of transition from state i to state j in a single time step.
Now a transition matrix of order n, denoted P^{n}is once again a matrix of size N x N where a (i, j) element is P(X_n=j|X_0=i), i.e. the probability of transition from state i to state j in n time steps.
A wonderful result says: P^{n} = P^n, i.e. taking n powers of single-step transition matrix gives you the n-step transition matrix.
Now with this recap, all that is needed is to estimate P from the given sequence, then to estimate P^{n} one can just use the already estimated P and take a n-th power of the matrix. So how to estimate the matrix P? Well if we denote N_{ij} the number of observations of transition from state i to state j and N_{i*} the number of observations being in state i, then P_{ij} = N_{ij} / N_{i*}.
Overall here in Python:
import numpy as np
def transition_matrix(arr, n=1):
""""
Computes the transition matrix from Markov chain sequence of order `n`.
:param arr: Discrete Markov chain state sequence in discrete time with states in 0, ..., N
:param n: Transition order
"""
M = np.zeros(shape=(max(arr) + 1, max(arr) + 1))
for (i, j) in zip(arr, arr[1:]):
M[i, j] += 1
T = (M.T / M.sum(axis=1)).T
return np.linalg.matrix_power(T, n)
transition_matrix(arr=a, n=1)
>>> array([[0.63636364, 0.18181818, 0.18181818],
>>> [0.4 , 0.4 , 0.2 ],
>>> [0.2 , 0.2 , 0.6 ]])
transition_matrix(arr=a, n=2)
>>> array([[0.51404959, 0.22479339, 0.26115702],
>>> [0.45454545, 0.27272727, 0.27272727],
>>> [0.32727273, 0.23636364, 0.43636364]])
transition_matrix(arr=a, n=3)
>>> array([[0.46927122, 0.23561232, 0.29511645],
>>> [0.45289256, 0.24628099, 0.30082645],
>>> [0.39008264, 0.24132231, 0.36859504]])
Interesting thing, when you set the order n to a fairly high number, the higher and higher powers of the P matrix seem to converge to some very specific values. That's known as stationary/invariant distribution of the Markov chain and it gives a very good indication of how the chain behaves over a long period of time/transitions. Also:
P = transition_matrix(a, 1)
P111 = transition_matrix(a, 111)
print(P)
print(P111.dot(P))
EDIT: Now to the tweaked solution based on your comment, I'd suggest to have higher dimensional matrices for higher orders instead of exploding the number of rows. One way would be like this:
def cal_tr_matrix(arr, order):
_shape = (max(arr) + 1,) * (order + 1)
M = np.zeros(_shape)
for _ind in zip(*[arr[_x:] for _x in range(order + 1)]):
M[_ind] += 1
return M
res1 = cal_tr_matrix(a, 1)
res2 = cal_tr_matrix(a, 2)
Now the element res1[i, j] says how many times transition i->j happened, while the element res2[i, j, k] says how many times transition i->j->k happened.

Python Sympy dot with the conjugate not same as norm squared

Why does sympy give false on the second Boolean? It correctly gives true on the first Boolean. I thought this last line would be the definition of the norm.
from sympy import *
eta_1, eta_2, m = 1, 1, 3
theta_1, theta_2 = symbols("theta_1 theta_2", real=True)
sigma_x = Matrix([[0, 1], [1, 0]])
sigma_y = Matrix([[0, -I], [I, 0]])
sigma_z = Matrix([[1, 0], [0, -1]])
H = eta_1*sin(theta_1)*sigma_x + eta_2*sin(theta_2)*sigma_y + (m-eta_1*cos(theta_1)-eta_2*cos(theta_2))*sigma_z
v = H.eigenvects()
l = v[0][0]
v = v[0][2][0]
n_normal = v/v.norm()
print(simplify(n_normal.norm()**2) == 1)
print(simplify(n_normal.dot(n_normal.H)))
print(simplify(n_normal.dot(n_normal.H)) == 1)
I think this has to do with the fact that sympy fails in that
simplify(abs(x**2)-x*conjugate(x))==0
gives false. Is there some other way to go around this problem, another way to define an inner product that does behave correctly. I'm doing some complicated physics calculations for my thesis and I would really like to check my results with sympy.
PS. I'm using sympy version 1.4dev.
Edit: I think the problem is with the fact that simplify doesn't realize that
$2*\cos(\theta_1)*cos(\theta_2) - 6*\cos(\theta_1) - 6*\cos(\theta_2) + 11>0\:.$
If I replace this in $n_normal$ with its absolute value it works. I think it is weird that the norm function does this correctly and the simplify of what essentially should be the norm doesn't.

How to find the eigenvalues and eigenvectors of a matrix with SymPy?

I want to calculate the eigenvectors x from a system A by using this: A x = λ x
The problem is that I don't know how to solve the eigenvalues by using SymPy.
Here is my code. I want to get some values for x1 and x2 from matrix A
from sympy import *
x1, x2, Lambda = symbols('x1 x2 Lambda')
I = eye(2)
A = Matrix([[0, 2], [1, -3]])
equation = Eq(det(Lambda*I-A), 0)
D = solve(equation)
print([N(element, 4) for element in D]) # Eigenvalus in decimal form
print(pretty(D)) # Eigenvalues in exact form
X = Matrix([[x1], [x2]]) # Eigenvectors
T = A*X - D[0]*X # The Ax = %Lambda X with the first %Lambda = D[0]
print(pretty(solve(T, x1, x2)))

The methods eigenvals and eigenvects is what one would normally use here.
A.eigenvals() returns {-sqrt(17)/2 - 3/2: 1, -3/2 + sqrt(17)/2: 1} which is a dictionary of eigenvalues and their multiplicities. If you don't care about multiplicities, use list(A.eigenvals().keys()) to get a plain list of eigenvalues.
The output of eigenvects is a bit more complicated, and consists of triples (eigenvalue, multiplicity of this eigenvalue, basis of the eigenspace). Note that the multiplicity is algebraic multiplicity, while the number of eigenvectors returned is the geometric multiplicity, which may be smaller. The eigenvectors are returned as 1-column matrices for some reason...
For your matrix, A.eigenvects() returns the eigenvector [-2/(-sqrt(17)/2 + 3/2), 1] for the eigenvalue -3/2 + sqrt(17)/2, and eigenvector [-2/(3/2 + sqrt(17)/2), 1] for eigenvalue -sqrt(17)/2 - 3/2.
If you want the eigenvectors presented as plain lists of coordinates, the following
[list(tup[2][0]) for tup in A.eigenvects()]
would output [[-2/(-sqrt(17)/2 + 3/2), 1], [-2/(3/2 + sqrt(17)/2), 1]]. (Note this just picks one eigenvector for each eigenvalue, which is not always what you want)

sympy has a very convenient way of getting eigenvalues and eigenvectors: sympy-doc
Your example would simply become:
from sympy import *
A = Matrix([[0, 2], [1, -3]])
print(A.eigenvals()) #returns eigenvalues and their algebraic multiplicity
print(A.eigenvects()) #returns eigenvalues, eigenvects

This answer will help you when you all eignvectors, the solution above doesnt always give you all eienvectos for example this matrix A used below
# the matrix
A = Matrix([
[4, 0, 1],
[2, 3, 2],
[1, 0, 4]
])
sym_eignvects = []
for tup in sMatrix.eigenvects():
for v in tup[2]:
sym_eignvects.append(list(v))

Calculating "generating functions" with numpy

In mathematics, a "generating function" is defined from a sequence of numbers c0, c1, c2, ..., cn by c0+c1*x+c2*x^2 + ... + cn*x^n. These come as "moment generating functions", "probability generating functions" and various other types, depending on the source of the coefficient.
I have an array of the coefficients and I'd like a quick way to create the corresponding generating function.
I could do
import numpy as np
myArray = np.array([1,2,3,4])
x=0.2
sum([c*x**k for k,c in enumerate myArray])
or I could have an array having c[k] in the kth entry. It seems there should be a fast numpy way to do this.
Unfortunately attempts to look this up are complicated by the fact that "generate" and "function" are common words in programming, as is the combination "generating function" so I haven't had any luck with search engines.

x = .2
coeffs = np.array([1,2,3,4])
Make an array of the degree of each term
degrees = np.arange(len(coeffs))
Raise x the each degree
terms = np.power(x, degrees)
Multiply the coefficients and sum
result = np.sum(coeffs*terms)
>>> coeffs
array([1, 2, 3, 4])
>>> degrees
array([0, 1, 2, 3])
>>> terms
array([ 1. , 0.2 , 0.04 , 0.008])
>>> result
1.552
>>>
As a function:
def f(coeffs, x):
degrees = np.arange(len(coeffs))
terms = np.power(x, degrees)
return np.sum(coeffs*terms)
Or simply us the Numpy Polynomial Package
from numpy.polynomial import Polynomial as P
p = P(coeffs)
result = p(x)

If you are looking for performance, using np.einsum could be suggested too -
np.einsum('i,i->',myArray,x**np.arange(myArray.size))

>>> coeffs = np.random.random(5)
>>> coeffs
array([ 0.70632473, 0.75266724, 0.70575037, 0.49293719, 0.66905641])
>>> x = np.random.random()
>>> x
0.7252944971757169
>>> powers = np.arange(0, coeffs.shape[0], 1)
>>> powers
array([0, 1, 2, 3, 4])
>>> result = coeffs * x ** powers
>>> result
array([ 0.70632473, 0.54590541, 0.37126147, 0.18807659, 0.18514853])
>>> np.sum(result)
1.9967167252487628

Using numpys Polynomial class is probably the easiest way.
from numpy.polynomial import Polynomial
coefficients = [1,2,3,4]
f = Polynomial( coefficients )
You can then use the object like any other function.
import numpy as np
import matplotlib.pyplot as plt
print f( 0.2 )
x = np.linspace( -5, 5, 51 )
plt.plot( x , f(x) )