Develop Reference

Implement variational approach for budget closure with 2 constraints in python

I'm new to Python and am quite helpless with a problem I have to solve:
I have two budget equations, let's say a+b+c+d=Res1 and a+c+e+f=Res2, now every term has a specific standard deviation a_std, b_std,... and I want to distribute the budget residuals Res1 and Res2 onto the individual terms relative to their uncertainty (see eqution below), to get a_new+b_new+c_new+d_new=0 and a_new+c_new+e_new+f_new=0
Regarding only 1 budget equation I'm able to solve the problem and get the terms a_new, b_new, c_new and d_new. But how can I add the second constraint to also get e_new and f_new?
e.g. I calculate a_new = a + (a_std^2/(a_std+b_std+c_std))*Res1 , however this is only dependent of the first equation, but I want a to be modified that way to also satisfy the second equation..
I appreciate any help/any ideas on how to approach this problem.
Thanks in advance,
Sue
Edit:
What I have so far:
def var_close(a,a_std,b,b_std,c,c_std,d,d_std,e,e_std,f,f_std,g,g_std):
x=[a,b,c,d,e]
Res1=np.sum([x])
std_ges1=a_std*a_std+b_std*b_std+c_std*c_std+d_std*d_std+e_std*e_std
y=[a,c,f,g]
Res2=np.sum([y])
std_ges2=a_std*a_std+c_std*c_std+f_std*f_std+g_std*g_std
a_new=a-((a_std*a_std)/std_ges1)*Res1
b_new=b-((b_std*b_std)/std_ges1)*Res1
c_new=c-((c_std*c_std)/std_ges1)*Res1
d_new=d-((d_std*d_std)/std_ges1)*Res1
e_new=e-((e_std*e_std)/std_ges1)*Res1
a_new2=a-((a_std*a_std)/std_ges2)*Res2
c_new2=c-((c_std*c_std)/std_ges2)*Res2
f_new=f-((f_std*f_std)/std_ges2)*Res2
g_new=g-((g_std*g_std)/std_ges2)*Res2
return a_new,b_new,c_new,d_new,e_new,a_new2,c_new2,f_new,g_new
But like this e.g. a_new and a_new2 are slightly different, but I want them to be equal and the other terms modified correspondng to their uncertainty..

Finding a large rank positive semidefinte matrix *C* whose entries give rise to a singular matrix *L*

My goal in a nutshell:
Given an 8x8 matrix C I have an algorithm which constructs another 8x8 matrix L=L(C) in physically relevant way, wherein each entry of L is given by a particular (possibly irrational) linear combination of up to 8 entries of C. I want to find a particular choice of C which is positive semidefinite and has large rank (7 or 8) but gives rise to a singular L.
Facts:
Every positive semidefinite C can be written as C=U*U for some U (where U* denotes the complex conjugate transpose of U).
In this case ker U=ker C, and so the rank of U and C are the same.
If U has a nonsingular 7x7 principle submatrix, then the rank of U is at least 7.
Naive solution:
Declare U as an 8x8 matrix of symbols and create C=U*U. This guarantees C is positive semidefinite.
Define V to be the upper-left 7x7 principle submatrix of U. Note if V is nonsingular the rank of C is at least 7.
Compute L from C.
Try to solve([ det(V)~=0, det(L)==0 ], [vars]), where vars=entries of U.
The difficulty:
One immediate problem is that not every choice of U which is rank 7 or 8 will have V nonsingular, so the matrix C I'm seeking might be missed in this regime. Ignoring this here, my issue for this forum is more to the complexity of the problem:
By setting C=U*U and using the entries of U as the variables, each entry of C becomes a sum of 8 terms, each degree 2. E.g., the c21 entry is given by
u11*conj(u12) + u21*conj(u22) + u31*conj(u32) + u41*conj(u42) + u51*conj(u52) + u61*conj(u62) + u71*conj(u72) + u81*conj(u82)
Since each entry of L is a linear combination of between 2 and 8 entries of C, we have that each entry of L is a linear combination of between 2*8=16 and 8*8=64 degree 2 terms from the entries of U (or U*). Thus det(L) is a uniform degree 2*8=16 polynomial with between 8!*16^8≈10^14 and 8!*64^8≈10^19 terms before simplification. I need this polynomial to be zero while simultaneously det(V) is nonzero (a uniform degree 7 polynomial with 7!=5040 terms).
Note that if one were to avoid using C=U*U and instead let the entries of C be the variables, then det(L) would be a uniform degree 8 polynomial with between 8!*2^8≈10^7 and 8!*8^8≈10^11 terms before simplification. I would need this polynomial to be zero while simultaneously det(V) is nonzero and some extra conditions are placed so that C is positive semidefinite (Sylvester's Criterion, etc).
My question:
Is there a smarter way to do this? Certainly the determinant is not the most efficient way to determine if L is singular, but ideally I would like an exact answer for C, rather then a numerical approximation.
I am most familiar with Matlab, but any suggestions using any system (Python, Macaulay2, ...) would be greatly appreciated. For computing power, I have access to several supercomputer clusters.
Edits:
Perhaps a bit lofty a question. More digestible sub-questions:
Is there a computationally easier, ideally symbolic, algorithm for determining if a matrix is singular (opposed to computing the determinant)?
Is there a computationally easier way of demanding the answer be positive semidefinite (opposed to setting C=U*U and using the entries of U as the variables)?
Is there a less restrictive (but still computationally easy) way to demand that C has rank 7 or 8?

Sparsity reduction

I have to factorize a big sparse matrix ( 6.5mln rows representing users* 6.5mln columns representing items) to find users and items latent vectors. I chose the als algorithm in spark framework(pyspark).
To boost the quality I have to reduce the sparsity of my matrix till 98%. (current value is 99.99% because I have inly 356mln of filled entries).
I can do it by dropping rows or columns, but I must find the optimal solution maximizing number of rows(users).
The main problem is that I must find some subsets of users and items sets, and dropping some row can drop some columns and vice versa, the second problem is that function that evaluates sparsity is not linear.
Which way I can solve this problem? which libraries in python can help me with it?
Thank you.

This is a combinatorial problem. There is no easy way to drop an optimal set of columns to achieve max number of users while reducing sparsity. A formal approach would be to formulate it as a mixed-integer program. Consider the following 0-1 matrix, derived from your original matrix C.
A(i,j) = 1 if C(i,j) is nonzero,
A(i,j) = 0 if C(i,j) is zero
Parameters:
M : a sufficiently big numeric value, e.g. number of columns of A (NCOLS)
N : total number of nonzeros in C (aka in A)
Decision vars are
x(j) : 0 or 1 implying whether column j is dropped or not
nr(i): number of nonzeros covered in row i
y(i) : 0 or 1 implying whether row i is dropped or not
Constraints:
A(i,:) x = nr(i) for i = 1..NROWS
nr(i) <= y(i) * M for i = 1..NROWS
#sum(nr(i)) + e = 0.98 * N # explicit slack 'e' to be penalized in the objective
y(i) and x(j) are 0-1 variables (binary variables) for i,j
Objective:
maximize #sum(y(i)) - N.e
Such a model would be extremely difficult to solve as an integer model. However, barrier methods should be able to solve the linear programming relaxations (LP) Possible solvers are Coin/CLP (open-source), Lindo (commercial) etc... It may then be possible to use the LP solution to compute approximate integer solutions by simple rounding.
In the end, you will definitely require an iterative approach which will require solving MF problem several times each time factoring a different submatrix of C, computed with above approach, until you are satisfied with the solution.

Finding the solutions of a complicated equation

I have a function (x^x)*((1-x)^(1-x))*(k^(x/2)) = 1 which has a unique solution in 0 < x < 1 for a given natural number k.
Can I use Python to find these solutions, or is my equation too complicated?

Yes, you can use Python to solve this equation.
I suggest you fix k=2 to simplify. Wolfram Alpha can verify your results: https://www.wolframalpha.com/input/?i=(x%5Ex)((1-x)%5E(1-x))(2%5E(x%2F2))+%3D+1
Depending on how you do your root search, you may have to take a first derivative with respect to x and place that into Python, as well.

Building dict combinations to match a target sum in Python

I have a dict made of N integer values like this:
units = {'trooper':2, 'tank':10, 'helicopter':12}
And I also have a target value... say 120.
I am trying to find all possible results of the equation:
a*units['trooper'] + b*units['tank'] + c*units['helicopter'] = 120
So the result would look something like:
60*trooper
55*trooper + 1*tank
54*trooper + 1*helicopter
And so on with all possible combinations of the N keys in the dict...
How can I go about building this ?

Searching for solutions to these kinds of problems is easiest if you know what they are called. Google for Diophantine equations.
In the Python world, you can use the Sympy package which includes a Diophantine equation solver. That package makes short work of your problem:
from sympy import symbols
from sympy.solvers.diophantine import diop_solve
trooper, tank, helicopter = symbols('trooper tank helicopter', integer=True)
print diop_solve(2*trooper + 10*tank + 12*helicopter - 120)
It outputs:
(5*t - trooper + 60, -6*t + trooper - 60, trooper)
You can also search for "ways to make change" which is another way of expressing the problem. A related problem is called called The Knapsack Problem and it is famously difficult to solve. The math behind solving general systems of linear Diophantine equations is a bit involved. Here are some resources:
http://www.math.udel.edu/~lazebnik/papers/dior1.pdf
http://www.dcc.fc.up.pt/~apt/onlinepapers/epia97_final.pdf
https://en.wikipedia.org/wiki/Diophantine_equation#System_of_linear_Diophantine_equations
http://www.math.utah.edu/~carlson/hsp2004/PythonShortCourse.pdf

Something like this would work for small values of target value:
units = {'trooper':2, 'tank':10, 'helicopter':12}
total = 120
for i in range(int(total/units['helicopter'])):
for j in range(int(total/units['tank'])):
if (total-units['helicopter']*i-units['tank']*j)%2==0 and (total-units['helicopter']*i-units['tank']*j)>0:
print ((total-units['helicopter']*i-units['tank']*j)/2,j,i)