Develop Reference

trying to do ward clustering on n by n matrix in scipy

I have a similarity score between 0 and 1 from each entry to every other entry in an 100 by 100 matrix. So e.g. [0,0] would be 1, [0,1] might be .54 etc. The similarity score was generated using Shannon Jensen divergence.
I want to use ward clustering (but am open to other suggestions) to cluster these together and I tried the following code:
x = np.array(mylist)
print x.shape#(100,100)
clustering = scipy.cluster.hierarchy.ward(x)
scipy.cluster.hierarchy.dendrogram(clustering)
but I am getting the error:
Traceback (most recent call last):
File "C:/Python27/ward.py", line 38, in <module>
clustering = scipy.cluster.hierarchy.ward(y)
File "C:\Python27\lib\site-packages\scipy\cluster\hierarchy.py", line 465, in ward
return linkage(y, method='ward', metric='euclidean')
File "C:\Python27\lib\site-packages\scipy\cluster\hierarchy.py", line 620, in linkage
y = _convert_to_double(np.asarray(y, order='c'))
File "C:\Python27\lib\site-packages\scipy\cluster\hierarchy.py", line 928, in _convert_to_double
X = X.astype(np.double)
ValueError: setting an array element with a sequence.
Do I need to do some transformation on my array first or use some other method?

scipy: Qhull error when using bilinear interpolation

I am interpolating data from satellite imagery. My initial data is not on a regular grid. The data e.g. ch1 refers to positions given by longitude and latitude. When I try nearest neighbour interpolation the result I get is good.
from scipy import interpolate
lats = np.arange(latitude.max(), latitude.min(),-.1)
lons = np.arange(longitude.min(),longitude.max(),.1)
all_lon,all_lat = np.meshgrid(lons,lats)
ch1_all = interpolate.griddata((longitude.reshape(-1),latitude.reshape(-1)),ch1.reshape(-1),(all_lon,all_lat),'nearest')
However when I request bilinear interpolation I raise a Qhull error.
ch1_all = interpolate.griddata((longitude.reshape(-1),latitude.reshape(-1)),ch1.reshape(-1),(all_lon,all_lat),'linear')
The error I get is:
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
File "C:\Python27\lib\site-packages\scipy\interpolate\ndgriddata.py", line 206, in griddata
rescale=rescale)
File "interpnd.pyx", line 239, in scipy.interpolate.interpnd.LinearNDInterpolator.__init__ (scipy\interpolate\interpnd.c:4549)
File "qhull.pyx", line 1736, in scipy.spatial.qhull.Delaunay.__init__ (scipy\spatial\qhull.c:13719)
File "qhull.pyx", line 328, in scipy.spatial.qhull._Qhull.__init__ (scipy\spatial\qhull.c:3602)
QhullError: Qhull error
I have read the post at griddata runtime error -- Python / SciPy (Interpolation) but in my case, it interpolates data with one method but not with the other.
What am I doing wrong?
In the image below I have plotted the final point positions (red) and overlaid the initial positions (blue)

Numpy random memory error

Hi I have a simple line that creates a random array for a rather large dataset:
import numpy as np
import random
N=276233
L=138116
np.random.random([L,N])
But i get this error:
Traceback (most recent call last):
File "<string>", line 3 (23), in <module>
File "mtrand.pyx", line 760, in mtrand.RandomState.random_sample (numpy\random\mtrand\mtrand.c:5713)
File "mtrand.pyx", line 137, in mtrand.cont0_array (numpy\random\mtrand\mtrand.c:1300)
MemoryError
What is the solution and what is the limit of the array ?

You are trying to create an array that would require 284GB of memory:
In [16]: L * N * 8 / (1024. ** 3)
Out[16]: 284.25601890683174
Either buy a lot more RAM (and make sure your system can handle it) or find a way to not have to generate a 276,233x138,116 matrix.

python sparse gmres messes with input arguments

I have a simple code to solve a sparse linear system using scipy.sparse.linalg.gmres
W, S = load_data()
M = normalize(W.T.astype('float64'),'l1')
S = normalize(S.astype('float64'),'l1')
rhs = S[cat_id,:].T
print M.shape
print rhs.shape
p = gmres(M, rhs)
function load_data loads two sparse matrices from matlab's .mat files and omitted.
The output is surprising:
(150495, 150495)
(150495, 1)
Traceback (most recent call last):
File "explain.py", line 54, in <module>
pr(1)
File "explain.py", line 42, in pr
p = gmres(M, rhs)
File "<string>", line 2, in gmres
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/isolve/iterative.py", line 85, in non_reentrant
return func(*a, **kw)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/isolve/iterative.py", line 418, in gmres
A,M,x,b,postprocess = make_system(A,M,x0,b,xtype)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/isolve/utils.py", line 78, in make_system
raise ValueError('A and b have incompatible dimensions')
ValueError: A and b have incompatible dimensions
But I've run gmres in accordance with documentation
A : {sparse matrix, dense matrix, LinearOperator}
The real or complex N-by-N matrix of the linear system.
b : {array, matrix}
Right hand side of the linear system. Has shape (N,) or (N,1).
I simply don't understand what is wrong with this code and would like any ideas.

The argument b of gmres must not be a sparse matrix; it can be a numpy array or matrix. Try
p = gmres(M, rhs.A)

Create a sparse diagonal matrix from row of a sparse matrix

I process rather large matrices in Python/Scipy. I need to extract rows from large matrix (which is loaded to coo_matrix) and use them as diagonal elements. Currently I do that in the following fashion:
import numpy as np
from scipy import sparse
def computation(A):
for i in range(A.shape[0]):
diag_elems = np.array(A[i,:].todense())
ith_diag = sparse.spdiags(diag_elems,0,A.shape[1],A.shape[1], format = "csc")
#...
#create some random matrix
A = (sparse.rand(1000,100000,0.02,format="csc")*5).astype(np.ubyte)
#get timings
profile.run('computation(A)')
What I see from the profile output is that most of the time is consumed by get_csr_submatrix function while extracting diag_elems. That makes me think that I use either inefficient sparse representation of initial data or wrong way of extracting row from a sparse matrix. Can you suggest a better way to extract a row from a sparse matrix and represent it in a diagonal form?
EDIT
The following variant removes bottleneck from the row extraction (notice that simple changing 'csc' to csr is not sufficient, A[i,:] must be replaced with A.getrow(i) as well). However the main question is how to omit the materialization (.todense()) and create the diagonal matrix from the sparse representation of the row.
import numpy as np
from scipy import sparse
def computation(A):
for i in range(A.shape[0]):
diag_elems = np.array(A.getrow(i).todense())
ith_diag = sparse.spdiags(diag_elems,0,A.shape[1],A.shape[1], format = "csc")
#...
#create some random matrix
A = (sparse.rand(1000,100000,0.02,format="csr")*5).astype(np.ubyte)
#get timings
profile.run('computation(A)')
If I create DIAgonal matrix from 1-row CSR matrix directly, as follows:
diag_elems = A.getrow(i)
ith_diag = sparse.spdiags(diag_elems,0,A.shape[1],A.shape[1])
then I can neither specify format="csc" argument, nor convert ith_diags to CSC format:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.6/profile.py", line 70, in run
prof = prof.run(statement)
File "/usr/local/lib/python2.6/profile.py", line 456, in run
return self.runctx(cmd, dict, dict)
File "/usr/local/lib/python2.6/profile.py", line 462, in runctx
exec cmd in globals, locals
File "<string>", line 1, in <module>
File "<stdin>", line 4, in computation
File "/usr/local/lib/python2.6/site-packages/scipy/sparse/construct.py", line 56, in spdiags
return dia_matrix((data, diags), shape=(m,n)).asformat(format)
File "/usr/local/lib/python2.6/site-packages/scipy/sparse/base.py", line 211, in asformat
return getattr(self,'to' + format)()
File "/usr/local/lib/python2.6/site-packages/scipy/sparse/dia.py", line 173, in tocsc
return self.tocoo().tocsc()
File "/usr/local/lib/python2.6/site-packages/scipy/sparse/coo.py", line 263, in tocsc
data = np.empty(self.nnz, dtype=upcast(self.dtype))
File "/usr/local/lib/python2.6/site-packages/scipy/sparse/sputils.py", line 47, in upcast
raise TypeError,'no supported conversion for types: %s' % args
TypeError: no supported conversion for types: object`

Here's what I came up with:
def computation(A):
for i in range(A.shape[0]):
idx_begin = A.indptr[i]
idx_end = A.indptr[i+1]
row_nnz = idx_end - idx_begin
diag_elems = A.data[idx_begin:idx_end]
diag_indices = A.indices[idx_begin:idx_end]
ith_diag = sparse.csc_matrix((diag_elems, (diag_indices, diag_indices)),shape=(A.shape[1], A.shape[1]))
ith_diag.eliminate_zeros()
Python profiler said 1.464 seconds versus 5.574 seconds before. It takes advantage of the underlying dense arrays (indptr, indices, data) that define sparse matrices. Here's my crash course: A.indptr[i]:A.indptr[i+1] defines which elements in the dense arrays correspond to the non-zero values in row i. A.data is a dense 1d array of non-zero the values of A and A.indptr is the column where those values go.
I would do some more testing to make very certain this does the same thing as before. I only checked a few cases.