I am working with a large matrix of size m * n for m,n>100000. Since my data is huge I want to store the matrix in memory and work with HDF5, and PyTables.
However, the elements of my matrix are small matrices of real values of dimension 5*5.
I have already looked at the following post, but I would like to know if there is any other way of storing this type of data in tables?
(Create a larger matrix from smaller matrices in numpy)
Thank you in advance
In numpy there are two relevant structures.
One is a 4dimensional array, e.g. np.zeros((100,100,5,5),int). The other is an 2 dimensional array of objects. np.zeros((100,100),dtype=object). With object array, the elements can be anythings - strings, numbers, lists, your 5x5 arrays, other 7x3 array, None, etc).
It is easiest to do math on the 4d array, for example taking the mean across all the 5x5 subarrays, or finding the [:,:,0,0] corner of all.
If your subarrays are all 5x5, it can be tricky to create and fill that object array. np.array(...) tries to create that 4dim array if possible.
With h5py you can chunk the file, and access portions of the larger array. But you still have to have a workable numpy representation to do anything with them.
Related
Couldn't find how Numpy and PyTorch store arrays (tensors), so I'm asking here.
If you need to store multidimensional arrays like those two, should you store all values in a single-contionuous array and store the shape of the array to determine how you parse it (which is also beneficial when reshaping the array), like the following int arr[];, or should you store the multidimensional array as is, so for example storing a 2d integer array like so in C for example: int arr[][];.
Which should be the way for maximum performance and memory efficiency?
Also, any reference on how to go on developing such a thing is appreciated.
P.S That's for a school exam, so no libraries.
I'm comparing in python the reading time of a row of a matrix, taken first in dense and then in sparse format.
The "extraction" of a row from a dense matrix costs around 3.6e-05 seconds
For the sparse format I tried both csr_mtrix and lil_matrix, but both for the row-reading spent around 1-e04 seconds
I would expect the sparse format to give the best performance, can anyone help me understand this ?
arr[i,:] for a dense array produces a view, so its execution time is independent of arr.shape. If you don't understand the distinction between view and copy, you need to do more reading about numpy basics.
csr and lil formats allow indexing that looks a lot like ndarray's, but there are key differences. For the most part the concept of a view does not apply. There is one exception. M.getrowview(i) takes advantage of the unique data structure of a lil to produce a view. (Read its doc and code)
Some indexing of a csr format actually uses matrix multiplication, using a specially constructed 'extractor' matrix.
In all cases where sparse indexing produces sparse matrix, actually constructing the new matrix from the data takes time. Sparse does not use compiled code nearly as much as numpy. It's strong point, relative to numpy is matrix multiplication of matrices that are 10% sparse (or smaller).
In the simplest format (to understand), coo, each nonzero element is represented by 3 values - data, row, col. Those are stored in 3 1d arrays. So it has to have a sparsity of less than 30% to even break even with respect to memory use. coo does not implement indexing.
I'm trying to understand the differences between what people call matrices and what people call lists within lists.
Are they the same in that, once created, you can do identical things to them (reference elements the same way within them, etc).
Examples:
Making lists within a list:
ListsInLists = [[1,2],[3,4],[5,6]]
Making a multidimensional array:
np.random.rand(3,2)
Stacking arrays to make a matrix:
Array1 = [1,2,3,4]
Array2 = [5,6,7,8]
CompleteArray = vstack((Array1,Array2))
A list of list is very different from a two-dimensional Numpy array.
A list has dynamic size and can hold any type of object, whereas an array has a fixed size and entries of uniform type.
In a list of lists, each sublist can have different sizes. An array has fixed dimensions along each axis.
An array is stored in a contiguous block of memory, whereas the objects in a list can be stored anywhere on the heap.
Numpy arrays are more restrictive, but offer greater performance and memory efficiency. They also provide convenient functions for vectorised mathematical operations.
Internally, a list is represented as an array of pointers that point to arbitrary Python objects. The array uses exponential over-allocation to achieve linear performance when appending repeatedly at the end of the list. A Numpy array on the other hand is typically represented as a C array of numbers.
(This answer does not cover the special case of Numpy object arrays, which can hold any kind of Python object as well. They are rarely used, since they have the restrictions of Numpy arrays, but don't have the performance advantages.)
They are not the same. Arrays are more memory efficient in python than lists, and there are additional functions that can be performed on arrays thanks the to numpy module that you cannot perform on lists.
For calculations, working with arrays in numpy tends to be a lot faster than using built in list functions.
You can read a bit more into it if you want in the answers to this question.
I need to form a 2D matrix with total size 2,886 X 2,003,817. I try to use numpy.zeros to make a 2D zero element matrix and then calculate and assign each element of Matrix (most of them are zero son I need to replace few of them).
but when I try numpy.zero to initialize my matrix I get the following memory error:
C=numpy.zeros((2886,2003817)) "MemoryError"
I also try to form the matrix without initialization. Basically I calculate the element of each row in each iteration of my algorithm and then
C=numpy.concatenate((C,[A]),axis=0)
in which C is my final matrix and A is the calculated row at the current iteration. But I find out this method takes a lots of time, I am guessing it is because of using numpy.concatenate(?)
could you please let me know if there is a way to avoid memory error in initializing my matrix or is there any better method or suggestion to form the matrix in this size?
Thanks,
Amir
If your data has a lot of zeros in it, you should use scipy.sparse matrix.
It is a special data structure designed to save memory for matrices that have a lot of zeros. However, if your matrix is not that sparse, sparse matrices start to take up more memory. There are many kinds of sparse matrices, and each of them is efficient at one thing, while inefficient at another thing, so be careful with what you choose.
I am working on an FEM project using Scipy. Now my problem is, that
the assembly of the sparse matrices is too slow. I compute the
contribution of every element in dense small matrices (one for each
element). For the assembly of the global matrices I loop over all
small dense matrices and set the matrice entries the following way:
[i,j] = someList[k][l]
Mglobal[i,j] = Mglobal[i,j] + Mlocal[k,l]
Mglobal is a lil_matrice of appropriate size, someList maps the
indexing variables.
Of course this is rather slow and consumes most of the matrice
assembly time. Is there a better way to assemble a large sparse matrix
from many small dense matrices? I tried scipy.weave but it doesn't
seem to work with sparse matrices
I posted my response to the scipy mailing list; stack overflow is a bit easier
to access so I will post it here as well, albeit a slightly improved version.
The trick is to use the IJV storage format. This is a trio of three arrays
where the first one contains row indicies, the second has column indicies, and
the third has the values of the matrix at that location. This is the best way
to build finite element matricies (or any sparse matrix in my opinion) as access
to this format is really fast (just filling an an array).
In scipy this is called coo_matrix; the class takes the three arrays as an
argument. It is really only useful for converting to another format (CSR os
CSC) for fast linear algebra.
For finite elements, you can estimate the size of the three arrays by something
like
size = number_of_elements * number_of_basis_functions**2
so if you have 2D quadratics you would do number_of_elements * 36, for example.
This approach is convenient because if you have local matricies you definitely
have the global numbers and entry values: exactly what you need for building
the three IJV arrays. Scipy is smart enough to throw out zero entries, so
overestimating is fine.