Develop Reference

MemoryError while working with large files in Python

So I'm trying to convert a few hundred 120MB-2GB sized .svs files (high-resolution microscopy photos at 40 times magnification) to jpeg tiles, and I keep getting MemoryError at the step where my script allocates large amounts of data to RAM. Do I simply need larger RAM or are there other tricks I can try?
My laptop has a 64-bit i3 processor and 4GB RAM (3.8 GB available, usually). I run Python 3.8.2. Even when I input only 1 of the smaller slides at once, this error occurs (see below), so I guess smaller batches are not possible. Downsampling will influence the scanning resolution, so that is not an option.
output_jpeg_tiles('D:/TCGA slides/TCGA-O1-A52J-01A.svs', 'D:/JPEG tiles')
converting D:/TCGA slides/TCGA-O1-A52J-01A.svs with width 66640 and height 25155
Traceback (most recent call last):
File "<pyshell#29>", line 1, in <module>
output_jpeg_tiles('D:/TCGA slides/TCGA-O1-A52J-01A.svs', 'D:/JPEG tiles')
File "<pyshell#23>", line 22, in output_jpeg_tiles
patch = img.read_region((begin_x, begin_y), 0, (patch_width, patch_height))
File "C:\Users\...\site-packages\openslide\__init__.py", line 222, in read_region
return lowlevel.read_region(self._osr, location[0], location[1],
File "C:\Users\...\site-packages\openslide\lowlevel.py", line 258, in read_region
buf = (w * h * c_uint32)()
MemoryError
Any suggestions? :)

Python sklearn ValueError: array is too big

I made simple script on Python (ver.3.7) that classifies satellite image, but It can classify only clip of the satellite image. When I'm trying to classify the whole satellite image, it returns this:
Traceback (most recent call last):
File "v0-3.py", line 219, in classification_tool
File "sklearn\cluster\k_means_.py", line 972, in fit
File "sklearn\cluster\k_means_.py", line 312, in k_means
File "sklearn\utils\validation.py", line 496, in check_array
File "numpy\core\_asarray.py", line 85, in asarray
ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.
I tried using MiniBatchKMeans instead of KMeans (from Sklearn.KMeans : how to avoid Memory or Value Error?), but It still doesn't work. How I can avoid or solve this error? Maybe there are some mistakes in my code?

Oh I'm idiot because I used x32 version of Python instead of x64.
Maybe reinstalling Python to x64 version will solve your problem, user

Saving vector with complex components in a nifti file

I generate a 4D complex numpy array
for example:
numpy_array = np.tile(np.array([3, 4, 0], dtype=np.complex64), (100,100,100,1))
i want to save the numpy array as a nifti file
I tried using vtk and SimpleITK, but both of them don't support complex numbers (only vectors which are real numbers)
It seems that only nibabel support complex vectors and i manage to save the file, but i can only load it with nibabel, when i try to load it with ITK-SNAP or with Slicer it doesn't open. I know that ITK-SNAP can open complex vector of nifti files because i already have those files saved with another script using matlab
import numpy as np
import nibabel as nib
import SimpleITK as sitk
import vtk
from vtk.util.numpy_support import numpy_to_vtk
numpy_array = np.tile(np.array([3, 4, 0], dtype=np.complex64), (100,100,100,1))
nibabel save try:
image = nib.Nifti2Image(numpy_array, affine=np.eye(4))
nib.save(image , r'file_name.nii')
SimpleITK save try:
image = sitk.GetImageFromArray(numpy_array)
sitk.writeimage(image, r'file_name.nii')
vtk save try:
array = np.concatenate((np.real(numpy_array), np.imag(numpy_array)), axis=3)
stacked_array = array.reshape(-1, array.shape[-1])
vtk_array = numpy_to_vtk(stacked_array, deep=True,
array_type=vtk.VTK_FLOAT)
vtk_image = vtk.vtkImageData()
vtk_image.SetDimensions(numpy_array.shape[0], numpy_array.shape[1],
numpy_array.shape[2])
vtk_image.GetPointData().SetScalars(vtk_array)
writer = vtk.vtkNIFTIImageWriter()
writer.SetFileName(file_name)
writer.SetInputData(vtk_image)
writer.Write()
nibabel output:
nibabel creates a nifti file with vector but with other programs like ITK-SNAP it doens't open
ITK-SNAP error:
Error: Unsupported or missing image file format. ITK-SNAP failed to create an
ImageIO object for the image 'file_name.nii' using format ''.
SimpleITK error:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\SimpleITK\SimpleITK.py", line 3366, in _get_sitk_vector_pixelid
return _np_sitk[numpy_array_type.dtype]
KeyError: dtype('complex64')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\site-packages\SimpleITK\SimpleITK.py", line 3431, in GetImageFromArray
id = _get_sitk_vector_pixelid( z )
File "C:\ProgramData\Anaconda3\lib\site-packages\SimpleITK\SimpleITK.py", line 3371, in _get_sitk_vector_pixelid
raise TypeError('dtype: {0} is not supported.'.format(numpy_array_type.dtype))
TypeError: dtype: complex64 is not supported.
vtk output:
vtk creates a vector nifti file but with 6 components instead of 3
(consider the imaginary part as components too), i saw in the
documentation of numpy_to_vtk that it doesn't support copmlex arrays,
maybe someone know about a workaround.

The SimpleITK 1.2 release supports writing 4D complex images. The missing feature is support for complex number in stk.GetImageFromArray, but it's already added to the development branch and your code works if you a compiled version of the current SimpleITK master branch or wait for the 1.3 release scheduled for October 2019.
In the mean time you can convert the complex number components to SimpleITK separately like this:
image = sitk.RealAndImaginaryToComplexImageFilter(sitk.GetImageFromArray(numpy_array.real),
sitk.GetImageFromArray(numpy_array.imag))
sitk.WriteImage(image, r'file_name.nii')

Memory Error: Numpy.random.normal

In theano the following code snippet is throwing Memory error:
self.w = theano.shared(
np.asarray(
np.random.normal(
loc=0.0, scale=np.sqrt(1.0/n_out), size=(n_in, n_out)),
dtype=theano.config.floatX),
name='w', borrow=True)
Just to mention the size n_in=64*56*56 and n_out=4096. The snippet is taken from the init method of a Fully Connected Layer. See the traceback:
Traceback (most recent call last):
File "<stdin>", line 8, in <module>
File "final.py", line 510, in __init__
loc=0.0, scale=np.sqrt(1.0/n_out), size=(n_in, n_out)),
File "mtrand.pyx", line 1636, in mtrand.RandomState.normal (numpy/random/mtrand/mtrand.c:20676)
File "mtrand.pyx", line 242, in mtrand.cont2_array_sc (numpy/random/mtrand/mtrand.c:7401)
MemoryError
Is there any way we can get around the problem?

A MemoryError is Pythons way of saying: "I tried getting enough memory for that operation but your OS says it doesn't have enough".
So there's no workaround. You have to do it another way (or buy more RAM!). I don't know what your floatX is, but your array contains 64*56*56*4096 elements that translates to:
6.125 GB if you use float64
3.063 GB if you use float32
1.531 GB if you use float16 (not sure if float16 is supported for your operations though)
But the problem with MemoryErrors is that just avoiding them once generally isn't enough. If you don't change your approach you'll get problems again as soon as you do an operation that requires an intermediate or new array (then you have two huge arrays) or that coerces to a higher dtype (then you have two huge arrays and the new one is of higher dtype so requires more space).
So the only viable workaround is to change the approach, maybe you can start by calculating subsets (map-reduce approach)?

Easy.py (libsvm) crashing before creating model

I've been trying to run the easy.py script provided by libsvm-3.17 however it is crashing before creating the model. It does generate the range, scale, output file and gnuplot image but not model, and I believe I do need this to classify the test data. Any input is greatly appreciated :) Thanks.
The error is :
Traceback (most recent call last):
File "tools\easy.py", line 61, in
c,g,rate = map(float,last_line.split())
ValueError: could not convert string to float: b'[0x7FFFB2243810]'
I've tried several data sets, this error pops up everytime.