Develop Reference

Why the form of matrix taken from matlab change when reading them by to python

I have a big file saved from matlab with version -v7.3, when reading it by python, the shape of matrix change !! is that normal ?
For example, let's have the below matrix in MATLAB,
clear all, clcl
A = randn(10,3) + randn(10,3)*i;
save('example.mat','-v7.3'); %% The saved file is example.mat with version 7.3
above, the saved file is example.mat a matrix of size (10,3)
so, let's go to python to read that file :
import numpy as np
import h5py as h5
data_try = h5.File('example.mat', 'r')
A = np.array(data_try)
A = A.view(np.complex) #here the matrix equivalent to that one in matlab
but what i find that A in python is of size (3,10) !! and also when having matrix of three dimensions, the shape is changing !!
Is that normal that python reads the transpose of matrix coming from matlab ??!! or something wrong is happening !
However when using the other way as below:
import scipy.io as spio
Data = spio.loadmat('example.mat', squeeze_me=True)
A = Data[‘A’]
in that case, everything is really nice, but unfortunately we can not use that way for big matrices !!!
please, any solution for that issue ?

You might face a problem with different memory alignment in Matlab (column-major) and Numpy (row-major)... check e.g. this question for related discussion and a solution (reshaping in Fortran-style, which is also column-major).
SciPy's .mat interface automatically takes care of this reinterpretation, which is why you don't encounter the problem when using it.

Displaying a large .dat binary file in python

I have a large 40 mb (about 173,397 lines) .dat file filled with binary data (random symbols). It is an astronomical photograph. I need to read and display it with Python. I am using a binary file because I will need to extract pixel value data from specific regions of the image. But for now I just need to ingest it into Python. Something like the READU procedure in IDL. Tried numpy and matplotlib but nothing worked. Suggestions?

You need to know the data type and dimensions of the binary file. For example, if the file contains float data, use numpy.fromfile like:
import numpy as np
data = np.fromfile(filename, dtype=float)
Then reshape the array to the dimensions of the image, dims, using numpy.reshape (the equivalent of REFORM in IDL):
im = np.reshape(data, dims)

third dimension must be 3 or 4 in matplotlib while i want to save 2

I have an image converted from rgb to L*a*b space.
I can save L part separately but not ab part together.
plt.imsave(path,ab_part) gives error. The last dimension should be 3 or 4
where in my case: ab_part.ndim=3 and ab_part.shape=a x b x 2 one dimension is saved as L part.
So how can I save it? I have also tried to save as savefig after doing plt.figure.fromarray('.')
So how can I save it as image or if not then a flattened array would be ok. any suggestion?
p.s. Please suggest only scipy, numpy, matplotlib but not cv2

What you're trying to save is not an image (at least, not one with a common file format), it's just data. Use np.save, or even just the builtin pickle module, to save arrays.

Converting PNG file to bitmap array in Python

I would like to convert a PNG image to a 2 dimensional array where each array holds a list of the RGB values of that specific pixel. How could one create a program to read-in a *.png file and convert to this type of data structure?

If you have PIL installed then you can create an image with Image.open and get the colors like so:
data = [image.getpixel((x, y)) for x in range(image.width) for y in range(image.height)]

You can use the existing pygame module. Import a file into a Surface using pygame.image.load. You can then access the bit array from this using pygame.surfarray.array2d. Please see the Pygame docs for more information.

You can use wand for such basic tasks. The syntax is very easy to read unlike other ImageMagik libs. Basically you'd do something like:
from wand.image import Image
from wand.display import display
array = []
with Image(filename='yourfile.png') as img:
array.append(img.channel_images) # this is most likely wrong, but it should be something similar
It will be along those lines. Once I leave the office I will try this out.

Best dtype for creating large arrays with numpy

I am looking to store pixel values from satellite imagery into an array. I've been using
np.empty((image_width, image_length)
and it worked for smaller subsets of an image, but when using it on the entire image (3858 x 3743) the code terminates very quickly and all I get is an array of zeros.
I load the image values into the array using a loop and opening the image with gdal
img = gdal.Open(os.path.join(fn + "\{0}".format(fname))).ReadAsArray()
but when I include print img_array I end up with just zeros.
I have tried almost every single dtype that I could find in the numpy documentation but keep getting the same result.
Is numpy unable to load this many values or is there a way to optimize the array?
I am working with 8-bit tiff images that contain NDVI (decimal) values.
Thanks

Not certain what type of images you are trying to read, but in the case of radarsat-2 images you can the following:
dataset = gdal.Open("RADARSAT_2_CALIB:SIGMA0:" + inpath + "product.xml")
S_HH = dataset.GetRasterBand(1).ReadAsArray()
S_VV = dataset.GetRasterBand(2).ReadAsArray()
# gets the intensity (Intensity = re**2+imag**2), and amplitude = sqrt(Intensity)
self.image_HH_I = numpy.real(S_HH)**2+numpy.imag(S_HH)**2
self.image_VV_I = numpy.real(S_VV)**2+numpy.imag(S_VV)**2
But that is specifically for that type of images (in this case each image contains several bands, so i need to read in each band separately with GetRasterBand(i), and than do ReadAsArray() If there is a specific GDAL driver for the type of images you want to read in, life gets very easy
If you give some more info on the type of images you want to read in, i can maybe help more specifically
Edit: did you try something like this ? (not sure if that will work on tiff, or how many bits the header is, hence the something:)
A=open(filename,"r")
B=numpy.fromfile(A,dtype='uint8')[something:].reshape(3858,3743)
C=B*1.0
A.close()
Edit: The problem is solved when using 64bit python instead of 32bit, due to memory errors at 2Gb when using the 32bit python version.