MemoryError when trying to normalize an array of images - python

I have a folder containing 110k images with shape (256, 256, 3) each. I'm reading one by one, converting to a numpy array and storing in a list. After that, I convert the list to a numpy array. The shape of the numpy array is (110000, 256, 256, 3). Then, when I try to normalize the images with images = images / float(255), this error is displayed:
File "loading_images.py", line 25, in <module>
images = images / float(255)
MemoryError: Unable to allocate 161. GiB for an array with shape (110000, 256, 256, 3) and data type float64
Is there any other way to do that?
My current code is this:
files = glob.glob(dir + "*.png")
images = []
for f in files
im = cv2.imread(f)
img_arr = np.asarray(im)
images.append(img_arr)
images = np.asarray(images)
images = images / float(255)

Think your issue is that cv2 gives an int8 (correct me if I'm wrong), and you're trying to cast the values into float64's
import numpy as np
print(np.float64(255).itemsize)
print(np.int8(255).itemsize)
Which means that after the typecast, you're left with approximately 8 times the bytes. you have 110000×256×256×3=21GB of image data to begin with, which is probably just within your RAM limitation. After the conversion to float, you get 8x21 = 168GB of data, which is above the RAM limit of anyone I know haha.
This is no solution however, do you really need to load all the images at the same time?

Related

load images into a numpy matrix array from a dataset and apply conversion

I need to load images into a numpy matrix array from a dataset which contains 800 images of each being 64 by 64 pixels. I need to convert each 64 by 64 image into a row of the matrix which has 4096 columns. Below I have shown how I am approaching the code. I recieve a ValueError: cannot reshape array of size 4096 into shape (64,). Please help thank you.
array = np.zeros((800, 64))
for i in range(800):
path = “some path”
img = mpimg.imread(path)
array[i] = img.reshape(64)
Your original array should be of shape 800, 4096, since each sub-array represents a (64, 64) image, which requires 4096 elements.
Therefore, I think you want this instead:
array = np.zeros((800, 4096))
paths = [...] # set paths here
for i, path in enumerate(paths):
array[i] = mpimg.imread(path).reshape(4096)

Why is it showing different images in 4d numpy array?

I'm trying to create a 4D array for a bunch of 3D images. I can load the image and show the image correctly, but after storing it to the 4D array and show the image from the array, it shows gibberish.
I tried to compare if the image loaded and the one read from the 4D array is equal, and it prints True.
import os
from glob import glob
import numpy as np
from PIL import Image
IMG_PATH = '32x32'
img_paths = glob(os.path.join(IMG_PATH, '*.jpg'))
images = np.empty((len(img_paths), 32, 32, 3))
for i, path_i in enumerate(img_paths):
img_i = np.array(Image.open(path_i))
Image.fromarray(img_i, 'RGB').show() # showing correct image
images[i] = img_i
Image.fromarray(images[i], 'RGB').show() # showing gibberish
print(np.array_equal(img_i, images[i])) # True
if i == 0:
break
I expect to show the exact same image as I run images[i] = img_i.
This line is performing a cast:
images[i] = img_i
Since images.dtype == np.float64, but img_i.dtype is probably np.uint8.
You can catch this type of mistake by specifying a casting rule:
np.copy_to(images[i], img_i, casting='no')
# TypeError: Cannot cast scalar from dtype('uint8') to dtype('float64') according to the rule 'no'
You can fix this by allocating the array with the right type:
images = np.empty((len(img_paths), 32, 32, 3), dtype=np.uint8)
Or you can let numpy do the allocation for you, but this will temporarily use almost twice the memory:
images = np.stack([
Image.open(path_i)
for path_i in img_paths
], axis=0)

Numpy ignoring frames of TIF file when converting from PIL

I have 3 diemnsional image saved as a multi page tif file. I tried reading it in using PIL(low) and it detected the correct number of frames but when I convert it to numpy it ignores the frames and only converts the single page/layer.
from PIL import Image
import numpy as np
pil_ = Image.open(path)
pil_.size # this outputs (1024, 512)
pil_.n_frames # this outputs the correct number of frames i.e. 21
num = np.array(pil_)
num.shape # this outputs (512, 1024)
Shouldnt the numpy array be a 3D array? How do I convert it so that the frames are also considered?
Thanks
There's a thread here (https://mail.python.org/pipermail/python-list/2007-May/419217.html) which suggests that you may be able to manually seek through the frames and assign each one into the 3rd dimension of your numpy array.

(Loading .mat file as image in OpenCV) - Reshaping numpy array with 128 channels to 3 channels

I am trying to load .mat image exported from Tensorflow with Scipy.io using OpenCV.
I can modify the Tensorflow code to export the .mat image with only 3 channels directly but I will lose a lot of data and it doesn't look correct even.
And that's why I am trying to export the raw data as it is.
In my case I load the .mat file with scipy.io and get the numpy array which looks like this
(640, 640, 128)
and I want to reshape it because OpenCV cannot load an image with 128 channels.
(640, 640, 3)
I am not fully understanding the concept of reshaping and I think I am doing it wrong.
I am getting this error:
ValueError: cannot reshape array of size 52428800 into shape
(640,640,3)
Thank you and have a good day,
Hesham
Edit 1:
That's the code:
import cv2
import scipy.io as sio
import numpy as np
matfile = 'docia.mat'
img = sio.loadmat(matfile)
img_reshaped = img['embedmap'].reshape(640, 640, 3)
cv2.imshow("segmented_map", img['embedmap'])
cv2.waitKey(0)`
Re-shaping is using when you want to retain all of the data but in a different shape. I believe that you are trying to drop 125 of the 128 channels. To do this you can just use indexing to get the first 3 channels:
img_reshaped = img['embedmap'][:, :, :3]
Also you are passing img['embedmap'], not the reshaped img_reshaped into cv2.imshow().
Although I would recommend looking at them 1 by 1 in grey scale.
for i in range(128):
cv2.imshow("segmented_map", img['embedmap'][:, :, i])
cv2.waitKey(0)

Opening a single image

I'm trying to open an image with size (520,696) but when I use this
array = np.array([np.array(Image.open(folder_path+folders+'/'+'images'+'/'+image))], np.int32).shape`
I'm getting the shape as
(1, 520, 696, 4)
The problem is with this shape I can't convert it to image using toimage(array); I get
'arr' does not have a suitable array shape for any mode.
Any suggestions on how may I read that image using only (520,696)?
The problem is the additional dumb dimension. You can remove it using:
arr = np.squeeze(arr)
You should load the picture as a single picture instead of loading it as a stack and then removing the irrelevant stack dimension. The basic procedure could be something like this:
from PIL import Image
pic = Image.open("test.jpg")
pic.show() #yup, that's the picture
arr = np.array(pic) #convert it to a numpy array
print(arr.shape, arr.dtype) #dimension and data type
arr //= 2 #now manipulate this array
new_pic = Image.fromarray(arr) #and keep it for later
new_pic.save("newpic.bmp") #maybe in a different format

Categories

Resources