I have a bunch of images in a directory and want to make a single ndarray of those images like CIFAR-10. I have written a brute-force way to do so. However it will get really slow if the number of images are large.
def dataLoader(data_path):
data = np.empty([1,1200,900,3]) # size of an image is 1200*900 with RGB
i = 0
for filename in os.listdir(data_path):
if filename.endswith(".jpg") or filename.endswith(".png"):
target = cv2.imread(os.path.join(data_path, filename))
data = np.concatenate([data, target.reshape([1,1200,900,3])], axis=0)
print(i, end='\r')
i += 1
return data
I am checking the progression by printing loop count. As I see that, it is fairly quick at first 50 iterations but it gets slower and slower as it iterates. I suppose it is due to the numpy concatenation. Is there better way to do so?
I also really appreciate if there is a method to save the ndarray so that I don't need to build the ndarray every time. Currently, I'm using numpy.save.
Related
I have a problem and don't know how to solve:
I'm learning how to analyze DICOM files with Python and, so,
I got a patient exam, on single patient and one single exam, which is 200 DICOM files all of the size 512x512 each archive representing a different layer of him and I want to turn them into a single archive .npy so I can use in another tutorial that I found online.
Many tutorials try to convert them to jpg or png using opencv first, but I don't want this since I'm not interested in a friendly image to see right now, I need the array. Also, this step screw all the quality of images.
I already know that using:
medical_image = pydicom.read_file(file_path)
image = medical_image.pixel_array
I can grab the path, turn 1 slice in a pixel array and them use it, but the thing is, it doesn't work in a for loop.
The for loop I tried was basically this:
image = [] # to create an empty list
for f in glob.iglob('file_path'):
img = pydicom.dcmread(f)
image.append(img)
It results in a list with all the files. Until here it goes well, but it seems it's not the right way, because I can use the list and can't find the supposed next steps anywhere, not even answers to the errors that I get in this part, (so I concluded it was wrong)
The following code snippet allows to read DICOM files from a folder dir_path and to store them into a list. Actually, the list does not consist of the raw DICOM files, but is filled with NumPy arrays of Hounsfield units (by using the apply_modality_lut function).
import os
from pathlib import Path
import pydicom
from pydicom.pixel_data_handlers import apply_modality_lut
dir_path = r"path\to\dicom\files"
dicom_set = []
for root, _, filenames in os.walk(dir_path):
for filename in filenames:
dcm_path = Path(root, filename)
if dcm_path.suffix == ".dcm":
try:
dicom = pydicom.dcmread(dcm_path, force=True)
except IOError as e:
print(f"Can't import {dcm_path.stem}")
else:
hu = apply_modality_lut(dicom.pixel_array, dicom)
dicom_set.append(hu)
You were well on your way. You just have to build up a volume from the individual slices that you read in. This code snippet will create a pixelVolume of dimension 512x512x200 if your data is as advertised.
import dicom
import numpy
images = [] # to create an empty list
# Read all of the DICOM images from file_path into list "images"
for f in glob.iglob('file_path'):
image = pydicom.dcmread(f)
images.append(image)
# Use the first image to determine the number of rows and columns
repImage = images[0]
rows=int(repImage.Rows)
cols=int(repImage.Columns)
slices=len(images)
# This tuple represents the dimensions of the pixel volume
volumeDims = (rows, cols, slices)
# allocate storage for the pixel volume
pixelVolume = numpy.zeros(volumeDims, dtype=repImage.pixel_array.dtype)
# fill in the pixel volume one slice at a time
for image in images:
pixelVolume[:,:,i] = image.pixel_array
#Use pixelVolume to do something interesting
I don't know if you are a DICOM expert or a DICOM novice, but I am just accepting your claim that your 200 images make sense when interpreted as a volume. There are many ways that this may fail. The slices may not be in expected order. There may be multiple series in your study. But I am guessing you have a "nice" DICOM dataset, maybe used for tutorials, and that this code will help you take a step forward.
Let's assume there are two images. One is called small image and another one is called big image. I want to randomly generate the small image inside the different parts of the big image one at a time everytime I run.
So, currently I have this image. Let's call it big image
I also have smaller image:
def mask_generation(blob_index,image_index):
experimental_image = markup_images[image_index]
h, w = cropped_images[blob_index].shape[:2]
x = np.random.randint(experimental_image.shape[0] - w)
y = np.random.randint(experimental_image.shape[1] - h)
experimental_image[y:y+h, x:x+w] = cropped_images[blob_index]
return experimental_image
I have created above function to generate the small image in big image everytime I call this function. Note: blob index is the index that I use to call specific 'small image' since I have a collection of those small images and image_index is the index to call specific 'big images'. Big images are stored in the list called experimental_image and small images are stored in list called markup images
However, when I run this, I do get the small image randomly generated but the previously randomly generated image never gets deleted and I am not too sure how to proceed with it,
Example: When I run it once
When I run it twice
How do I fix this? Any help will be appreciated. Thank you
I tried the above code but didn't work as I wanted it to work
I suppose you only want the small random image you generated in this iteration in your code.
The problem you have, is due to the modification of your calling args.
When you call your function multiple times with the same big image
markup_image = ...
result_1 = mask_generation(blob_index=0, image_index=0)
result_2 = mask_generation(blob_index=1, image_index=0)
You get in result_2 both small images.
This is due to the writing to the original image in
experimental_image[y:y+h, x:x+w] = cropped_images[blob_index]
This adds the small image to your original image in your list of images.
When getting this image the next time, the small image is already there.
To fix:
Do not alter your images, e.g. by first copying the image and then adding the small image in your function
Probably even better: Only give your function a big and small image, and make sure that they always receive a copy
I'm having trouble loading a lot of small image files (aprox. 90k png images) into a single 3D np.array.
The current solution takes couple of hours which is unacceptable.
The images are size 64x128.
I have a pd.DataFrame called labels with the names of the images and want to import whose images in the same order as in the labels variable.
My current solution is:
dataset = np.empty([1, 64, 128], dtype=np.int32)
for file_name in labels['file_name']:
array = cv.imread(f'{IMAGES_PATH}/{file_name}.png', cv.COLOR_BGR2GRAY)
dataset = np.append(dataset, [array[:]], axis=0)
From what I have timed, the most time consuming operation is the dataset = np.append(dataset, [array[:]], axis=0), which takes around 0.4s per image.
Is there any better way to import such files and store them in a np.array?
I was thinking about multiprocessing, but I want the labels and dataset to be in the same order.
Game developers typically concatenate bunches of small images into a single big file and then use sizes and offsets to slice out the currently needed piece. Here's example of how this can be done with imagemagick:
montage -mode concatenate -tile 1x *.png out.png
But then again it will not get around the reading of 90k of small files. And magick has it's own peculiarities which may or may not surface in your case.
Also, I haven't originally noticed that you are having problem with np.append(dataset, [array[:]], axis=0).
That is very bad line. Appending in a loop is never a performant code.
Either preallocate the array and write to it. Or use numpy's functions for concatenating many arrays at once:
dataset = np.empty([int(90e3), 64, 128], dtype=np.int32)
for i,file_name in enumerate(labels['file_name']):
array = cv.imread(f'{IMAGES_PATH}/{file_name}.png', cv.COLOR_BGR2GRAY)
dataset[i,:,:] = array
I need to perform a function on images in less than 1 second. I have a problem on a 1000x1000 image that, just to load it as a matrix in the program, takes 1 second.
The function I use to load it is as follows:
import png
def load(fname):
with open(fname, mode='rb') as f:
reader = png.Reader(file=f)
w, h, png_img, _ = reader.asRGB8()
img = []
for line in png_img:
l = []
for i in range(0, len(line), 3):
l+=[(line[i], line[i+1], line[i+2])]
img+=[l]
return img
How can I modify it in such a way that, when opening the image, it takes a little more than a few milliseconds?
IMPORTANT NOTE: I cannot import other functions outside of this (this is a university exercise and therefore there are rules -.-). So I have to get one myself
you can use PIL to do this for you, it's highly optimized and fast
from PIL import Image
def load(path):
return Image.open(path)
Appending to a list is inherently slow - read about Shlemiel the painter’s algorithm. You can replace it with a generator expression and slicing.
for line in png_img:
img += list(zip(line[0::3], line[1::3], line[2::3])
I'm not sure it is remotely possible to run a python script that opens a file, etc. in just a few ms. On my computer, the simplest program takes several 10ms
Without knowing more about the specifics of your problem and the reasons for your constraint, it is hard to answer. You should consider what you are trying to do, in the context of the way your program really works, and then formulate a strategy to achieve your goal.
The total context here is, you're asking the computer to:
run python, load your code and interpret it
load any modules you want to use
find your image file and read it from disk
give those bytes some meaning as an image abstraction - parse, etc these bytes
do some kind of transform or "work" on the image
export your result in some way
You need to figure out which of those steps is it that really needs to be lightning fast. After that, maybe someone can make a suggestion.
I am looking to store pixel values from satellite imagery into an array. I've been using
np.empty((image_width, image_length)
and it worked for smaller subsets of an image, but when using it on the entire image (3858 x 3743) the code terminates very quickly and all I get is an array of zeros.
I load the image values into the array using a loop and opening the image with gdal
img = gdal.Open(os.path.join(fn + "\{0}".format(fname))).ReadAsArray()
but when I include print img_array I end up with just zeros.
I have tried almost every single dtype that I could find in the numpy documentation but keep getting the same result.
Is numpy unable to load this many values or is there a way to optimize the array?
I am working with 8-bit tiff images that contain NDVI (decimal) values.
Thanks
Not certain what type of images you are trying to read, but in the case of radarsat-2 images you can the following:
dataset = gdal.Open("RADARSAT_2_CALIB:SIGMA0:" + inpath + "product.xml")
S_HH = dataset.GetRasterBand(1).ReadAsArray()
S_VV = dataset.GetRasterBand(2).ReadAsArray()
# gets the intensity (Intensity = re**2+imag**2), and amplitude = sqrt(Intensity)
self.image_HH_I = numpy.real(S_HH)**2+numpy.imag(S_HH)**2
self.image_VV_I = numpy.real(S_VV)**2+numpy.imag(S_VV)**2
But that is specifically for that type of images (in this case each image contains several bands, so i need to read in each band separately with GetRasterBand(i), and than do ReadAsArray() If there is a specific GDAL driver for the type of images you want to read in, life gets very easy
If you give some more info on the type of images you want to read in, i can maybe help more specifically
Edit: did you try something like this ? (not sure if that will work on tiff, or how many bits the header is, hence the something:)
A=open(filename,"r")
B=numpy.fromfile(A,dtype='uint8')[something:].reshape(3858,3743)
C=B*1.0
A.close()
Edit: The problem is solved when using 64bit python instead of 32bit, due to memory errors at 2Gb when using the 32bit python version.