I'm doing some fast image processing in Python (with Numpy/Scipy + OpenCV).
There are several thousand images which are exactly the same shape - once I read the first one, I know exactly how all others will look like.
The problem is that reading every next image from disk causes allocation of new memory (which is slow). Is there a way to avoid it by reading every next image directly into some already existing memory (ndarray)? I know of cv2.imdecode which in C++ can accept a pointer to pre-allocated Mat, but it does not seem to have a Python binding (the only option is to return a whole new array).
I need this for multiprocessing - I'd like to read images into a shared memory, then do some heavy work on them in worker processes. Right now I'm forced to copy data from the array allocated and returned by cv2.imread into that shared memory, which again takes time. I'd like to be able to write there directly.
height, width = (50, 50)
image = np.zeros((height, width))
id(image)
# outputs: 140411457307552
image[:, :] = np.ones((height, width))
id(image)
# outputs: 140411457307552
image = np.ones((height, width))
id(image)
# outputs -> 140411437723280
# when reading from disk (assuming your images are 50x50 pixels)
image[:, :] = cv2.imread("/home/.../your_im_50x50.png")
By addressing each image's dimensions python will try to store the given array into the existing one. This results in a memory assignation to the pre-allocated memory zone. If arrays do not have the same shape it will raise a ValueError.
When mentioning only the variable name, a new reference to the array is created, resulting in a new object in memory (cf ids)
Related
I'm having trouble loading a lot of small image files (aprox. 90k png images) into a single 3D np.array.
The current solution takes couple of hours which is unacceptable.
The images are size 64x128.
I have a pd.DataFrame called labels with the names of the images and want to import whose images in the same order as in the labels variable.
My current solution is:
dataset = np.empty([1, 64, 128], dtype=np.int32)
for file_name in labels['file_name']:
array = cv.imread(f'{IMAGES_PATH}/{file_name}.png', cv.COLOR_BGR2GRAY)
dataset = np.append(dataset, [array[:]], axis=0)
From what I have timed, the most time consuming operation is the dataset = np.append(dataset, [array[:]], axis=0), which takes around 0.4s per image.
Is there any better way to import such files and store them in a np.array?
I was thinking about multiprocessing, but I want the labels and dataset to be in the same order.
Game developers typically concatenate bunches of small images into a single big file and then use sizes and offsets to slice out the currently needed piece. Here's example of how this can be done with imagemagick:
montage -mode concatenate -tile 1x *.png out.png
But then again it will not get around the reading of 90k of small files. And magick has it's own peculiarities which may or may not surface in your case.
Also, I haven't originally noticed that you are having problem with np.append(dataset, [array[:]], axis=0).
That is very bad line. Appending in a loop is never a performant code.
Either preallocate the array and write to it. Or use numpy's functions for concatenating many arrays at once:
dataset = np.empty([int(90e3), 64, 128], dtype=np.int32)
for i,file_name in enumerate(labels['file_name']):
array = cv.imread(f'{IMAGES_PATH}/{file_name}.png', cv.COLOR_BGR2GRAY)
dataset[i,:,:] = array
Obviously this line is wrong.
matOut = cv2::Mat::Mat(height, width, cv2.CV_8UC4)
this one too.
Mat matOut(height, width, cv2.CV_8UC4)
How do I create an modern openCV empty matrix with a given size, shape, format? The latest openCV docs on topic don't seem to be helping... I gleaned the formats used above directly from that document. Note: I'm assuming OpenCV (import cv2) Python3, import numpy, etc...
I was looking to create an empty matrix with the intent of copying content from a different buffer into it...
edit, more failed attempts...
matOut = cv2.numpy.Mat(height, width, cv2.CV_8UC4)
matOut = numpy.array(height, width, cv2.CV_8UC4)
So user696969 called out a largely successful solution to the question asked. You create a new shaped area via:
matOut = numpy.zeros([height, width, 4], dtype=numpy.uint8)
note I have replaced the desired content, cv2.CV_8UC4, with its expected response, the number 4. There are 4 eight bit bytes in a simple RGBA pixel descriptor. I would have preferred if OpenCV tooling had performed that response as a function call, but that didn't seem to work...
I do want to share my use case. I originally was going to create an empty shaped matrix so I could transfer data from a single dimensional array there. As I worked this problem I realized there was a better way. I start the routine where I have received a file containing 8 bit RGBA data without any prefix metadata. Think raw BMP without any header info.
matContent = numpy.frombuffer(fileContent, numpy.uint8)
matContentReshaped = matContent.reshape(height, width, 4)
cv2.imshow("Display Window", matContentReshaped)
k = cv2.waitKey(0)
Thats it. Easy, Peasy.... Thanks to user696969 and eldesgraciado for help here.
I want to create a png or tiff image file from a very large h5py dataset that cannot be loaded into memory all at once. So, I was wondering if there is a way in python to write to a png or tiff file in patches? (I can load the h5py dataset in slices to a numpy.ndarray).
I've tried using the pillow library and doing PIL.Image.paste giving the box coordinates, but for large images it goes out of memory.
Basically, I'm wondering if there's a way to do something like:
for y in range(0, height, patch_size):
for x in range(0, width, patch_size):
y2 = min(y + patch_size, height)
x2 = min(x + patch_size, width)
# image_arr is an h5py dataset that cannot be loaded completely
# in memory, so load it in slices
image_file.write(image_arr[y:y2, x:x2], box=(y, x, y2, x2))
I'm looking for a way to do this, without having the whole image loaded into memory. I've tried the pillow library, but it loads/keeps all the data in memory.
Edit: This question is not about h5py, but rather how extremely large images (that cannot be loaded into memory) can we written out to a file in patches - similar to how large text files can be constructed by writing to it line by line.
Try tifffile.memmap:
from tifffile import memmap
image_file = memmap('temp.tif', shape=(height, width), dtype=image_arr.dtype,
bigtiff=True)
for y in range(0, height, patch_size):
for x in range(0, width, patch_size):
y2 = min(y + patch_size, height)
x2 = min(x + patch_size, width)
image_file[y:y2, x:x2] = image_arr[y:y2, x:x2]
image_file.flush()
This creates a uncompressed BigTIFF file with one strip. Memory-mapped tiles are not implemented yet. Not sure how many libraries can handle that kind of file, but you can always directly read from the strip using the meta data in the TIFF tags.
Short answer to "if there is a way in Python to write to a png or tiff file in patches?". Well, yes - everything is possible in Python, given enough time and skill to implement it. On the other hand, NO, there is no ready-made solution for this - because it doesn't appear to be very useful.
I don't know about TIFF and a comment here says it is limited to 4GB, so this format is likely not a good candidate. PNG has no practical limit and can be written in chunks, so it is doable in theory - on the condition that at least one scan line of your resulting image does fit into memory.
If you really want to go ahead with this, here is the info that you need:
A PNG file consists of a few metadata chunks and a series of image data chunks. The latter are independent of each other and you can therefore construct a big image out of several smaller images (each of which contains a whole number of rows, a minimum of one row) by simply concatenating their image data chunks (IDAT) together and adding the needed metadata chunks (you can pick those from the first small image, except for the IHDR chunk - that one will need to be constructed to contain the final image size).
So, here is how I'd do it, if I had to (NOTE you will need some understanding of Python's bytes type and the methods of converting byte sequences to and from Python data types to pull this off):
find how many rows I can fit into memory and make that the height of my "small image chunk". The width is the width of the entire final image. let's call those width and small_height
go through my giant data set in h5py one chunk at a time (width * small_height), convert it to PNG and save it to disk in a temporary file, or if your image conversion library allows it - directly to a bytes string in memory. Then process the byte data as follows and delete it at the end:
-- on the first iteration: walk through the PNG data one record at a time (see the PNG spec: http://www.libpng.org/pub/png/spec/1.2/png-1.2-pdg.html, it is in length-tag-value form and very easy to write code that efficiently walks over the file record by record), save ALL the records into my target file, except: modify IHDR to have the final image size and skip the IEND record.
-- on all subsequent iterations: scan through the PNG data and pick only the IDAT records, write those out to the output file.
append an IEND record to the target file.
All done - you should now have a valid humongous PNG. I wonder who or what could read that, though.
I have some trouble converting some amount (in this case, 153) of Numpy 2D arrays into a 3D array (these 2D arrays represent gray images - i.e. 2048x2048x1 - in order to deal with an image sequence instead of a set of 2D images). I need this to obtain the signal formed by each pixel value over time (which should be convenient with Numpy, once this problem is solved).
My code is (pretty much) the following :
zdim = len(imglist) # 'imglist' a Python list of the path for each image I need to process
windowspan = 512
xmin = ymin = 2
xmax = ymax = xmin + windowspan
sequence = []
for i in range(zdim):
hdulist = fits.open(imglist[i],'readonly') # allow to open FITS image files
hdr = hdulist[0].header['DATE-OBS'] # fetch the image date/time
img = fc.readfitsimg(imglist[i]) # return a np ndarray (2D)
patch = img[ymin:ymax, xmin:xmax] # take a small of the original image
print("patchSize : " + str(patch.size*4))
sequence.append(patch) # adding to the list
print("it : " + str(i))
sequence = np.array(sequence) # transform to numpy array
The interpreter returns a MemoryError after about 85 iterations...
Anyone would have any hints of what's happening ? (See some details below)
Some others details :
- I'm using WinPython 32 bits (portable), because I cannot install a 'proper' Python distribution (I switched between Python 2.7.9.4 and 3.4.3.3 for testing purpose)
- I'm forced to use a 32-bits Windows 7, on a PC which has 4GB, so 3.5GB usable / I've tried executing my script on another computer (Win7 64bits, 16GB of RAM)
Thanks for any help you could provide me with.
The MemoryError happens when your computer runs out of RAM memory. In this case, you seem to run out while adding all the images into a cube, when hitting the limit 85x512x512. If this was the only problem of the code, I would advice to use memmap to save the results directly to the hard drive instead of in the RAM. The memmap option is also available when you open a fits file fits.open(..., memmap=True). In that case, you only open the image in the disc, and read the parts you need, instead of loading the whole image in RAM.
But the real problem here, I suspect, is that you have been opening fits files without any closing at the end of the loop (hdu.close() in your case).
I am looking to store pixel values from satellite imagery into an array. I've been using
np.empty((image_width, image_length)
and it worked for smaller subsets of an image, but when using it on the entire image (3858 x 3743) the code terminates very quickly and all I get is an array of zeros.
I load the image values into the array using a loop and opening the image with gdal
img = gdal.Open(os.path.join(fn + "\{0}".format(fname))).ReadAsArray()
but when I include print img_array I end up with just zeros.
I have tried almost every single dtype that I could find in the numpy documentation but keep getting the same result.
Is numpy unable to load this many values or is there a way to optimize the array?
I am working with 8-bit tiff images that contain NDVI (decimal) values.
Thanks
Not certain what type of images you are trying to read, but in the case of radarsat-2 images you can the following:
dataset = gdal.Open("RADARSAT_2_CALIB:SIGMA0:" + inpath + "product.xml")
S_HH = dataset.GetRasterBand(1).ReadAsArray()
S_VV = dataset.GetRasterBand(2).ReadAsArray()
# gets the intensity (Intensity = re**2+imag**2), and amplitude = sqrt(Intensity)
self.image_HH_I = numpy.real(S_HH)**2+numpy.imag(S_HH)**2
self.image_VV_I = numpy.real(S_VV)**2+numpy.imag(S_VV)**2
But that is specifically for that type of images (in this case each image contains several bands, so i need to read in each band separately with GetRasterBand(i), and than do ReadAsArray() If there is a specific GDAL driver for the type of images you want to read in, life gets very easy
If you give some more info on the type of images you want to read in, i can maybe help more specifically
Edit: did you try something like this ? (not sure if that will work on tiff, or how many bits the header is, hence the something:)
A=open(filename,"r")
B=numpy.fromfile(A,dtype='uint8')[something:].reshape(3858,3743)
C=B*1.0
A.close()
Edit: The problem is solved when using 64bit python instead of 32bit, due to memory errors at 2Gb when using the 32bit python version.