I want to create a png or tiff image file from a very large h5py dataset that cannot be loaded into memory all at once. So, I was wondering if there is a way in python to write to a png or tiff file in patches? (I can load the h5py dataset in slices to a numpy.ndarray).
I've tried using the pillow library and doing PIL.Image.paste giving the box coordinates, but for large images it goes out of memory.
Basically, I'm wondering if there's a way to do something like:
for y in range(0, height, patch_size):
for x in range(0, width, patch_size):
y2 = min(y + patch_size, height)
x2 = min(x + patch_size, width)
# image_arr is an h5py dataset that cannot be loaded completely
# in memory, so load it in slices
image_file.write(image_arr[y:y2, x:x2], box=(y, x, y2, x2))
I'm looking for a way to do this, without having the whole image loaded into memory. I've tried the pillow library, but it loads/keeps all the data in memory.
Edit: This question is not about h5py, but rather how extremely large images (that cannot be loaded into memory) can we written out to a file in patches - similar to how large text files can be constructed by writing to it line by line.
Try tifffile.memmap:
from tifffile import memmap
image_file = memmap('temp.tif', shape=(height, width), dtype=image_arr.dtype,
bigtiff=True)
for y in range(0, height, patch_size):
for x in range(0, width, patch_size):
y2 = min(y + patch_size, height)
x2 = min(x + patch_size, width)
image_file[y:y2, x:x2] = image_arr[y:y2, x:x2]
image_file.flush()
This creates a uncompressed BigTIFF file with one strip. Memory-mapped tiles are not implemented yet. Not sure how many libraries can handle that kind of file, but you can always directly read from the strip using the meta data in the TIFF tags.
Short answer to "if there is a way in Python to write to a png or tiff file in patches?". Well, yes - everything is possible in Python, given enough time and skill to implement it. On the other hand, NO, there is no ready-made solution for this - because it doesn't appear to be very useful.
I don't know about TIFF and a comment here says it is limited to 4GB, so this format is likely not a good candidate. PNG has no practical limit and can be written in chunks, so it is doable in theory - on the condition that at least one scan line of your resulting image does fit into memory.
If you really want to go ahead with this, here is the info that you need:
A PNG file consists of a few metadata chunks and a series of image data chunks. The latter are independent of each other and you can therefore construct a big image out of several smaller images (each of which contains a whole number of rows, a minimum of one row) by simply concatenating their image data chunks (IDAT) together and adding the needed metadata chunks (you can pick those from the first small image, except for the IHDR chunk - that one will need to be constructed to contain the final image size).
So, here is how I'd do it, if I had to (NOTE you will need some understanding of Python's bytes type and the methods of converting byte sequences to and from Python data types to pull this off):
find how many rows I can fit into memory and make that the height of my "small image chunk". The width is the width of the entire final image. let's call those width and small_height
go through my giant data set in h5py one chunk at a time (width * small_height), convert it to PNG and save it to disk in a temporary file, or if your image conversion library allows it - directly to a bytes string in memory. Then process the byte data as follows and delete it at the end:
-- on the first iteration: walk through the PNG data one record at a time (see the PNG spec: http://www.libpng.org/pub/png/spec/1.2/png-1.2-pdg.html, it is in length-tag-value form and very easy to write code that efficiently walks over the file record by record), save ALL the records into my target file, except: modify IHDR to have the final image size and skip the IEND record.
-- on all subsequent iterations: scan through the PNG data and pick only the IDAT records, write those out to the output file.
append an IEND record to the target file.
All done - you should now have a valid humongous PNG. I wonder who or what could read that, though.
Related
I'm having trouble loading a lot of small image files (aprox. 90k png images) into a single 3D np.array.
The current solution takes couple of hours which is unacceptable.
The images are size 64x128.
I have a pd.DataFrame called labels with the names of the images and want to import whose images in the same order as in the labels variable.
My current solution is:
dataset = np.empty([1, 64, 128], dtype=np.int32)
for file_name in labels['file_name']:
array = cv.imread(f'{IMAGES_PATH}/{file_name}.png', cv.COLOR_BGR2GRAY)
dataset = np.append(dataset, [array[:]], axis=0)
From what I have timed, the most time consuming operation is the dataset = np.append(dataset, [array[:]], axis=0), which takes around 0.4s per image.
Is there any better way to import such files and store them in a np.array?
I was thinking about multiprocessing, but I want the labels and dataset to be in the same order.
Game developers typically concatenate bunches of small images into a single big file and then use sizes and offsets to slice out the currently needed piece. Here's example of how this can be done with imagemagick:
montage -mode concatenate -tile 1x *.png out.png
But then again it will not get around the reading of 90k of small files. And magick has it's own peculiarities which may or may not surface in your case.
Also, I haven't originally noticed that you are having problem with np.append(dataset, [array[:]], axis=0).
That is very bad line. Appending in a loop is never a performant code.
Either preallocate the array and write to it. Or use numpy's functions for concatenating many arrays at once:
dataset = np.empty([int(90e3), 64, 128], dtype=np.int32)
for i,file_name in enumerate(labels['file_name']):
array = cv.imread(f'{IMAGES_PATH}/{file_name}.png', cv.COLOR_BGR2GRAY)
dataset[i,:,:] = array
I'm doing some fast image processing in Python (with Numpy/Scipy + OpenCV).
There are several thousand images which are exactly the same shape - once I read the first one, I know exactly how all others will look like.
The problem is that reading every next image from disk causes allocation of new memory (which is slow). Is there a way to avoid it by reading every next image directly into some already existing memory (ndarray)? I know of cv2.imdecode which in C++ can accept a pointer to pre-allocated Mat, but it does not seem to have a Python binding (the only option is to return a whole new array).
I need this for multiprocessing - I'd like to read images into a shared memory, then do some heavy work on them in worker processes. Right now I'm forced to copy data from the array allocated and returned by cv2.imread into that shared memory, which again takes time. I'd like to be able to write there directly.
height, width = (50, 50)
image = np.zeros((height, width))
id(image)
# outputs: 140411457307552
image[:, :] = np.ones((height, width))
id(image)
# outputs: 140411457307552
image = np.ones((height, width))
id(image)
# outputs -> 140411437723280
# when reading from disk (assuming your images are 50x50 pixels)
image[:, :] = cv2.imread("/home/.../your_im_50x50.png")
By addressing each image's dimensions python will try to store the given array into the existing one. This results in a memory assignation to the pre-allocated memory zone. If arrays do not have the same shape it will raise a ValueError.
When mentioning only the variable name, a new reference to the array is created, resulting in a new object in memory (cf ids)
I've just translated some CT reconstruction software from IDL into Python, and this is my first experience ever with Python. The code works fine except that it's much, much slower. This is due, in part, to the fact that IDL allows me to save memory and time by reading in just one row of an image at a time, using the following:
image = read_tiff(filename, sub_rect = [0, slice, x, 1])
I need to read one row each from 1800 different projection images, but as far as I can tell I can only create an image array by reading in the entire image and then converting it to an array. Is there any trick to just read in one row from the start, since I don't need the other 2047 rows?
It looks like tifffile.py by Christoph Gohlke (http://www.lfd.uci.edu/~gohlke/code/tifffile.py.html) can do the job.
from tiffile import TiffFile
with TiffFile('test.tiff') as tif:
for page in tif:
image = page.asarray(memmap=True)
print image[0,:,:]
If I interpret the code correctly this will extract the first row of every page in the file without loading the whole file into memory (through numpy.memmap).
I am looking to store pixel values from satellite imagery into an array. I've been using
np.empty((image_width, image_length)
and it worked for smaller subsets of an image, but when using it on the entire image (3858 x 3743) the code terminates very quickly and all I get is an array of zeros.
I load the image values into the array using a loop and opening the image with gdal
img = gdal.Open(os.path.join(fn + "\{0}".format(fname))).ReadAsArray()
but when I include print img_array I end up with just zeros.
I have tried almost every single dtype that I could find in the numpy documentation but keep getting the same result.
Is numpy unable to load this many values or is there a way to optimize the array?
I am working with 8-bit tiff images that contain NDVI (decimal) values.
Thanks
Not certain what type of images you are trying to read, but in the case of radarsat-2 images you can the following:
dataset = gdal.Open("RADARSAT_2_CALIB:SIGMA0:" + inpath + "product.xml")
S_HH = dataset.GetRasterBand(1).ReadAsArray()
S_VV = dataset.GetRasterBand(2).ReadAsArray()
# gets the intensity (Intensity = re**2+imag**2), and amplitude = sqrt(Intensity)
self.image_HH_I = numpy.real(S_HH)**2+numpy.imag(S_HH)**2
self.image_VV_I = numpy.real(S_VV)**2+numpy.imag(S_VV)**2
But that is specifically for that type of images (in this case each image contains several bands, so i need to read in each band separately with GetRasterBand(i), and than do ReadAsArray() If there is a specific GDAL driver for the type of images you want to read in, life gets very easy
If you give some more info on the type of images you want to read in, i can maybe help more specifically
Edit: did you try something like this ? (not sure if that will work on tiff, or how many bits the header is, hence the something:)
A=open(filename,"r")
B=numpy.fromfile(A,dtype='uint8')[something:].reshape(3858,3743)
C=B*1.0
A.close()
Edit: The problem is solved when using 64bit python instead of 32bit, due to memory errors at 2Gb when using the 32bit python version.
I have used PIL to convert and resize JPG/BMP file to PNG format. I can easily resize and convert it to PNG, but the file size of the new image is too big.
im = Image.open('input.jpg')
im_resize = im.resize((400, 400), Image.ANTIALIAS) # best down-sizing filter
im.save(`output.png')
What do I have to do to reduce the image file size?
PNG Images still have to hold all data for every single pixel on the image, so there is a limit on how far you can compress them.
One way to further decrease it, since your 400x400 is to be used as a "thumbnail" of sorts, is to use indexed mode:
im_indexed = im_resize.convert("P")
im_resize.save(... )
*wait *
Just saw an error in your example code:
You are saving the original image, not the resized image:
im=Image.open(p1.photo)
im_resize = im.resize((400, 400), Image.ANTIALIAS) # best down-sizing filter
im.save(str(merchant.id)+'_logo.'+'png')
When you should be doing:
im_resize.save(str(merchant.id)+'_logo.'+'png')
You are just saving back the original image, that is why it looks so big. Probably you won't need to use indexed mode them.
Aother thing: Indexed mode images can look pretty poor - a better way out, if you come to need it, might be to have your smalle sizes saved as .jpg instead of .png s - these can get smaller as you need, trading size for quality.
You can use other tools like PNGOUT