I am trying to convert large (+50 GB) BigTiff images into JPEG 2000 format for post analysis using Python. I have succeeded on converting the large BigTiff files into JPEG using libvips; however, libvips does not have direct support for JPEG 2000 - what a bummer
I have been able to write JPEG 2000 images using glymur but the problem with glymur is that writing JPEG 2000 images is currently limited to images that can fit in memory. Since my workstation haves only 8 GB of RAM it would be impossible to convert a +50 GB file into JPEG 2000.
If anyone could point me into the right direction on converting a BigTiff into JPEG 2000 efficiently using a RAM limited work station, I would like to hear about it.
Cheers,
-Frank
Related
import pygame
from pygame.locals import * \
image = pygame.image.load('.\\w.jpg')
I'm trying to load an 59MB jpg file and it fails saying 'Out of memory'
I'm using windows 64bit and python 3.10.4 64bit.
My computer had more than 10GB of free ram left when the program ran.
As seen in the Wikipedia entry for JPEG, JPEG is a method of compression for digital images. So even if your jpg file is only 59MB the uncompressed image could take much more than that, depending on the amount of redundancy in the original image. The Wikipedia article asserts that JPEG typically achieves 10:1 compression and at first one might think, based on this assertion about typical compression rates, that even in uncompressed form that the image would not be too large. However, JPEG also uses Huffman coding and the amount of compression in Huffman coding can be extremely high if the uncompressed data is sufficiently redundant.
For some reason, on my Ubuntu 20.04 machine when I use OpenCV in Python like:
cv2.imwrite("myfile.webp", cv2image)
in my code, the file of 800x600 px has about 300 KiB while if I do:
cv2.imwrite("myfile.jpg", cv2image)
the output file of the same pixel size has about 100 KiB.
Why is that, if webp should have 25% less in size than jpg?
Or do I have to set some options first?
P.S. for png:
cv2.imwrite("myfile.png", cv2image)
the size is about 500 KiB.
Webp has 2 forms of saving data. Lossy (what JPEG does) where information get lost to reduce data, and lossless (what png does) with no data loss.
By default opencv uses its cv2.IMWRITE_WEBP_QUALITY to determin the quality. If this is set at 100 it means no compression (lossless)
https://docs.opencv.org/master/d8/d6a/group__imgcodecs__flags.html#gga292d81be8d76901bff7988d18d2b42aca7d3f848cc45d3138de0a2053d213a54a
SOLVED! It should be like this to work:
cv2.imwrite("myfile.webp", cv2image, [int(cv2.IMWRITE_WEBP_QUALITY), 20])
Now, the file has 4 kB ;D
I have to convert large CZI microscopy images about +50GB in size into compress JP2 images for post analysis. The JP2 images need to be compress in order to save disk space and be post analyzed using software. My current setup only has 8 GB of ram available, so I need to be able to process this large images with my ram limited workstation.
I have managed to write scripts that convert smaller CZI images about 5 GB into JP2. I do this by reading a compress representation of the image into memory. However, when I try to do the same trick with the 50 GB images everything comes crashing down.
The follow is a representation of my workflow:
Read the CZI image into memory and store it in a numpy array
Save the numpy array into jp2 format using glymur. To be able to write an image in JP2 format with glymur the whole image needs to be loaded into memory. This is obviously a huge limitation when working with large images.
I would like to read a chunk of the CZI image and then write it into a JP2 image. This process should be repeated until the CZI image has been fully converted into its JP2 representation. If someone can show me how to write a JP2 image in chunks that would be enough to get the ball rolling, since I have seen documentation on reading chunks of CZI images into memory.
I appreciate any help or suggestions. Thank you in advance for your time.
I'm trying to process some images and obtain numerical output. The skimage library only works with jpg format images. I only have tiff images on hand. Most converting functions work by loading a tiff image and saving it in jpg format. I do agree that the easiest way is
PIL.Image.open('pic.tiff').save('pic.jpg','jpeg')
I'm, on the other hand, trying to abstain from using hard drive for several reasons, but mainly due to the complexity file handling on heroku. Hence the question.
I am trying to read JPEG2000 images which are roughly 86kB (size = (4096,21504)) using Python. I used skimage and pillow modules to read them. The time it takes to read either of these files is ~1s.
Here are snippets on how I read the images.
SKIMAGE
im = io.imread(filename,plugin='freeimage')
PILLOW
im = Image.open(filename)
Using profiling I can see that the JPEG2000 decoder is responsible for the slowdown.
On the other hand, reading the same image using MATLAB's imread takes ~0.5s.
Is it possible to speed up the reading of JPEG2000 using Python libraries? Are there other libraries that could speed decoding process? I tried searching for ways to speed it up and couldn't find anything on this. I would appreciate links/reports that will point me to the right direction.
Computer Specs
OS: Windows 8.1
Processor: Intel i7-4930K#3.40GHz
RAM: 32.0 GB