I want to store my image directory in memory, then load the images into a numpy array.
The normative way to load images that are not in memory is as follows:
import PIL.Image
import numpy as np
image = PIL.Image.open("./image_dir/my_image_1.jpg")
image = np.array(image)
However, I am not sure how to do this when the images are in memory. So far, I have been able to setup the following starter code:
import fs
import fs.memoryfs
import fs.osfs
image_dir = "./image_dir"
mem_fs = fs.memoryfs.MemoryFS()
drv_fs = fs.osfs.OSFS(image_path)
fs.copy.copy_fs(drv_fs, mem_fs)
print(mem_fs.listdir('.'))
Returns:
['my_image_1.jpg', 'my_image_2.jpg']
How do I load images that are in memory into numpy?
I am also open to alternatives to the fs package.
As per the documentation, Pillow's Image.open accepts a file object instead of a file name, so as long as your in-memory file package provides Python file objects (which it most likely does), you can just use them. If it doesn't, you could even just wrap them in a class that provides the required methods. Assuming you are using PyFilesystem, according to its documentation you should be fine.
So, you want something like:
import numpy as np
import PIL.Image
import fs.memoryfs
import fs.osfs
import fs.copy
mem_fs = fs.memoryfs.MemoryFS()
drv_fs = fs.osfs.OSFS("./image_dir")
fs.copy.copy_file(drv_fs, './my_image_1.jpg', mem_fs, 'test.jpg')
with mem_fs.openbin('test.jpg') as f:
image = PIL.Image.open(f)
image = np.array(image)
(note I just used copy_file because I tested with a single file, you can use copy_fs if you need to copy the entire tree - it's the same principle)
Related
I need to read a folder full of multi-page tiffs generated by the Suite2p neurobiology package.
From the Suite2p source code the multi-tiffs are created as follows:
import numpy as np
from tifffile import TiffWriter
# fake example images
img_5_tiles = np.random.randint(0,65535,(5,10,20), dtype='uint16')
img_4_tiles = img_5_tiles[1:,...]
# save fake images
with TiffWriter(r'D:\5tiles.tiff') as tif:
for frame in img_5_tiles:
tif.save(frame)
with TiffWriter(r'D:\4tiles.tiff') as tif:
for frame in img_4_tiles:
tif.save(frame)
When I try to read them into dask, skimage.io default tifffile plugin fails to get the correct shape:
from dask.array.image import imread
from skimage import io
def read_with_tifffile(path):
return io.imread(path, plugin='tifffile')
# should produce (5,10,20) instead of (1,10,20)
imread(r'D:\5tiles.tiff', imread=read_with_tifffile).shape
I can overcome this by using the non-default skimage.io.imread plugin 'pil'
def read_with_pil(path):
return io.imread(path, plugin='pil', mode='L')
# gives (1,5,10,20) which is acceptable
imread(r'D:\5tiles.tiff', imread=read_with_pil).shape
Unfortunately, if the number of tiles equals to 4, skimage starts to handle the shape differently:
# gives (1,10,20,4) instead of (1,4,10,20)
imread(r'D:\4tiles.tiff', imread=read_with_pil).shape
From reading skimage docs, it's probably trying to interpret my image as RGBA and then fails.
Is there a solution to force 'uint16' multipage read for all image shapes?
Any advice is appreciated!
Based on cgohlke's comment:
from dask.array.image import imread
import numpy as np
from skimage import io
import tifffile
# fake data
img_with_4_tiles = np.random.randint(0,65535,(4,10,20), dtype='uint16')
with tifffile.TiffWriter(r'D:\4tiles.tiff') as tif:
for frame in img_with_4_tiles:
tif.save(frame)
# my original bad solution
def read_with_pil(path):
return io.imread(path, plugin='pil', mode='L')
print(imread(r'D:\4tiles.tiff', imread=read_with_pil).shape)
# a good solution from the horse's mouth (https://pypi.org/user/cgohlke/)
def read_with_tifffile(path):
return tifffile.imread(path, is_shaped=False)
print(imread(r'D:\4tiles.tiff', imread=read_with_tifffile).shape)
I am trying to convert the celebA dataset(https://www.kaggle.com/jessicali9530/celeba-dataset) images folder into a numpy array for later to be converted into a .pkl file(for using the data as simply as mnist or cifar).
I am willing to find a better way of converting since this method is absolutely consuming the whole RAM.
from PIL import Image
import pickle
from glob import glob
import numpy as np
TARGET_IMAGES = "img_align_celeba/*.jpg"
def generate_dataset(glob_files):
dataset = []
for _, file_name in enumerate(sorted(glob(glob_files))):
img = Image.open(file_name)
pixels = list(img.getdata())
dataset.append(pixels)
return np.array(dataset)
celebAdata = generate_dataset(TARGET_IMAGES)
I am rather curious on how the mnist authors did this themselves but any approach that works is welcome.
You can transform any kind of data on the fly in Keras and load in memory one batch at the time during training.
See documentation, search for 'Example of using .flow_from_directory(directory)'.
How to load pixels of multiple images in a directory in a numpy array . I have loaded a single image in a numpy array . But can not figure out how to load multiple images from a directory . Here what i have done so far
image = Image.open('bn4.bmp')
nparray=np.array(image)
This loads a 32*32 matrices . I want to load 100 of the images in a numpy array . I want to make 100*32*32 size numpy array . How can i do that ? I know that the structure would look something like this
for filename in listdir("BengaliBMPConvert"):
if filename.endswith(".bmp"):
-----------------
else:
continue
But can not find out how to load the images in numpy array
Getting a list of BMP files
To get a list of BMP files from the directory BengaliBMPConvert, use:
import glob
filelist = glob.glob('BengaliBMPConvert/*.bmp')
On the other hand, if you know the file names already, just put them in a sequence:
filelist = 'file1.bmp', 'file2.bmp', 'file3.bmp'
Combining all the images into one numpy array
To combine all the images into one array:
x = np.array([np.array(Image.open(fname)) for fname in filelist])
Pickling a numpy array
To save a numpy array to file using pickle:
import pickle
pickle.dump( x, filehandle, protocol=2 )
where x is the numpy array to be save, filehandle is the handle for the pickle file, such as open('filename.p', 'wb'), and protocol=2 tells pickle to use its current format rather than some ancient out-of-date format.
Alternatively, numpy arrays can be pickled using methods supplied by numpy (hat tip: tegan). To dump array x in file file.npy, use:
x.dump('file.npy')
To load array x back in from file:
x = np.load('file.npy')
For more information, see the numpy docs for dump and load.
Use OpenCV's imread() function together with os.listdir(), like
import numpy as np
import cv2
import os
instances = []
# Load in the images
for filepath in os.listdir('images/'):
instances.append(cv2.imread('images/{0}'.format(filepath),0))
print(type(instances[0]))
class 'numpy.ndarray'
This returns you a list (==instances) in which all the greyscale values of the images are stored. For colour images simply set .format(filepath),1.
I just would like to share two sites where one can split a dataset into train, test and validation sets: split_folder
and create numpy arrays out of images residing in respective folders code snippet from medium by muskulpesent
I am trying to use SVMs to classify a set if images I have on my computer into 3 categories :
I am just facing a problem of how to load the data as in the following example , he uses a data set that is already saved.
http://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html
Me I have all the images in png format saved in a folder on my pc
You can load data as numpy arrays using Pillow, in this way:
from PIL import Image
import numpy as np
data = np.array(Image.open('yourimg.png')) # .astype(float) if necessary
couple it with os.listdir to read multiple files, e.g.
import os
for file in os.listdir('your_dir/'):
img = Image.open(os.path.join('your_dir/', file))
data = np.array(img)
your_model.train(data)
I have about 200 grayscale PNG images stored within a directory like this.
1.png
2.png
3.png
...
...
200.png
I want to import all the PNG images as NumPy arrays.
How can I do this?
According to the doc, scipy.misc.imread is deprecated starting SciPy 1.0.0, and will be removed in 1.2.0. Consider using imageio.imread instead.
Example:
import imageio
im = imageio.imread('my_image.png')
print(im.shape)
You can also use imageio to load from fancy sources:
im = imageio.imread('http://upload.wikimedia.org/wikipedia/commons/d/de/Wikipedia_Logo_1.0.png')
Edit:
To load all of the *.png files in a specific folder, you could use the glob package:
import imageio
import glob
for im_path in glob.glob("path/to/folder/*.png"):
im = imageio.imread(im_path)
print(im.shape)
# do whatever with the image here
Using just scipy, glob and having PIL installed (pip install pillow) you can use scipy's imread method:
from scipy import misc
import glob
for image_path in glob.glob("/home/adam/*.png"):
image = misc.imread(image_path)
print image.shape
print image.dtype
UPDATE
According to the doc, scipy.misc.imread is deprecated starting SciPy 1.0.0, and will be removed in 1.2.0. Consider using imageio.imread instead. See the answer by Charles.
This can also be done with the Image class of the PIL library:
from PIL import Image
import numpy as np
im_frame = Image.open(path_to_file + 'file.png')
np_frame = np.array(im_frame.getdata())
Note: The .getdata() might not be needed - np.array(im_frame) should also work
Using a (very) commonly used package is prefered:
import matplotlib.pyplot as plt
im = plt.imread('image.png')
If you are loading images, you are likely going to be working with one or both of matplotlib and opencv to manipulate and view the images.
For this reason, I tend to use their image readers and append those to lists, from which I make a NumPy array.
import os
import matplotlib.pyplot as plt
import cv2
import numpy as np
# Get the file paths
im_files = os.listdir('path/to/files/')
# imagine we only want to load PNG files (or JPEG or whatever...)
EXTENSION = '.png'
# Load using matplotlib
images_plt = [plt.imread(f) for f in im_files if f.endswith(EXTENSION)]
# convert your lists into a numpy array of size (N, H, W, C)
images = np.array(images_plt)
# Load using opencv
images_cv = [cv2.imread(f) for f in im_files if f.endswith(EXTENSION)]
# convert your lists into a numpy array of size (N, C, H, W)
images = np.array(images_cv)
The only difference to be aware of is the following:
opencv loads channels first
matplotlib loads channels last.
So a single image that is 256*256 in size would produce matrices of size (3, 256, 256) with opencv and (256, 256, 3) using matplotlib.
To read in one image:
import PIL.Image
im = PIL.Image.open('path/to/your/image')
im = np.array(im)
Iterate to read in multiple images.
This answer is similar to this but simpler (no need for .getdata()).
I changed a bit and it worked like this, dumped into one single array, provided all the images are of same dimensions.
png = []
for image_path in glob.glob("./train/*.png"):
png.append(misc.imread(image_path))
im = np.asarray(png)
print 'Importing done...', im.shape
I like the build-in pathlib libary because of quick options like directory= Path.cwd()
Together with opencv it's quite easy to read pngs to numpy arrays.
In this example you can even check the prefix of the image.
from pathlib import Path
import cv2
prefix = "p00"
suffix = ".png"
directory= Path.cwd()
file_names= [subp.name for subp in directory.rglob('*') if (prefix in subp.name) & (suffix == subp.suffix)]
file_names.sort()
print(file_names)
all_frames= []
for file_name in file_names:
file_path = str(directory / file_name)
frame=cv2.imread(file_path)
all_frames.append(frame)
print(type(all_frames[0]))
print(all_frames[0] [1][1])
Output:
['p000.png', 'p001.png', 'p002.png', 'p003.png', 'p004.png', 'p005.png', 'p006.png', 'p007.png', 'p008.png', 'p009.png']
<class 'numpy.ndarray'>
[255 255 255]
If you prefer the standard library:
#IMPORTANT: This Code only works with Python>=3.6
Directory="."#Your directory
import os
import tkinter
import numpy
tk=tkinter.Tk()
tk.overrideredirect(1)
tk.geometry("0x0")
Items=[]
for i in os.listdir(Directory):
fn=Directory+os.sep+i
imgArray=[]
image=tkinter.PhotoImage(file=fn)
for w in range(image.width()):
entry=[]
for h in range(image.height()):
entry.append(image.get(w,h))
imgArray.append(entry)
imgArray=numpy.array(imgArray)
Items.append(imgArray)
tk.destroy()