Develop Reference

Render NumPy array in FastAPI

I have found How to return a numpy array as an image using FastAPI?, however, I am still struggling to show the image, which appears just as a white square.
I read an array into io.BytesIO like so:
def iterarray(array):
output = io.BytesIO()
np.savez(output, array)
yield output.get_value()
In my endpoint, my return is StreamingResponse(iterarray(), media_type='application/octet-stream')
When I leave the media_type blank to be inferred a zipfile is downloaded.
How do I get the array to be displayed as an image?

Option 1 - Return image as bytes
The below examples show how to convert an image loaded from disk, or an in-memory image (in the form of numpy array), into bytes (using either PIL or OpenCV libraries) and return them using a custom Response. For the purposes of this demo, the below code is used to create the in-memory sample image (numpy array), which is based on this answer.
# Function to create a sample RGB image
def create_img():
w, h = 512, 512
arr = np.zeros((h, w, 3), dtype=np.uint8)
arr[0:256, 0:256] = [255, 0, 0] # red patch in upper left
return arr
Using PIL
Server side:
You can load an image from disk using Image.open, or use Image.fromarray to load an in-memory image (Note: For demo purposes, when the case is loading the image from disk, the below demonstrates that operation inside the route. However, if the same image is going to be served multiple times, one could load the image only once at startup and store it on the app instance, as described in this answer). Next, write the image to a buffered stream, i.e., BytesIO, and use the getvalue() method to get the entire contents of the buffer. Even though the buffered stream is garbage collected when goes out of scope, it is generally better to call close() or use the with statement, as shown here and below.
from fastapi import Response
from PIL import Image
import numpy as np
import io
#app.get('/image', response_class=Response)
def get_image():
# loading image from disk
# im = Image.open('test.png')
# using an in-memory image
arr = create_img()
im = Image.fromarray(arr)
# save image to an in-memory bytes buffer
with io.BytesIO() as buf:
im.save(buf, format='PNG')
im_bytes = buf.getvalue()
headers = {'Content-Disposition': 'inline; filename="test.png"'}
return Response(im_bytes, headers=headers, media_type='image/png')
Client side:
The below demonstrates how to send a request to the above endpoint using Python requests module, and write the received bytes to a file, or convert the bytes back into PIL Image, as described here.
import requests
from PIL import Image
url = 'http://127.0.0.1:8000/image'
r = requests.get(url=url)
# write raw bytes to file
with open('test.png', 'wb') as f:
f.write(r.content)
# or, convert back to PIL Image
# im = Image.open(io.BytesIO(r.content))
# im.save('test.png')
Using OpenCV
Server side:
You can load an image from disk using cv2.imread() function, or use an in-memory image, which—if it is in RGB order, as in the example below—needs to be converted, as OpenCV uses BGR as its default colour order for images. Next, use cv2.imencode() function, which compresses the image data (based on the file extension you pass that defines the output format, i.e., .png, .jpg, etc.) and stores it in an in-memory buffer that is used to transfer the data over the network.
import cv2
#app.get('/image', response_class=Response)
def get_image():
# loading image from disk
# arr = cv2.imread('test.png', cv2.IMREAD_UNCHANGED)
# using an in-memory image
arr = create_img()
arr = cv2.cvtColor(arr, cv2.COLOR_RGB2BGR)
# arr = cv2.cvtColor(arr, cv2.COLOR_RGBA2BGRA) # if dealing with 4-channel RGBA (transparent) image
success, im = cv2.imencode('.png', arr)
headers = {'Content-Disposition': 'inline; filename="test.png"'}
return Response(im.tobytes() , headers=headers, media_type='image/png')
Client side:
On client side, you can write the raw bytes to a file, or use the numpy.frombuffer() function and cv2.imdecode() function to decompress the buffer into an image format (similar to this)—cv2.imdecode() does not require a file extension, as the correct codec will be deduced from the first bytes of the compressed image in the buffer.
url = 'http://127.0.0.1:8000/image'
r = requests.get(url=url)
# write raw bytes to file
with open('test.png', 'wb') as f:
f.write(r.content)
# or, convert back to image format
# arr = np.frombuffer(r.content, np.uint8)
# img_np = cv2.imdecode(arr, cv2.IMREAD_UNCHANGED)
# cv2.imwrite('test.png', img_np)
Useful Information
Since you noted that you would like the image displayed similar to a FileResponse, using a custom Response to return the bytes should be the way to do this, instead of using StreamingResponse (as shown in your question). To indicate that the image should be viewed in the browser, the HTTP response should include the following header, as described here and as shown in the above examples (the quotes around the filename are required, if the filename contains special characters):
headers = {'Content-Disposition': 'inline; filename="test.png"'}
Whereas, to have the image downloaded rather than viewed (use attachment instead):
headers = {'Content-Disposition': 'attachment; filename="test.png"'}
If you would like to display (or download) the image using a JavaScript interface, such as Fetch API or Axios, have a look at the answers here and here.
As for the StreamingResponse, if the numpy array is fully loaded into memory from the beginning, StreamingResponse is not necessary at all. StreamingResponse streams by iterating over the chunks provided by your iter() function (if Content-Length is not set in the headers—unlike StreamingResponse, other Response classes set that header for you, so that the browser will know where the data ends). As described in this answer:
Chunked transfer encoding makes sense when you don't know the size of
your output ahead of time, and you don't want to wait to collect it
all to find out before you start sending it to the client. That can
apply to stuff like serving the results of slow database queries, but
it doesn't generally apply to serving images.
Even if you would like to stream an image file that is saved on disk (which you should rather not, unless it is a rather large file that can't fit into memory. Instead, you should use use FileResponse), file-like objects, such as those created by open(), are normal iterators; thus, you can return them directly in a StreamingResponse, as described in the documentation and as shown below (if you find yield from f being rather slow when using StreamingResponse, please have a look at this answer for solutions):
#app.get('/image')
def get_image():
def iterfile():
with open('test.png', mode='rb') as f:
yield from f
return StreamingResponse(iterfile(), media_type='image/png')
or, if the image was loaded into memory instead, and was then saved into a BytesIO buffered stream in order to return the bytes, BytesIO is a file-like object (like all the concrete classes of io module), which means you could return it directly in a StreamingResponse:
from fastapi import BackgroundTasks
#app.get('/image')
def get_image(background_tasks: BackgroundTasks):
arr = create_img()
im = Image.fromarray(arr)
buf = BytesIO()
im.save(buf, format='PNG')
buf.seek(0)
background_tasks.add_task(buf.close)
return StreamingResponse(buf, media_type='image/png')
Thus, for your case scenario, it is best to return a Response with your custom content and media_type, as well as setting the Content-Disposition header, as described above, so that the image is viewed in the browser.
Option 2 - Return image as JSON-encoded numpy array
The below should not be used for displaying the image in the browser, but it is rather added here for the sake of completeness, showing how to convert an image into a numpy array (preferably, using asarray() function), then return the data in JSON format, and finally, convert the data back to image on client side, as described in this and this answer. For faster alternatives to the standard Python json library, see this answer.
Using PIL
Server side:
from PIL import Image
import numpy as np
import json
#app.get('/image')
def get_image():
im = Image.open('test.png')
# im = Image.open('test.png').convert('RGBA') # if dealing with 4-channel RGBA (transparent) image
arr = np.asarray(im)
return json.dumps(arr.tolist())
Client side:
import requests
from PIL import Image
import numpy as np
import json
url = 'http://127.0.0.1:8000/image'
r = requests.get(url=url)
arr = np.asarray(json.loads(r.json())).astype(np.uint8)
im = Image.fromarray(arr)
im.save('test_received.png')
Using OpenCV
Server side:
import cv2
import json
#app.get('/image')
def get_image():
arr = cv2.imread('test.png', cv2.IMREAD_UNCHANGED)
return json.dumps(arr.tolist())
Client side:
import requests
import numpy as np
import cv2
import json
url = 'http://127.0.0.1:8000/image'
r = requests.get(url=url)
arr = np.asarray(json.loads(r.json())).astype(np.uint8)
cv2.imwrite('test_received.png', arr)

is there a way to manipulate image without saving to a file first?

I'm trying to collect img data for my ML/DL project.
I need facial data so, I have to detect the face and crop around it.
I have a bunch of img URLs that I searched online.
Normally I would save them in a file using requests library, but would it be possible to do it in-memory?
response = requests.get(IMG_URL)
img_byte = response.content
# Do image processing without saving to a file
I looked at some image libraries in python such as PIL or OpenCV,
but they all seems to get images loaded from a file in first place.
I think I could speed up the process if I don't save temporary files. (I/O is a big bottleneck)
I tried playing around with BytesIO functions but I couldn't figure it out.

BytesIO is the way to go!. You can store the binary data returned by the request to the In-memory byte buffer(BytesIO) and pass it to Image.open
>>> from PIL import Image
>>> from io import BytesIO
>>> img_obj = Image.open(BytesIO(r.content))
The only requirement of the Image.open API is that the fp argument must be a filename (string), pathlib.Path object or a file object. The file object must implement file.read, file.seek, and file.tell methods, and be opened in binary mode.
BytesIO implements all these methods.
>>> from io import BytesIO
>>>
>>> buffer = BytesIO()
>>> hasattr(buffer, 'read') and hasattr(buffer, 'tell') and hasattr(buffer, 'seek')
True

Pass PIL Image to google cloud vision without saving and reading

UPDATE BELOW
Is there a way to pass a PIL Image to google cloud vision?
I tried to use io.Bytes, io.String and Image.tobytes() but I always get:
Traceback (most recent call last):
"C:\Users\...\vision_api.py", line 20, in get_text
image = vision.Image(content)
File "C:\...\venv\lib\site-packages\proto\message.py", line 494, in __init__
raise TypeError(
TypeError: Invalid constructor input for Image:b'Ma\x81Ma\x81La\x81Ma\x81Ma\x81Ma\x81Ma\x81Ma\x81Ma\x81Ma\x81Ma\x81La\x81Ma\x81Ma\x81Ma\x81Ma\x80Ma\x81La\x81Ma\x81Ma\x81Ma\x80Ma\x81Ma\x81Ma\x81Ma\x8 ...
or this if I pass the PIL-Image directly:
TypeError: Invalid constructor input for Image: <PIL.Image.Image image mode=RGB size=480x300 at 0x1D707131DC0>
This is my code:
image = Image.open(path).convert('RGB') # Opening the saved image
cropped_image = image.crop((30, 900, 510, 1200)) # Cropping the image
vision_image = vision.Image(# I passed the different options) # Here I need to pass the image, but I don't know how
client = vision.ImageAnnotatorClient()
response = client.text_detection(image=vision_image) # Text detection using google-vision-api
FOR CLARITY:
I want google text detection to only analyse a certain part of an image saved on my disk. So my idea was to crop the image using PIL and then pass the cropped image to google-vision. But it is not possible to pass an PIL-Image to vision.Image, as I get the error above.
The documentation from Google.
This can be found in the vision.Image class:
Attributes:
content (bytes):
Image content, represented as a stream of bytes. Note: As
with all ``bytes`` fields, protobuffers use a pure binary
representation, whereas JSON representations use base64.
Currently, this field only works for BatchAnnotateImages
requests. It does not work for AsyncBatchAnnotateImages
requests.
A working option is to save the PIL-Image as a PNG/JPG on my disk and load it using:
with io.open(file_name, 'rb') as image_file:
content = image_file.read()
vision_image = vision.Image(content=content)
But this is slow and seems unnecessary. And the whole point for me behind using google-vision-api is the speed comaped to open-cv.
UPDATE as of 25/9/2021
from PIL import Image
from io import BytesIO
from google.cloud import vision
with open('images/screenshots/screenshot.png', 'rb') as image_file:
data = image_file.read()
try:
image = vision.Image(content=data)
print('worked')
except TypeError:
print('failed')
im = Image.open('images/screenshots/screenshot.png')
buffer = BytesIO()
im.save(buffer, format='PNG')
try:
image = vision.Image(buffer.getvalue())
print('worked')
except TypeError:
print('failed')
The first version works as expected, but I can't get the second one to work as #Mark Setchell recommended. The first few characters (~50) are the same, the rest is completely different.
UPDATE as of 26/9/2021
Both inputs are of type <class 'bytes'>. The complete error stack can be seen at the top of the question.
Using this code:
print(input_data[:200])
print(type(input_data))
i get the following output:
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x048\x00\x00\x07\x80\x08\x06\x00\x00\x00+a\xe7\n\x00\x00\x00\x04sBIT\x08\x08\x08\x08|\x08d\x88\x00\x00 \x00IDATx\x9c\xec\xbdy\xd8-\xc7Y\x1f\xf8\xab\xea>\xe7\xdb\xef\xaa\xbbk\xb3%\xcb\x8b\x16[\x12\xc6\xc8\xbb,\x1b\x03\x06\xc6\x8111\x93#2y\xc2381\x8b1\x90\x10\x9e\xf18\x93\x10\x0811\x84\x192\x0c3\x9e\x1020\x03\x03\xc3\xb0\x04\xf0C0\xc6\x96m\xc9\x96m\xed\xb2dI\x96\xaetu\xf7\xed\xdb\xcf\xe9\xae\x9a?j\xe9\xea\xbd\xba\xbb\xbaO\x9f\xef\x9e\xd7\xd6\xfd\xfat\xbf\xf5Vu-o\xbd\xf5\xeb\xb7\xde"\xef\xff\xc7\'8\x1c\x13\x07\x00\xd2\x82\xcc6\xe5\xc6\xa8B&'
<class 'bytes'>
for the working input.
And:
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x048\x00\x00\x07\x80\x08\x06\x00\x00\x00+a\xe7\n\x00\x01\x00\x00IDATx\x9c\xec\xbdw\x80$\xc7u\x1f\xfc\xab\xea\xeeI\x9bw/\'\x1cr\xce\x04#\x10\x04A\x82`\x84\x95%J"\x95,\xcb\x1f%\x91T\xb0$*}\x1fM\xd9\x96\x95EY\x94(\xc9\xb6\x92i+\x90\x12\x83(3)0\x82\x08$rN\x07\\\xce\xb7\xb7yBw\xd5\xf7G\x85\xaeN3\xdd=\xdd\xb3\xb3{\xfb\xc8\xc3\xceLW\xbd\xca\xaf\xde\xfb\xf5\xabW\xe4{\xdeu\x84\xa3`\xe2\x00#J\xe0Y&\xdf\x00e($\x94\x94\'p\xcc\xc3\xda\xe7Y\x0c\xf1Te\x13\xbf\xcc>\xfa:]Y=x\x84\x7f\xe8\xc23u\x1f\x91l\xfd\x99'
<class 'bytes'>
for the failing input.

As far as I can tell, you start off with a PIL Image and you want to obtain a PNG image in memory without going to disk. So you need this:
#!/usr/bin/env python3
from PIL import Image
from io import BytesIO
# Create PIL Image like you have - filled with red
im = Image.new('RGB', (320,240), (255,0,0))
# Create in-memory PNG - like you want for Google Cloud Vision
buffer = BytesIO()
im.save(buffer, format="PNG")
# Look at first few bytes
PNG = buffer.getvalue()
print(PNG[:20])
It prints this, which is exactly what you would get if you wrote the image to disk as a PNG and then read it back as binary - except this does it in memory without going to disk:
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01#'

It would be good to have whole error stack and more accurate code snippet. But form presented information this seems to be confusion of two different "Images". Probably the some copy/paste error, as the tutorials have exactly the same line:
response = client.text_detection(image=image)
But mentioned tutorials image is created by vision.Image() so I think in presented code this should be:
response = client.text_detection(image=vision_image)
As, at least if I understand correctly the code snippet, image is PIL Image, while vision_image is Vision Image that should be passed to text_detection method. So whatever is done in vision.Image() does not have effect on the error massage.

How do I add an image from a list in python using docx?

I wrote a code that takes a screenshot that I want to paste into a word document using docx. So far I have to save the image as a png file. The relevant part of my code is:
from docx import Document
import pyautogui
import docx
doc = Document()
images = []
img = pyautogui.screenshot(region = (some region))
images.append(img)
img.save(imagepath.png)
run =doc.add_picture(imagepath.png)
run
I would like to be able to add the image without saving it. Is it possible to do this using docx?

Yes, according to add_picture — Document objects — python-docx 0.8.10 documentation, add_picture can import data from a stream as well.
As per Screenshot Functions — PyAutoGUI 1.0.0 documentation, screenshot() produces a PIL/Pillow image object which can be save()'d with a BytesIO() as destination to produce a compressed image data stream in memory.
So that'll be:
import io
imdata = io.BytesIO()
img.save(imdata, format='png')
imdata.seek(0)
doc.add_picture(imdata)
del imdata # cannot reuse it for other pictures, you need a clean buffer each time
# can use .truncate(0) then .seek(0) instead but this is probably easier

How to read raw png from an array in python opencv?

I'm streaming a png image from my iPhone to my MacBook over tcp. The MacBook code is from http://docs.python.org/library/socketserver.html#requesthandler-objects. How can the image be converted for use with OpenCV? A png was selected because they are efficient, but other formats could be used.
I wrote a test program that reads the rawImage from a file, but not sure how to convert it:
# Read rawImage from a file, but in reality will have it from TCPServer
f = open('frame.png', "rb")
rawImage = f.read()
f.close()
# Not sure how to convert rawImage
npImage = np.array(rawImage)
matImage = cv2.imdecode(rawImage, 1)
#show it
cv.NamedWindow('display')
cv.MoveWindow('display', 10, 10)
cv.ShowImage('display', matImage)
cv. WaitKey(0)

#Andy Rosenblum's works, and it might be the best solution if using the outdated cv python API (vs. cv2).
However, because this question is equally interesting for users of the latest versions, I suggest the following solution. The sample code below may be better than the accepted solution because:
It is compatible with newer OpenCV python API (cv2 vs. cv). This solution is tested under opencv 3.0 and python 3.0. I believe only trivial modifications would be required for opencv 2.x and/or python 2.7x.
Fewer imports. This can all be done with numpy and opencv directly, no need for StringIO and PIL.
Here is how I create an opencv image decoded directly from a file object, or from a byte buffer read from a file object.
import cv2
import numpy as np
#read the data from the file
with open(somefile, 'rb') as infile:
buf = infile.read()
#use numpy to construct an array from the bytes
x = np.fromstring(buf, dtype='uint8')
#decode the array into an image
img = cv2.imdecode(x, cv2.IMREAD_UNCHANGED)
#show it
cv2.imshow("some window", img)
cv2.waitKey(0)
Note that in opencv 3.0, the naming convention for the various constants/flags changed, so if using opencv 2.x, you will need to change the flag cv2.IMREAD_UNCHANGED. This code sample also assumes you are loading in a standard 8-bit image, but if not, you can play with the dtype='...' flag in np.fromstring.

another way,
also in the case of a reading an actual file this will work for a unicode path (tested on windows)
with open(image_full_path, 'rb') as img_stream:
file_bytes = numpy.asarray(bytearray(img_stream.read()), dtype=numpy.uint8)
img_data_ndarray = cv2.imdecode(file_bytes, cv2.CV_LOAD_IMAGE_UNCHANGED)
img_data_cvmat = cv.fromarray(img_data_ndarray) # convert to old cvmat if needed

I figured it out:
# Read rawImage from a file, but in reality will have it from TCPServer
f = open('frame.png', "rb")
rawImage = f.read()
f.close()
# Convert rawImage to Mat
pilImage = Image.open(StringIO(rawImage));
npImage = np.array(pilImage)
matImage = cv.fromarray(npImage)
#show it
cv.NamedWindow('display')
cv.MoveWindow('display', 10, 10)
cv.ShowImage('display', matImage)
cv. WaitKey(0)

This works for me (these days):
import cv2
import numpy as np
data = open('016e263c726a.raw').read()
x = np.frombuffer(data, dtype='uint8').reshape(2048,2448)
cv2.imshow('x',x); cv2.waitKey(); cv2.destroyAllWindows()
But it reads a RAW image saved without any specific format.

(Your question seems to be tagged objective-c but you ask for Python and so is your example, so I'll use that.)
My first post on Stack Overflow!
The cv.LoadImageM method seems to be what you are looking for.
http://opencv.willowgarage.com/documentation/python/reading_and_writing_images_and_video.html
Example use:
http://opencv.willowgarage.com/wiki/PythonInterface/
LoadImage(filename, iscolor=CV_LOAD_IMAGE_COLOR) → None
Loads an image from a file as an IplImage.
Parameters:
filename (str) – Name of file to be loaded.
iscolor (int) –
Specific color type of the loaded image:
CV_LOAD_IMAGE_COLOR the loaded image is forced to be a 3-channel color image
CV_LOAD_IMAGE_GRAYSCALE the loaded image is forced to be grayscale
CV_LOAD_IMAGE_UNCHANGED the loaded image will be loaded as is.
The function cvLoadImage loads an image from the specified file and
returns the pointer to the loaded image. Currently the following file
formats are supported:
Windows bitmaps - BMP, DIB
JPEG files - JPEG, JPG, JPE
Portable Network Graphics - PNG
Portable image format - PBM, PGM, PPM
Sun rasters - SR, RAS
TIFF files - TIFF, TIF
Note that in the current implementation the alpha channel, if any, is
stripped from the output image, e.g. 4-channel RGBA image will be
loaded as RGB.

When you have to load from file, this simple solution does the job (tested with opencv-python-3.2.0.6):
import cv2
img = cv2.imread(somefile)