Develop Reference

How to read all screen pixels into a python numpy array at 60fps+

First time asking a question on SO.
I am trying to find a fast way to read the screen live (60fps+). Screenshot to numpy is a fast method, but does not match that speed. There is a brilliant answer in this question for pixels: Most efficient/quickest way to parse pixel data with Python?
I tried changing GetPixel to this long form for BMP, but that reduces it to 5fps:
t1 = time.time()
count = 0
width = win32api.GetSystemMetrics(win32con.SM_CXVIRTUALSCREEN)
height = win32api.GetSystemMetrics(win32con.SM_CYVIRTUALSCREEN)
left = win32api.GetSystemMetrics(win32con.SM_XVIRTUALSCREEN)
top = win32api.GetSystemMetrics(win32con.SM_YVIRTUALSCREEN)
while count < 1000:
hwin = win32gui.GetDesktopWindow()
hwindc = win32gui.GetWindowDC(hwin)
srcdc = win32ui.CreateDCFromHandle(hwindc)
memdc = srcdc.CreateCompatibleDC()
bmp = win32ui.CreateBitmap()
bmp.CreateCompatibleBitmap(srcdc, width, height)
memdc.SelectObject(bmp)
memdc.BitBlt((0, 0), (width, height), srcdc, (left, top), win32con.SRCCOPY)
bmpinfo = bmp.GetInfo()
bmpInt = bmp.GetBitmapBits(False)
count +=1
t2 = time.time()
tf = t2-t1
it_per_sec = int(count/tf)
print (str(it_per_sec) + " iterations per second")
I watched a youtube video of a guy working on C# where he said GetPixel opens and closes memory and that's why doing a GetPixel on each individual pixel has a lot of overhead. He suggested to lock the entire data field and only then do getpixel. I don't know how to do that, so any help will be appreciated. (EDIT: this link might refer to that Unsafe Image Processing in Python like LockBits in C# )
There is also another method which gets a memory address of the bitmap, but I don't know what to do with it. The logic there is that I should be able to read memory from that point into any numpy array, but I have not been able to do that.
Any other option to read the screen fast will also be appreciated.
There must be a way, the GPU knows what pixels to draw at each location, that means there must be a memory bank somehere or a data stream we can tap into.
P.S. why a highspeed requirement? I am working on work automation tools that have a lot of overhead already and I am hoping to optimize screen data stream to help that part of the project.

The code below uses MSS, which if modified to show no output can reach 44fps for 1080p. https://python-mss.readthedocs.io/examples.html#opencv-numpy
import time
import cv2
import mss
import numpy
with mss.mss() as sct:
# Part of the screen to capture
monitor = {'top': 40, 'left': 0, 'width': 800, 'height': 640}
while 'Screen capturing':
last_time = time.time()
# Get raw pixels from the screen, save it to a Numpy array
img = numpy.array(sct.grab(monitor))
# Display the picture
#cv2.imshow('OpenCV/Numpy normal', img)
# Display the picture in grayscale
# cv2.imshow('OpenCV/Numpy grayscale',
# cv2.cvtColor(img, cv2.COLOR_BGRA2GRAY))
print('fps: {0}'.format(1 / (time.time()-last_time)))
# Press "q" to quit
if cv2.waitKey(25) & 0xFF == ord('q'):
cv2.destroyAllWindows()
break
Still not perfect though as it is not 60fps+ and using a raw repackaged buffer from the GPU would be a better solution if possible.

Fill image up to a size with Python

I am writing a handwriting recognition app and my inputs have to be of a certain size (128x128). When I detect a letter it looks like this:
That image for instance has a size of 40x53. I want to make it 128x128, but simply resizing it lowers the quality especially for smaller images. I want to somehow fill the rest up to 128x128 with the 40x53 in the middle. The background color should also stay relatively the same. I am using Python's opencv but I am new to it. How can I do this, and is it even possible?

Here you can get what you have asked using outputImage. Basically I have added a border using copyMakeBorder method. You can refer this for more details. You have to set the color value as you want in the value parameter. For now it is white [255,255,255].
But I would rather suggest you to resize the original image, seems like it is the better option than what you have asked. Get the image resized you can use resized in the following code. For your convenience I have added both methods in this code.
import cv2
import numpy as np
inputImage = cv2.imread('input.jpg', 1)
outputImage = cv2.copyMakeBorder(inputImage,37,38,44,44,cv2.BORDER_CONSTANT,value=[255,255,255])
resized = cv2.resize(inputImage, (128,128), interpolation = cv2.INTER_AREA)
cv2.imwrite('output.jpg', outputImage)
cv2.imwrite('resized.jpg', resized)

I believe you want to scale your image.
This code might help:
import cv2
img = cv2.imread('name_of_image', cv2.IMREAD_UNCHANGED)
# Get original size of image
print('Original Dimensions: ',img.shape)
# Percentage of the original size
scale_percent = 220
width = int(img.shape[1] * scale_percent / 100)
height = int(img.shape[0] * scale_percent / 100)
dim = (width, height)
# Resize/Scale the image
resized = cv2.resize(img, dim, interpolation = cv2.INTER_AREA)
# The new size of the image
print('Resized Dimensions: ',resized.shape)
cv2.imshow("Resized image", resized)
cv2.waitKey(0)
cv2.destroyAllWindows()

Python Resize image with ratio of maximum side of the image

I am very new in Python and this is going to be a very basic question.I have a website which is image based and i am developing it using Django.Now i want to resize the image or you can say i want to minimize the size of the images.There are different size of images are avaible,some images are largest in width,some images are largest in height and i want to resize images without changing there shape.
Here are some example what dimensions images are using in my website.
Here the First image is largest in width and the second image is largest in height and they are really big in Dimension.so they need to be resized or rather these images are need to be minimized in size.So i have used the PIL as below.
from PIL import Image,ImageDraw, ImageFont, ImageEnhance
def image_resize(request,image_id):
photo = Photo.objects.get(pk=image_id)
img = Image.open(photo.photo.file)
image = img.resize((1000, 560), Image.ANTIALIAS)
image.save()
so this function returns all the images with width of 1000 and height of 560.But i don't want to resize all the images with same width and height,rather i want to resize each images maintaining there own shape. That is there shape will be same but the images will be resized.How can i do this? i am really new in python.

Do you want to have all images with same width 1000? Try this code. It will resize to at most 1000 as width (if the image's width is less than 1000, nothing changes)
def image_resize(request,image_id):
photo = Photo.objects.get(pk=image_id)
image = Image.open(photo.photo.file)
(w,h) = image.size
if (w > 1000):
h = int(h * 1000. / w)
w = 1000
image = image.resize((w, h), Image.ANTIALIAS)
image.save()

I recall doing this sometime back without any problem except that I used thumbnail method rather than resize. Try it. You need not assign img to image. You can process img and save the same.
# open img
img.thumbnail((1000,560), Image.ANTIALIAS)
# save img

Python adding extra area to image

So I have a table with image sizes. There are multiple images of different sizes (66x66, 400x400, etc.). I have one example of image (the original) that always has a size of 600x532, and on this image is a product (a TV, a PC, etc.).
I have to resize this image, which isn't a problem. But if I do this with proportion I get something like 66x55. If I don't do this with proportion the image doesn't look good.
So the background of the original is always white. Is there a way to extend the area of the image and filling the rest with white?
So like this: 600x532 -> 600x600 -> 66x66 etc etc.
It should be like a anti-crop.
EDIT: I found out that if I use crop() from PIL and instead of "minimizing" using a value above the actual image-size it creates my extra area. but it is going to be black.
Any idea how I could fill this area white?
EDIT2: I guess it has something to do with ImageDraw.
EDIT3: After finding out that ImageDraw was the solution, my problem was solved. Please close this.
Here my solution:
import Image, ImageDraw
img1 = Image.open("img.jpg")
img2 = img1.crop((0,0,600,600))
draw = ImageDraw.Draw(img2)
draw.rectangle( (0,532,600,600), fill='white' )
del draw
img2.save("img2.jpg","JPEG", quality=75)
The next thing I will do is to make the extra crop above and under. So the picture stays in the middle.
EDIT4: final solution
img1 = Image.open("img1.jpg")
img2 = img1.crop( (0,-34,600,566) )
draw = ImageDraw.Draw(img2)
draw.rectangle( (0,0,600,34), fill="white" )
draw.rectangle( (0,566,600,600), fill="white" )
del draw
img2.save("img2.jpg", "JPEG", quality=75)

Supposing we use PIL to process the image
from PIL import Image
def white_bg_square(img):
"return a white-background-color image having the img in exact center"
size = (max(img.size),)*2
layer = Image.new('RGB', size, (255,255,255))
layer.paste(img, tuple(map(lambda x:(x[0]-x[1])/2, zip(size, img.size))))
return layer
You could resize a PIL Image object, img for example
img.resize((width, height), resample=Image.ANTIALIAS)
Thus in the python shell, it looks like
>>> from PIL import Image
>>> img = Image.open('path/to/image')
>>> square_one = white_bg_square(img)
>>> square_one.resize((100, 100), Image.ANTIALIAS)
>>> square_one.save('path/to/result')
There are nice examples inside PIL document and sorl-thumbnail 3.2.5
http://effbot.org/imagingbook/image.htm
http://pypi.python.org/pypi/sorl-thumbnail/3.2.5

My final solution
img1 = Image.open("img1.jpg")
img2 = img1.crop( (0,-34,600,566) )
draw = ImageDraw.Draw(img2)
draw.rectangle( (0,0,600,34), fill="white" )
draw.rectangle( (0,566,600,600), fill="white" )
del draw
img2.save("img2.jpg", "JPEG", quality=75)

If we use opencv to process the image.
import cv2
import numpy as np
def make_square(self, image_in):
size = image_in.shape[:2]
max_dim = max(size)
delta_w = max_dim - size[1]
delta_h = max_dim - size[0]
top, bottom = delta_h//2, delta_h-(delta_h//2)
left, right = delta_w//2, delta_w-(delta_w//2)
color = [255, 255, 255]
#image_out = cv2.copyMakeBorder(image_in, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)
image_out = cv2.copyMakeBorder(image_in, top, bottom, left, right, cv2.BORDER_REPLICATE, value=color)
return image_out
image_in = cv2.imread(image_path)

python PIL - background displayed opaque instead of transparent

I want to generate 32x32 sized thumbnails from uploaded images (actually avatars).
To prevent a thumbnail from being smaller than that size, I want to create a transparent 32x32 background and paste the thumbnail on it.
The code below tries to do so. However, the avatar is displayed on a black and opaque background; I lose transparency information somewhere through the process. Where am I doing wrong?
def handle_image(img):
size = SMALL_AVATAR_IMAGE_SIZE
img.thumbnail(size, Image.ANTIALIAS)
img = img.convert('RGBA')
background = Image.new('RGBA', size, (255, 255, 255, 0))
background.paste(img, (0, (size[1] - img.size[1]) / 2), img)
img = background
processed_image_small = ContentFile(img.tostring('jpeg', img.mode))
targetpath = str(self.user.id) + '_S' + '.jpg'
self.img_small.save(targetpath, processed_image_small,save=False)

That is because JPEG cannot save transparency informations which are contained in a RGBA image. You may want to save the avatar to a format like PNG which is able to keep these informations.

You're generating a JPG image. JPEGs don't support background transparency. You need to generate a PNG image to support transparencies.