Develop Reference

How to convert screen x,y (cartesian coordinates) to 3D world space crosshair movement angles (screenToWorld)?

Recently I've been playing around with computer vision and neural networks.
And came across experimental object detection within a 3D application.
But, surprisingly to me - I've faced an issue of converting one coordinates system to another (AFAIK cartesian to polar/sphere).
Let me explain.
For example, we have a screenshot of a 3D application window (some 3D game):
Now, using Open-CV or neural network I'm able to detect the round spheres (in-game targets).
As well as their X, Y coordinates within the game window (x, y offsets).
And if I will programmatically move a mouse cursor within the given X, Y coordinates in order to aim one of the targets.
It will work only when I'm in desktop environment (moving the cursor in desktop).
But when I switch to the 3D game and thus, my mouse cursor is now within 3D game world environment - it does not work and does not aim the target.
So, I did a decent research on the topic.
And what I came across, is that the mouse cursor is locked inside 3D game.
Because of this, we cannot move the cursor using MOUSEEVENTF_MOVE (0x0001) + MOUSEEVENTF_ABSOLUTE (0x8000) flags within the mouse_event win32 call.
We are only able to move the mouse programmatically using relative movement.
And, theoretically, in order to get this relative mouse movement offsets, we can calculate the offset of detections from the middle of the 3D game window.
In such case, relative movement vector would be something like (x=-100, y=0) if the target point is 100px left from the middle of the screen.
The thing is, that the crosshair inside a 3D game will not move 100px to the left as expected.
And will not aim the given target.
But it will move a bit in a given direction.
After that, I've made more research on the topic.
And as I understand, the crosshair inside a 3D game is moving using angles in 3D space.
Specifically, there are only two of them: horizontal movement angles and vertical movement angles.
So the game engine takes our mouse movement and converts it to the movement angles within a given 3D world space.
And that's how the crosshair movement is done inside a 3D game.
But we don't have access to that, all we can is move the mouse with win32 calls externally.
Then I've decided to somehow calculate pixels per degree (amount of pixels we need to use with win32 relative mouse movement in order to move the crosshair by 1 degrees inside the game).
In order to do this, I've wrote down a simple calculation algorithm.
Here it is:
As you can see, we need to move our mouse relatively with win32 by 16400 pixels horizontally, in order to move the crosshair inside our game by 360 degrees.
And indeed, it works.
16400/2 will move the crosshair by 180 degrees respectively.
What I did next, is I tried to convert our screen X, Y target offset coordinates to percentages (from the middle of the screen).
And then convert them to degrees.
The overall formula looked like (example for horizontal movement only):
w = 100 # screen width
x_offset = 10 # target x offset
hor_fov = 106.26
degs = (hor_fov/2) * (x_offset /w) # 5.313 degrees
And indeed, it worked!
But not quite as expected.
The overall aiming precision was different, depending on how far the target is from the middle of the screen.
I'm not that great with trigonometry, but as I can say - there's something to do with polar/sphere coordinates.
Because we can see only some part of the game world both horizontally & vertically.
It's also called the FOV (Field of view).
Because of this, in the given 3D game we are only able to view 106.26 degrees horizontally.
And 73.74 degrees vertically.
My guess, is that I'm trying to convert coordinates from linear system to something non-linear.
As a result, the overall accuracy is not good enough.
I've also tried to use math.atan in Python.
And it works, but still - not accurate.
Here is the code:
def point_get_difference(source_point, dest_point):
# 1000, 1000
# source_point = (960, 540)
# dest_point = (833, 645)
# result = (100, 100)
x = dest_point[0]-source_point[0]
y = dest_point[1]-source_point[1]
return x, y
def get_move_angle__new(aim_target, gwr, pixels_per_degree, fov):
game_window_rect__center = (gwr[2]/2, gwr[3]/2)
rel_diff = list(point_get_difference(game_window_rect__center, aim_target))
x_degs = degrees(atan(rel_diff[0]/game_window_rect__center[0])) * ((fov[0]/2)/45)
y_degs = degrees(atan(rel_diff[1] / game_window_rect__center[0])) * ((fov[1]/2)/45)
rel_diff[0] = pixels_per_degree * x_degs
rel_diff[1] = pixels_per_degree * y_degs
return rel_diff, (x_degs+y_degs)
get_move_angle__new((900, 540), (0, 0, 1920, 1080), 16364/360, (106.26, 73.74))
# Output will be: ([-191.93420990140876, 0.0], -4.222458785413539)
# But it's not accurate, overall x_degs must be more or less than -4.22...
Is there a way to precisely convert 2D screen X, Y coordinates into 3D game crosshair movement degrees?
There must be a way, I just can't figure it out ...

The half-way point between the center and the edge of the screen is not equal to the field of view divided by four. As you noticed, the relationship is nonlinear.
The angle between a fractional position on the screen (0-1) and the middle of the screen can be calculated as follows. This is for the horizontal rotation (i.e, around the vertical axis), so we're only considering the X position on the screen.
# angle is the angle in radians that the camera needs to
# rotate to aim at the point
# px is the point x position on the screen, normalised by
# the resolution (so 0.0 for the left-most pixel, 0.5 for
# the centre and 1.0 for the right-most
# FOV is the field of view in the x dimension in radians
angle = math.atan((x-0.5)*2*math.tan(FOV/2))
For a field of view of 100 degrees and an x of zero, that gives us -50 degrees of rotation (exactly half the field of view). For an x of 0.25 (half-way between the edge and middle), we get a rotation of around -31 degrees.
Note that the 2*math.tan(FOV/2) part is constant for any given field of view, so you can calculate it in advance and store it. Then it just becomes (assuming we named it z):
angle = math.atan((x-0.5)*z)
Just do that for both x and y and it should work.
Edit / update:
Here is a complete function. I've tested it, and it seems to work.
import math
def get_angles(aim_target, window_size, fov):
"""
Get (x, y) angles from center of image to aim_target.
Args:
aim_target: pair of numbers (x, y) where to aim
window_size: size of area (x, y)
fov: field of view in degrees, (horizontal, vertical)
Returns:
Pair of floating point angles (x, y) in degrees
"""
fov = (math.radians(fov[0]), math.radians(fov[1]))
x_pos = aim_target[0]/(window_size[0]-1)
y_pos = aim_target[1]/(window_size[1]-1)
x_angle = math.atan((x_pos-0.5)*2*math.tan(fov[0]/2))
y_angle = math.atan((y_pos-0.5)*2*math.tan(fov[1]/2))
return (math.degrees(x_angle), math.degrees(y_angle))
print(get_angles(
(0, 0), (1920, 1080), (100, 67.67)
), "should be around -50, -33.835")
print(get_angles(
(1919, 1079), (1920, 1080), (100, 67.67)
), "should be around 50, 33.835")
print(get_angles(
(959.5, 539.5), (1920, 1080), (100, 67.67)
), "should be around 0, 0")
print(get_angles(
(479.75, 269.75), (1920, 1080), (100, 67.67)
), "should be around 30.79, 18.53")

Windows: Draw rectangle on camera preview

Similar questions have been asked a lot for Android, but so far I haven´t been able to find resources related to Windows OS. So basically, as the topic suggests, I would like to draw a rectangle on my camera preview. Some work has been done, but there´s still some problem in my program. Due to some limits, I would like to avoid using opencv as much as possible. Following is my approach:
Open Window´s built-in camera app
Run Python code that draws rectangle on screen, pixel by pixel (see below)
Click on screen with mouse to move rectangle with its upper-left corner
As you can see in the code, I´m not actually drawing on the camera preview but rather drawing on my screen, where the camer preview runs on one layer lower.
Here´s the python code:
import win32gui, win32ui, win32api, win32con
from win32api import GetSystemMetrics
dc = win32gui.GetDC(0)
dcObj = win32ui.CreateDCFromHandle(dc)
hwnd = win32gui.WindowFromPoint((0,0))
monitor = (0, 0, GetSystemMetrics(0), GetSystemMetrics(1))
red = win32api.RGB(255, 0, 0) # Red
past_coordinates = monitor
rec_x = 200 # width of rectangle
rec_y = 100 # height of rectangle
m = (100, 100) # initialize start coordinate
def is_mouse_down():
key_code = win32con.VK_LBUTTON
state = win32api.GetAsyncKeyState(key_code)
return state != 0
while True:
if(is_mouse_down() == True):
m = win32gui.GetCursorPos()
for x in range(rec_x):
win32gui.SetPixel(dc, m[0]+x, m[1], red)
win32gui.SetPixel(dc, m[0]+x, m[1]+rec_y, red)
for y in range(rec_y):
win32gui.SetPixel(dc, m[0], m[1]+y, red)
win32gui.SetPixel(dc, m[0]+rec_x, m[1]+y, red)
As a result, I´m able to draw a red rectangle. However, because the screen is constantly being refreshed, the two horizontal lines of my rectangle (see gif below) are shown as running dots that go from left to right. I can´t find or think of a way to improve this, whilst keeping the possibility to move the rectangle around per ckick.
PS. Ignore white rectangle. It´s a built-in thing of the camera app when you click anywhere on the preview.
Here are the references I used to get to this step:
How to draw an empty rectangle on screen with Python
https://python-forum.io/Thread-How-to-trigger-a-function-by-clicking-the-left-mouse-click

How to get rect of a pygame Surface without transparent parts?

I am implementing collision detection and want to check if a rectangular object is touching the player. My wall uses .set_colorkey(background) where background is the specified background colour. The problem is that when I get my wall's rectangle with .get_rect(), it gets the full image's size which includes the transparent parts instead of just the opaque parts.
I thought about making the wall image file smaller in size to remove the background but that would be inconvenient as I would need to do this for each partially transparent image I have. I also thought about using arrays to get the colour and checking if it matches the background colour and getting the rectangle's size from there but that would be slow and cumbersome.
for x, y in ((i, j) for i in land_x for j in land_y):
# land_x, land_y hold the tiles to be checked
try:
tx1, ty1, tx2, ty2 = \
texture[land[y][x]].get_rect()
# tx1, ty1 coordinates of top-left corner
# tx2, ty2 width and height respectively
if tx2 == 0 and ty2 == 0:
continue # skip to other objects
tx1 = x*64 - tx2/2
ty1 = y*64 - ty2/2
px1, py1, px2, py2 = \
PLAYER.get_rect()
px1 = player_x - px2/2
py1 = -player_y - py2/2
if p.Rect(px1, py1, px2, py2).colliderect(
p.Rect(tx1, ty1, tx2, ty2)
):
player_x -= direction_x
break # go outside loop to start checking y
except IndexError: # incase player is outside map
pass # skip to other objects
The .get_rect() outputs a rectangle the size of the whole image whereas I want a rectangle that doesn't include the transparent parts.
Example:
texture is a 64x64 image with a 48x48 block in the centre.
The background colour is removed and a 48x48 solid coloured block is left (even though the image size is still 64x64).
Expected Output:
texture.get_rect() should output a rectangle of size 48x48.
Actual Output:
texture.get_rect() instead outputs a rectangle of size 64x64.
Any help on this would be appreciated :D

If you want to ignore the transparent pixels in your collision detection, you're talking about pixel-perfect collision.
To do this in pygame, pygame offers the Mask class. You usually create your masks with pygame.mask.from_surface and use it together with pygame.sprite.spritecollide and pygame.sprite.collide_mask.
Maybe think about using pygame's Sprite class to make use of all the features it offers.
Even if you don't want to use pygame's build-in collision detection, you can take a look at the source to see how it works.

You are making this too hard. You know the size of your objects. Add a smaller collision rect to each of your objects at creation time and use that for collision. Or use a circle if that is better for the object.
tile.crect = Rect(whatever)
Or just multiply the existing rect dimensions by some scale factor for your collision rect. Don't do all of these calculations. Store a Rect for each collideable object and have a rect for the player.
tx1 = x*64 - tx2/2
ty1 = y*64 - ty2/2
px1, py1, px2, py2 = \
PLAYER.get_rect()
px1 = player_x - px2/2
py1 = -player_y - py2/2
Then just test collision directly:
for t in tiles:
if player.rect.colliderect( t.rect ):
If the player is a sprite its rect moves around. Look at the example code in the doc.
https://www.pygame.org/docs/ref/sprite.html

Python/Pygame: Drawing graph origin mathematically?

I am working on a graphing program that I am calling PyGraph.
It allows you to create a graph of any size and draw on it, and later in development I will provide coordinates and things, but for now I have one question: How can I draw a intersecting lines through the center to represent the origin?
Here is what I have so far:
#pygraph
import pygame
from pygame.locals import *
pygame.init()
screen=pygame.display.set_mode((640,480))
x=0
y=0
size=16
screen.fill((255,255,255))
pygame.draw.line(screen, (0,0,0), (screen.get_width()/2,0),(screen.get_width()/2,screen.get_height()),5)
pygame.draw.line(screen, (0,0,0), (0,screen.get_height()/2),(screen.get_width(),screen.get_height()/2),5)
while True:
while y<480:
pygame.draw.rect(screen,(0,0,0),(x,y,size,size),1)
if x>640:
x=0
y+=size
pygame.draw.rect(screen,(0,0,0),(x,y,size,size),1)
x+=size
for e in pygame.event.get():
if e.type==QUIT:
exit()
if e.type==KEYUP:
if e.key==K_SPACE:
x=0
y=0
screen.fill((255,255,255))
pygame.draw.line(screen, (0,0,0), (screen.get_width()/2,0),(screen.get_width()/2,screen.get_height()),5)
pygame.draw.line(screen, (0,0,0), (0,screen.get_height()/2),(screen.get_width(),screen.get_height()/2),5)
size=input('Enter size: ')
pygame.display.flip()
The lines go though the center, but it doesn't work for every size graph. I'm not the best at math, but I hope this isn't obvious.. any advice?

The problem is that you draw the grid using the top left corner as your anchor. That is, all your grid rectangles have one corner in the top left. This becomes a problem when the distance between the center line and the screen edge is not divisible by the size - you can't divide a line of 640 units into even divisions of 15, for example.
A far better solution would be to use the center as the anchor. So basically, all the grid rectangles have one corner in the center of the graph, which means you will never get any "remainder" on the center line, and the "remainder" will instead be on the border of the graph, which looks much nicer.
Here is code for anchoring your rectangles at the center (should replace your original while y<480 loop):
while y<=480/2+size:
pygame.draw.rect(screen,(0,0,0),(640/2+x, 480/2+y,size,size),1)
pygame.draw.rect(screen,(0,0,0),(640/2-x, 480/2+y,size,size),1)
pygame.draw.rect(screen,(0,0,0),(640/2+x, 480/2-y,size,size),1)
pygame.draw.rect(screen,(0,0,0),(640/2-x, 480/2-y,size,size),1)
x+=size
if x>=640/2+size:
x=0
y+=size
Brief explanation:
I change the anchor of the rectangle (the point you pass into pygame.draw.rect) to the center of the graph, and instead of drawing one rectangle, I draw four - one in each quadrant of the graph.
I also fixed the code a bit to not need to call pygame.draw.rect() in the if statement.
A minor style tip:
Replace 480 and 640 with "screen.width" and "screen.height", so you can adjust the width and height later without problems.

How can I make my circles fly off the screen in pygame?

I am a begginner at python and I'm trying to make a circle game. So far it draws a circle at your mouse with a random color and radius when you click.
Next, I would like the circle to fly off the screen in a random direction. How would I go about doing this? This is the main chunk of my code so far:
check1 = None
check2 = None
while True:
for event in pygame.event.get():
if event.type == QUIT:
pygame.quit()
sys.exit
if event.type == MOUSEBUTTONDOWN:
last_mouse_pos = pygame.mouse.get_pos()
if last_mouse_pos:
global check
color1 = random.randint(0,255)
color2 = random.randint(0,255)
color3 = random.randint(0,255)
color = (color1,color2,color3)
radius = random.randint (5,40)
posx,posy = last_mouse_pos
if posx != check1 and posy != check2:
global check1, check2
screen.lock()
pygame.draw.circle(screen, color, (posx,posy), radius)
screen.unlock()
check1,check2 = posx,posy
pygame.display.update()
Again, I want the circle to fly off the screen in a random direction.
I have made a few attempts but no successes yet.
Also, thanks to jdi who helped me s

When you create the circle (on click), generate 2 random numbers. These will be your (x,y) components for a two dimensional Euclidean velocity vector:
# interval -1.0 to 1.0, adjust as necessary
vx, vy = ( (random.random()*2) -1, (random.random()*2) - 1 )
Then after the ball has been created, on each iteration of the game loop, increment your ball's position by the velocity vector:
posx, posy = posx + vx, posy + vy
Note that the overall speed might be variable. If you want to always have a speed of 1.0 per seconds, normalize the vector:
To normalize the vector, you divide it by its magnitude:
So in your case:
And hence:
So in Python, after importing math with import math:
mag = math.sqrt(vx*vx + vy*vy)
v_norm = vx/mag, vy/mag
# use v_norm instead of your (vx, vy) tuple
Then you can multiply this with some speed variable, to get reliable velocity.
Once you progress to having multiple objects moving around and potentially off screen, it is useful to remove the offscreen objects which have no chance of coming back, and have nothing to do with your program anymore. Otherwise, if you keep tracking all those offscreen objects while creating more, you get essentially a memory leak, and will run out of memory given enough time/actions.
While what you are doing right now is quite simple, assuming you haven't already, learning some basic vector math will pay itself off very soon. Eventually you may need to foray into some matrix math to do certain transformations. If you are new to it, its not as hard as it first looks. You can probably get away with not studying it, but you will save yourself effort later if you start attempting to do more ambitious things.

Right now, you are doing the following (drastically simplifying your code)...
while True:
if the mouse was clicked:
draw a circle on the screen where the mouse was clicked
Let's make things a little easier, and build up gradually.
Start with the circle without the user clicking
To keep things simple, let's make the circle near the top left of the screen, that way we can always assume there will be a circle (making some of the logic easier)
circle_x, circle_y = 10,10
while True:
draw the circle at circle_x, circle_y
pygame.display.update()
Animate the circle
Before going into too much about "random directions", let's just make it easy and go in one direction (let's say, always down and to the right).
circle_x, circle_y = 0,0
while True:
# Update
circle_x += 0.1
circle_y += 0.1
# Draw
draw the circle at circle_x, circle_y
update the display
Now, every time through the loop, the center of the circle moves a bit, and then you draw it in its new position. Note that you might need to reduce the values that you add to circle_x and y (in my code, 0.1) in case the circle moves too fast.
However, you'll notice that your screen is now filling up with circles! Rather than one circle that is "moving", you're just drawing the circle many times! To fix this, we're going to "clear" the screen before each draw...
screen = ....
BLACK = (0,0,0) # Defines the "black" color
circle_x, circle_y = 0,0
while True:
# Update
circle_x += 0.1
circle_y += 0.1
# Draw
screen.fill(BLACK)
draw the circle at circle_x, circle_y
update the display
Notice that we are "clearing" the screen by painting the entire thing black right before we draw our circle.
Now, you can start work the rest of what you want back into your code.
Keep track of multiple circles
You can do this by using a list of circles, rather than two circle variables
circles = [...list of circle positions...]
while True:
# Update
for circle in circles:
... Update the circle position...
# Draw
screen.fill(BLACK)
for circle in circles:
draw the circle at circle position # This will occur once for each circle
update the display
One thing to note is that if you keep track of the circle positions in a tuple, you won't be able to change their values. If you're familiar with Object Oriented Programming, you could create a Circle class, and use that to keep track of the data relating to your circles. Otherwise, you can every loop create a list of updated coordinates for each circle.
Add circle when the user clicks
circles = []
while True:
# event handling
for event in pygame.event.get():
if event.type == MOUSEBUTTONDOWN:
pos = pygame.mouse.get_pos()
circles.append( pos ) # Add a new circle to the list
# Update all the circles
# ....
# Draw
clear the screen
for circle_position in circles:
draw the circle at circle_position # This will occur once for each circle
update the display
Have the circle move in a random direction
This is where a good helping of math comes into play. Basically, you'll need a way to determine what to update the x and y coordinate of the circle by each loop. Keep in mind it's completely possible to just say that you want it to move somewhere between -1 and 1 for each axis (X, y), but that isn't necessarily right. It's possible that you get both X and Y to be zero, in which case the circle won't move at all! The next Circle could be 1 and 1, which will go faster than the other circles.
I'm not sure what your math background is, so you might have a bit of learning to do in order to understand some math behind how to store a "direction" (sometimes referred to as a "vector") in a program. You can try Preet's answer to see if that helps. True understanding is easier with a background in geometry and trigonometry (although you might be able to get by without it if you find a good resource).
Some other thoughts
Some other things you'll want to keep in mind:
Right now, the code that we're playing with "frame rate dependent". That is, the speed in which the circles move across the screen is entirely dependent on how fast the computer can run; a slower computer will see the circles move like snails, while a faster computer will barely see the circles before they fly off the screen! There are ways of fixing this, which you can look up on your own (do a search for "frame rate dependence" or other terms in your favorite search engine).
Right now, you have screen.lock() and screen.unlock(). You don't need these. You only need to lock/unlock the screen's surface if the surface requires it (some surfaces do not) and if you are going to manually access the pixel data. Doing things like drawing circles to the screen, pygame in lock/unlock the surfaces for you automatically. In short, you don't need to deal with lock/unlock right now.