I am trying to project a 3D point from a pointcloud to a 2D panorama image. I have manually picked 8 points on the 2D image and the corresponding 3D pointcloud points and used the OpenCV library to find the camera matrix, translation, and rotation vectors that are then used in projection method.
#initial camera matrix
camera_matrix = cv2.initCameraMatrix2D([objectPoints_3D],[imagePoints_2D], image_size_2D)
#calibrate cameras
intrinsics_error, intrinsic_matrix, intrinsics_distortion_coeffs, rotation_vectors, translation_vectors = cv2.calibrateCamera([objectPoints_3D], [imagePoints_2D], image_size_2D, camera_matrix, None, flags=cv2.CALIB_USE_INTRINSIC_GUESS) #camera_matrix and intrinsic_matrix are the same
#project a 3D point in the pointcloud onto a 2D image
projected_point_2D = cv2.projectPoints(selected_point_3D, rotation_vectors[0], translation_vectors[0], intrinsic_matrix, intrinsics_distortion_coeffs)
The projected_point_2D seems reasonable for the same image that I used when calibrating, but fails in another set of 2D and 3D images. Am I doing this wrong?
Images contain buildings and trees
Related
I'm looking for a solution to estimate head poses from a stereo camera setup.
I calibrated my cameras via opencv and estimated all important matrices to estimate the disparity of my tracked face points to get the 3D and 2D coordinates to solve the PnP-Problem via opencv.
The idea was to track the landamrks on both images Left and Right (I used the unsdistorted images via cv.remap(image_L, stereoMap_L)).
On this images I estimated the landmarks, see image.
Landmarks left and right
With the estimated landmarks I estimated the disparity of all landmarks.
For Disparity I used this function:
def disparity(points_L, points_R):
dispar = landmarks_left[:,0] - landmarks_right[:,0]
return dispar
I followd this paper to estimate all important matrices and points for my setup.
Three-Dimensional Head Pose Estimation Using a Stereo Camera Arrangement
face_2d_L = np.array(face_2d_L, dtype=np.float64) #landmarks left 2D
face_2d_R = np.array(face_2d_R, dtype=np.float64) # landmarks right 2D
dispar = disparity(face_2d_L[:,0], face_2d_R[:,0])
face_3d_L[:,2] = dispar
Q_matrix = np.load('Q_matrix.npy') # estimated via stereo camera calibration
face_3d_Left = face_3d_L # Q_matrix.T
face_3d_Left = (face_3d_Left[:,0:3].T / face_3d_Left[:,3]).T
cam_matrix_L = np.load('cam_matrix_L.npy')
cam_matrix_R = np.load('cam_matrix_R.npy')
dist_matrix_L = np.load('distorsion_left.npy')
dist_matrix_R = np.load('distorsion_right.npy')
success, rot_vec_L, trans_vec_L = cv2.solvePnP(face_3d_Left, face_2d_L, cam_matrix_L, dist_matrix_L)
To solve the PnP-Problem I used the 3D estimated points from the landmarks and the disaprity, the 2D landmarks, left cam matrix and the distorsion matrix from the stereo calibration.
Then i thought I'm more or less done and have all import matrices to estimate the rotation vectors.
rmat_L, jac_L = cv2.Rodrigues(rot_vec_L)
angles_L, mtxR_L, mtxQ_L, Qx_L, Qy_L, Qz_L = cv2.RQDecomp3x3(rmat_L)
Unfortunately I think that I made some mistakes or understand something wrong.
Perhaps you can help to understand this topic a bit better.
Thank you!
Is there a way how I could transform 3D points to 2D points corresponding to the positions of a 2D image?
Is there a function in Python/OpenCV that has following properties:
input: 3D points, camera position in 3D, direction of the camera, maybe additional features of the camera
output: corresponding 2D points on the 2D image of the camera?
Have you seen cv2.projectPoints method?
Python: cv2.projectPoints(objectPoints, rvec, tvec, cameraMatrix, distCoeffs[, imagePoints[, jacobian[, aspectRatio]]]) → imagePoints, jacobian
The function computes projections of 3D points to the image plane given intrinsic and extrinsic camera parameters.
more details here
I'm currently working with some 3D models from the ModelNet40 dataset and I'd like to place a virtual spherical camera centered inside of the 3D object - which is a 3D triangular mesh - in order to trace rays from the camera and store informations related to each triangle that the ray hits.
I've only computed some basic informations such as vertices, normals and triangles:
mesh = o3d.io.read_triangle_mesh(r"...\airplane_0627.off")
print('Vertices:')
print(np.asarray(mesh.vertices),"\n")
print('Triangles:')
print(np.asarray(mesh.triangles),"\n")
print("Computing normal and rendering it:")
mesh.compute_vertex_normals()
print(np.asarray(mesh.triangle_normals))
meshVertices = np.asarray(mesh.vertices)
triangleNormals = np.asarray(mesh.triangle_normals)
meshTriangles = np.asarray(mesh.triangles)
o3d.visualization.draw_geometries([mesh], mesh_show_wireframe=True)
And tried to use the camera positioning from Open3D with some random pose parameters to see where the camera is placed:
vis = o3d.visualization.VisualizerWithKeyCallback()
vis.create_window()
view_ctl = vis.get_view_control()
vis.add_geometry(mesh)
pose = np.array([ 0.23522,-7.0289e-17,-0.97194,-129.54,
0.97194,-7.3988e-17,0.23522,59.464,
-8.8446e-17,-1,5.0913e-17,2.11,
0,0,0,1]).reshape(4,4)
cam = view_ctl.convert_to_pinhole_camera_parameters()
cam.extrinsic = pose
view_ctl.convert_from_pinhole_camera_parameters(cam)
vis.run()
vis.destroy_window()
What is the best way to place the camera inside the centre of a 3D object/triangular mesh?
I have a binary mask of 3D medical images in nii format which is reconstructed from 2D slices and I need to extract surface mesh from the mask. AFAK, a surface mesh is constructed from the surface points of 3D surface, but I do not have the point sets (vertices).
Do you know any function or software that I can extract the mesh from the binary mask?
Thanks
The opencv function cv2.projectPoints() takes 3d points, rotation vector, translation vector, camera matrix and distortion coefficients and aspect ratio and generate the 2d projection of the 3d point in the image plane. Is there some way to get the full projection matrix which does the same to any 3d point so that i can do the projection without using the projectPoints fucntion