Develop Reference

TypeError: unhashable type 'numpy.ndarray' followed by SympifyError

I am following the steps on https://www.sympy.org/scipy-2017-codegen-tutorial/notebooks/22-lambdify.html in order to produce a plot of a surface. The example shown on the link works perfectly to me.
Now, the problem appears to me when trying to produce a plot of my own: I define my functions also making use of lambdify and, as far as I am concerned, following the same steps as in the link mentioned before. Here is my code:
import numpy as np
import plotly.graph_objects as go # for the surface
import sympy as sym
sym.init_printing()
## definition of the surface and its normal unitary field
u,v = sym.symbols('u v') # we define the symbols
surface_expr = -sym.sqrt(1-u**2-v**2) # expresion for the surface
surface = sym.lambdify([u,v], surface_expr, modules = ['sympy']) # symbolic surface
## plot the surface
uplot = np.outer(np.linspace(-0.5, 0.5, 100), np.ones(100)) # plotting mesh
vplot = uplot.T
zplot = surface(uplot,vplot) # z value over the mesh
colorscale = 'oryel' # color scale for the surface plot
fig = go.Figure() # this creates the figure object
fig.add_surface(x=uplot, y=vplot, z=zplot, colorscale = colorscale) # we add a surfaceplot to it
fig.update_traces(showscale=False)
When running this code, I get the following error:
TypeError Traceback (most recent call last)
File ~/opt/anaconda3/lib/python3.9/site-packages/sympy/core/cache.py:70, in __cacheit.<locals>.func_wrapper.<locals>.wrapper(*args, **kwargs)
69 try:
---> 70 retval = cfunc(*args, **kwargs)
71 except TypeError as e:
TypeError: unhashable type: 'numpy.ndarray'
During handling of the above exception, another exception occurred:
SympifyError Traceback (most recent call last)
/Users/jzaragoza/PhD/Codes/offset_surface.ipynb Cell 4 in <cell line: 5>()
3 uplot = np.outer(np.linspace(-0.5, 0.5, 100), np.ones(100)) # plotting mesh
4 vplot = uplot.T
----> 5 zplot = surface(uplot,vplot) # z value over the mesh
7 colorscale = 'oryel' # color scale for the surface plot
10 fig = go.Figure() # this creates the figure object
File <lambdifygenerated-1>:2, in _lambdifygenerated(u, v)
1 def _lambdifygenerated(u, v):
----> 2 return -sqrt(-u**2 - v**2 + 1)
File ~/opt/anaconda3/lib/python3.9/site-packages/sympy/functions/elementary/miscellaneous.py:154, in sqrt(arg, evaluate)
70 """Returns the principal square root.
...
0.5197939 ],
[0.50999898, 0.51999796, 0.52979288, ..., 0.52979288, 0.51999796,
0.50999898],
[0.5 , 0.50999898, 0.5197939 , ..., 0.5197939 , 0.50999898,
0.5 ]])
You can finde an image of the error here
Anyone knows why this happens and/or how to fix it and make it work? Thank you in advance!!!
PS: number lines in the error message do not match with the lines of code posted here since my actual code has more lines but they are not relevant for the question.

The problem is this line:
surface = sym.lambdify([u,v], surface_expr, modules = ['sympy'])
here, you have created a lambda function that will be evaluated by SymPy. Then, with zplot = surface(uplot,vplot) you are passing in Numpy arrays, but SymPy doesn't know what to do with them.
Instead, you should create a lambda function that will be evaluated with Numpy/Scipy:
surface = sym.lambdify([u,v], surface_expr)
Now, everything will work as expected.
EDIT: if you want to create plots of symbolic expressions with Plotly, I suggest to take a look at this module, SymPy Plot Backends. Then, you can simply write:
from spb import *
plot3d(surface_expr, (u, -0.5, 0.5), (v, -0.5, 0.5), backend=PB, use_cm=True)

Struggling to create watertight meshes out of point cloud data using Open3D in Python

I am trying to create a watertight mesh out of point cloud representing organ contour data from cone beam CT images. My goal is to take two meshes and calculate the volume of intersection between the two of them.
I have tried using each of the methods shown here
Poisson Reconstruction
point_cloud = np.genfromtxt('ct_prostate_contour_data.csv', delimiter=',')
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(point_cloud)
pcd.compute_convex_hull()
pcd.estimate_normals()
pcd.orient_normals_consistent_tangent_plane(10)
mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson(pcd, depth=10, width=0, scale=20, linear_fit=True)[0]
mesh.compute_vertex_normals()
mesh.paint_uniform_color([0.5, 0.5, 0.5])
mesh.remove_degenerate_triangles()
o3d.visualization.draw_geometries([pcd, mesh], mesh_show_back_face=True)
While this method seemingly leads to a watertight mesh to my eye, the result of mesh.is_watertight() is False, however for the Bladder data it returns True. Furthermore, the algorithm extends the mesh above and below the vertical limits of the data. Wile this isn't a deal breaking issue if there were a way to minimize it that would be great.
Poisson Mesh Image
Ball Pivoting
point_cloud = np.genfromtxt('ct_prostate_contour_data.csv', delimiter=',')
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(point_cloud)
pcd.compute_convex_hull()
pcd.estimate_normals()
pcd.orient_normals_consistent_tangent_plane(30)
distances = pcd.compute_nearest_neighbor_distance()
avg_dist = np.mean(distances)
radii = [0.1*avg_dist, 0.5*avg_dist, 1*avg_dist, 2*avg_dist]
r = o3d.utility.DoubleVector(radii)
rec_mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_ball_pivoting(pcd, r)
o3d.visualization.draw_geometries([pcd, rec_mesh], mesh_show_back_face=True)
This would be my preferred method if I were able to fill the holes as it simply connects vertices without interpolation. Perhaps if I were able to get this into a state where the only remaining holes were large I could convert this mesh into a Pyvista compatible mesh and use Pymeshfix to patch the holes.
Ball Pivoting Mesh Image
Alpha Shapes
point_cloud = np.genfromtxt('ct_prostate_contour_data.csv', delimiter=',')
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(point_cloud)
alpha = 8
tetra_mesh, pt_map = o3d.geometry.TetraMesh.create_from_point_cloud(pcd)
mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_alpha_shape(pcd, alpha, tetra_mesh, pt_map)
mesh.compute_vertex_normals()
mesh.paint_uniform_color([0.5, 0.5, 0.5])
mesh.remove_degenerate_triangles()
o3d.visualization.draw_geometries([pcd, mesh])
The results from this are similar to ball pivoting but worse.
Alpha Shapes Mesh Image
Sample Data
ct_prostate_contour_data.csv
ct_rectum_contour_data.csv
ct_bladder_contour_data.csv

I am one of the authors of the PyVista module. We've introduced the vtkSurfaceReconstructionFilter within PyVista in pull request #1617.
import pymeshfix
import numpy as np
import pyvista as pv
pv.set_plot_theme('document')
array = np.genfromtxt('ct_prostate_contour_data.csv', delimiter=',')
point_cloud = pv.PolyData(array)
surf = point_cloud.reconstruct_surface(nbr_sz=20, sample_spacing=2)
mf = pymeshfix.MeshFix(surf)
mf.repair()
repaired = mf.mesh
pl = pv.Plotter()
pl.add_mesh(point_cloud, color='k', point_size=10)
pl.add_mesh(repaired)
pl.add_title('Reconstructed Surface')
pl.show()

reproject mesh points of vtu file using pyvista and pyproj

Is there a chance I can replace coordinates of a pyvista_ndarray by a ndarray?
My objective is to reproject points coordinates of a vtu file (unstructured grid). The current coordinates of the mesh object are in the coordinate reference system EPSG 27672 and I want them in EPSG 4326 (WGS84).
For this, I open the vtu file using the pyvista module :
mesh = pv.read("arg_Tm2__t0002.pvtu")
type(mesh.points)
>pyvista.core.pyvista_ndarray.pyvista_ndarray
As a result, mesh.points gives the 3 spatial coordinates. Then, I use pyproj module to reproject the 3 coordinates into EPSG 4326. By combining the 3 resulting x,y,z numpy.ndarray, I now get a NumPy array with a shape and size similar to mesh.points.
# set the pyproj transformer
crs1 = CRS.from_epsg(27562)
crs2 = CRS.from_epsg(4326)
reproj = Transformer.from_crs(crs1, crs2)
# Reprojection
reproj_dataY,reproj_dataX,altitude = reproj.transform(mesh.points[:,0],mesh.points[:,1],mesh.points[:,2])
reprojData = np.column_stack((reproj_dataX,reproj_dataY,altitude))
#compare objects
print('original Mesh points -> ', mesh.points)
print('original Mesh type: ', type(mesh.points))
print('Reprojected points-> ', reprojData)
print('Reprojected type: ', type(reprojData))
Original Mesh points -> [[958427.33 119680.95 2396.288549 ]
[957754.39 120023.85 2430.1833881 ]
[957256.56 120241.02 2112.22953263]
...
[963366.748527 115096.364632 3054.75408138]
[963401.840285 113351.753238 3024.50286566]
[963497.913738 113339.696062 3048.83674197]]
Original Mesh type: <class 'pyvista.core.pyvista_ndarray.pyvista_ndarray'>
Reprojected points-> [[ 6.96487903 45.9823843 2396.288549 ]
[ 6.95646936 45.98581994 2430.1833881 ]
[ 6.95021969 45.98803333 2112.22953263]
...
[ 7.02498443 45.93857775 3054.75408138]
[ 7.02409542 45.92289079 3024.50286566]
[ 7.02532248 45.92273099 3048.83674197]]
Reprojected type: <class 'numpy.ndarray'>enter code here
Now, It's time to replace the coordinates of the vtu object:
mesh.points = reprojData
Finally, I check the modified mesh: The X bounds and Y bounds have been modified and the ranges are correct. However, the plot shows a line of points instead a nice 3d object. :(.
Do you have any idea what is wrong? Do you see another way to manage reprojection?

The range of values of XY and Z after the transformation are significantly different:
>>>np.array(mesh.bounds).reshape((3,-1)).ptp(axis=1)
array([1.22515302e-01, 7.78657599e-02, 2.47978788e+03])
XY are indeed in degrees and Z still in meter. The visual representation of this data is unrealistic and data should be adapted to do so.

open3d compute distance between mesh and point cloud

For a study project, I try to get into point cloud comparison.
to keep it short, I have a CAD file (.stl) and several point clouds created by a laser scanner.
now I want to calculate the difference between the CAD file and each point cloud.
first I started with Cloud Compare which helps a lot to get a basic understanding. (reduction of points, remove duplicates, create a mesh and compare distances)
In python, I was able to import the files and do some basic calculations. However, I am not able to calculate the distance.
here is my code:
import numpy as np
import open3d as o3d
#read point cloud
dataname_pcd= "pcd.xyz"
point_cloud = np.loadtxt(input_path+dataname_pcd,skiprows=1)
#read mesh
dataname_mesh = "cad.stl"
mesh = o3d.io.read_triangle_mesh(input_path+dataname_mesh)
print (mesh)
#calulate the distance
mD = o3d.geometry.PointCloud.compute_point_cloud_distance([point_cloud],[mesh])
#calculate the distance gives me this error:
"TypeError: compute_point_cloud_distance(): incompatible function arguments. The following argument types are supported:
1. (self: open3d.cpu.pybind.geometry.PointCloud, target: open3d.cpu.pybind.geometry.PointCloud) -> open3d.cpu.pybind.utility.DoubleVector"
Questions:
what pre transformations for mesh and point clouds are needed to calculate their distances?
is there a recommended way to display the differences?
so far I just used the visualization line below
o3d.visualization.draw_geometries([pcd],
zoom=0.3412,
front=[0.4257, -0.2125, -0.8795],
lookat=[2.6172, 2.0475, 1.532],
up=[-0.0694, -0.9768, 0.2024])

You need 2 point clouds for the function "compute point cloud distance()", but one of your geometries is a mesh, which is made of polygons and vertices. Just convert it to a point cloud:
pcd = o3d.geometry.PointCloud() # create a empty geometry
pcd.points = mesh.vertices # take the vertices of your mesh
I'll illustrate how you can visualize the distances between 2 clouds, both captured on a moving robot (a Velodyne LIDAR) separeted by 1 meter in average. Consider 2 cloud before and after the registration, the distances between them should decrease, right? Here is some code:
import copy
import pandas as pd
import numpy as np
import open3d as o3d
from matplotlib import pyplot as plt
# Import 2 clouds, paint and show both
pc_1 = o3d.io.read_point_cloud("scan_0.pcd") # 18,421 points
pc_2 = o3d.io.read_point_cloud("scan_1.pcd") # 19,051 points
pc_1.paint_uniform_color([0,0,1])
pc_2.paint_uniform_color([0.5,0.5,0])
o3d.visualization.draw_geometries([pc_1,pc_2])
# Calculate distances of pc_1 to pc_2.
dist_pc1_pc2 = pc_1.compute_point_cloud_distance(pc_2)
# dist_pc1_pc2 is an Open3d object, we need to convert it to a numpy array to
# acess the data
dist_pc1_pc2 = np.asarray(dist_pc1_pc2)
# We have 18,421 distances in dist_pc1_pc2, because cloud pc_1 has 18,421 pts.
# Let's make a boxplot, histogram and serie to visualize it.
# We'll use matplotlib + pandas.
df = pd.DataFrame({"distances": dist_pc1_pc2}) # transform to a dataframe
# Some graphs
ax1 = df.boxplot(return_type="axes") # BOXPLOT
ax2 = df.plot(kind="hist", alpha=0.5, bins = 1000) # HISTOGRAM
ax3 = df.plot(kind="line") # SERIE
plt.show()
# Load a previos transformation to register pc_2 on pc_1
# I finded it with the Fast Global Registration algorithm, in Open3D
T = np.array([[ 0.997, -0.062 , 0.038, 1.161],
[ 0.062, 0.9980, 0.002, 0.031],
[-0.038, 0.001, 0.999, 0.077],
[ 0.0, 0.0 , 0.0 , 1.0 ]])
# Make a copy of pc_2 to preserv the original cloud
pc_2_copy = copy.deepcopy(pc_2)
# Aply the transformation T on pc_2_copy
pc_2_copy.transform(T)
o3d.visualization.draw_geometries([pc_1,pc_2_copy]) # show again
# Calculate distances
dist_pc1_pc2_transformed = pc_1.compute_point_cloud_distance(pc_2_copy)
dist_pc1_pc2_transformed = np.asarray(dist_pc1_pc2_transformed)
# Do as before to show diferences
df_2 = pd.DataFrame({"distances": dist_pc1_pc2_transformed})
# Some graphs (after registration)
ax1 = df_2.boxplot(return_type="axes") # BOXPLOT
ax2 = df_2.plot(kind="hist", alpha=0.5, bins = 1000) # HISTOGRAM
ax3 = df_2.plot(kind="line") # SERIE
plt.show()

problem with hierarchical clustering in Python

I am doing a hierarchical clustering a 2 dimensional matrix by correlation distance metric (i.e. 1 - Pearson correlation). My code is the following (the data is in a variable called "data"):
from hcluster import *
Y = pdist(data, 'correlation')
cluster_type = 'average'
Z = linkage(Y, cluster_type)
dendrogram(Z)
The error I get is:
ValueError: Linkage 'Z' contains negative distances.
What causes this error? The matrix "data" that I use is simply:
[[ 156.651968 2345.168618]
[ 158.089968 2032.840106]
[ 207.996413 2786.779081]
[ 151.885804 2286.70533 ]
[ 154.33665 1967.74431 ]
[ 150.060182 1931.991169]
[ 133.800787 1978.539644]
[ 112.743217 1478.903191]
[ 125.388905 1422.3247 ]]
I don't see how pdist could ever produce negative numbers when taking 1 - pearson correlation. Any ideas on this?
thank you.

There are some lovely floating point problems going on. If you look at the results of pdist, you'll find there are very small negative numbers (-2.22044605e-16) in them. Essentially, they should be zero. You can use numpy's clip function to deal with it if you would like.

If you were getting error
KeyError: -428
and your code was on the lines of
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
from scipy.cluster.hierarchy import ward, dendrogram
linkage_matrix = ward(dist) #define the linkage_matrix using ward clustering pre-computed distances
fig, ax = plt.subplots(figsize=(35, 20),dpi=400) # set size
ax = dendrogram(linkage_matrix, orientation="right",labels=queries);
`
It is due to the mismatch in indexes of queries.
Might want to update to
ax = dendrogram(linkage_matrix, orientation="right",labels=list(queries));