As i am starting to build some basic plotting methods for 3D visualization with VTK for some Data-Visualization, i ran over the following issue:
My Dataset is usually about the Size ob 200e6-1000e6 Datapoints (Sensor Values) with its corresponding coordinates, points, (X,Y,Z).
My Visualization method works fine, but there is at least one Bottleneck. Beside the rest of the code, the schown example with the 2 for loops, is the most time consuming part of the whole method.
I am not happy with adding the Coordinates (points, numpy(n,3) ) and Sensor Values (intensity, numpy(n,1) ) via foor loops to the VTK Objects
The spicific code example:
vtkpoints = vtk.vtkPoints() # https://vtk.org/doc/nightly/html/classvtkPoints.html
vtkpoints.SetNumberOfPoints(self.points.shape[0])
# Bottleneck - Faster way?
self.start = time.time()
for i in range(self.points.shape[0]):
vtkpoints.SetPoint(i, self.points[i])
self.vtkpoly = vtk.vtkPolyData() # https://vtk.org/doc/nightly/html/classvtkPolyData.html
self.vtkpoly.SetPoints(vtkpoints)
self.elapsed_time_normal = (time.time() - self.start)
print(f" AddPoints took : {self.elapsed_time_normal}")
# Bottleneck - Faster way?
vtkcells = vtk.vtkCellArray() # https://vtk.org/doc/nightly/html/classvtkCellArray.html
self.start = time.time()
for i in range(self.points.shape[0]):
vtkcells.InsertNextCell(1)
vtkcells.InsertCellPoint(i)
map(vtkcells.InsertNextCell(1),self.points)
self.elapsed_time_normal = (time.time() - self.start)
print(f" AddCells took : {self.elapsed_time_normal}")
# Inserts Cells to vtkpoly
self.vtkpoly.SetVerts(vtkcells)
Times:
Convert DataFrame took: 6.499739646911621
AddPoints took : 58.41245102882385b
AddCells took : 48.29743027687073
LookUpTable took : 0.7522616386413574
All Input Data is of type int, its basicly a Dataframe converted to vtknumpy objects by numpy_to_vtk method.
I am very happy, if someone has an idea of speeding this up.
BR
Bastian
For the first Loop #Mathieu Westphal gave me a nice solution:
vtkpoints = vtk.vtkPoints()
vtkpoints.SetData(numpy_to_vtk(self.points))
self.vtkpoly = vtk.vtkPolyData()
self.vtkpoly.SetPoints(vtkpoints)
New Time to Add Points:
AddPoints took : 0.03202845573425293
For the second Loop it took me a bit longer to find that solution, bit with some hints of the vtk community, i got it running too. Heres the Code i used to set the Verts of the vtkpoly:
vtkcells = vtk.vtkCellArray()
cells_array_init = np.arange(len(self.points)).reshape(len(self.points),1)
cells_array_set = np.repeat(cells_array_init, repeats=3, axis=1)
cells_npy = np.column_stack([np.full(len(self.points), 3, dtype=np.int64), cells_array_set.astype(np.int64)]).ravel()
vtkcells.SetCells(len(self.points), numpy_to_vtkIdTypeArray(cells_npy)
# Inserts Cells to vtkpoly
self.vtkpoly.SetVerts(vtkcells)
New Time to Add Cells/Verts:
AddCells took : 2.73202845573425293
(17-18 x faster)
Needed to do some numpy stuff to get a "Cellarray" and get it in the right shape: (3,0,0,0, ..., 3, n-1, n-1, n-1) with n as len(points), and convert this to numpy_to_vtkIDTypeArray.
Pretty sure this can be improved to, but i am pretty happy for now. If someone has a quick idea to even speed this up, i'd love to hear!
BR
Related
I'm looking for some help with OSRM.
First of all I'm using PyCharm and installed the packge osrm, which is a wrapper around the OSRM-API.
I have a list of GPS-points, but some of them are with noisy, so i want to match all of them to the road they belong to (see picture)
I tried the match-function as mentioned in the documentation with the following code:
chunk_size = 99
coordinates = [coordinates[i:i+chunk_size] for i in range(0, len(coordinates), chunk_size)]
osrm.RequestConfig.host = "https://router.project-osrm.org"
coord=[]
for i in coordinates:
result = osrm.match(i, steps=False, overview="simplified")
#print(result)
routes = polyline.decode(result['matchings'][0]['geometry'], geojson=True)
#print(routes)
for j in routes:
coord.append((j[0],j[1]))
First Question: Is this the right way in doing this and and is it possible to plot that right away?
Because after that i but these coord in a dataframe to plot them:
df = pd.DataFrame(coord, columns=['Lon','Lat']) # Lat=51.xxx Lon=12.xxx
print(df)
df = df.sort_values(by=['Lat'])
fig_route = px.line_mapbox(df, lon=df['Lon'], lat=df['Lat'],
width=1200, height=900, zoom=13)
fig_route.update_layout(mapbox_style='open-street-map')
fig_route.update_layout(margin={'r': 0, 't': 0, 'l': 0, 'b': 0})
fig_route.show()
And if I do that the following happens:
These points might have been matched, but look at the whole plot:
This whole plot is a mess :D
And second question:
I've got the feeling, that it takes to long to run the whole code for such a "little task". It takes roughly 10 seconds for the whole code to read the points from excel into a parquet-file (7800 GPS points) - put them into a list and delete duplicates (list(set())) and do the request. Are these 10 seconds ok or is there a mistake in my code?
Thank you in advance for the help!
My previous code provides me for each entry of a reference list borders of a interesting region with indices of another list.
So for example I have a listA, which should be assigned to values in another listB. For each entry it should be possible to get the indices, so that is valid:
listA[:]-d/2 <= listB[indices to find] <= listA[:]+d/2
I solved the problem with a list comprehension and use the python builtin range() method with the border indices as argument to get all required values. So I loop over my border list and produce a list with all inddices. So e.g.:
borders[0,:] = [1,4] becomes indices[0] = [1,2,3].
arr = [values[range(borders[i,0], borders[i,1])] for i in range(borders.shape[0])]
It works, but it is too slow for large datasets. I located the list comprehension to be the problem. Is there a numpy/pandas/... method which I can use that it is a kind of matrix operation?
The dataset is similar to the following:
no_points = 10000
no_groups = 3
meas_duration = 60
df_AT = pd.DataFrame(np.transpose([np.sort(np.random.rand(no_points)*meas_duration) for _ in range(no_groups)]), columns = ['AT {}'.format(i+1) for i in range(no_groups)])
df_TT = pd.DataFrame(np.transpose([np.random.rand(no_points) for _ in range(no_groups)]), columns = ['TT {}'.format(i+1) for i in range(no_groups)])
df = pd.concat([df_AT, df_TT], axis=1)
filterCoincidence(df, window=1e-3)
\\ edit
Unfortunately, I'm still working on it. I will just copy a piece of my code:
# process coincidence
borders = [list() for _ in range(len(AT_cols)-1)]
test = np.empty((AT_df.shape[0],3), dtype=object)
test[:,0] = np.arange(AT_df.shape[0])
for i, [AT, TT] in enumerate(zip(AT_cols[np.where(AT_cols != AT_cols[used_ref])], TT_cols[np.where(AT_cols != AT_cols[used_ref])])):
AT_ix = np.argwhere(AT_cols == AT).flatten()[0]
neighbors_lower = np.searchsorted(AT_df[AT].values, AT_df[AT_cols[used_ref]]-window, side='left')
neighbors_upper = np.searchsorted(AT_df[AT].values, AT_df[AT_cols[used_ref]]+window, side='left')
borders[i] = np.transpose([neighbors_lower, neighbors_upper])
coinc_ix = np.where(np.diff(borders[i], axis=1).flatten() != 0)[0]
test[coinc_ix,i+1]=np.asarray([np.arange(borders[i][j][0], borders[i][j][1], dtype=int) for j in coinc_ix])
test = test[~np.any(pd.isnull(test), axis=1)]
So now this part is fast enough for my purpose. With the hint from Drecker and nakor is is still slightly faster. The problem is now I have candidates for my samples, but I still have to do the following task:
Order the samples regarding the first criterion: Which the most similar? So I have to compare the transit time and the arrival time (the two columns AT und TT). I could do it with sorted(key=my_fun), but this is really time consuming
Check if all samples are within the time window. This is fulfilled compared with the reference data, but are the measurements from the two non-reference groups also within the time window? I could simplify the problem by using only +- window/2 in the code, but this is a really strong assumption because the reference measurements should always be in the middle of the time window. Therefore I used scipy.spatial.distance.cdist() and checked for the distances
Use direct slices:
arr = [values[border_pair[0]:border_pair[1]] for border_pair in borders]
If values is numpy array this could be at least somewhat faster.
However, this is just simple copy of your data and if this is taking too long (i think) there is not so much you can do with it. But if this is really the bottleneck, maybe you do not need copy all these data but you could use then read-only inside the values array? It is hard to tell as you do not mention what you do with them in the question.
To complement Drecker's comment, I did some tests on my machine, and it's about 10 times as fast on a 10,000x10,000 matrix if values is a numpy array:
# Generate random data
import numpy as np
N=10000
values = np.random.randint(0,100,[N,N]).astype(int)
borders = []
for _ in range(N):
inf = np.random.randint(0,99)
sup = np.random.randint(inf,100)
borders.append([inf,sup])
borders = np.array(borders)
In [1]: %time arr = [values[range(borders[i,0], borders[i,1])] for i in range(borders.shape[0])]
CPU times: user 7.97 s, sys: 7.27 s, total: 15.2 s
Wall time: 17.5 s
In [2]: %time arr=[values[borders[i,0]: borders[i,1]] for i in range(borders.shape[0])]
CPU times: user 30.7 ms, sys: 1.4 s, total: 1.43 s
Wall time: 1.43 s
"""Some simulations to predict the future portfolio value based on past distribution. x is
a numpy array that contains past returns.The interpolated_returns are the returns
generated from the cdf of the past returns to simulate future returns. The portfolio
starts with a value of 100. portfolio_value is filled up progressively as
the program goes through every loop. The value is multiplied by the returns in that
period and a dollar is removed."""
portfolio_final = []
for i in range(10000):
portfolio_value = [100]
rand_values = np.random.rand(600)
interpolated_returns = np.interp(rand_values,cdf_values,x)
interpolated_returns = np.add(interpolated_returns,1)
for j in range(1,len(interpolated_returns)+1):
portfolio_value.append(interpolated_returns[j-1]*portfolio_value[j-1])
portfolio_value[j] = portfolio_value[j]-1
portfolio_final.append(portfolio_value[-1])
print (np.mean(portfolio_final))
I couldn't find a way to write this code using numpy. I was having a look at iterations using nditer but I was unable to move ahead with that.
I guess the easiest way to figure out how you can vectorize your stuff would be to look at the equations that govern your evolution and see how your portfolio actually iterates, finding patterns that could be vectorized instead of trying to vectorize the code you already have. You would have noticed that the cumprod actually appears quite often in your iterations.
Nevertheless you can find the semi-vectorized code below. I included your code as well such that you can compare the results. I also included a simple loop version of your code which is much easier to read and translatable into mathematical equations. So if you share this code with somebody else I would definitely use the simple loop option. If you want some fancy-pants vectorizing you can use the vector version. In case you need to keep track of your single steps you can also add an array to the simple loop option and append the pv at every step.
Hope that helps.
Edit: I have not tested anything for speed. That's something you can easily do yourself with timeit.
import numpy as np
from scipy.special import erf
# Prepare simple return model - Normal distributed with mu &sigma = 0.01
x = np.linspace(-10,10,100)
cdf_values = 0.5*(1+erf((x-0.01)/(0.01*np.sqrt(2))))
# Prepare setup such that every code snippet uses the same number of steps
# and the same random numbers
nSteps = 600
nIterations = 1
rnd = np.random.rand(nSteps)
# Your code - Gives the (supposedly) correct results
portfolio_final = []
for i in range(nIterations):
portfolio_value = [100]
rand_values = rnd
interpolated_returns = np.interp(rand_values,cdf_values,x)
interpolated_returns = np.add(interpolated_returns,1)
for j in range(1,len(interpolated_returns)+1):
portfolio_value.append(interpolated_returns[j-1]*portfolio_value[j-1])
portfolio_value[j] = portfolio_value[j]-1
portfolio_final.append(portfolio_value[-1])
print (np.mean(portfolio_final))
# Using vectors
portfolio_final = []
for i in range(nIterations):
portfolio_values = np.ones(nSteps)*100.0
rcp = np.cumprod(np.interp(rnd,cdf_values,x) + 1)
portfolio_values = rcp * (portfolio_values - np.cumsum(1.0/rcp))
portfolio_final.append(portfolio_values[-1])
print (np.mean(portfolio_final))
# Simple loop
portfolio_final = []
for i in range(nIterations):
pv = 100
rets = np.interp(rnd,cdf_values,x) + 1
for i in range(nSteps):
pv = pv * rets[i] - 1
portfolio_final.append(pv)
print (np.mean(portfolio_final))
Forget about np.nditer. It does not improve the speed of iterations. Only use if you intend to go one and use the C version (via cython).
I'm puzzled about that inner loop. What is it supposed to be doing special? Why the loop?
In tests with simulated values these 2 blocks of code produce the same thing:
interpolated_returns = np.add(interpolated_returns,1)
for j in range(1,len(interpolated_returns)+1):
portfolio_value.append(interpolated_returns[j-1]*portfolio[j-1])
portfolio_value[j] = portfolio_value[j]-1
interpolated_returns = (interpolated_returns+1)*portfolio - 1
portfolio_value = portfolio_value + interpolated_returns.tolist()
I assuming that interpolated_returns and portfolio are 1d arrays of the same length.
really not long ago I had my first dumb question answered here so... there I am again, with a hopefully less dumb and more interesting headscratcher. Keep in my mind I am still making my baby steps in scripting !
There it is : I need to rig a feathered wing, and I already have all the feathers in place. I thought of mimicking another rig I animated recently that had the feathers point-constrained to the arm and forearm, and orient-constrained to three other controllers on the arm : each and every feather was constrained to two of those controllers at a time, and the constraint's weights would shift as you went down the forearm towards the wrist, so that one feather perfectly at mid-distance between the elbow and the forearm would be equally constrained by both controllers... you get the picture.
My reasoning was as follows : let's make a loop that iterates over every feather, gets its world position, finds the distance from that feather to each of the orient controllers (through Pythagoras), normalize that and feed the values into the weight attribute of an orient constraint. I could even go the extra mile and pass the normalized distance through a sine function to get a nice easing into the feathers' silhouette.
My pseudo-code is ugly and broken, but it's a try. My issues are inlined.
Second try !
It works now, but only on active object, instead of the whole selection. What could be happening ?
import maya.cmds as cmds
# find world space position of targets
base_pos = cmds.xform('base',q=1,ws=1,rp=1)
tip_pos = cmds.xform('tip',q=1,ws=1,rp=1)
def relative_dist_from_pos(pos, ref):
# vector substract to get relative pos
pos_from_ref = [m - n for m, n in zip(pos, ref)]
# pythagoras to get distance from vector
dist_from_ref = (pos_from_ref[0]**2 + pos_from_ref[1]**2 + pos_from_ref[2]**2)**.5
return dist_from_ref
def weight_from_dist(dist_from_base, dist_to_tip):
normalize_fac = (1/(dist_from_base + dist_to_tip))
dist_from_base *= normalize_fac
dist_to_tip *= normalize_fac
return dist_from_base, dist_to_tip
sel = cmds.ls(selection=True)
for obj in sel:
# find world space pos of feather
feather_pos = cmds.xform(obj, q=1, ws=1, rp=1)
# call relative_dist_from_pos
dist_from_base = relative_dist_from_pos(feather_pos, base_pos)
dist_to_tip = relative_dist_from_pos(feather_pos, tip_pos)
# normalize distances
weight_from_dist(dist_from_base, dist_to_tip)
# constrain the feather - weights are inverted
# because the smaller the distance, the stronger the constraint
cmds.orientConstraint('base', obj, w=dist_to_tip)
cmds.orientConstraint('tip', obj, w=dist_from_base)
There you are. Any pointers are appreciated.
Have a good night,
Hadriscus
When I run this script:
import bpy, time
t0 = time.time()
for i in range(1000):
bpy.ops.mesh.primitive_uv_sphere_add()
if i % 100 == 0:
print(time.time()-t0)
t0 = time.time()
This is the output (exponential growth vs. time):
1.1920928955078125e-05
0.44658803939819336
0.46373510360717773
0.5661759376525879
0.7258329391479492
0.9994637966156006
1.381392002105713
1.8257861137390137
2.4634311199188232
3.2817111015319824
Why does this happen? Is there a better approach?
I am running this on a server with ample memory, and I know Blender can expand to use most of it (it does in rendering).
The quick answer:
bpy.ops.object.select_all(action='DESELECT')
bpy.ops.mesh.primitive_uv_sphere_add()
sphere = bpy.context.object
for i in range(1000):
ob = sphere.copy()
ob.data = sphere.data.copy()
bpy.context.scene.objects.link(ob)
bpy.context.scene.update()
Explanation:
Anything in bpy.ops.* causes a scene redraw with each call. You want to avoid calling these in loops. The above script calls lower-level copy() methods, which don't redraw. If you want linked duplicates, you can remove the sphere.data.copy() line.
This solution is not my own. Kudos go to CoDEmanX over at BlenderArtists for this answer!