This question is likely to be more about the right terminology and subjects to search for more than anything else. It feels like a simple enough concept to the point where tools should be available in Python/NumPy to do it, but I just have no idea what to look for.
I recently watched a video on Space Carving and I would like to implement the basic concept using video game sprites to attempt voxelizing them. I have sprites of characters from 8 equally spaced angles (front on, camera 45 degress right, fully right, back, etc.)
I haven't found a library dedicated to this concept, but I think it should be fairly simple to implement. My thinking is I can make a 3D array that is the max size of the sprite in all dimensions that is a solid block of "clay". Then I need a 2D representation of that 3D array at each rotation angle. For each pixel in that representation I need to be able to iterate down into the 3D array each block that a "laser" fired from that position would hit.
The first step would be removing the clay where the sprite is just alpha layer, (aka setting a bool to false). The next would be "painting" the shape where possible.
The problem is I just have no idea what mathematical or programmatic terms are associated with these concepts. I could get so far as to make a 3D block of clay. But how do I get 2D representations of a 3D array at several rotated angles about one axis that I can essentially fire lasers at?
Related
I intend to make a 3D model based on multi view stereo images ( basically 2D plane images of the same object from different angles and orientation) inside Blender from scratch.However, I am new to Blender.
I wanted to know if there are any tutorials of how to project a single pixel or point in the space of Blender's 3D environment using python. If not tutorial, any documentation. I am still learning about this whole 3D construction thing and pretty new to this, so I am not sure maybe these points are displayed using a 3 dimensional matrix/array ?
Basically I want to implement 3D construction based on a paper written by some researchers. Mostly every such project is in C++. I want to do it in Python in Blender, and if I am capable enough, make these libraries open source.
Suggest me any pre-requisite if you think that shall help me. I have just started my 3rd year of BSc Computer Science course, and very new to the world of Computer Graphics.
(My skillset is C, Java and Python.)
I would be very glad and appreciate any help.
Thank You
[Link to websitehttps://vision.in.tum.de/research/image-based_3d_reconstruction/multiviewreconstruction[][1]]
image2
Yes, it can very likely be done in Blender, and in Python at least for small geometries / low resolution.
A valid approach for the kind of scenarios you seem to want to play with is based on the idea of "space carving" or "silhouette projection". A good description in is an old paper by Kutulakos and Seitz, which was based in part on earlier work by Szelisky.
Given a good estimation of the silhouettes, these methods can correctly reconstruct all convex portions of the object's surface, and the subset of concavities that are resolved in the photo hull. The remaining concavities are "patched" over and need to be reconstructed using a different method (e.g. stereo, or structured light). For the surfaces that can be reconstructed, space carving is generally more robust than stereo (since it is insensitive to the color and surface texture of the object), and can work on surfaces where structured light struggles (e.g. surfaces with specularities, or very dark objects with low reflectance for a laser stripe)
The basic idea is to use the silhouettes of the projection of the object in cameras around it to "remove" mass from an initial volume (e.g. a box) encompassing the object, a bit like a sculptor carving a statue by removing material from a block of marble.
Computationally, you can do it representing the volume of space of interest using an octree, initialized with a minimal level of subdivision, and then progressively refined. The refinement consists of projecting the vertices of the octree leaves in the cameras, and identifying which leaves are completely outside or partially inside the silhouettes. The former are pruned, while the latter are split, and the process continues until no more leaves can be split or a maximul level of subdivision is reached. The hull of the octree is then extracted as a "watertight" mesh using standard methods.
Apart from the above paper, a way more detailed description can be found on an old patent by Geometrix - it sold a scanner based on the above ideas around year 2000. Here is what it looked like:
I'm a wee bit stuck.
I have a 3D point cloud (an array of (n,3) vertices), in which I am trying to generate a 3D triangular mesh from. So far I have had no luck.
The format my data comes in:
(x,y) values in regularly spaced (z) intervals. Think of the data as closed loop planar contours stored slice by slice in the z direction.
The vertices in my data must be absolute positions for the mesh triangles (i.e. I don't want them to be smoothed out such that the volume begins to change shape, but linear interpolation between the layers is fine).
Illustration:
Z=2. : ..x-------x... <- Contour 2
Z=1.5: ...\......|... <- Join the two contours into a mesh.
Z=1. : .....x----x... <- Contour 1
Repeat for n slices, end up with an enclosed 3D triangular mesh.
Things I have tried:
Using Open3D:
The rolling ball (pivot) method can only get 75% of the mesh completed and leaves large areas incomplete (despite a range of ball sizes). It has particular problems at the top and bottom slices where there tends to be large gaps in the middle (i.e. a flat face).
The Poisson reconstruction method smooths out the volume too much and I no longer have an accurate representation of the volume. This occurs at all depths from 3-12.
CGAL:
I cannot get this to work for the life of me. SWIG is not very good, the implementation of CGAL using SWIG is also not very good.
There are two PyBind implementations of CGAL however they have not incorporated the 3D triangulation libraries from CGAL.
Explored other modules like PyMesh, TriMesh, TetGen, Scikit-Geometry, Shapely etc. etc. I may have missed the answer somewhere along the line.
Given that my data is a list of closed-loop planar contours, it seems as though there must be some simple solution to just "joining" adjacent slice contours into one big 3d mesh. Kind of like you would in blender.
There are non-python solutions (like MeshLab) that may well solve these problems, but I require a python solution. Does anyone have any ideas? I've had a bit of a look into VTK and ITK but haven't found exactly what I'm looking for as of yet.
I'm also starting to consider that maybe I can interpolate intermediate contours between slices, and fill the contours on the top and bottom with vertices to make the data a bit more "pivot ball" method friendly.
Thank you in advance for any help, it is appreciated.
If there is a good way of doing this that isn't coded yet, I promise to code it and make it available for people in my situation :)
Actually there are two ways of having meshlab functionality in python:
The first is MeshLabXML (https://github.com/3DLIRIOUS/MeshLabXML ) a third party, is a Python scripting interface to meshlab scripting interface
the second is PyMeshLab (https://github.com/cnr-isti-vclab/PyMeshLab ) an ongoing effort done by the MeshLab authors, (currently in alpha stage) to have a direct Python bindings to all the meshlab filters
There is a very neat paper titled "Technical Note: an algorithm and software for conversion of radiotherapy contour‐sequence data to ready‐to‐print 3D structures" in the Journal of Medical Physics that describes this problem quite nicely. No python packages are required, however it is more easily implemented with numpy. No need for any 3D packages.
A useful excerpt is provided:
...
The number of slices (2D contours) constituting the specified structure is determined.
The number of points in each slice is determined.
Cartesian coordinates of each of the points in each slice are extracted and stored within dedicated data structures...
Numbers of points in each slice (curve) are re‐arranged in such a way, that the starting points (points with indices 0) are the closest points between the subsequent slices. Renumeration starts at point 0, slice 0 (slice with the lowest z coordinate).
Orientation (i.e., the direction determined by the increasing indices of points with relation to the interior/exterior of the curve) of each curve is determined. If differences between slices are found, numbering of points in non‐matching curves (and thus, orientation) is reversed.
The lateral surface of the considered structure is discretized. Points at the neighboring layers are arranged into threes, constituting triangular facets for the STL file. For each triangle the closest points with the subsequent indices from each layer are connected.
Lower and upper base surfaces of the considered structure are discretized. The program iterates over every subsequent three points on the curve and checks if they belong to a convex part of the edge. If yes, they are connected into a facet, and the middle point is removed from further iterations.
So basically it's a problem of aligning datasets in each slice to the nearest value of each slice. Then aligning the orientation of each contour. Then joining the points between two layers based on distance.
The paper also provides code to do this (for a DICOM file), however I re-wrote it myself and it works a charm.
I hope this helps others! Make sure you credit the author's in any work you do that uses this.
A recent feature of pymadcad can do things like this, not sure through if it fits your exact expectation in term of "pivot ball" or such things, checkout the doc for blending
Starting from a list of outlines, it can generate blended surfaces to join them:
For your purpose, I guess the best is one of:
blendpair(line1, line2)
junction(*lines)
I have a question regarding python scripting in Blender, and I'd really appreciate it if someone could give me at least some conceptual guidelines to how I could do this:
Basically I have around 100 planes (simple primitive planes) and each of them has its own material and each material has it's own transparency map applied to it.
I need a way to snap each of those plane's respective pivots to their opaque parts. I.e. if there is a way to tell the following to blender through python language - "hey, go over every one of these planes, and do the following for each - snap a 3D cursor to an opaque part of the plane (it doesn't matter where exactly, as long as it's inside of an opaque part of the plane) and then snap plane's pivot point to the 3D cursor".
Of course I don't expect anyone to write me a full algorithm for this, I am just asking for a little help and a push in the right direction, as I do have experience with python, but not with blender :/
Any help would be appreciated.
You can find documentation on blender's python api here.
Within blender's image class you can access the pixel data at image.pixels as an array of floats, 4 floats per pixel (RGBA I think). image.size[0] is the width in pixels image.size[1] for height.
Given bpy.data.objects['Plane'].bound_box is an [8][3] array of points defining the outer extremes of the plane, you can locate a point on the plane for the pixel location to get the target point for the origin. You will also find bpy.data.objects['Plane'].matrix_world useful to translate the object coordinates to global.
bpy.context.scene.cursor_location = Vector((x,y,z)) will move the cursor to where you want.
bpy.ops.object.origin_set(type='ORIGIN_CURSOR') will set the active objects origin to the cursor. Note that this works on the active object, so you will need to alter your selection as you go.
I have a scene, and I need to be able to overlay the scene with translucent polygons (which can be done easily using pygame.gfxdraw.filled_polygon which supports drawing with alpha), but the catch is that the amount of translucency has to fade over a distance (so for example, if the alpha value is 255 at one end of the polygon, then it is 0 at the other end and it blends from 255 to 0 through the polygon). I've implemented drawing shapes with gradients by drawing the gradient and then drawing a mask on top, but I've never come across a situation like this, so I have no clue what to do. I need a solution that can run in real time. Does anyone have any ideas?
It is possible that you have already thought of this and have decided against it, but it would obviously run far better in real time if the polygons were pre-drawn. Presuming there aren't very many different types of polygons, you could even resize them however you need and you would be saving CPU.
Also, assuming that all of the polygons are regular, you could just have several different equilateral triangles with gradients going in various directions on them to produce the necessary shapes.
Another thing you could do is define the polygon you are drawing, than draw an image of a gradient saved on your computer inside that shape.
The final thing you could do is to build your program (or certain, CPU intensive parts of your program) in C or C++. Being compiled and automatically optimized during compiling, these languages are significantly faster than python and better suited to what you are trying to do.
Currently, if I want to compare pressure under each of the paws of a dog, I only compare the pressure underneath each of the toes. But I want to try and compare the pressures underneath the entire paw.
But to do so I have to rotate them, so the toes overlap (better). Because most of the times the left and right paws are slightly rotated externally, so if you can't simply project one on top of the other. Therefore, I want to rotate the paws, so they are all aligned the same way.
Currently, I calculate the angle of rotation, by looking up the two middle toes and the rear one using the toe detection then I calculate the the angle between the yellow line (axis between toe green and red) and the green line (neutral axis).
Now I want to rotate the array would rotate around the rear toe, such that the yellow and green lines are aligned. But how do I do this?
Note that while this image is just 2D (only the maximal values of each sensor), I want to calculate this on a 3D array (10x10x50 on average). Also a downside of my angle calculation is that its very sensitive to the toe detection, so if somebody has a more mathematically correct proposal for calculating this, I'm all ears.
I have seen one study with pressure measurements on humans, where they used the local geometric inertial axis method, which at least was very reliable. But that still doesn't help me explain how to rotate the array!
If someone feels the need to experiment, here's a file with all the sliced arrays that contain the pressure data of each paw. To clarfiy: walk_sliced_data is a dictionary that contains ['ser_3', 'ser_2', 'sel_1', 'sel_2', 'ser_1', 'sel_3'], which are the names of the measurements. Each measurement contains another dictionary, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] (example from 'sel_1') which represent the impacts that were extracted.
Why would you do it that way? Why not simply integrate the whole region and compare? In this case you'll get a magnitude of the force and you can simply compare scalars which would be much easier.
If you need to somehow compare regions(and hence that's why you need to align them) then maybe attempt a feature extraction and alignment. But this would seem to fail if the pressure maps are not similar(say someone is not putting much wait on one foot).
I suppose you can get really complex but it sounds like simply calculating the force is what you want?
BTW, you can use a simple correlation test to find the optimal angle and translation if the images are similar.
To do this you simply compute the correlation between the two different images for various translations and rotations.
Using the Python Imaging Library, you can rotate an array with for example:
array(Image.fromarray(<data>).rotate(<angle>, resample=Image.BICUBIC))
From there, you can just create a for loop over the different layers of your 3D array.
If you have your first dimension as the layers, then array[<layer>] would return a 2D layer, thus:
for x in range(<amount of layers>):
layer = <array>[i]
<array>[i] = (Image.fromarray(layer).rotate(<angle>, resample=Image.BICUBIC))
Results by #IvoFlipse, with a conversation suggesting:
Putting the array in a bigger array to remedy the darker background.
Look into resampling, perhaps scale the array first.
Moving the rear toe towards the middle allows you to rotate around that instead.
A smaller image can be determined by finding the borders and positioning them in a 15x15 again.