I have unstructured (taken in no regular order) point cloud data (x,y,z) for a surface. This surface has bulges (+z) and depressions (-z) scattered around in an irregular fashion. I would like to generate some surface that is a function of the original data points and then be able to input a specific (x,y) and get the surface roughness value from it (z value). How would I go about doing this?
I've looked at scipy's interpolation functions, but I don't know if creating a single function for the entire surface is the correct approach? Is there a technical name for what I am trying to do? I would appreciate any suggestions/direction.
I don't know if creating a single function for the entire surface is the correct approach?
I guess this depends on your data. Let's assume the base form of your surface is spherical. Then you can model it as such.
If your surface is more complex then a sphere you might can still model the neighborhood of (x,y) as such. Maybe you could even consider your surface as plain in the near neighborhood of (x,y).
What you are trying to do, can be called surface fitting, or two-dimensional curve fitting. You would be able to find lots of available algorithms by searching for those terms. Now, the choice of the particular algorithm/method should be dictated:
by the origin of your data (there are specialized algorithms or variations of more common ones that are tailored for certain application areas)
by the future use of your data (depending on what you are going to do with it, maybe you need to be able to calculate derivatives easily, etc)
It is not easy to represent complicated data (especially the noisy one) using a single function. Thus there is a lot of research about it. However, in a lot of applications curve-fitting is very successful and very widely used.
Related
If I take a picture with a camera, so I know the distance from the camera to the object, such as a scale model of a house, I would like to turn this into a 3D model that I can maneuver around so I can comment on different parts of the house.
If I sit down and think about taking more than one picture, labeling direction, and distance, I should be able to figure out how to do this, but, I thought I would ask if someone has some paper that may help explain more.
What language you explain in doesn't matter, as I am looking for the best approach.
Right now I am considering showing the house, then the user can put in some assistance for height, such as distance from the camera to the top of that part of the model, and given enough of this it would be possible to start calculating heights for the rest, especially if there is a top-down image, then pictures from angles on the four sides, to calculate relative heights.
Then I expect that parts will also need to differ in color to help separate out the various parts of the model.
As mentioned, the problem is very hard and is often also referred to as multi-view object reconstruction. It is usually approached by solving the stereo-view reconstruction problem for each pair of consecutive images.
Performing stereo reconstruction requires that pairs of images are taken that have a good amount of visible overlap of physical points. You need to find corresponding points such that you can then use triangulation to find the 3D co-ordinates of the points.
Epipolar geometry
Stereo reconstruction is usually done by first calibrating your camera setup so you can rectify your images using the theory of epipolar geometry. This simplifies finding corresponding points as well as the final triangulation calculations.
If you have:
the intrinsic camera parameters (requiring camera calibration),
the camera's position and rotation (it's extrinsic parameters), and
8 or more physical points with matching known positions in two photos (when using the eight-point algorithm)
you can calculate the fundamental and essential matrices using only matrix theory and use these to rectify your images. This requires some theory about co-ordinate projections with homogeneous co-ordinates and also knowledge of the pinhole camera model and camera matrix.
If you want a method that doesn't need the camera parameters and works for unknown camera set-ups you should probably look into methods for uncalibrated stereo reconstruction.
Correspondence problem
Finding corresponding points is the tricky part that requires you to look for points of the same brightness or colour, or to use texture patterns or some other features to identify the same points in pairs of images. Techniques for this either work locally by looking for a best match in a small region around each point, or globally by considering the image as a whole.
If you already have the fundamental matrix, it will allow you to rectify the images such that corresponding points in two images will be constrained to a line (in theory). This helps you to use faster local techniques.
There is currently still no ideal technique to solve the correspondence problem, but possible approaches could fall in these categories:
Manual selection: have a person hand-select matching points.
Custom markers: place markers or use specific patterns/colours that you can easily identify.
Sum of squared differences: take a region around a point and find the closest whole matching region in the other image.
Graph cuts: a global optimisation technique based on optimisation using graph theory.
For specific implementations you can use Google Scholar to search through the current literature. Here is one highly cited paper comparing various techniques:
A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms.
Multi-view reconstruction
Once you have the corresponding points, you can then use epipolar geometry theory for the triangulation calculations to find the 3D co-ordinates of the points.
This whole stereo reconstruction would then be repeated for each pair of consecutive images (implying that you need an order to the images or at least knowledge of which images have many overlapping points). For each pair you would calculate a different fundamental matrix.
Of course, due to noise or inaccuracies at each of these steps you might want to consider how to solve the problem in a more global manner. For instance, if you have a series of images that are taken around an object and form a loop, this provides extra constraints that can be used to improve the accuracy of earlier steps using something like bundle adjustment.
As you can see, both stereo and multi-view reconstruction are far from solved problems and are still actively researched. The less you want to do in an automated manner the more well-defined the problem becomes, but even in these cases quite a bit of theory is required to get started.
Alternatives
If it's within the constraints of what you want to do, I would recommend considering dedicated hardware sensors (such as the XBox's Kinect) instead of only using normal cameras. These sensors use structured light, time-of-flight or some other range imaging technique to generate a depth image which they can also combine with colour data from their own cameras. They practically solve the single-view reconstruction problem for you and often include libraries and tools for stitching/combining multiple views.
Epipolar geometry references
My knowledge is actually quite thin on most of the theory, so the best I can do is to further provide you with some references that are hopefully useful (in order of relevance):
I found a PDF chapter on Multiple View Geometry that contains most of the critical theory. In fact the textbook Multiple View Geometry in Computer Vision should also be quite useful (sample chapters available here).
Here's a page describing a project on uncalibrated stereo reconstruction that seems to include some source code that could be useful. They find matching points in an automated manner using one of many feature detection techniques. If you want this part of the process to be automated as well, then SIFT feature detection is commonly considered to be an excellent non-real-time technique (since it's quite slow).
A paper about Scene Reconstruction from Multiple Uncalibrated Views.
A slideshow on Methods for 3D Reconstruction from Multiple Images (it has some more references below it's slides towards the end).
A paper comparing different multi-view stereo reconstruction algorithms can be found here. It limits itself to algorithms that "reconstruct dense object models from calibrated views".
Here's a paper that goes into lots of detail for the case that you have stereo cameras that take multiple images: Towards robust metric reconstruction
via a dynamic uncalibrated stereo head. They then find methods to self-calibrate the cameras.
I'm not sure how helpful all of this is, but hopefully it includes enough useful terminology and references to find further resources.
Research has made significant progress and these days it is possible to obtain pretty good-looking 3D shapes from 2D images. For instance, in our recent research work titled "Synthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes With Deep Generative Networks" took a big step in solving the problem of obtaining 3D shapes from 2D images. In our work, we show that you can not only go from 2D to 3D directly and get a good, approximate 3D reconstruction but you can also learn a distribution of 3D shapes in an efficient manner and generate/synthesize 3D shapes. Below is an image of our work showing that we are able to do 3D reconstruction even from a single silhouette or depth map (on the left). The ground-truth 3D shapes are shown on the right.
The approach we took has some contributions related to cognitive science or the way the brain works: the model we built shares parameters for all shape categories instead of being specific to only one category. Also, it obtains consistent representations and takes the uncertainty of the input view into account when producing a 3D shape as output. Therefore, it is able to naturally give meaningful results even for very ambiguous inputs. If you look at the citation to our paper you can see even more progress just in terms of going from 2D images to 3D shapes.
This problem is known as Photogrammetry.
Google will supply you with endless references, just be aware that if you want to roll your own, it's a very hard problem.
Check out The Deadalus Project, althought that website does not contain a gallery with illustrative information about the solution, it post several papers and info about the working method.
I watched a lecture from one of the main researchers of the project (Roger Hubbold), and the image results are quite amazing! Althought is a complex and long problem. It has a lot of tricky details to take into account to get an approximation of the 3d data, take for example the 3d information from wall surfaces, for which the heuristic to work is as follows: Take a photo with normal illumination of the scene, and then retake the picture in same position with full flash active, then substract both images and divide the result by a pre-taken flash calibration image, apply a box filter to this new result and then post-process to estimate depth values, the whole process is explained in detail in this paper (which is also posted/referenced in the project website)
Google Sketchup (free) has a photo matching tool that allows you to take a photograph and match its perspective for easy modeling.
EDIT: It appears that you're interested in developing your own solution. I thought you were trying to obtain a 3D model of an image in a single instance. If this answer isn't helpful, I apologize.
Hope this helps if you are trying to construct 3d volume from 2d stack of images !! You can use open source tool such as ImageJ Fiji which comes with 3d viewer plugin..
https://quppler.com/creating-a-classifier-using-image-j-fiji-for-3d-volume-data-preparation-from-stack-of-images/
I have two sets of data points; effectively, one is from a preimage and the other from its image, but I do not know the rule between the two. This rule/function is nonlinear.
I've collected many data points of corresponding locations on both images, and I was wondering if anyone knew of a way to find a more complete mapping. That is, does anyone know the best way to find a mapping from R^2 to R^2 with an extensive set of sample points. This mapping is one-to-one and onto.
My goal is to use the data I've found to find a polynomial function that takes in some x,y coordinate from the preimage, and outputs the shifted coordinates.
edit: I have sample points along the domain and their corresponding points in the image, but not for every point in the domain. I want to be able to input any point (only integer values) in the domain and output the shifted point.
I don't think polynomial is easy (or easy to guarantee is a bijection). The obvious thing to do is to
Construct the delaunay triangulation of the known points in the domain.
For each delaunay triangle the mapping is just the linear mapping which interpolates the map on the vertices.
Then, when you have a random point, look up its delaunay triangle, and apply the requisite map.
I believe that all of the above can be done via scipy.spatial.delaunay.
The transformation you're trying to find sounds a lot like what's accomplished in Geographic Information Systems using a technique called rubber-sheeting https://en.wikipedia.org/wiki/Rubbersheeting
Igor Rivin's description of a process using a Delaunay triangulation is pretty much the solution that's used in such systems. Some systems will use a Barycentric coordinate system rather than a linear mapping to try to reduce the appearance of triangle-related artifacts in the transformed image.
What you are describing also sounds a bit like the "morphing" special effect used in video. Maybe a web search on that topic would turn up some leads for you.
I'd like to create synthetic training data for DL models for segmentation and classification in point clouds. The ground truth / real data comprise LiDAR point clouds. I scripted a simple mesh sampling model in python/open3d and I'm able to quickly transfer 3D scenes to point clouds (see fig 1), but I need to include certain characteristics of LiDAR sensors.
Blensor (https://www.blensor.org/) works the way I need it (fig 2), but I don't want to use blender atm. Also the results don't have a sufficient quality for my use case.
In the first step I'd just like to cut off the points, which are not reachable by a certain position of a LiDAR sensor, mainly to create the "shadows", which are important to make the training data more realistic. Do you have any suggestions for a simple and fast workaround? My point cloud is saved in a pandas dataframe including x,y,z and nx,ny,nz values.
Thx in advance,
reiti
If your 3D scene can be described in the form of distance functions (essentially consisting of a range of simple geometric shapes opposed to point cloud data) you may be good to go with an easily modified ray tracing algorithm that emulates a LiDAR sensor.
For each LiDAR "ray" (i.e. for every direction) you only need to save the first scene collision's xyz coordinates. This also gives you full freedom to match the original real world sensor properties (like angles and number of points).
How easy the calculation of the distance between scene and sensor-ray will be depends on the scene you have set up and how it is represented. Sorry for not being able to provide you with a ready-to-use implementation, but this might give you some direction.
I am using OpenCV and Python.
Let say I have this sequence video of the car. And I have tracked some 'interesting points' of the car with the cv2.goodFeaturesToTrack and cv2.calcOpticalFlowPyrLK. Now, given the traced points, I want to estimate a very rough shape (maybe a 3D box) of the car and its distance from the camera. It doesn't need to be that accurate.
On top of that, I want it to be keep updating in real time. The closest youtube video I can find that can give a view of what I am trying to achieve is this. I have found a new Structure from Motion module in OpenCV, but it is more on building a 3D model from a collection of points.
The question is, what is the best way of achieving this and what kind of library I can use (especially in order to construct the 3D space)?
And it is also OK if somehow I need to use C++ for this (although I am still not good in it yet).
Thanks.
I know that this problem has been solved before, but I've been great difficulty finding any literature describing the algorithms used to process this sort of data. I'm essentially doing some edge finding on a set of 2D data. I want to be able to find a couple points on an eye diagram (generally used to qualify high speed communications systems), and as I have had no experience with image processing I am struggling to write efficient methods.
As you can probably see, these diagrams are so called because they resemble the human eye. They can vary a great deal in the thickness, slope, and noise, depending on the signal and the system under test. The measurements that are normally taken are jitter (the horizontal thickness of the crossing region) and eye height (measured at either some specified percentage of the width or the maximum possible point). I know this can best be done with image processing instead of a more linear approach, as my attempts so far take several seconds just to find the left side of the first crossing. Any ideas of how I should go about this in Python? I'm already using NumPy to do some of the processing.
Here's some example data, it is formatted as a 1D array with associated x-axis data. For this particular example, it should be split up every 666 points (2 * int((1.0 / 2.5e9) / 1.2e-12)), since the rate of the signal was 2.5 GB/s, and the time between points was 1.2 ps.
Thanks!
Have you tried OpenCV (Open Computer Vision)? It's widely used and has a Python binding.
Not to be a PITA, but are you sure you wouldn't be better off with a numerical approach? All the tools I've seen for eye-diagram analysis go the numerical route; I haven't seen a single one that analyzes the image itself.
You say your algorithm is painfully slow on that dataset -- my next question would be why. Are you looking at an oversampled dataset? (I'm guessing you are.) And if so, have you tried decimating the signal first? That would at the very least give you fewer samples for your algorithm to wade through.
just going down your route for a moment, if you read those images into memory, as they are, wouldn't it be pretty easy to do two flood fills (starting centre and middle of left edge) that include all "white" data. if the fill routine recorded maximum and minimum height at each column, and maximum horizontal extent, then you have all you need.
in other words, i think you're over-thinking this. edge detection is used in complex "natural" scenes when the edges are unclear. here you edges are so completely obvious that you don't need to enhance them.