I was wondering if I could build an augmented reality system in Python using OpenCV and SLAM. If so, do you have any tutorials or documentation that you could recommend? I've been scratching my head for awhile now trying to find resources to start with, any help would be greatly appreciated!
If I were to be a bit more specific, it would be on how would I be able to integrate SLAM and AR together. SLAM acting as a form of mapping so that the AR would know where to place objects in.
if I want to be honest, python is not strong enough to bring you "RealTime Monocular SLAM System". So, firstly you should consider writing your SLAM system in C++ that is highly recommended for RealTime Systems!
Secondly, You Can see some openSource SLAM systems here (Stella-VSLAM, ORBSLAM3, PTAM). But Consider that for developing a SLAM system, you should gain knowledge in wide range of computer science related topics! and the main reason of why ARCore and ARKit are working great is their efficient SLAM system. you can also read this resource for more info on SLAM systems. If you have more questions, please don't hesitate to ask!
Where to Start
In order to gain some knowledge on SLAM and Computer Vision I would recommend watching Cyrill Stachniss' SLAM course and reading the papers ORB-SLAM, ORB-SLAM2, ORB-SLAM3, and DSO. For Computer Vision I recommend reading R. Szeliski book.
Which Language to Use
I wrote my thesis on SLAM and AR systems, and the outcome is the following: State-of-the-art SLAM systems which achieve the best accuracy are still using machine learning techniques: SURF, ORB descriptors, Bag of Words (BoW) etc. All of the systems (ORB-SLAM3, DM-VIO, DSO) are written in C++.
I'm always using C++ for programming SLAM, and only sometimes I use Python to write scripts for example to fix the recovered trajectory.
SLAM + AR
There's no much resources on this subject, although the idea is simple. SLAM system has to give you the camera location, usually as the 4x4 transformation matrix, where the first 3x3 matrix is the rotation matrix, and the last 3x1 column is the translation part. Example of the transformation matrix.
Having the camera location, you can use the projective geometry to project the AR objects on the camera frame. ORB-SLAM2 has a nice AR demo to study; basically they display a 2D image, and put the 3D rendered image on top of that.
They use Pangolin, so you need to know how to use OpenGL, Pangolin. I recommend studying Pangolin by its' examples, as it mostly documented through them.
Related
I'm producing an ugv prototype. The goal is to perform the desired actions to the targets set within the maze. When I surf the Internet, the mere right to navigate in the labyrinth is usually made with a distance sensor. I want to consult more ideas than the question.
I want to navigate the labyrinth by analyzing the image from the 3d stereo camera. Is there a resource or successful method you can suggest for this? As a secondary problem, the car must start in front of the entrance of the labyrinth, see the entrance and go in, and then leave the labyrinth after it completes operations in the labyrinth.
I would be glad if you suggest a source for this problem. :)
The problem description is a bit vague, but i'll try to highlight some general ideas.
An useful assumption is that labyrinth is a 2D environment which you want to explore. You need to know, at every moment, which part of the map has been explored, which part of the map still needs exploring, and which part of the map is accessible in any way (in other words, where are the walls).
An easy initial data structure to help with this is a simple matrix, where each cell represents a square in the real world. Each cell can be then labelled according to its state, starting in an unexplored state. Then you start moving, and exploring. Based on the distances reported by the camera, you can estimate the state of each cell. The exploration can be guided by something such as A* or Q-learning.
Now, a rather subtle issue is that you will have to deal with uncertainty and noise. Sometimes you can ignore it, sometimes you don't. The finer the resolution you need, the bigger is the issue. A probabilistic framework is most likely the best solution.
There is an entire field of research of the so-called SLAM algorithms. SLAM stands for simultaneous localization and mapping. They build a map using some sort of input from various types of cameras or sensors, and they build a map. While building the map, they also solve the localization problem within the map. The algorithms are usually designed for 3d environments, and are more demanding than the simpler solution indicated above, but you can find ready to use implementations. For exploration, something like Q-learning still have to be used.
I'd like to improve my little robot with machine learning.
Up to now it uses simple while and if then decisions in its main function to act as a lawn mowing robot.
My idea is to use SKLearn for that purpose.
Please help me to find the right first steps.
i have a few sensors that tell about the world otside:
World ={yaw, pan, tilt, distance_to_front_obstacle, ground_color}
I have a state vector
State = {left_motor, right_motor, cutter_motor}
that controls the 3 actors of the robot.
I'd like to build a dataset of input and output values to teach sklearn the wished behaviour, after that the input values should give the correct output values for the actors.
One example: if the motors are on and the robot should move forward but the distance meter tells constant values, the robot seems to be blocked. Now it should decide to draw back and turn and move to another direction.
First of all, do you think that this is possible with sklearn and second how should i start?
My (simple) robot control code is here: http://github.com/bgewehr/RPiMower
Please help me with the first steps!
I would suggest to use Reinforcement Learning. Here you have a tutorial of Q-Learning that fits well into your problem.
If you want code in python, right now I think there is no implementation of Q-learning in scikit-learn. However, I can give you some examples of code in python that you could use: 1, 2 and 3.
Also please have in mind that reinforcement learning is set to maximize the sum of all future rewards. You have to focus on the general view.
Good luck :-)
The sklearn package contains a lot of useful tools for machine learning so I dont think thats a problem. If it is, then there are definitely other useful python packages. I think collecting data for the supervised learning phase will be the challenging part, and wonder if it would be smart to make a track with tape within a grid system. That would make it be easier to translate the track to labels (x,y positions in the grid). Each cell in the grid should be small if you want to make complex tracks later on I think. It may be very smart to check how they did in the self-driving google car.
I hope you don't mind if I ask for a bit of advice regarding modelling robotic systems. I've recently become rather interested in using inverse kinematics (IK) to control a 5 dof robotic manipulator. I have a solid foundation in IK but what I'm having trouble with is a way to visualize how the manipulator moves with respect to joint angles.
I've looked into using 3D toolkits (such as Blender, Panda3D, vPython) to create a 3d model of the arm, but I'm not sure if I should be looking something with physics support. I'm also not sure how well I can model motion with these packages. Anyone have any suggestions? What I'm NOT looking for is a full blown robotic simulator like Microsoft's Robotic Studio, I'd like to start with the basics and learn how everything works first, ie code the IK in Python, then visualize the motion in 3D. I'm very familiar with Python, so something that interfaces with Python would be preferable.
Thanks!
Well the great thing about Blender is that its API is actually in python!
In addition, it supports inverse kinematics (IK) quite well in addition to many other modeling tools.
Blender Cookie is a great resource.
Here is a tutorial on making IK rigs in Blender.
Blenders python api is documented quite extensively, and it even has an interactive python shell built right in so that you can see the effects of your script as you go along.
The physics engine that blender uses is the popular bullet physics engine, which has been used in many commercial games as well as a few feature films (2012 among them).
This isn't really a hard problem, is it? Presumably you're working out the math on your own; so if your robotic arm is visualized as, say, a few rectangular solids then all you need is something that will render these at the x,y,z coordinates and with the orientation vector you supply, updating when need be. OpenGL should do just fine for this, you could probably do it in <50 lines.
Python-Ogre is a wrapper around the very mature C++ engine OGRE 3D. I have used it for a pretty significant project and I would give the Python wrapper a mixed but mostly positive review. It's very capable and does a good job of wrapping all C++ functionality in a mostly Pythonic interface. You can get the physics support through one of the many OGRE extension modules which provide those capabilities.
The biggest negative is that the documentation for the Python interface is poor, and you will have to rely on the C++ library's documentation to make sense of some things. To give you a sense of what's possible, I was able to implement a scripting language around OGRE using Python and PyV8 (the language was Javascript), as I prototyped a C++ project.
The resulting prototype was very mature and robust, and I was able to wrap the internals of OGRE with high-level Javascript. With the scripting language I was able to script animation, materials, rendering and physics.
I took a scientific programming course this semester that I really enjoyed and experimented with a lot. We used python, and all the related modules. I am taking a physics lab next semester and I just wanted to hear from some of you how python can help me in ways that excel can't or in ways that are better than excel's capabilities. I use Mathematica for symbolic stuff so I would use python for data purposes.
Off the top of my head, here are the related things I can do:
All of the things you would expect in a intro course (loops, arrays, slicing arrays, etc).
Reading data from a text file.
Plotting scatter, line, and bar graphs.
Learning how to plot linear regression but haven't totally figured it out.
I have done 7 of the problems on Project Euler (nothing to brag about, but it might give you a better idea of where I stand in skills).
Looking forward to hearing from some of you. You don't have to explain how to use the things you mention, I could look up the documentation.
The paper Python all a scientist needs comes to mind. I hope you can make the needed transformations from Biology to Physics.
Scipy will also be useful to you, as it includes many more advanced analysis tools. For example, Scipy includes a linear regression, and gets more interesting from there. Along with the other tools you mentioned, you'll probably find most of your needs covered.
Other notes on tool selection:
Mathematica is a great tool, if you can afford it. I've played around with the other options, like Sympy, and sadly, they don't come close to being as useful as Mathematica.
I can't imagine using Excel for any serious scientific work. If you're planning to continue forward using the tools that you learn in class, you might as well start with tools that offer you that potential.
Don't reject Excel outright. It's still great for doing simple data analysis and plotting. Excel also has the considerable advantage of being installed on most engineer and scientist's computers, making it a lot easier to share your work with colleagues.
That said, I do use Python when Excel just won't cut it; times when I've had to:
color the points in a scatter plot based on a third column
plot a field of vectors
extract a few values from each of several thousand data files to do statistical process control
generate dozens of scatter plots over different dimensions of a large data set to find which variables are important
solve a nonlinear equation at several intermediate points of a calculation, not just as the final result.
accept variable length input from a user to define a problem
VBA in Excel can do a lot of those things too, but it becomes painful fast in such a primitive language. I dream that Microsoft will make IronPython a first-class scripting language in the next version of Excel. Until then, you might want to try Resolver One
I can recall 2 presentations by Jan Martinek on EuroScipy 2008, he's PhD candidate and presented some fun experiments with Physics in the background. Abstracts are here and I'm sure he would't mind to share more if you contact him directly. Also, take a look at other presentation from EuroScipy, there are some more Physics-related ones.
I would like to know similar, concrete simulations, as the simulation about watering a field here.
What is your favorite library/internet page for such simulations in Python?
I know little Simpy, Numpy and Pygame. I would like to get examples about them.
If you are looking for some game physics (collisions, deformations, gravity, etc.) which looks real and is reasonably fast consider re-using some physics engine libraries.
As a first reference, you may want to look into pymunk, a Python wrapper of Chipmunk 2D physics library. You can find a list of various Open Source physics engines (2D and 3D) in Wikipedia.
If you are looking for physically correct simulations, no matter what language you want to use, it will be much slower (almost never real-time), and you need to use some numerical analysis software (and probably to write something yourself). Exact answer depends on the problem you want to solve. It is a fairly complicated field (of math).
For example, if you need to do simulations in continuum mechanics or electromagnetism, you probably need Finite Difference, Finite Volume or Finite Element methods. For Python, there are some ready-to-use libraries, for example: FiPy (FVM), GetFem++ (FEM), FEniCS/DOLFIN (FEM), and some other.
Here is some simple astronomy related python. And here is a hardcore code from the same guy.
And Eagleclaw solves and plots various hyperbolic equations using some python. However, most of the code is written in Fortran to do the computations and python to plot the results. If you are studying physics though you may have to get used to this kind of Fortran wrapped code. It is a reality. But this isn't really what your looking for I guess. The good thing it that it is documented in a literate programming style so it should be understandable.
Maybe PyODE?
I've heard of PyBox2D, which is a port of the really nice Box2D. To quote the site:
Box2D is a feature rich 2d rigid body physics engine, written in C++ by Erin Catto. It has been used in many games, including Crayon Physics Deluxe, winner of the 2008 Independent Game Festival Grand Prize.