I am attempting to draw a very large networkx graph that has approximately 5000 nodes and 100000 edges. It represents the road network of a large city. I cannot determine if the computer is hanging or if it simply just takes forever. The line of code that it seems to be hanging on is the following:
##a is my network
pos = networkx.spring_layout(a)
Is there perhaps a better method for plotting such a large network?
Here is the good news. Yes it wasn't broken, It was working and you wouldn't want to wait for it even if you could.
Check out my answer to this question to see what your end result would look like.
Drawing massive networkx graph: Array too big
I think the spring layout is an n^3 algorithm which would take 125,000,000,000 calculations to get the positions for your graph. The best thing for you is to choose a different layout type or plot the positions yourself.
So another alternative is pulling out the relevant points yourself using a tool called gephi.
As Aric said, if you know the locations, that's probably the best option.
If instead you just know distances, but don't have locations to plug in, there's some calculation you can do that will reproduce locations pretty well (up to a rotation). If you do a principal component analysis of the distances and project into 2 dimensions, it will probably do a very good job estimating the geographic locations. (It was an example I saw in a linear algebra class once)
Related
I would like to represent a bunch of particles (~100k grains), for which I have the position (including rotation) and a discretized level set function (i.e. the signed distance of every voxel from surface). Due to the large sample, I'm searching for eficient solutions to visualize it.
I first went for vtk, using its python interface, but I'm not really sure if it's the best (and simplest) way to do it since, as far as I know, there is no direct implementation for getting an isosurface from a 3D data set. In the beginning I was thinking usind marching cubes, but then I still would have to use a threshold or to interpolate in order to get the voxels that are on the surface and label them in order to be used by the marching cubes.
Now I found mayavi which has a python function
mlab.pipeline.iso_surface()
I however did not find much documentation on it and was wondering how it behaves in terms of performance.
Does someone have experience with this kind of tools? Which would be the best solution (in terms of efficiency and, secondly, in terms of simplicity - I do not know the vtk library, but if there is a huge difference in performance I can dig into it, also without python interface).
Lowest/Highest Combined Surface(s)
I'm looking for a methodology (and/or preferably a software approach) to create what I'm calling the Lowest (or highest) combined surface for a set of polygons.
So if our input was these two polygons that partially overlap and definitely intersect
My Lowest Combined output would be these three polygons
Given a number of "surfaces" (3d polygons)
We've gone through a variety of approaches and the best solution we could come up with involved applying a point grid to each polygon and performing calculations to return the lowest sets of points at each grid location. The problem is that the original geometry is lost in this approach which doesn't give us a working solution.
Background
I'm looking at a variety of "surfaces" that can be represented by 3d faces (cad Speak) or polygons and usually are distributed in a shapefile (.shp). When there are two surfaces that interact I'm interested in taking either the lowest combined or highest combined surface. I'm able to do this in CAD by manually tracing out new polygons for the interaction zones - but once I get into more than a handful of surfaces this becomes too labor intensive.
The current Approach
My current approach which falls somewhere in the terrible category is to generate a point cloud from each surface on a 1m grid and then do a grid cell based comparison of the points.
I do this by using AutoCAD Civl 3D's surface Generation Tools to create a TIN from each polygon surface and then using its Surface. This is then exported to a 1m DEM file which I believe is a gridded output format.
Next each DEM file is brought into Global Mapper where I generate a single point at the center of each "elevation grid cell". Next this data is exported to a .csv file where each point contains a variety of attributes such as what the name of the surface this point came from and what its altitude is
Next once I have a set of CSV files I run them through a python script that will export the lowest point (and associated attributes) at each grid. I do everything in UTM because the UTM grid is based on meters and it makes everything easier.
Lastly we bring the point file back into global mapper - coloring each point by what surface it started from.
There a variety of issues with this approach - sometimes things don't line up perfectly and there is a variety of cleanup I have to do
Also the edges end up being jagged - as is the case because I've converted nice straight lines into a point cloud
Alternatively we came up with a similar approach in Arc GIS using the Surface Comparison tool, however it had similar limitations to what we ran into with my approach.
What I'm looking for is a way to do this automatically with a variable number of inputs. I'm willing to use just about any tool to have this done, as it seems like it shouldn't be too difficult a process
Software?
When I look at this problem from a programmers point of view it looks rather straight forward - but I'm at a total loss how to proceed. I'm assuming Stack Overflow is the correct stack exchange for this question - but if it should be somewhere else - I'm happy to move it to a different exchange.
I wasn't sure if something like Mathematica (which i have zero experience) with could handle this situation or whether there was some fancy 3d math library in python that could chop polygons up by how they interact and then give me the lowest for co-located polys.
In any case I'm willing to try anything out so please if you have an idea of what tools and/or libraries I can use to do this please share! I have to assume that there is SOMETHING out there that can handle this type of 3d geometric processing
Thanks
EDIT
Because the commenters seem confused I am not asking for code - I am asking for methodologies, libraries, support tools, or even software packages that can perform these operations. I plan to write software to do this, however, I am hoping I don't need to pull out my trig books and write all these operations by hand. I have to assume there is somebody out there that has dealt with something similar before.
I've an application where I have a graph and I need to count the number of triangles in the graph using MrJob (MapReduce in Python). However, I'm having some trouble wrapping my head around the mapping and the reducing steps needed.
What is the best Map Reduce pipeline for computing the triangles of a network graph?
Well, it would help to answer this to have a bit more context. Do you have a single graph or a large number of graphs, a tree? How many nodes are we talking about in your graph?
But in general, I would try to build a solution that uses the networkx package, specifically the triangles method at the core.
An issue you may face is filtering duplicates, as the triangles are reported relative to a node.
So a bit more context here on the specifics here would help narrow down the answer.
I'm working on a kind of geographic problem, specifically finding the fastest path for an Electric Vehicle.
So to facilitate the experimentations of the algorithms I've created I need some kind of road network dataset. I have been looking at some real world data sets such as OpenStreetMap, but this seems like an awfully complicated thing to integrate.
All I really need is a road network dataset that contains distances and speed limits, being able to work with it in python is preferable.
Researchers frequently use graphs from the 9th DIMACS Implementation Challenge for experiments with their shortest-path algorithms. Coordinates, distances, and estimated travel times all are provided. The format is simple and textual; I estimate that a dozen lines of Python would suffice to read them.
I am working on some code that needs to recognize some fairly basic geometry based on a cloud of nodes. I would be interested in detecting:
plates (simple bounded planes)
cylinders (two node loops)
half cylinders (arc+line+arc+line)
domes (n*loop+top node)
I tried searching for "geometry from node cloud", "get geometry from nodes", but I cant find a nice reference. There is probably a whole field on this, can someone point me the way? i already started coding something, but I feel like re-inventing the wheel...
A good start is to just get the convex hull (the tightest fitting polygon that can surround your node cloud) of the nodes, use either Grahams algorithm or QuickHull. Note that QuickHull is easier to code and probably faster, unless you are really unlucky. There is a pure python implementation of QuickHull here. But I'm sure a quick Google search will show many other results.
Usually the convex hull is the starting point for most other shape recognition algorithms, if your cloud can be described as a sequence of strokes, there are many algorithms and approaches:
Recognizing multistroke geometric shapes: an experimental evaluation
This may be even better, once you have the convex hull, break down the polygon to pairs of vertices and run this algorithm to match based on similarity to training data:
Hierarchical shape recognition using polygon approximation and dynamic alignment
Both of these papers are fairly old, so you can use google scholar to see who cites these papers and there you have a nice literature trail of attempts to solve this problem.
There are a multitude of different methods and approaches, this has been well studied in the literature, what method you take really depends on the level of accuracy you hope to achieve, and the amount of shapes you want to recognize, as well as your input data set.
Either way, using a convex hull algorithm to produce polygons out of point clouds is the very first step and usually input to the more sophisticated algorithmms.
EDIT:
I did not consider the 3D case, for that their is a lot of really interesting work in computer graphics that has focused on this, for instance this paper Efficient RANSAC for Point-Cloud Shape Detection
Selections from from Abstract:
We present an automatic algorithm to detect basic shapes in unorganized point clouds. The algorithm decomposes the point cloud into a concise, hybrid structure of inherent shapes and a set of remaining points. Each detected shape serves as a proxy for a set of corresponding points. Our method is based on random sampling and detects planes, spheres, cylinders, cones and tori...We demonstrate that the algorithm is robust even in the presence of many outliers and a high degree of noise...Moreover the algorithm is conceptually simple and easy to implement...
To complement Josiah's answer -- since you didn't say whether there is a single such object to be detected in your point cloud -- a good solution can be to use a (generalized) Hough transform.
The idea is that each point will vote for a set of candidates in the parameter space of the shape you are considering. For instance, if you think the current object is a cylinder, you have a 7D parameter space consisting of the cylinder center (3D), direction (2D), height (1D) and radius (1D), and each point in your point cloud will vote for all parameters that agree with the observation of that point. Doing so allows to find the parameters of the actual cylinder by taking the set of parameters who have the highest number of votes.
Doing the same thing for planes, spheres etc.., will give you the best matching shape.
A strength of this method is that it allows for multiple objects in the same point cloud.