I have a task to receive names of highways according to the list of coordinates of cars. Cars drive all over the world. Every day I get a list of about 800 thousand new coordinates.
I see two solutions:
Using third party API servers for reverse geocoding. Obvious problems:
google is too expensive
nominatim from OSM has a limit of 1 request per second. It is not possible to use your own nominate instance
Get the coordinates of all highways in the world as polygonal geofences to then use mpltPath.Path to count the number of occurrences of coordinates in the geofence. It works fast, I checked. Problems:
I don't have a single standardized source of highway geofences from all over the world.
How would you solve this problem?
What are fast and inexpensive API other than google and osm?
Where can I get geofences for all highways in the world?
Related
I try to consume some GTFS Feeds and work with them.
I created a MySQL Database and a Python Script, which downloads GTFS - Files and import them in the right tables.
Now I am using the LeafLet Map - Framework, to display the Stops on the map.
The next step is to display the routes of a bus or tram line on the map.
In the GTFS - Archive is no shapes.txt.
Is there a way to display the routes without the shapes.txt ?
Thanks!
Kali
You will have to generate your own shape using underlying street data or public transit lines. See detailed post by Anton Dubrau (he is an angel for writing this blog post).
https://medium.com/transit-app/how-we-built-the-worlds-prettiest-auto-generated-transit-maps-12d0c6fa502f
Specifically:
Here’s an example. In the diagram below, we have a trip with three
stops, and no shape information whatsoever. We extract the set of
tracks the trip uses from OSM (grey lines). Our matching algorithm
then finds a trajectory (black line) that follows the OSM, while
minimizing its length and the errors to the stops (e1, e2, e3).
The only alternative to using shapes.txt would be to use the stops along the route to define shapes. The laziest way would be to pick a single trip for that route, get the set of stops from stop_times.txt, and then get the corresponding stop locations from stops.txt.
If you wanted or needed to, you could get a more complete picture by finding the unique ordered sets of stops among all of the trips on that route, and define a shape for each ordered set in the same way.
Of course, these shapes would only be rough estimates because you don't have any information about the path taken by the vehicles between stops.
I need to find latitudes and longitudes of all points that lie along the route between two points (start point and end point, both represented as lat-long pair) in the google map. I have 1 million start points and the corresponding 1 million end points. So, basically I would have 1 million routes and I need to find all points that lie along each of these 1 million routes, separately. I do not want to call google javascript api, because there is a restriction on the no of calls that one can make per day. Also, this will not scale. Can anyone please suggest, how can I do it using python and some offline api/tool for google map?
You say you represent a route as start/end point, you have to consider that for each pair of start and end points you can have more than one route (depending on the mode of travel, etc.).
This is why you need routing. If you don't want to use Google's directions API you can check OpenStreetMap's routing APIs or roll your own router using existing libraries and engines on OSM data.
Similar question: Calculating shortest path on maps (Google Maps, Openstreetmaps, etc)
A user signs up for my site and enters in their zip code. I want to query for other users, and sort by distance.
I have a database full of zip codes with lat/lon points for each zip code.
zip_code (char)
lat (float)
lon (float)
I have a method which will calculate the distance between two sets of lat/lons, but to run this on every other zip code in my db is expensive. I'd need to run this on every zip code combination. I suppose I can do it once and store it somewhere, but where would I store it? Seems strange to have a table for every zip code which would contain the distance to every other zip code. Is there a clean way to do this?
Doing it once and storing it somewhere sounds good to me. Here are some ideas that might give good performance with some consideration to storage space without sacrificing accuracy:
There are something like 43,191 zip codes, so the full would be 1,865,462,481. But the distances are of course symmetrical and the self-to-self ones are useless, which immediately cuts it down to 932,709,645 entries. We might also cut the space by realizing that a bunch of zip codes are either the same as each other, or one contains the other (e.g. 10178 seems to be inside 10016, and they're both geographically small). Many zip codes will have no users at all, so we might avoid populating those until they're needed (i.e. lazy load the cache). And finally, you can probably throw away large-distance results, where large is defined as a distance greater than is useful for your users.
For a more algorithmic view, see this previous question: Calculate distance between zip codes and users
Bonus tip: don't forget about non-US users. Poor non-US users.
Here's a solution with a fair amount of overhead, but which will pay off as your dataset size, user base, and/or number of transactions grow:
If you don't already have one, use a database that supports spatial types and spatial indexing. I recommend the PostGIS extension for PostGres, but most of these steps apply to other spatially-enabled databases:
Store your zip code location as Point geometry type instead of a two columns for lat and long.
Create a spatial index against the Point geometry column. Every time you add a new zip code, its location will automatically be added to the spatial index.
Assuming you don't want to show "nearest" neighbors that are thousands of miles away, use a Within function (ST_DWithin in PostGIS) to filter out those zip codes that are too far away. This will significantly reduce the search space for close neighbors.
Finally use a Distance function (ST_Distance in PostGIS) to calculate the distance between your zip code of interest and its closer neighbors, and use the DB to return results sorted by distance.
By using a database with spatial index and a filtering function that uses that index, you can significantly speed up your search. And when the time comes to do more spatial analysis or show maps, you'll already have a framework in place to support that new functionality.
I'm working on Python, but I suppose that doesn't affect the question itself.
I'm working on a game, and I need to store entities, each of which has an [x,y,z] in the world. I need to be able to run a "All entities within X euclidean distance of point Y".
These entities will be moving fairly often.
What would be the most efficient way to store the entities to make this as fast as possible?
As an alternative to what has been suggested already, if you don't need an exact distance, you could also use spatial hashing, which is quite easy to implement.
In summary, you have to think of your world as a grid where each cell in the grid would correspond to one bucket in the hash table. Since your entities are moving often, on each new frame you could clear and reconstruct the whole table and put the entities to their corresponding bucket depending on their position. Then, for any given entity you could just check the cells nearby and get the entities lists.
You can use a kd-tree (link has photo and code and examples) or an octree (this link is a C++ class template which you can use). Real-usage can be seen in this open-source game engine
I have a large collection (and growing) of geospatial data (lat, lon) points (stored in mongodb, if that helps).
I'd like to generate a choropleth map (http://vis.stanford.edu/protovis/ex/choropleth.html), which requires knowing the state which contains that point. Is there a database or algorithm that can do this without requiring call to external APIs (i.e. I'm aware of things like geopy and the google maps API).
Actually, the web app you linked to contains the data you need -
If you look at http://vis.stanford.edu/protovis/ex/us_lowres.js for each state, borders[] contains a [lat,long] polyline which outlines the state. Load this data and check for point-in-polygon - http://en.wikipedia.org/wiki/Point_in_polygon
Per Reverse Geocoding Without Web Access you can speed it up a lot by pre-calculating a bounding box on each state and only testing point-in-polygon if point-in-bounding-box.
Here's how to do it in FORTRAN. Remember FORTRAN? Me neither. Anyway, it looks pretty simple, as every state has its own range.
EDIT It's been point out to me that your starting point is LAT-LONG, not the zipcode.
The algorithm for converting a lat-long to a political division is called "a map". Seriously, that's allan ordinary map is, a mapping of every point in some range to the division it belongs to. A detailed digital map of all 48 contiguous states would be a big database, and then you would need some (fairly simple) code to determine for each state (described as a series of line segments outlining the border) whether a given point was inside it or out.
you can try using Geonames database. It has long/lat as well as city, postal and other location type data. It is free as well.
but If you need to host it locally or import it into your own database , the USGS and NGA provide a comprehensive list of cities with lat/lon. It's updated reguarly, free, and reliable.
http://geonames.usgs.gov
http://earth-info.nga.mil/gns/html/index.html
Not sure the quality of the data, but give this a shot: http://www.boutell.com/zipcodes/
If you don't mind a very crude solution, you could adapt the click-map here.