Develop Reference

extract .dxf data using ezdxf to add to a pandas dataframe

The end goal is to extract the text contained on a specific layer, inside of a named view, from the model space. I have the layer restriction (the text in yellow for visual), but I can't seem to figure out the syntax (if possible) to limit the query to either items inside one of the named views, or within a bounding box that I define to match the named view (orange box). The text being queried is single text. It is (and will always be) an exploded table. Each text item has a unique .insert value.
Ultimately this above mentioned query loop would be put inside a loop to iterate over all named views (bounding box) inside of the model space. Each of the iterations creating a unique list containing the query results. The list would then be input into a pandas dataframe for further manipulations.
import ezdxf
filepath = "C:/MBS_JOBS-16/8741-22/8741-22_(DTL).dxf"
doc = ezdxf.readfile(filepath)
msp = doc.modelspace()
ls = []
for e in msp.query('TEXT MTEXT[layer=="text"]'):
ls.append(e.dxf.text)
print(ls)

The ezdxf package does not have a feature for selecting entities based on location and size, but a bounding box based implementation is relatively easy.
It is important to know that bounding boxes of text based entities (TEXT, MTEXT) are inaccurate because matplotlib (used by ezdxf to render text) renders TTF fonts differently than AutoCAD.
I created an example DXF file with 6 views, called "v1", "v2", ...:
The following code prints the the content of view "v1": ["Text1a", "Text1b"]
import ezdxf
from ezdxf.math import BoundingBox, Vec2
from ezdxf import bbox
def get_view_box(view):
center = Vec2(view.dxf.center)
size = Vec2(view.dxf.width, view.dxf.height)
bottom_left = center - (size / 2)
top_right = center + (size / 2)
return BoundingBox([bottom_left, top_right])
doc = ezdxf.readfile("views.dxf")
msp = doc.modelspace()
view = doc.views.get("v1")
view_box = get_view_box(view)
for e in msp.query("TEXT MTEXT"):
text_box = bbox.extents([e]) # expects a list of entities!
if view_box.contains(text_box):
print(e.dxf.text)

Convert LineString / MultiLineString geometries to lat lon

I am using this Mapillary endpoint: https://tiles.mapillary.com/maps/vtp/mly1_public/2/{zoom_level}/{x}/{y}?access_token={} and getting such responses back (see photo). Also, here is the Mapillary documentation.
It is not quite clear to me what the nested coordinate lists in the response represent. By the looks of it, I initially thought it may have to do with pixel coordinates. But judging by the context (the API documentation) and the endpoint I am using, I would say that is not the case. Also, I am not sure if the json response you see in the picture is valid geojson. Some online formatters did not accept it as valid.
I would like to find the bounding box of the "sequence". For context, that would be the minimal-area rectangle defined by two lat, lon positions that fully encompasses the geometry of the so-called "sequence"; and a "sequence" is basically a series of photos taken during a vehicle/on-foot trip, together with the metadata associated with the photos (metadata is available using another endpoint, but that is just for context).
My question is: is it possbile to turn the coordinates you see in the pictures into (lat,lon)? Having those, it would be easy for me to find the bounding box of the sequence. And if so, how? Also, please notice that some of the nested lists are of type LineString while others are MultiLineString (which I read about the difference here: help.arcgis.com, hope this helps)
Minimal reproducible code snippet:
import json
import requests
import mercantile
import mapbox_vector_tile as mvt
ACCESS_TOKEN = 'XXX' # can be provided from here: https://www.mapillary.com/dashboard/developers
z6_tiles = list(mercantile.tiles( #us_west_coast_bbox
west=-125.066423,
south=42.042594,
east=-119.837770,
north=49.148042,
zooms=6
))
# pprint(z6_tiles)
vector_tiles_url = 'https://tiles.mapillary.com/maps/vtp/mly1_public/2/{}/{}/{}?access_token={}'
for tile in z6_tiles:
res = requests.get(vector_tiles_url.format(tile.z,tile.x,tile.y,ACCESS_TOKEN))
res_json = mvt.decode(res.content)
with open('idea.json','w+') as f:
json.dump(res_json, f, indent=4)

I think this get_normalized_coordinates is the solution I was looking for. Please take this with a grain of salt, as I did not fully test it yet. Will try to and then I will update my answer. Also, please be cautious, because for tiles closer to either the South or the North Pole, the Z14_TILE_DMD_WIDTH constant will not be the one you see, but something more like: 0.0018958715374282065.
Z14_TILE_DMD_WIDTH = 0.02197265625
Z14_TILE_DMD_HEIGHT = 0.018241950298914844
def get_normalized_coordinates(bbox: mercantile.LngLatBbox,
target_lat: int,
target_lon: int,
extent: int=4096): # 4096 is Mapillary's default
"""
Returns lon,lat tuple representing real position on world map of a map feature.
"""
min_lon, min_lat, _, _ = bbox
return min_lon + target_lon / extent * Z14_TILE_DMD_WIDTH,
min_lat + target_lat / extent * Z14_TILE_DMD_HEIGHT
And if you are wondering how I came with the constants that you see, I simply iterated over the list of tiles that I am interested in and checked to make sure they all have the same width/height size (this might have not been the case, keeping in mind what I mentioned above about tiles closer to one of the poles - I think this is called "distortion", not sure). Also, for context: these tiles I iterated over are within this bbox: (-125.024414, 31.128199, -108.896484, 49.152970) (min_lon, min_lat, max_lon, max_lat; US west coast) which I believe is also why all the tiles have the same width/height sizes.
set_test = set()
for tile in relevant_tiles_set:
curr_bbox = mercantile.bounds(list_relevant_tiles_set[i])
dm_width_diff: float = curr_bbox.east - curr_bbox.west
dm_height_diff: float = curr_bbox.north - curr_bbox.south
set_test.add((dm_width_diff, dm_height_diff))
set_test
output:
{(0.02197265625, 0.018241950298914844}
UPDATE: forgot to mention that you actually do not need to compute those WIDTH, HEIGHT constants. You just replace those with (max_lon - min_lon) and (max_lat - min_lat) respectively. What I did with those constants was something for testing purposes only

Generating Random Cordinates for Specific Country

Am Trying to Generate Random Coordinates for a Country
I used this library Faker
def geo_point():
"""make random cordinates"""
faker = factory.Faker('local_latlng', country_code = 'IN')
coords = faker.generate()
return (coords[1], coords[0])
But the problem in this is, it has a very limited set of coordinates around 30-40 we require at least 10,000 for testing.
I tried a simple approach
def random_geo_cordinate():
"""make random geocordinates"""
x, y = uniform(-180,180), uniform(-90, 90)
return (y, x)
But then only 10-20 coordinates for Specific Country Comes.
There were a lot of references I found that through shape_files we can generate but in all of them only geom parameters are only available.
I found a method through which I can check that these coordinates lie in that country or not via the Geom column.
But am still missing something in generating random coordinates for a country.
Is there any simple and direct approach.
Am using
POST GIS Database
GeoDjango Server
Note:
I used GDAL for getting shapefiles for a country

You could use Overpass API, which queries the OSM database, so you get real coordinates.
For example fetching all villages in India:
import requests
import json
overpass_url = "http://overpass-api.de/api/interpreter"
overpass_query = """
[out:json];area[name="India"];(node[place="village"](area););out;
"""
response = requests.get(
overpass_url,
params={'data': overpass_query}
)
coords = []
if response.status_code == 200:
data = response.json()
places = data.get('elements', [])
for place in places:
coords.append((place['lat'], place['lon']))
print ("Got %s village coordinates!" % len(coords))
print (coords[0])
else:
print("Error")
Output:
Got 102420 village coordinates!
(9.9436615, 77.8978759)
Note: Overpass API is rate limited, so you should save the all coordinates locally and extract your random set from there!
Additionally, you can play around with places parameter fetching just cities or towns, or fetch restaurant locations for a specific district, ...

https://3geonames.org/randomland.IN is a free API that returns random locations in any country of the world.

appending an index to laspy file (.las)

I have two files, one an esri shapefile (.shp), the other a point cloud (.las).
Using laspy and shapefile modules I've managed to find which points of the .las file fall within specific polygons of the shapefile. What I now wish to do is to add an index number that enables identification between the two datasets. So e.g. all points that fall within polygon 231 should get number 231.
The problem is that as of yet I'm unable to append anything to the list of points when writing the .las file. The piece of code that I'm trying to do it in is here:
outFile1 = laspy.file.File("laswrite2.las", mode = "w",header = inFile.header)
outFile1.points = truepoints
outFile1.points.append(indexfromshp)
outFile1.close()
The error I'm getting now is: AttributeError: 'numpy.ndarray' object has no attribute 'append'. I've tried multiple things already including np.append but I'm really at a loss here as to how to add anything to the las file.
Any help is much appreciated!

There are several ways to do this.
Las files have classification field, you could store the indexes in this field
las_file = laspy.file.File("las.las", mode="rw")
las_file.classification = indexfromshp
However if the Las file has version <= 1.2 the classification field can only store values in the range [0, 35], but you can use the 'user_data' field which can hold values in the range [0, 255].
Or if you need to store values higher than 255 / you need a separate field you can define a new dimension (see laspy's doc on how to add extra dimensions).
Your code should be close to something like this
outFile1 = laspy.file.File("laswrite2.las", mode = "w",header = inFile.header)
# copy fields
for dimension in inFile.point_format:
dat = inFile.reader.get_dimension(dimension.name)
outFile1.writer.set_dimension(dimension.name, dat)
outFile1.define_new_dimension(
name="index_from_shape",
data_type=7, # uint64_t
description = "Index of corresponding polygon from shape file"
)
outFile1.index_from_shape = indexfromshp
outFile1.close()

dynamic plotting in wxpython

I have been developing a GUI for reading continuous data from a serial port. After reading the data, some calculations are made and the results will be plotted and refreshed (aka dynamic plotting). I use the wx backend provided in the matplotlib for this purposes. To do this, I basically use an array to store my results, in which I keep appending it to, after each calculation, and replot the whole graph. To make it "dynamic", I just set the x-axis lower and upper limits for each iteration. Something like found in:
http://eli.thegreenplace.net/2008/08/01/matplotlib-with-wxpython-guis/
The problem, however, is that since the data is continuous, and if I keep plotting it, eventually the system memory will run out and system will crash. Is there any other way I can plot my result continuously?

To do this, I basically use an array
to store my results, in which I keep
appending it to
Try limiting the size of this array, either by deleting old data or by deleting every n-th entry (the screen resolution will prevent all entries to be displayed anyway). I assume you write all the data to disk so you won't lose anything.
Also, analise your code for memory leaks. Stuff you use and don't need anymore but that doesn't get garbage-collected because you still have a reference to it.

I have created such a component with pythons Tkinter. The source is here.
Basically, you have to keep the plotted data somewhere. You cannot keep an infinite amount of data points in memory, so you either have to save it to disk or you have to overwrite old data points.

Data and representation of data are two different things. You might want to store your data to disk if it's important data to be analyzed later, but only keep a fixed period of time or the last N points for display purposes. You could even let the user pick the time frame to be displayed.

I actually ran into this problem (more of a mental block, actually...).
First of all I copy-pasted some wx Plot code from wx Demo Code.
What I do is keep a live log of a value, and compare it to two markers (min and max, shown as red and green dotted lines) (but I will make these 2 markers optional - hence the optional parameters).
In order to implement the live log, I first wanted to use the deque class, but since the data is in tuple mode (x,y coordinates) I gave up and just tried to rewrite the entire parameter list of tuples: see _update_coordinates.
It works just fine for keeping track of the last 100-10,000 plots. Would have also included a printscreen, but I'm too much of a noob at stackoverflow to be allowed :))
My live parameter is updated every 0.25 seconds over a 115kbps UART.
The trick is at the end, in the custom refresh method!
Here is most of the code:
class DefaultPlotFrame(wx.Frame):
def __init__(self, ymin=0, ymax=MAXIMUM_PLOTS, minThreshold=None,
maxThreshold=None, plotColour='blue',
title="Default Plot Frame",
position=(10,10),
backgroundColour="yellow", frameSize=(400,300)):
self.minThreshold = minThreshold
self.maxThreshold = maxThreshold
self.frame1 = wx.Frame(None, title="wx.lib.plot", id=-1, size=(410, 340), pos=position)
self.panel1 = wx.Panel(self.frame1)
self.panel1.SetBackgroundColour(backgroundColour)
self.ymin = ymin
self.ymax = ymax
self.title = title
self.plotColour = plotColour
self.lines = [None, None, None]
# mild difference between wxPython26 and wxPython28
if wx.VERSION[1] < 7:
self.plotter = plot.PlotCanvas(self.panel1, size=frameSize)
else:
self.plotter = plot.PlotCanvas(self.panel1)
self.plotter.SetInitialSize(size=frameSize)
# enable the zoom feature (drag a box around area of interest)
self.plotter.SetEnableZoom(False)
# list of (x,y) data point tuples
self.coordinates = []
for x_item in range(MAXIMUM_PLOTS):
self.coordinates.append((x_item, (ymin+ymax)/2))
self.queue = deque(self.coordinates)
if self.maxThreshold!=None:
self._update_max_threshold()
#endif
if self.lockThreshold!=None:
self._update_min_threshold()
#endif
self.line = plot.PolyLine(self.coordinates, colour=plotColour, width=1)
self.lines[0] = (self.line)
self.gc = plot.PlotGraphics(self.lines, title, 'Time', 'Value')
self.plotter.Draw(self.gc, xAxis=(0, MAXIMUM_PLOTS), yAxis=(ymin, ymax))
self.frame1.Show(True)
def _update_max_threshold(self):
if self.maxThreshold!=None:
self.maxCoordinates = []
for x_item in range(MAXIMUM_PLOTS):
self.maxCoordinates.append((x_item, self.maxThreshold))
#endfor
self.maxLine = plot.PolyLine(self.maxCoordinates, colour="green", width=1)
self.maxMarker = plot.PolyMarker(self.maxCoordinates, colour="green", marker='dot')
self.lines[1] = self.maxMarker
#endif
def _update_live_param(self, liveParam, minParam, maxParam):
if minParam!=None:
self.minThreshold = int(minParam)
self._update_min_threshold()
#endif
if maxParam!=None:
self.maxThreshold = int(maxParam)
self._update_max_threshold()
#endif
if liveParam!=None:
self._update_coordinates(int(liveParam))
#endif
def _update_coordinates(self, newValue):
newList = []
for x,y in self.coordinates[1:]:
newList.append((x-1, y))
#endfor
newList.append((x, newValue))
print "New list", newList
self.line = (plot.PolyLine(newList, colour=self.plotColour, width=1))
self.lines[0] = self.line
self.coordinates = newList
def _MyLIVE_MAGIC_refresh__(self, liveParam=None, minParam=None, maxParam=None):
self._update_live_param(liveParam, minParam, maxParam)
self.gc = plot.PlotGraphics(self.lines, self.title, 'Time', 'Value')
self.plotter.Draw(self.gc, xAxis=(0, MAXIMUM_PLOTS), yAxis=(self.ymin, self.ymax))
self.plotter.Refresh()
self.frame1.Refresh()