According to the GeoJSON Format Specification
"If a feature has a commonly used identifier, that identifier should be included as a member of the feature object with the name "id"."
My question is how do I add this to my GeoJSON?
If I create it as an attribute and then save it as GeoJSON in QGIS it ends up in Properties instead of in Features.
This is what I want to do:
{
"type": "FeatureCollection",
"crs": { "type": "name", "properties": { "name":"urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "id":"1", "properties": { "Namn":.................
This is what QGIS produces:
{
"type": "FeatureCollection",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { "id": 1, "Name"..................
I have also tried PyGeoj https://github.com/karimbahgat/PyGeoj. This has a function to add a unique id but it also adds it under properties.
If I open the GeoJSON and write it in by hand then it works but I don't want to do this for all of my layers, some of which contain many features.
Thought I would write here how I have solved it in case anyone else comes across the same problem.
A simple python script:
#create empty file to be writen to
file = open("kommun_id.geojson", "w")
count = 0
#read original file
with open('kommun.geojson', 'r')as myfile:
for line in myfile:
#lines that don't need to be edited with an 'id'
if not line.startswith('{ "type": '):
file.write(line)
else:
#lines that need to be edited
count = count +1
idNr = str(count)
file.write(line[0:20] + '"id":'+ '"'+ idNr + '",' +line[21:])
file.close()
It might not work for all geoGJSONs but it works for the ones I have created with QGIS
You can fix this by using the preserve_fid flag in ogr2ogr. You're going to "convert" the bad GeoJSON that QGIS dumps to the format you want, with the id field exposed.
ogr2ogr -f GeoJSON fixed.geojson qgis.geojson -preserve_fid
Here qgis.geojson is the one the QGIS created and fixed.geojson is the new version with the exposed id field.
This is due to ogr2ogr not knowing which QGIS field to use as the geoJSON ID.
Per this QGIS issue (thank you #gioman), a workaround is to manually specify the ID field in Custom Options --> Layer Settings when you do the export:
If you need to do this to geoJSONs that have already been created, you can use the standalone ogr2ogr command-line program:
ogr2ogr output.geojson input.geojson -lco id_field=id
If you need to convert the IDs from properties inside multiple files, you can use the command-line tool jq inside a Bash for-loop:
for file in *.geojson; do jq '(.features[] | select(.properties.id != null)) |= (.id = .properties.id)' $file > "$file"_tmp; mv "$file"_tmp $file;done
I had the same problem and i managed with this little python script.
import json
with open('georef-spain-municipio.geojson') as json_file:
data = json.load(json_file)
i = 0
for p in data['features']:
i+=1
p['id'] = i
with open('georef-spain-municipio_id.geojson', 'w') as outfile:
json.dump(data, outfile)
I hope it helps
Related
I am helping my teacher by creating a game where the k-2 kids can learn their passwords, I added this json file to create a save for the teacher so he doesn't have to re-add all the computers names and passwords... But when I run my code, the json file gets wiped, and I lose all of my code... Luckily I had backups but I cant have it erase for my teacher.
Python Code:
import json
with open("accounts.json", "w") as f:
accountData = json.dumps(f)
type(accountData)
JSON:
{ // not real names and passwords for security
"accounts": [
{
"id": "1",
"uname": "scarlett",
"pword": "k,",
"points": "0"
},
{
"id": "2",
"uname": "santiago",
"pword": "k,",
"points": "0"
},
{
"id": "3",
"uname": "harper",
"pword": "k,",
"points": "0"
}
]
}
It's a wrong use of the json.dump method.
Here is part of the help to json.dump in python at version 3.7.6.
Help on function dump in module json:
dump(obj, fp, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False, **kw)
Serialize ``obj`` as a JSON formatted stream to ``fp`` (a
``.write()``-supporting file-like object).
The first parameter should be the dict object which contains the names and passwords, and you missed this parameter.
And the parameter f should be in the second place.
Is there any easy way to generate such a JSON? I found os.walk() and os.listdir(), so I may do recursive descending into directories and build a python object, well, but it sounds like reinventing a wheel, maybe someone knows working code for such a task?
{
"type": "directory",
"name": "hello",
"children": [
{
"type": "directory",
"name": "world",
"children": [
{
"type": "file",
"name": "one.txt"
},
{
"type": "file",
"name": "two.txt"
}
]
},
{
"type": "file",
"name": "README"
}
]
}
I don't think that this task is a "wheel" (so to speak). But it is something which you can easily achieve by means of the tools you mentioned:
import os
import json
def path_to_dict(path):
d = {'name': os.path.basename(path)}
if os.path.isdir(path):
d['type'] = "directory"
d['children'] = [path_to_dict(os.path.join(path,x)) for x in os.listdir\
(path)]
else:
d['type'] = "file"
return d
print json.dumps(path_to_dict('.'))
On Linux, the command-line tool tree can be used, although it is not installed by default. The output is almost identical to that required by the OP, using the flag -J for JSON output (which can then be streamed to a file for example):
tree -J folder
On OSX, this tool can be installed via Homebrew.
I just had to do this (well, almost) so hit this page but the above doesn't recurse into subdirectories.
So this version only handles directories, not files, but you could add those.
First generate a nested python dict:
def fs_tree(root):
results = {}
for (dirpath, dirnames, filenames) in os.walk(root):
parts = dirpath.split(os.sep)
curr = results
for p in parts:
curr = curr.setdefault(p, {})
return results
Secondly dump that to json using the json module.
I have json files in S3 containing array of objects in each file, like shown below.
[{
"id": "c147162a-a304-11ea-aa90-0242ac110028",
"clientId": "xxx",
"contextUUID": "1bb6b39e-b181-4a6d-b43b-4040f9d254b8",
"tags": {},
"timestamp": 1592855898
}, {
"id": "c147162a-a304-11ea-aa90-0242ac110028",
"clientId": "yyy",
"contextUUID": "1bb6b39e-b181-4a6d-b43b-4040f9d254b8",
"tags": {},
"timestamp": 1592855898
}]
I used crawler to detect and load the schema to catalog. It was successful and it created a schema with a single column named array with data type array<struct<id:string,clientId:string,contextUUID:string,tags:string,timestamp:int>>.
Now, I tried to load the data using glueContext.create_dynamic_frame.from_catalog function, but I could not see any data. I tried printing schema and data as shown below.
ds = glueContext.create_dynamic_frame.from_catalog(
database = "dbname",
table_name = "tablename")
ds.printSchema()
root
ds.schema()
StructType([], {})
ds.show()
empty
ds.toDF().show()
++
||
++
++
Any idea, what I am doing wrong? I am planning to extract each object in array and transform the object to a different schema.
You can try to give regex in format_options to tell glue how it should read the data. Following code has worked for me:
glueContext.create_dynamic_frame_from_options('s3',
{
'paths': ["s3://glue-test-bucket-12345/events/101-1.json"]
},
format="json",
format_options={"jsonPath": "$[*]"}
).toDF()
I hope it solves the problem.
I have a .geojson file (call it data.geojson) which I use to manually update a dataset on mapbox.
Suppose that my data.geojson file is structured as follows:
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"suburb": "A",
"unemployed": 10
},
"geometry": {
"type": "Point",
"coordinates": [
0,
0
]
}
},
{
"type": "Feature",
"properties": {
"suburb": "B",
"unemployed": 20
},
"geometry": {
"type": "Point",
"coordinates": [
1,
1
]
}
data.geojson is stored locally, and every 12 hours the 'unemployed' property of each feature is updated using another python script that scrapes data from the web.
Currently, in order to update these properties within the online dataset (stored at mapbox.com) I am manually navigating to the Mapbox website and reuploading the data.geojson file. I am looking for a way to accomplish this task pythonically.
Any help would be greatly appreciated!
you can setup a timer of some sort to automatically update the data using javascript functions. Here I am using a source and layer named "STI", which is just geoJSON line data.
The function would first add the source of the data as well as the layer :
var STI_SOURCE = 'json/sti/STI.json'; // declare URL for data
map.addSource('sti', { type: 'geojson', data: STI1 }); // Add source using URL
// Add the actual layer using the source
map.addLayer({
"id": "sti",
"type": "line",
"source": "sti",
"layout": {
"line-join": "miter",
"line-cap": "round"
},
"paint": {
"line-color": "#fff",
"line-width": 1,
"line-dasharray": [6, 2]
}
});
Then, when you want to refresh the data - remove them :
map.removeLayer('sti');
map.removeSource('sti');
Then, you can re-add them by starting at the beginning. There are other ways (and better) to do this, but this is just one way that works. I think there is a setData() function that does this better. But hopefully this can get you started.
My solution, in the end, was simply to point the source of the Mapbox layer to the locally stored dataset.geojson file rather than the corresponding dataset stored online at mapbox.com.
I was able to edit the locally stored dataset.geojson using the 'json' python package. Since the Mapbox layer source was pointing directly to the local dataset, all updates to this local file would then be reflected in the Mapbox layer. This way, there is no need to upload any data to Mapbox.
#David also posted a helpful solution if you wish to go down that route.
I would like to use automation to create the hocon configuration with python 3 scripting. I read that lightbend (https://github.com/lightbend/config) recommends pyhocon (https://github.com/chimpler/pyhocon).
I am having problems figuring out how to create an Hocon object and write the data to a file as hocon. It is important to me that the syntax for the substitution are in the result.
For example I expect the output of the file myconfig.conf to look something like this:
{
Environment: "dev"
JobName: ${Environment}"-hello-bob"
}
So, I assumed that there was a way to do something like this:
config2 = ConfigFactory.parse_string("{}")
config2.put("Environment", "dev")
#Some type of object for references or special syntax for ${Environment}
config2.put("JobName", "${Environment}")
Then after creating the stuffed object there should be a simple way to write out to a file or files:
filename = "myconfig.conf"
print("Write to disk as {}".format(filename))
with open(filename, "w") as fd:
fd.write(config2.to_hocon_str)
Has anyone figured a way to do this? It seems odd that the library can only be used for reading data only.
Probably this example will answer your question
from pyhocon.converter import HOCONConverter
import pyhocon
string = '{"Environment": "Dev","Test": ${Environment}}'
factory = pyhocon.ConfigFactory.parse_string(string, resolve=True)
factory.put('somekey','somevalue')
print(HOCONConverter().to_hocon(factory))
returns
Environment = "Dev"
Test = "Dev"
somekey = "somevalue"
So, I decided to look at documentation for JVM (Java/Scala) library (https://github.com/lightbend/config). After reading the documentation, there was a clear section on hocon examples (https://github.com/lightbend/config#examples-of-hocon). In this documentation, they categorized 7 valid hocon styles. I call these styles because if I was to automate the generation of these files, I would be picking one way to write out and sticking with it.
All of these are valid HOCON.
1.Start with valid JSON:
{
"foo" : {
"bar" : 10,
"baz" : 12
}
}
2.Drop root braces:
"foo" : {
"bar" : 10,
"baz" : 12
}
3.Drop quotes:
foo : {
bar : 10,
baz : 12
}
4.Use = and omit it before {:
foo {
bar = 10,
baz = 12
}
5.Remove commas:
foo {
bar = 10
baz = 12
}
6.Use dotted notation for unquoted keys:
foo.bar=10
foo.baz=12
7.Put the dotted-notation fields on a single line:
foo.bar=10, foo.baz=12
Because I will be using the pyhocon library, I needed to look for write solutions within the library. I found some help from chimpler's git (https://github.com/chimpler/pyhocon). What I found was that they have two hocon styles which can be simply written out. One is json and the other is something that wasn't on the list which was describe above by lightbend.
Style 1: pure JSON, witch can be written out in two ways:
HOCONConverter.to_json
#Using HOCONConverter.to_json
confTree = ConfigFactory.parse_string("{}")
confTree.put("Environment","Dev")
confTree.put("Test","${Environment}")
filename = "./json_coverted.conf"
print("Write to disk as {}".format(filename))
with open(filename, "w") as fd:
fd.write(HOCONConverter.to_json(confTree))
HOCONConverter.to_json Result
{
"Environment": "Dev",
"Test": "${Environment}"
}
OR Using json.dump
#Using json.dump
confTree = ConfigFactory.parse_string("{}")
confTree.put("Environment","Dev")
confTree.put("Test","${Environment}")
filename = "./json_dumped.conf"
print("Write to disk as {}".format(filename))
with open(filename, "w") as fd:
fd.write(json.dumps(confTree,indent=4))
Using json.dump Result
{
"Environment": "Dev",
"Test": "${Environment}"
}
Pyhocon's other Style, not listed by lightbend
# HOCONConverter.to_hocon
confTree = ConfigFactory.parse_string("{}")
confTree.put("Environment","Dev")
confTree.put("Test","${Environment}")
filename = "./hocon_coverted.txt"
print("Write to disk as {}".format(filename))
with open(filename, "w") as fd:
fd.write(HOCONConverter.to_hocon(confTree))
Pyhocon's other Style, not listed by lightbend Result
Environment = "Dev"
Test = "${Environment}"
So, to answer my own question the only dependable way to generate a hocon conf file dynamically using pyhocon in Python 3 is by using one of the json methods (converter or dumps). But this still leaves an open question. The question being, will reading a json to a pyhocon ConfTree object be able dereference the substitutions when they are in the json?
For example if I read the file
{
"Environment": "Dev",
"Test": "${Environment}"
}
Will the ConfTree object get "Dev" as the value for Test?
No, it will not. Here is my test
filename = "json_coverted.conf"
print("Reading file{}".format(filename))
conf = ConfigFactory.parse_file(filename)
key="Test"
value=conf.get(key)
print("Key:{} Value:{}".format(key,value))
Test Result Out to screen
Reading filejson_coverted.conf
Key:Test Value:${Environment}
So, then how does one use pyhocon with substitutions?
It just can't hence, I will not use either library for writing out confs. It has to be a manual process if I want to use substitutions. So, I am only using this library for reading confs.