Develop Reference

Receiving Error message: JSONDecodeError when attempting to use API

I am following along with Python for Data Analysis and am on Chapter 6 looking at using APIs.
I wish to connect to sources provided by National Grid on their Data Portal. They provide a number of URLs (e.g. several found here, https://data.nationalgrideso.com/ancillary-services/obligatory-reactive-power-service-orps-utilisation). I want to read these directly into pandas rather than download the Excel/csv file and then open that.
I am receiving the error message
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
after attempting the following:
import requests
import codecs
import json
url = 'https://data.nationalgrideso.com/backend/dataset/7e142b03-8650-4f46-8420-7ce1e84e1e5b/resource/a61c6c26-62ec-41e1-ae25-ed95f4562274/download/reactive-utilisation-data-apr-2020-mar-2022.csv'
resp = request.get(url)
decoded_data = codecs.decode(resp.text.encode(), 'utf-8-sig')
data = json.loads(decoded_data)
I understand that I need to use 'utf-8-sig' due to a particular BOM appearing on the first line otherwise.
I have looked at answers regarding the same error message but nothing is working for me at present. The API is working in the browser and I am receiving a response of 200 and data is being returned. Perhaps I am missing something more fundamental in the approach?

Choropleth map with OpenStreetMap data

My goal is to get a so-called "choropleth map" (I guess) of the zip code areas in Germany. I have found the python package "folium" but it seems like it takes a .json file as input:
https://github.com/python-visualization/folium
On OpenStreetMap I only see shp.zip and .osm.pbf files. Inside the shp.zip archive I find all sorts of file endings which I have never heard of but no .json file. How do I use the data from OpenStreetMap to feed folium? Am I running into the wrong direction?

If you want to create a choropleth map you must follow these steps:
First you need a file containing info about the regions of that country. A sample .json file has been supplied with this answer, however, there are actually many file formats commonly used for maps. In your case, you need to convert your OSM shape file (.shp) into a more modern file type like .geojson. Thankfully we have ogr2ogr to do this last part:
ogr2ogr -f GeoJSON -t_srs EPSG:4326 -simplify 1000 [name].geojson [name].shp
Advice: You can also extract the administrative borders from these web sites:
* [OSM Boundaries Map 4.2][2]
* [Mapzen][3]
* [Geofabrik][4]
Download data based on it (a .csv file, for example). Obviously, the file must have a column with the ZIP Codes of that country.
Once you get these files the rest is straightforward, Follium will create the choropleth map automatically.
I wrote a simple example of this about the unemployment rate in the US:
Code:
import folium
import pandas as pd
osm = folium.Map([43, -100], zoom_start=4)
osm.choropleth(
geo_str = open('US_states.json').read(),
data = pd.read_csv("US_unemployment.csv"),
columns = ['State', 'Unemployment'],
key_on = 'feature.id',
fill_color = 'YlGn',
)
Output:

I haven't done this myself but there are various solutions for converting OSM files (.osm or .pbf) to (geo)json. For example osmtogeojson. More tools can be found at the GeoJSON page in the OSM wiki.

I went to https://overpass-turbo.eu/ (which retrieves data from openstreetmap via a specific Query Language QL) and hit run on the following code:
[timeout:900];
area[name="Deutschland"][admin_level=2][boundary=administrative]->.myarea;
rel(area.myarea)["boundary"="postal_code"];
out geom;
You can "export to geojson" but in my case that didn't work because it's too much data which cannot be processed inside the browser. But exporting the "raw data" works. So I did that and then I used "osmtogeojson" to get the right format. After that I was able to feed my openstreetmap data to folium as described in the tutorial of folium.
This answer was posted as an edit to the question Choropleth map with OpenStreetMap data by the OP user3182532 under CC BY-SA 3.0.

GeoJSON data not displaying in Python folium map

I am trying to display the following geojson file in a folium map in Python but it just shows an empty map with none of the data.
Here are the steps I have tried:
I tried using the python code below but nothing shows up.
I tried other geojson files in the github repository below using the same code and the data show up without any issue, so it looks like my python code is fine
I opened the "census_tracts_2010.geojson" file in github and Mapshaper, the data showed up perfectly without any issue, so it doesn't look like the geojson file is corrupted
Could anyone please let me know how I can fix it?
Geojson file:
https://github.com/dwillis/nyc-maps/blob/master/census_tracts_2010.geojson
Python code:
import folium
m = folium.Map(location=[40.66393072,-73.93827499], zoom_start=13)
m.choropleth(geo_path="census_tracts_2010.geojson")
m.save(outfile='datamap.html')
Thanks a lot!

That file is not a GeoJson it is a TopoJson. You need to use folium.TopoJson instead.
import folium
m = folium.Map(location=[40.66393072,-73.93827499], zoom_start=13)
folium.TopoJson(
open('census_tracts_2010.geojson'),
object_path='objects.nyct2010',
).add_to(m)
m

You need to open the geojson file.
m.choropleth(open("census_tracts_2010.geojson"))
Take a look at the examples https://folium.readthedocs.io/en/latest/quickstart.html

Try this: m.add_child(folium.GeoJson(data = open("census_tracts_2010.geojson"))) and then call m.save() fun

working with geojson and vincent on python

I want to import a geojson file into python so I can map it with a visualization package vincent and merge with other data in a pandas data frame.
To be specific, the said geojson file is: http://ec2-54-235-58-226.compute-1.amazonaws.com/storage/f/2013-05-12T03%3A50%3A18.251Z/dcneighorhoodboundarieswapo.geojson. It's a map of DC with neighborhoods, put together by Justin Grimes.
Right now, I'm just trying to visualize this map on notebook. Here's my code:
import vincent
map=r'http://ec2-54-235-58-226.compute-1.amazonaws.com/storage/f/2013-05-12T03%3A50%3A18.251Z/dcneighorhoodboundarieswapo.geojson'
geo_data = [{'name': 'countries',
'url': map,
'feature': "features"}]
vis = vincent.Map(geo_data=geo_data, scale=5000)
vis
but I keep getting an error message, local host says: [Vega err] loading failed.
What am I doing wrong here?
I don't yet know much about GIS and Python so I ask you be specific in your explanation. Thank you in advance.

At this moment you can't use for you maps with vincent anything but topojson file format (see https://github.com/mbostock/topojson/wiki).
You can convert geojson into topojson using web tools like https://mapshaper.org/ or using command-line utility (https://github.com/mbostock/topojson/wiki/Command-Line-Reference) with command like this:
topojson -p -o <target-file>.topo.json -- <input-file>.json
(-p says utility to keep properties of geometries.)

Opening a geojson file in RhinoPython

I'm hoping my problem can be solved with some geojson expertise. The problem I'm having has to do with RhinoPython - the embedded IronPython engine in McNeel's Rhino 5 (more info here: http://python.rhino3d.com/). I don't think its necessary to be an expert on RhinoPython to answer this question.
I'm trying to load a geojson file in RhinoPython. Because you can't import the geojson module into RhinoPython like in Python I'm using this custom module GeoJson2Rhino provided here: https://github.com/localcode/rhinopythonscripts/blob/master/GeoJson2Rhino.py
Right now my script looks like this:
`import rhinoscriptsyntax as rs
import sys
rp_scripts = "rhinopythonscripts"
sys.path.append(rp_scripts)
import rhinopythonscripts
import GeoJson2Rhino as geojson
layer_1 = rs.GetLayer(layer='Layer 01')
layer_color = rs.LayerColor(layer_1)
f = open('test_3.geojson')
gj_data = geojson.load(f,layer_1,layer_color)
f.close()`
In particular:
f = open('test_3.geojson')
gj_data = geojson.load(f)
works fine when I'm trying to extract geojson data from regular python 2.7. However in RhinoPython I'm getting the following error message: Message: expected string for parameter 'text' but got 'file'; in reference to gj_data = geojson.load(f).
I've been looking at the GeoJson2Rhino script linked above and I think I've set the parameters for the function correctly. As far as I can tell it doesn't seem to recognize my geojson file, and wants it as a string. Is there an alternative file open function I can use to get the function to recognize it as a geojson file?

Judging by the error message, it looks like the load method requires a string as the first input but in the above example a file object is being passed instead. Try this...
f = open('test_3.geojson')
g = f.read(); # read contents of 'f' into a string
gj_data = geojson.load(g)
...or, if you don't actually need the file object...
g = open('test_3.geojson').read() # get the contents of the geojson file directly
gj_data = geojson.load(g)
See here for more information about reading files in python.