Develop Reference

Most efficient way of converting RESTful output to dataframe

I have output from a REST call that I've converted to JSON.
It's a highly nested collection of dicts and lists, but I'm eventually able to convert it to dataframe as follows:
import panads as pd
from requests import get
url = 'http://stats.oecd.org/SDMX-JSON/data/MEI_FIN/IR3TIB.GBR+USA.M/all'
params = {
'startTime' : '2008-06',
'dimensionAtObservation' : 'TimeDimension'
}
r = get(url, params = params)
x = r.json()
d = x['dataSets'][0]['series']
a = pd.DataFrame(d['0:0:0']['observations'])
b = pd.DataFrame(d['0:1:0']['observations'])
This works absent some manipulation to make it easier to work with, and as there are multiple time series, I can do a version of the same for each, but it goes without saying it's kind of clunky.
Is there a better/cleaner way to do this.

The pandasdmx library makes this super-simple:
import pandasdmx as sdmx
df = sdmx.Request('OECD').data(
resource_id='MEI_FIN',
key='IR3TIB.GBR+USA.M',
params={'startTime': '2008-06', 'dimensionAtObservation': 'TimeDimension'},
).write()

Absent any responses, here's the solution I came up with. I added a list comprehension to deal with getting each series into a dataframe, and then a transpose as this source resulted in the series being aligned across rows instead of down columns.
import panads as pd
from requests import get
url = 'http://stats.oecd.org/SDMX-JSON/data/MEI_FIN/IR3TIB.GBR+USA.M/all'
params = {
'startTime' : '2008-06',
'dimensionAtObservation' : 'TimeDimension'
}
r = get(url, params = params)
x = r.json()
d = x['dataSets'][0]['series']
df = [pd.DataFrame(d[i]['observations']).loc[0] for i in d]
df = pd.DataFrame(df).T

Can't figure out why I get error when trying to get JSON value into DF

Been looking online but can't figure out why I'm getting the error as the data is available in the JSON.
I'm trying to extract "pull_request_contributors" value from JSON and put into DF.
I get the error:
KeyError: "Try running with errors='ignore' as key 'pull_request_contributors' is not always present"
Code
cg = CoinGeckoAPI()
ts = '01-01-2017'
cs = 'bitcoin'
# get data
result = cg.get_coin_history_by_id(cs, ts)
#pull_request_contributors
df_pr = pd_json.json_normalize(data,
record_path='developer_data',
meta=['pull_request_contributors']).set_index(ts)
JSON
{'community_data': {'facebook_likes': 40055,
'reddit_accounts_active_48h': '4657.4',
'reddit_average_comments_48h': 186.5,
'reddit_average_posts_48h': 3.75,
'reddit_subscribers': 1014816,
'twitter_followers': 64099},
'developer_data': {'closed_issues': 3845,
'commit_count_4_weeks': 245,
'forks': 22024,
'pull_request_contributors': 564,
'pull_requests_merged': 6163,
'stars': 36987,
'subscribers': 3521,
'total_issues': 4478}...
Expectation
date bitcoin
01-01-2017 564

Since the field pull_request_contributors is not available in each object, pandas cannot build the dataframe. Run
df_pr = pd_json.json_normalize(data, record_path='developer_data', meta=['pull_request_contributors'], errors='ignore').set_index(ts)
to ignore missing fields.
EDIT
json_normalized creates a table with all fields as columns and their values make the rows. So for what you want to achieve, I wouldn't go with json_normalize, since you know which particular field you want to read. Here's how I would do it
ts = '01-01-2017'
cs = 'bitcoin'
df_pr = pd_json.json_normalize(data['developer_data'])
df = pd.DataFrame(data=[{'date': ts,
cs: data['developer_data']['pull_request_contributors']}]).set_index('date')
This way we simply construct the DataFrame, without first normalizing the response.
If the response is a string and not a dict, I don't know what the CoinGeckoAPI returns, you can decode it first with
import json
data = json.loads(json_string)
Hope this helps

Create Proper Dataframe from SDMX Response, Python 3.6

I want to prepare dataset from the data available in http://stat.data.abs.gov.au/Index.aspx?DataSetCode=ATSI_BIRTHS_SUMM
Data API:
http://stat.data.abs.gov.au/restsdmx/sdmx.ashx/GetData/ATSI_BIRTHS_SUMM/1+4+5+7+8+9+10+13+14+15+18+19+20.IM+IB.0+1+2+3+4+5+6+7.A/all
from pandasdmx import Request
Agency_Code = 'ABS'
Dataset_Id = 'ATSI_BIRTHS_SUMM'
ABS = Request(Agency_Code)
data_response = ABS.data(resource_id='ATSI_BIRTHS_SUMM')
print(data_response.url)
DF = data_response.write(data_response.data.obs(with_values=True, with_attributes=True), parse_time=False)
Above gives error: ValueError: Type names and field names cannot be a keyword: 'None'
DF = data_response.write(data_response.data.series, parse_time=False), This works but Dimension items coming in column wise.
Support Links:
http://stat.data.abs.gov.au/restsdmx/sdmx.ashx/GetDataStructure/all
http://stat.data.abs.gov.au/restsdmx/sdmx.ashx/GetDataStructure/ATSI_BIRTHS_SUMM
http://stat.data.abs.gov.au/Index.aspx?DataSetCode=ATSI_BIRTHS_SUMM
Please suggest better way to retrieve data.

Your example
DF = data_response.write(data_response.data.series, parse_time=False)
Produces a stacked DataFrame, by unstack().reset_index() you will get a "flat" DataFrame.
data_response.write().unstack().reset_index()
MEASURE INDIGENOUS_STATUS ASGS_2011 FREQUENCY TIME_PERIOD 0
0 1 IM 0 A 2001 8334.0
Is this what you are looking for?

Cannot Access dict with duplicate values dynamically from Api - Python for Binance

I am attempting to format this Api
https://www.binance.com/api/v1/ticker/allBookTickers
Here is an abbreviated version of the Api
[{"symbol":"ETHBTC","bidPrice":"0.07200500","bidQty":"0.67800000","askPrice":"0.07203200","askQty":"7.19200000"},{"symbol":"LTCBTC","bidPrice":"0.01281100","bidQty":"10.90000000","askPrice":"0.01282500","askQty":"1.01000000"}]
Each dict is saved as an index in the list my issue is with the fact that each dict starts with 'symbol' rather then the name like 'ETHBTC'
I can call the index number but as their is hundreds of dicts in the api I need to find a method of being able to type in for instance 'ETHBTC' to call that dict?
This is what it would look like in an ideal world but I have no idea how to achieve this any help would be greatly appreciated?
> data = requests.get('https://www.binance.com/api/v1/ticker/allBookTickers')
> data = data.json()
> ltc = data['LTCBTC']

Use the following code:-
import requests
#fetched data from url using requests
data = requests.get('https://www.binance.com/api/v1/ticker/allBookTickers')
# creating json object from response
dataJson = data.json()
# creating dictionary from json object using symbol value as key
dataDictionary = {d['symbol'] : d for d in dataJson}
# accessing dictionary object using symbol value
ltc = dataDictionary['LTCBTC']
print ltc
# now you can use ltc values by given keys as, and so on for other values
print ltc['askPrice']
In this code we created python dictionary from the response returned.

Pandas Google Distance Matrix API - Pass coordinates into URL

I am working with the Google Distance Matrix API, where I want to feed coordinates from a dataframe into the API and return the duration and distance between the two points.
Here is my dataframe:
import pandas as pd
import simplejson
import urllib
import numpy as np
Record orig_lat orig_lng dest_lat dest_lng
1 40.7484405 -74.0073127 40.7115242 -74.0145492
2 40.7421218 -73.9878531 40.7727216 -73.9863531
First, I need to combine the orig_lat & orig_lng and dest_lat & dest_lng into strings, which then pass into the url. So I've tried creating the variables orig_coord & dest_coord then passing them into the URL and returning values:
orig_coord = df[['orig_lat','orig_lng']].apply(lambda x: '{},{}'.format(x[0],x[1]), axis=1)
dest_coord = df[['dest_lat','dest_lng']].apply(lambda x: '{},{}'.format(x[0],x[1]), axis=1)
for row in df.itertuples():
url = "http://maps.googleapis.com/maps/api/distancematrix/json?origins={0}&destinations={1}&units=imperial&MYGOOGLEAPIKEY".format(orig_coord,end_coord)
result = simplejson.load(urllib.urlopen(url))
df['driving_time_text'] = result['rows'][0]['elements'][0]['duration']['text']
But I get the following error: "TypeError: () got an unexpected keyword argument 'axis'"
So my question is: how do I concatenate values from two columns into a string, then pass that string into a URL and output the result?
Thank you in advance!

Hmm, I am not sure how you constructed your data frame. Maybe post those details? But if you can live with referencing tuple elements positionally, this worked for me:
import pandas as pd
data = [{'orig_lat': 40.748441, 'orig_lng': -74.007313, 'dest_lat': 40.711524, 'dest_lng': -74.014549},
{'orig_lat': 40.742122, 'orig_lng': -73.987853, 'dest_lat': 40.772722, 'dest_lng': -73.986353}]
df = pd.DataFrame(data)
for row in df.itertuples():
orig_coord='{},{}'.format(row[1],row[2])
dest_coord='{},{}'.format(row[3],row[4])
url = "http://maps.googleapis.com/maps/api/distancematrix/json?origins={0}&destinations={1}&units=imperial&MYGOOGLEAPIKEY".format(orig_coord,dest_coord)
print url
produces
http://maps.googleapis.com/maps/api/distancematrix/json?origins=40.748441,-74.007313&destinations=40.711524,-74.014549&units=imperial&MYGOOGLEAPIKEY
http://maps.googleapis.com/maps/api/distancematrix/json?origins=40.742122,-73.987853&destinations=40.772722,-73.986353&units=imperial&MYGOOGLEAPIKEY
To update the data frame with the result, since row is a tuple and not writeable, you might want to keep track of the current index as you iterate. Maybe something like this:
data = [{'orig_lat': 40.748441, 'orig_lng': -74.007313, 'dest_lat': 40.711524, 'dest_lng': -74.014549, 'result': -1},
{'orig_lat': 40.742122, 'orig_lng': -73.987853, 'dest_lat': 40.772722, 'dest_lng': -73.986353, 'result': -1}]
df = pd.DataFrame(data)
i_row = 0
for row in df.itertuples():
orig_coord='{},{}'.format(row[1],row[2])
dest_coord='{},{}'.format(row[3],row[4])
url = "http://maps.googleapis.com/maps/api/distancematrix/json?origins={0}&destinations={1}&units=imperial&MYGOOGLEAPIKEY".format(orig_coord,dest_coord)
# Do stuff to get your result
df['result'][i_row] = result
i_row++