Develop Reference

API gives only the headers in Python but not the data

I am trying to access an API from this website. (https://www.eia.gov/opendata/qb.php?category=717234)
I am able to call the API but I am getting only headers. Not sure if I am doing correctly or any additions are needed.
Code:
import urllib
import requests
import urllib.request
locu_api = 'WebAPI'
def locu_search(query):
api_key = locu_api
url = 'https://api.eia.gov/category?api_key=' + api_key
locality = query.replace(' ', '%20')
response = urllib.request.urlopen(url).read()
json_obj = str(response, 'utf-8')
data = json.loads(json_obj)
When I try to print the results to see whats there in data:
data
I am getting only the headers in JSON output. Can any one help me figure out how to do extract the data instead of headers.

Avi!
Look, the data you posted seems to be an application/json response. I tried to reorganize your snippet a little bit so you could reuse it for other purposes later.
import requests
API_KEY = "insert_it_here"
def get_categories_data(api_key, category_id):
"""
Makes a request to gov API and returns its JSON response
as a python dict.
"""
host = "https://api.eia.gov/"
endpoint = "category"
url = f"{host}/{endpoint}"
qry_string_params = {"api_key": api_key, "category_id": category_id}
response = requests.post(url, params=qry_string_params)
return response.json()
print(get_categories_data(api_key=API_KEY, category_id="717234"))
As far as I can tell, the response contains some categories and their names. If that's not what you were expecting, maybe there's another endpoint that you should look for. I'm sure this snippet can help you if that's the case.
Side note: isn't your API key supposed to be private? Not sure if you should share that.
Update:
Thanks to Brad Solomon, I've changed the snippet to pass query string arguments to the requests.post function by using the params parameter which will take care of the URL encoding, if necessary.

You haven't presented all of the data. But what I see here is first a dict that associates category_id (a number) with a variable name. For example category_id 717252 is associated with variable name 'Import quantity'. Next I see a dict that associates category_id with a description, but you haven't presented the whole of that dict so 717252 does not appear. And after that I would expect to see a third dict, here entirely missing, associating a category_id with a value, something like {'category_id': 717252, 'value': 123.456}.
I think you are just unaccustomed to the way some APIs aggressively decompose their data into key/value pairs. Look more closely at the data. Can't help any further without being able to see the data for myself.

IBM Phoneme Detection in Python

I'm trying to use the watson developer cloud python library to interface with the IBM Speech to Text API in an effort to detect which phonemes or syllables exist in some text. But I'm running into some issues relating to a required parameter called customization_id, and I'm hoping someone might be able to provide more context on what value to pass in. I wasn't able to understand after reading the docs. Here is a code snippet:
from watson_developer_cloud import TextToSpeechV1, WatsonApiException
API_KEY = "<redacted>"
URL = "https://gateway-wdc.watsonplatform.net/text-to-speech/api"
client = TextToSpeechV1(iam_apikey=API_KEY, url=URL)
try:
#response = client.get_word(customization_id="1", word="HELLO WORLD")
#> Malformed GUID: '1'
#response = client.get_word(word="HELLO WORLD")
#> get_word() missing 1 required positional argument: 'customization_id'
#response = client.get_word(customization_id=None, word="HELLO WORLD")
#> ValueError: customization_id must be provided
#response = client.get_word(customization_id="GA", word="HELLO WORLD")
#> ERROR 400: Malformed GUID: 'GA'
# WHAT VALUE TO USE FOR CUSTOMIZATION_ID ??? ...
response = client.get_word(customization_id="_______", word="HELLO WORLD") #>
print("RESPONSE")
print(type(response))
except WatsonApiException as ex:
print(f"ERROR {str(ex.code)}: {ex.message}")
EDIT: It is possible the expected value is the identifier of a new custom voice model. I've started investigating that strategy here, but unfortunately I'm also running into issues on that one as well. The approach might be something like:
# ...
voice_model_response = client.create_voice_model(
name="My Custom Model",
language=LANG,
description="to get a valid 'customization_id' value..."
).get_result()
customization_id = voice_model_response["customization_id"]
response = client.get_word(customization_id=customization_id, word="HELLO WORLD")
# ...

I think you have misread the documentation for the Speech to Text service.
You can create a customisation to modify how a corpus uses pronunciation to detect words
https://cloud.ibm.com/apidocs/speech-to-text?code=python#add-custom-words
But to do that you need to create a customisation, which you can’t do with a lite account.
https://cloud.ibm.com/apidocs/speech-to-text?code=python#create-a-custom-language-model
You can use the API to list which customisations you have already created.
https://cloud.ibm.com/apidocs/speech-to-text?code=python#list-custom-language-models

It turns out I was using the wrong URL and wrong API Key. After fixing the URL to really be "https://gateway-wdc.watsonplatform.net/text-to-speech/api" and creating and upgrading a new standard-level text to speech service and using that service's API Key, I was able to implement the two-step process I mentioned in the updated part of my question.

Pythonic/efficient way to check organisation of public IP addresses

I have a DataFrame common_ips containing IPs as shown below.
I need to achieve two basic tasks:
Identify private and public IPs.
Check organisation for public IPs.
Here is what I am doing:
import json
import urllib
import re
baseurl = 'http://ipinfo.io/' # no HTTPS supported (at least: not without a plan)
def isIPpublic(ipaddress):
return not isIPprivate(ipaddress)
def isIPprivate(ipaddress):
if ipaddress.startswith("::ffff:"):
ipaddress=ipaddress.replace("::ffff:", "")
# IPv4 Regexp from https://stackoverflow.com/questions/30674845/
if re.search(r"^(?:10|127|172\.(?:1[6-9]|2[0-9]|3[01])|192\.168)\..*", ipaddress):
# Yes, so match, so a local or RFC1918 IPv4 address
return True
if ipaddress == "::1":
# Yes, IPv6 localhost
return True
return False
def getipInfo(ipaddress):
url = '%s%s/json' % (baseurl, ipaddress)
try:
urlresult = urllib.request.urlopen(url)
jsonresult = urlresult.read() # get the JSON
parsedjson = json.loads(jsonresult) # put parsed JSON into dictionary
return parsedjson
except:
return None
def checkIP(ipaddress):
if (isIPpublic(ipaddress)):
if bool(getipInfo(ipaddress)):
if 'bogon' in getipInfo(ipaddress).keys():
return 'Private IP'
elif bool(getipInfo(ipaddress).get('org')):
return getipInfo(ipaddress)['org']
else:
return 'No organization data'
else:
return 'No data available'
else:
return 'Private IP'
And applying it to my common_ips DataFrame with
common_ips['Info'] = common_ips.IP.apply(checkIP)
But it's taking longer than I expected. And for some IPs, it's giving incorrect Info.
For instance:
where it should have been AS19902 Department of Administrative Services as I cross-checked it by
and
What am I missing here ? And how can I achieve these tasks in a more Pythonic way ?

A blanket except: is basically always a bug. You are returning None instead of handling any anomalous or error response from the server, and of course the rest of your code has no way to recover.
As a first debugging step, simply take out the try/except handling. Maybe then you can find a way to put back a somewhat more detailed error handler for some cases which you know how to recover from.
def getipInfo(ipaddress):
url = '%s%s/json' % (baseurl, ipaddress)
urlresult = urllib.request.urlopen(url)
jsonresult = urlresult.read() # get the JSON
parsedjson = json.loads(jsonresult) # put parsed JSON into dictionary
return parsedjson
Perhaps the calling code in checkIP should have a try/except instead, and e.g. retry after sleeping for a bit if the server indicates that you are going too fast.
(In the absence of an authorization token, it looks like you are using the free version of this service, which is probably not in any way guaranteed anyway. Also maybe look at using their recommended library -- I haven't looked at it in more detail, but I would imagine it at the very least knows better how to behave in the case of a server-side error. It's almost certainly also more Pythonic, at least in the sense that you should not reinvent things which already exist.)

How to connect to weather API using requests?

I am trying to learn Requests and practicing by connecting to the weather API. But for some reason I can't get it to work? It is clear the weather API is wanting the param in a certain way that I cannot seem to figure out how it translates to Requests.
Here is my code:
r = requests.get('http://api.openweathermap.org/data/2.5/weather', q={'Anderson'})
Here is the link to the API page:
https://openweathermap.org/current
I see the weather pages wants the param in terms of q = city, but the error I get is:
TypeError: request() got an unexpected keyword argument 'q'
Also here the the Requests page that I am referring to: http://docs.python-requests.org/en/master/
Thanks for the help!

Please review the requests user manual, at least the quickstart guide.
The RESTful API you are about to use expects GET request with q="City Name" parameter. Moreover you must have the appid and add it for every request.
Register you application and choose a pricing plan https://openweathermap.org/price
Pass both q="City Name" and APPID=xxx parameters
Here is corresponding request:
api_url = 'http://api.openweathermap.org/data/2.5/weather'
appid = ...
r = requests.get(url=api_url, params=dict(q='Anderson', APPID=appid))

You need an appid for openweather, which for individual use is free (https://openweathermap.org/appid#get)
>>> import requests
>>> r = requests.get('http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1')
>>> loc_weather = r.content.strip()
>>> loc_weather
'{"coord":{"lon":-0.13,"lat":51.51},"weather":[{"id":300,"main":"Drizzle","description":"light intensity drizzle","icon":"09d"}],"base":"stations","main":{"temp":280.32,"pressure":1012,"humidity":81,"temp_min":279.15,"temp_max":281.15},"visibility":10000,"wind":{"speed":4.1,"deg":80},"clouds":{"all":90},"dt":1485789600,"sys":{"type":1,"id":5091,"message":0.0103,"country":"GB","sunrise":1485762037,"sunset":1485794875},"id":2643743,"name":"London","cod":200}'

Try changing your call to the one that follows (by adding the api key to the params):
r = requests.get('http://api.openweathermap.org/data/2.5/weather', params={'q': 'Anderson', 'APPID': YOUR_API_KEY})

This worked for weatherbit.
The url:
url = "https://api.weatherbit.io/v2.0/history/daily"
Create a payload:
payload = {'key':'12345fcfa0hjde13a7896fbdae1ghjkjh',
'city_id': '1261481', # This is New Delhi
'start_date': '2021-03-13',
'end_date': '2021-03-14'}
Call it using GET (the website specifies that GET call is supported):
data = requests.get(url, params=payload)
assert data.status_code == 200
assert 'error' not in data.json().keys()
Your data:
data.json() # Or text or content
My sys specs: Python 3.8, Request 2.25.1

geopy openmapquest for Python throws GeocoderInsufficientPrivileges error

I am running the following:
import geopy
geolocator = geopy.geocoders.OpenMapQuest(api_key='my_key_here')
location1 = geolocator.geocode('Madrid')
where my_key_here is my consumer key for mapquest, and I get the following error:
GeocoderInsufficientPrivileges: HTTP Error 403: Forbidden
Not sure what I am doing wrong.
Thanks!

I've also tried the same with the same result. After checking the Library, I found out, that the error is referring to the line, where the request ist build and it seems, that the API Key is not transmitted. If you add no key in the init statement, the api_key='' so I tried to change the line 66 in my own Library of the file: https://github.com/geopy/geopy/blob/master/geopy/geocoders/openmapquest.py to my key.
Still no success! The key itself works, I've tested it with calling the URL that is also called in the Library:
http://open.mapquestapi.com/nominatim/v1/search.php?key="MY_KEY"&format=json&json_callback=renderBasicSearchNarrative&q=westminster+abbey
no idea why this isn't working…
Cheers.kg

I made slight progress with fixing this one. I was able to get the query written correctly, but its the json parsing that kind of have me stumped. Maybe someone knows. I know the url is being sent correctly (I checked it in the browser and it returned a json object). Maybe someone knows how to parse the returned json object to get it to finally work.
Anyways, I had to go in the openmapquest.py source code, and starting from line 66, I made the following modifications:
self.api_key = api_key
self.api = "http://www.mapquestapi.com/geocoding/v1/address?"
def geocode(self, query, exactly_one=True, timeout=None): # pylint: disable=W0221
"""
Geocode a location query.
:param string query: The address or query you wish to geocode.
:param bool exactly_one: Return one result or a list of results, if
available.
:param int timeout: Time, in seconds, to wait for the geocoding service
to respond before raising a :class:`geopy.exc.GeocoderTimedOut`
exception. Set this only if you wish to override, on this call
only, the value set during the geocoder's initialization.
.. versionadded:: 0.97
"""
params = {
'key': self.api_key,
'location': self.format_string % query
}
if exactly_one:
params['maxResults'] = 1
url = "&".join((self.api, urlencode(params)))
print url # Print the URL just to make sure it's produced correctly
Now the task remains to get the _parse_json function working.