I have this api : https://api.publicapis.org/entries
And I wat to iterate key entries of it. I tried as it follows :
r = requests.get('https://api.publicapis.org/entries')
entries = [] #initializing the vector entries
for i in entries: #iterating it
return(i.text) #trying to print the entries
Then I received the following error :
TypeError: The view function did not return a valid response. The function either returned None or ended without a return statement.
How can I solve this problem ?
For that particular API endpoint, you should be fine with
resp = requests.get('https://api.publicapis.org/entries')
resp.raise_for_status() # raise exception on HTTP errors
entries = resp.json()["entries"] # parse JSON, dig out the entries list
# ... do something with entries.
You can use json.loads to parse response text.
Let me add the full code.
import requests
import json
r = requests.get('https://api.publicapis.org/entries')
entries = json.loads(r.text)['entries'] #initializing the vector entries
for i in entries: #iterating it
API = i['API']
Description = i['Description']
Related
I'm making a script with Python to search for competitors with a Google API.
Just for you to see how it works:
First I make a request and save data inside a Json:
# make the http GET request to Scale SERP
api_result = requests.get('https://api.scaleserp.com/search', params)
# Save data inside Json
dados = api_result.json()
Then a create some lists to get position, title, domain and things like that, then I create a loop for to append the position from my competitors inside my lists:
# Create the lists
sPositions = []
sDomains = []
sUrls = []
sTitles = []
sDescription = []
sType = []
# Create loop for to look for information about competitors
for sCompetitors in dados['organic_results']:
sPositions.append(sCompetitors['position'])
sDomains.append(sCompetitors['domain'])
sUrls.append(sCompetitors['link'])
sTitles.append(sCompetitors['title'])
sDescription.append(sCompetitors['snippet'])
sType.append(sCompetitors['type'])
The problem is that not every bracket of my Json is going to have the same values. Some of them won't have the "domain" value. So I need something like "when there is no 'domain' value, append 'no domain' to sDomains list.
I'm glad if anyone could help.
Thanks!!
you should use the get method for dicts so you can set a default value incase the key doesn't exist:
for sCompetitors in dados['organic_results']:
sPositions.append(sCompetitors.get('position', 'no position'))
sDomains.append(sCompetitors.get('domain', 'no domain'))
sUrls.append(sCompetitors.get('link', 'no link'))
sTitles.append(sCompetitors.get('title', 'no title'))
sDescription.append(sCompetitors.get('snippet', 'no snippet'))
sType.append(sCompetitors.get('type', 'no type'))
I am trying to parse a "complicated" JSON string that is returned to me by an API.
It looks like this:
{
"data":[
["Distance to last strike","23.0","miles"],
["Time of last strike","1/14/2022 9:23:42 AM",""],
["Number of strikes today","1",""]
]
}
While the end goal will be to extract the distance, date/time, as well as count, for right now I am just trying to successfully get the distance.
My python script is:
import requests
import json
response_API = requests.get('http://localhost:8998/api/extra/lightning.json')
data = response_API.text
parse_json = json.loads(data)
value = parse_json['Distance to last strike']
print(value)
This does not work. If I change the value line to
value = parse_json['data']
then the entire string I listed above is returned.
I am hoping it's just a simple formatting issue. Suggestions?
You have an object with a list of lists. If you fetch
value = parse_json['data']
Then you will have a list containing three lists. So:
print(value[0][1])
will print "23.0".
import requests
import json
r = requests.get("https://api.investing.com/api/search/?t=Equities&q=amd") # i get json text from this api
data = json.loads(r.text)
if data['articles'][0]['exchange'] == 'Sydney': # the error is here KeyError: 'exchange'
print('success')
else:
print('fail')
if i want to get the url '/equities/segue-resources-ltd' by checking if the 'exchange' is 'Sydney' which is stored in this part of the json text, {"id":948190,"url":"/equities/segue-resources-ltd","description":"Segue Resources Ltd","symbol":"AMD","exchange":"Sydney","flag":"AU","type":"Equities"}
If i'm understanding this correctly, the exchange identifier only appears in part of the json response. So, in order to get your result using the same data variable in your question, we can do this:
result = [val["url"] for val in data["quotes"] if val["exchange"] == "Sydney"]
We are using a list comprehension here, where the loop is only going through data["quotes"] instead of the whole json response, and for each item in that json subset, we're returning the value for key == "url" where the exchange == "Sydney". Running the line above should get you:
['/equities/segue-resources-ltd']
As expected. If you aren't comfortable with list comprehensions, the more conventional loop-version of it looks like:
result = []
for val in data["quotes"]:
if val["exchange"] == "Sydney":
result.append(val["url"])
print(result)
KeyError: 'exchange' means that the dictionary data['articles'][0] did not have a key 'exchange'.
Depending on your use case, you may want to iterate over the whole list of articles:
for article in data['articles']:
if 'exchange' in article and article['exchange'] == 'Sydney':
... # Your code here
If you only want to check the first article, then use data['articles'][0].get('exchange'). The dict.get() method will return None if the key is not present instead of throwing a KeyError.
I have just made a program to parse some data from an api. The api gives data back with a JSON format. When I try to parse it it gives me a key error
url = json.loads(r.text)["url"]
KeyError: 'url'
This is the part of the code
url = json.loads(r.text)["url"]
I am trying to get the data in the plain field. Here is the output from the API:
{"updates":[{"id":"a6aa-8bd","description":"Bug fixes and enhancemets","version":"8.1.30","type":"firmware","url":"https://con-man.company.com/api/v1/file-732e844b","updated":"2017-07-25"}]}
You cannot access url since it is inside update (list), therefore you need to Pass index and then key :
One liner:
>>> url = json.loads(r.text)['updates'][0]['url']
'https://con-man.company.com/api/v1/file-732e844b'
Explicit
>>> jobj = json.loads(r.text)
>>> url = jobj['updates'][0]['url']
'https://con-man.company.com/api/v1/file-732e844b'
try this,
url = json.loads(r.text)["updates"][0]["url"]
{
"updates": [
{
"id":"a6aa-8bd",
"description":"Bug fixes and enhancemets",
"version":"8.1.30",
"type":"firmware",
"url":"https://con-man.company.com/api/v1/file-732e844b",
"updated":"2017-07-25"
}
]
}
Try to visualize of your dict, it has only one key "update" in that key value it has another list and into that list, you has another dict
so if in your case
_dict = json.loads(r.text) # read file and load dict
_list = _dict['updates'] # read list inside dict
_dict_1 = _list[0] # read list first value and load dict
url = _dict_1['url'] # read 'url' key from dict
I used this and works now for me.
json_object = json.loads(response.content.decode("utf-8"))['list'][0]['localPercentDynamicObjectsUsed']
I am using the below code for scraping data from a website. but I am facing key error: 0
Kindly tell me the problems in my code.
Original JSON response from the webpage:
https://www.demo.com/api/user_details/22
Response:
{"user_details":{"user_id":"22","username":"Test","user_email":"test#gmail.com"}}
I wanna scrape the username, user_id and user_email.
What I have tried:
import json
import requests
import datetime
#data outputs to a CSV file in the current directory
csv_output = open("test.csv", "w")
end_page = 5;
#scan through pages 1 to end_page for data, 20 results per page
for page in range(1,end_page+1):
r = requests.get('https://www.demo.com/api/user_details/' + str(page))
data = r.json()
for index in range(len(data["user_details"])):
csv_output.write("\"%s\",%s\n" % (data["user_details"][index]["user_id"].encode('ascii', 'ignore'))),
data["user_details"][index]["user_id"]
csv_output.close()
data["user_details"] is a dict and not a list and you are getting the error because you are trying to access the values using an index:
data["user_details"][index] ....
You can get the entries by accessing specific keys from the dict:
user_id = data["user_details"]['user_id']
username = data["user_details"]['username']
user_email = data["user_details"]['user_email']
Exactly what AKS asnwered, but I really recommend you to use a framework called Scrapy tocreate crawlers. Much easier. :)
{"user_details":{"user_id":"22","username":"Test","user_email":"test#gmail.com"}}
User details is a dictionary here. On the other hand, index is an integer coming from the range call. The first value would be 0. Your code tries to load data["user_details"][0]. But there is no key 0 in that dictionary.
To iterate over a dictionary, you can call the items method which would give you a tuple with (key, value) pair.
d = {"user_id":"22","username":"Test","user_email":"test#gmail.com"}
for k,v in d.items():
print("Key: {}, Value: {}".format(k,v))